Re: [DISCUSS] KIP-1150 Diskless Topics

[Stanislav Kozlovski <st...@apache.org>]

I have some nits - it may be useful to

a) group all the KIP email threads in the main one (just a bunch of links to everything)
b) create the email threads

It's a bit hard to track it all - for example, I was searching for a discuss thread for KIP-1165 for a while; As far as I can tell, it doesn't exist yet.

Since the KIPs are published (by virtue of having the root KIP be published, having a DISCUSS thread and links to sub-KIPs where were aimed to move the discussion towards), I think it would be good to create DISCUSS threads for them all.

Best,
Stan

On 2025/04/16 11:58:22 Josep Prat wrote:
> Hi Kafka Devs!
> 
> We want to start a new KIP discussion about introducing a new type of
> topics that would make use of Object Storage as the primary source of
> storage. However, as this KIP is big we decided to split it into multiple
> related KIPs.
> We have the motivational KIP-1150 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics)
> that aims to discuss if Apache Kafka should aim to have this type of
> feature at all. This KIP doesn't go onto details on how to implement it.
> This follows the same approach used when we discussed KRaft.
> 
> But as we know that it is sometimes really hard to discuss on that meta
> level, we also created several sub-kips (linked in KIP-1150) that offer an
> implementation of this feature.
> 
> We kindly ask you to use the proper DISCUSS threads for each type of
> concern and keep this one to discuss whether Apache Kafka wants to have
> this feature or not.
> 
> Thanks in advance on behalf of all the authors of this KIP.
> 
> ------------------
> Josep Prat
> Open Source Engineering Director, Aiven
> josep.prat@aiven.io   |   +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Ivan,

Thanks for the reply.

JR 10.1 The producer state snapshot in tiered storage is only used to build
the producer state in the follower. If diskless doesn't need to maintain
producer state in the broker, this part is no longer needed. However, since
the producer state snapshot is part of the public API, a specific
implementation could depend on that. So, we probably still need to provide
it on the broker.  it would be useful to think through how to achieve that.
For example, does the batch coordinator need to do snapshotting of the
producer state?

JR 10.2 We want to be a bit careful with the coordination of the last
stable offset and the aborted index. In the classic approach, when a
replica advances the last stable offset, the aborted index needed to serve
any data up to the last stable offset is locally available. This makes sure
that the replica can return the correct aborted info in the fetch response.
We need a mechanism to achieve the same thing in diskless.

JR 10.3 Currently, the server only routes a consumer to a close replica if
it's in-sync with the leader. Do we want to keep the same logic in
diskless? If so, how does the leader know whether a replica is in-sync
since there is no classic replication?

JR 10.4 How does partition reassignment work? Does the new replica need to
rebuild the local log segments from the last tiered offset in remote
storage?

JR 11 Ok. That could work.

JR12 “lying” seems hacky and I am not sure if it works for producer either
since the current MetadataRequest doesn't carry rack.id. So, the server
doesn't know how to align with the producer. It's probably better to design
the right API for diskless and gate the feature on new clients with the
right API.

JR14 I gave a similar comment for KIP-1176. A read from an object store is
much cheaper than a write, but is not free. So, one can't afford to read
the same object 100 times. It would be useful to think through how to
prevent the same object from being read too many times in the broker in
diskless.

Jun

On Mon, Sep 15, 2025 at 9:40 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi Jun and all,
>
> Thank you for your questions and comments.
>
> > > JR 10. "Consumer fetches are now served from local segments, making
> use of
> > the
> > indexes, page cache, request purgatory, and zero-copy functionality
> already
> > built into classic topics."
> > JR 10.1 Does the broker build the producer state for each partition in
> > diskless topics?
>
> Preliminary we think the producer state itself won’t be needed locally.
> Consumers don’t need it even for classic topics. Brokers won’t use it to
> filter idempotent appends for diskless topics as all this logic will be
> handled by the batch coordinator. However, we probably should create a
> valid producer state snapshot for uploading a segment to tiered storage, if
> the downstream code that uses these snapshots (TierStateMachine,
> ReplicaFetcherThread) will ever be executed for “post-diskless” remote
> segments.
>
> > > JR 10.2 For transactional data, the consumer fetches need to know
> aborted
> > records. How is that achieved?
>
> We now propose to use local segments to serve fetch requests. These
> segments will have all the necessary auxiliary state, including the
> transaction index. So reading mechanics will be the same as with classic
> topics.
>
> > > JR 11. "The batch coordinator saves that the transaction is finished
> and
> > also inserts the control batches in the corresponding logs of the
> involved
> > Diskless topics. This happens only on the metadata level, no actual
> control
> > batches are written to any file. "
> > A fetch response could include multiple transactional batches. How does
> the
> > broker obtain the information about the ending control batch for each
> > batch? Does that mean that a fetch response needs to be built by
> > stitching record batches and generated control batches together?
>
> When the broker looks up batches, the batch coordinator will inform it
> about control batches in the returned range.
> When the broker constructs a fully valid local segment, all the necessary
> control batches will be inserted and indices, including the transaction
> index will be built to serve FetchRequests exactly as they are today.
>
> > > JR 12. Queues: Is there still a share partition leader that all
> consumers
> > are routed to?
>
> Thank you for pointing out this. We analyzed the matter deeper and
> realized that the share-partition leader does make it more difficult than
> we originally thought. Unfortunately, globalizing the share-partition
> leader state looks quite complicated and it’s probably not the right route
> to take. We think about the following approach:
>
> Keep share-partition leaders as they are: co-located with the partition
> leader. (Even though there’s no partition leader from the producer/consumer
> point of view, there will still be a valid Kafka leader replica.) Route
> ShareFetch through them, like with classic topics.
>
> We may need to reconsider our approach to clients and metadata to support
> this. The current way is that we “lie” to clients about who is the leader
> of the partition to route client traffic to the correct availability
> zone/rack. Here comes the problem: for normal consumers (and producers) we
> need to “lie”, but for ShareConsumer we need to tell the truth.
> Unfortunately, the broker doesn’t know what is the purpose of
> MetadataRequest and it’s even not guaranteed the purpose is single.
>
> This issue isn’t visible without share-groups, so probably our original
> approach with “lying” to clients in metadata will not work. We may need a
> way to tell the clients which broker hosts the share-partition leader and
> modify the share consumer to use this information. There may be different
> ways to solve this, for example:
> Add a new Metadata request/response version. In this version, add the
> is_diskless field to inform the client that the topic is diskless. The
> client will select which broker to consumer from and produce to based on
> the rack information. The share-group fetches will go to the leader. To
> support older clients, keep “lying” to them about partition leaders.
> Identify the older clients by the version of the Metadata request. As it
> seems, the older clients won’t be able to fetch from share-groups, but
> still will have all other benefits of diskless topics.
>
> This is not the final idea and before we develop it deeper, we’d like to
> ask the community how the idea of increasing the Metadata request/response
> version and changing the client sounds in general in the context of the KIP.
>
> > > JR 13. "Should the KIPs be modified to include this or it's too
> > implementation-focused?" It would be useful to include enough details to
> > understand correctness and performance impact.
>
> Sure, we’ll modify the KIP with this and also other feedback we may
> collect later.
>
> > > HC5. Henry has a valid point. Requests from a given producer contain a
> > sequence number, which is ordered. If a producer sends every Produce
> > request to an arbitrary broker, those requests could reach the batch
> > coordinator in different order and lead to rejection of the produce
> > requests.
>
> Yes, this is possible to happen. A producer sticks to one broker for the
> duration of metadata.max.age.ms (5 min by default,) and then another
> replica in the zone could be selected as the advertised leader. When
> metadata is rotated, a small possibility window for out-of-sequence produce
> opens, where a “new” broker may commit to the batch coordinator faster than
> the “old” one. This will lead to retried requests, but idempotence won’t be
> violated. We have seen this generate nuisance OUT_OF_SEQUENCE errors with
> the prior design, but these errors should be roughly as common as
> NOT_LEADER_OR_FOLLOWER errors on classic topics in the current design.
>
> Best,
> Ivan
>
> On Mon, Sep 8, 2025, at 19:36, Jun Rao wrote:
> > Hi, Greg and Ivan,
> >
> > Thanks for the update. A few comments.
> >
> > JR 10. "Consumer fetches are now served from local segments, making use
> of
> > the
> > indexes, page cache, request purgatory, and zero-copy functionality
> already
> > built into classic topics."
> > JR 10.1 Does the broker build the producer state for each partition in
> > diskless topics?
> > JR 10.2 For transactional data, the consumer fetches need to know aborted
> > records. How is that achieved?
> >
> > JR 11. "The batch coordinator saves that the transaction is finished and
> > also inserts the control batches in the corresponding logs of the
> involved
> > Diskless topics. This happens only on the metadata level, no actual
> control
> > batches are written to any file. "
> > A fetch response could include multiple transactional batches. How does
> the
> > broker obtain the information about the ending control batch for each
> > batch? Does that mean that a fetch response needs to be built by
> > stitching record batches and generated control batches together?
> >
> > JR 12. Queues: Is there still a share partition leader that all consumers
> > are routed to?
> >
> > JR 13. "Should the KIPs be modified to include this or it's too
> > implementation-focused?" It would be useful to include enough details to
> > understand correctness and performance impact.
> >
> > HC5. Henry has a valid point. Requests from a given producer contain a
> > sequence number, which is ordered. If a producer sends every Produce
> > request to an arbitrary broker, those requests could reach the batch
> > coordinator in different order and lead to rejection of the produce
> > requests.
> >
> > Jun
> >
> > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi all,
> > >
> > > We have also thought in a bit more details about transactions and
> queues,
> > > here's the plan.
> > >
> > > *Transactions*
> > >
> > > The support for transactions in *classic topics* is based on precise
> > > interactions between three actors: clients (mostly producers, but also
> > > consumers), brokers (ReplicaManager and other classes), and transaction
> > > coordinators. Brokers also run partition leaders with their local state
> > > (ProducerStateManager and others).
> > >
> > > The high level (some details skipped) workflow is the following. When a
> > > transactional Produce request is received by the broker:
> > > 1. For each partition, the partition leader checks if a non-empty
> > > transaction is running for this partition. This is done using its local
> > > state derived from the log metadata (ProducerStateManager,
> > > VerificationStateEntry, VerificationGuard).
> > > 2. The transaction coordinator is informed about all the partitions
> that
> > > aren’t part of the transaction to include them.
> > > 3. The partition leaders do additional transactional checks.
> > > 4. The partition leaders append the transactional data to their logs
> and
> > > update some of their state (for example, log the fact that the
> transaction
> > > is running for the partition and its first offset).
> > >
> > > When the transaction is committed or aborted:
> > > 1. The producer contacts the transaction coordinator directly with
> > > EndTxnRequest.
> > > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT
> to
> > > its log and responds to the producer.
> > > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
> leaders
> > > of the involved partitions.
> > > 4. The partition leaders write the transaction markers to their logs
> and
> > > respond to the coordinator.
> > > 5. The coordinator writes the final transaction state COMPLETE_COMMIT
> or
> > > COMPLETE_ABORT.
> > >
> > > In classic topics, partitions have leaders and lots of important state
> > > necessary for supporting this workflow is local. The main challenge in
> > > mapping this to Diskless comes from the fact there are no partition
> > > leaders, so the corresponding pieces of state need to be globalized in
> the
> > > batch coordinator. We are already doing this to support idempotent
> produce.
> > >
> > > The high level workflow for *diskless topics* would look very similar:
> > > 1. For each partition, the broker checks if a non-empty transaction is
> > > running for this partition. In contrast to classic topics, this is
> checked
> > > against the batch coordinator with a single RPC. Since a transaction
> could
> > > be uniquely identified with producer ID and epoch, the positive result
> of
> > > this check could be cached locally (for the double configured duration
> of a
> > > transaction, for example).
> > > 2. The same: The transaction coordinator is informed about all the
> > > partitions that aren’t part of the transaction to include them.
> > > 3. No transactional checks are done on the broker side.
> > > 4. The broker appends the transactional data to the current shared WAL
> > > segment. It doesn’t update any transaction-related state for Diskless
> > > topics, because it doesn’t have any.
> > > 5. The WAL segment is committed to the batch coordinator like in the
> > > normal produce flow.
> > > 6. The batch coordinator does the final transactional checks of the
> > > batches. This procedure would output the same errors like the partition
> > > leader in classic topics would do. I.e. some batches could be rejected.
> > > This means, there will potentially be garbage in the WAL segment file
> in
> > > case of transactional errors. This is preferable to doing more network
> > > round trips, especially considering the WAL segments will be relatively
> > > short-living (see the Greg's update above).
> > >
> > > When the transaction is committed or aborted:
> > > 1. The producer contacts the transaction coordinator directly with
> > > EndTxnRequest.
> > > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT
> to
> > > its log and responds to the producer.
> > > 3. *[NEW]* The transaction coordinator informs the batch coordinator
> that
> > > the transaction is finished.
> > > 4. *[NEW]* The batch coordinator saves that the transaction is finished
> > > and also inserts the control batches in the corresponding logs of the
> > > involved Diskless topics. This happens only on the metadata level, no
> > > actual control batches are written to any file. They will be
> dynamically
> > > created on Fetch and other read operations. We could technically write
> > > these control batches for real, but this would mean extra produce
> latency,
> > > so it's better just to mark them in the batch coordinator and save
> these
> > > milliseconds.
> > > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
> leaders
> > > of the involved partitions. – Now only to classic topics now.
> > > 6. The partition leaders of classic topics write the transaction
> markers
> > > to their logs and respond to the coordinator.
> > > 7. The coordinator writes the final transaction state COMPLETE_COMMIT
> or
> > > COMPLETE_ABORT.
> > >
> > > Compared to the non-transactional produce flow, we get:
> > > 1. An extra network round trip between brokers and the batch
> coordinator
> > > when a new partition appear in the transaction. To mitigate the impact
> of
> > > them:
> > >   - The results will be cached.
> > >   - The calls for multiple partitions in one Produce request will be
> > > grouped.
> > >   - The batch coordinator should be optimized for fast response to
> these
> > > RPCs.
> > >   - The fact that a single producer normally will communicate with a
> > > single broker for the duration of the transaction further reduces the
> > > expected number of round trips.
> > > 2. An extra round trip between the transaction coordinator and batch
> > > coordinator when a transaction is finished.
> > >
> > > With this proposal, transactions will also be able to span both classic
> > > and Diskless topics.
> > >
> > > *Queues*
> > >
> > > The share group coordination and management is a side job that doesn't
> > > interfere with the topic itself (leadership, replicas, physical
> storage of
> > > records, etc.) and non-queue producers and consumers (Fetch and Produce
> > > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > > reason why we can't make Diskless topics compatible with share groups
> the
> > > same way as classic topics are. Even on the code level, we don't
> expect any
> > > serious refactoring: the same reading routines are used that are used
> for
> > > fetching (e.g. ReplicaManager.readFromLog).
> > >
> > >
> > > Should the KIPs be modified to include this or it's too
> > > implementation-focused?
> > >
> > > Best regards,
> > > Ivan
> > >
> > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > Hi all,
> > > >
> > > > Thank you all for your questions and design input on KIP-1150.
> > > >
> > > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > > summarize
> > > > the changes:
> > > >
> > > > 1. The design prioritizes integrating with the existing KIP-405
> Tiered
> > > > Storage interfaces, permitting data produced to a Diskless topic to
> be
> > > > moved to tiered storage.
> > > > This lowers the scalability requirements for the Batch Coordinator
> > > > component, and allows Diskless to compose with Tiered Storage plugin
> > > > features such as encryption and alternative data formats.
> > > >
> > > > 2. Consumer fetches are now served from local segments, making use
> of the
> > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > already
> > > > built into classic topics.
> > > > However, local segments are now considered cache elements, do not
> need to
> > > > be durably stored, and can be built without contacting any other
> > > replicas.
> > > >
> > > > 3. The design has been simplified substantially, by removing the
> previous
> > > > Diskless consume flow, distributed cache component, and "object
> > > > compaction/merging" step.
> > > >
> > > > The design maintains leaderless produces as enabled by the Batch
> > > > Coordinator, and the same latency profiles as the earlier design,
> while
> > > > being simpler and integrating better into the existing ecosystem.
> > > >
> > > > Thanks, and we are eager to hear your feedback on the new design.
> > > > Greg Harris
> > > >
> > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Hi, Jan,
> > > > >
> > > > > For me, the main gap of KIP-1150 is the support of all existing
> client
> > > > > APIs. Currently, there is no design for supporting APIs like
> > > transactions
> > > > > and queues.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > <ja...@kentra.io.invalid> wrote:
> > > > >
> > > > > > Would it be a good time to ask for the current status of this
> KIP? I
> > > > > > haven't seen much activity here for the past 2 months, the vote
> got
> > > > > vetoed
> > > > > > but I think the pending questions have been answered since then.
> > > KIP-1183
> > > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > > >
> > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> > > > > > made, with a coordinator-based approach being by far the
> dominant one
> > > > > when
> > > > > > it comes to market adoption - but all these are standalone
> products.
> > > > > >
> > > > > > I'm a big fan of both approaches, but would hate to see a stall.
> So
> > > the
> > > > > > question is: can we get an update?
> > > > > >
> > > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > > questions
> > > > > > been answered? If not, is there anything I can do to help? I'm
> deeply
> > > > > > familiar with both architectures and have written about both?
> > > > > >
> > > > > > Kind regards,
> > > > > > Jan
> > > > > >
> > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > >
> > > > > > > I have some nits - it may be useful to
> > > > > > >
> > > > > > > a) group all the KIP email threads in the main one (just a
> bunch of
> > > > > links
> > > > > > > to everything)
> > > > > > > b) create the email threads
> > > > > > >
> > > > > > > It's a bit hard to track it all - for example, I was searching
> for
> > > a
> > > > > > > discuss thread for KIP-1165 for a while; As far as I can tell,
> it
> > > > > doesn't
> > > > > > > exist yet.
> > > > > > >
> > > > > > > Since the KIPs are published (by virtue of having the root KIP
> be
> > > > > > > published, having a DISCUSS thread and links to sub-KIPs where
> were
> > > > > aimed
> > > > > > > to move the discussion towards), I think it would be good to
> create
> > > > > > DISCUSS
> > > > > > > threads for them all.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stan
> > > > > > >
> > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > Hi Kafka Devs!
> > > > > > > >
> > > > > > > > We want to start a new KIP discussion about introducing a new
> > > type of
> > > > > > > > topics that would make use of Object Storage as the primary
> > > source of
> > > > > > > > storage. However, as this KIP is big we decided to split it
> into
> > > > > > multiple
> > > > > > > > related KIPs.
> > > > > > > > We have the motivational KIP-1150 (
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > )
> > > > > > > > that aims to discuss if Apache Kafka should aim to have this
> > > type of
> > > > > > > > feature at all. This KIP doesn't go onto details on how to
> > > implement
> > > > > > it.
> > > > > > > > This follows the same approach used when we discussed KRaft.
> > > > > > > >
> > > > > > > > But as we know that it is sometimes really hard to discuss on
> > > that
> > > > > meta
> > > > > > > > level, we also created several sub-kips (linked in KIP-1150)
> that
> > > > > offer
> > > > > > > an
> > > > > > > > implementation of this feature.
> > > > > > > >
> > > > > > > > We kindly ask you to use the proper DISCUSS threads for each
> > > type of
> > > > > > > > concern and keep this one to discuss whether Apache Kafka
> wants
> > > to
> > > > > have
> > > > > > > > this feature or not.
> > > > > > > >
> > > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > > >
> > > > > > > > ------------------
> > > > > > > > Josep Prat
> > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > Aiven Deutschland GmbH
> > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Jun and all,

Thank you for your questions and comments.

> > JR 10. "Consumer fetches are now served from local segments, making use of
> the
> indexes, page cache, request purgatory, and zero-copy functionality already
> built into classic topics."
> JR 10.1 Does the broker build the producer state for each partition in
> diskless topics?

Preliminary we think the producer state itself won’t be needed locally. Consumers don’t need it even for classic topics. Brokers won’t use it to filter idempotent appends for diskless topics as all this logic will be handled by the batch coordinator. However, we probably should create a valid producer state snapshot for uploading a segment to tiered storage, if the downstream code that uses these snapshots (TierStateMachine, ReplicaFetcherThread) will ever be executed for “post-diskless” remote segments.

> > JR 10.2 For transactional data, the consumer fetches need to know aborted
> records. How is that achieved?

We now propose to use local segments to serve fetch requests. These segments will have all the necessary auxiliary state, including the transaction index. So reading mechanics will be the same as with classic topics.

> > JR 11. "The batch coordinator saves that the transaction is finished and
> also inserts the control batches in the corresponding logs of the involved
> Diskless topics. This happens only on the metadata level, no actual control
> batches are written to any file. "
> A fetch response could include multiple transactional batches. How does the
> broker obtain the information about the ending control batch for each
> batch? Does that mean that a fetch response needs to be built by
> stitching record batches and generated control batches together?

When the broker looks up batches, the batch coordinator will inform it about control batches in the returned range.
When the broker constructs a fully valid local segment, all the necessary control batches will be inserted and indices, including the transaction index will be built to serve FetchRequests exactly as they are today.

> > JR 12. Queues: Is there still a share partition leader that all consumers
> are routed to?

Thank you for pointing out this. We analyzed the matter deeper and realized that the share-partition leader does make it more difficult than we originally thought. Unfortunately, globalizing the share-partition leader state looks quite complicated and it’s probably not the right route to take. We think about the following approach:

Keep share-partition leaders as they are: co-located with the partition leader. (Even though there’s no partition leader from the producer/consumer point of view, there will still be a valid Kafka leader replica.) Route ShareFetch through them, like with classic topics.

We may need to reconsider our approach to clients and metadata to support this. The current way is that we “lie” to clients about who is the leader of the partition to route client traffic to the correct availability zone/rack. Here comes the problem: for normal consumers (and producers) we need to “lie”, but for ShareConsumer we need to tell the truth. Unfortunately, the broker doesn’t know what is the purpose of MetadataRequest and it’s even not guaranteed the purpose is single.

This issue isn’t visible without share-groups, so probably our original approach with “lying” to clients in metadata will not work. We may need a way to tell the clients which broker hosts the share-partition leader and modify the share consumer to use this information. There may be different ways to solve this, for example:
Add a new Metadata request/response version. In this version, add the is_diskless field to inform the client that the topic is diskless. The client will select which broker to consumer from and produce to based on the rack information. The share-group fetches will go to the leader. To support older clients, keep “lying” to them about partition leaders. Identify the older clients by the version of the Metadata request. As it seems, the older clients won’t be able to fetch from share-groups, but still will have all other benefits of diskless topics.

This is not the final idea and before we develop it deeper, we’d like to ask the community how the idea of increasing the Metadata request/response version and changing the client sounds in general in the context of the KIP.

> > JR 13. "Should the KIPs be modified to include this or it's too
> implementation-focused?" It would be useful to include enough details to
> understand correctness and performance impact.

Sure, we’ll modify the KIP with this and also other feedback we may collect later.

> > HC5. Henry has a valid point. Requests from a given producer contain a
> sequence number, which is ordered. If a producer sends every Produce
> request to an arbitrary broker, those requests could reach the batch
> coordinator in different order and lead to rejection of the produce
> requests.

Yes, this is possible to happen. A producer sticks to one broker for the duration of metadata.max.age.ms (5 min by default,) and then another replica in the zone could be selected as the advertised leader. When metadata is rotated, a small possibility window for out-of-sequence produce opens, where a “new” broker may commit to the batch coordinator faster than the “old” one. This will lead to retried requests, but idempotence won’t be violated. We have seen this generate nuisance OUT_OF_SEQUENCE errors with the prior design, but these errors should be roughly as common as NOT_LEADER_OR_FOLLOWER errors on classic topics in the current design.

Best,
Ivan

On Mon, Sep 8, 2025, at 19:36, Jun Rao wrote:
> Hi, Greg and Ivan,
> 
> Thanks for the update. A few comments.
> 
> JR 10. "Consumer fetches are now served from local segments, making use of
> the
> indexes, page cache, request purgatory, and zero-copy functionality already
> built into classic topics."
> JR 10.1 Does the broker build the producer state for each partition in
> diskless topics?
> JR 10.2 For transactional data, the consumer fetches need to know aborted
> records. How is that achieved?
> 
> JR 11. "The batch coordinator saves that the transaction is finished and
> also inserts the control batches in the corresponding logs of the involved
> Diskless topics. This happens only on the metadata level, no actual control
> batches are written to any file. "
> A fetch response could include multiple transactional batches. How does the
> broker obtain the information about the ending control batch for each
> batch? Does that mean that a fetch response needs to be built by
> stitching record batches and generated control batches together?
> 
> JR 12. Queues: Is there still a share partition leader that all consumers
> are routed to?
> 
> JR 13. "Should the KIPs be modified to include this or it's too
> implementation-focused?" It would be useful to include enough details to
> understand correctness and performance impact.
> 
> HC5. Henry has a valid point. Requests from a given producer contain a
> sequence number, which is ordered. If a producer sends every Produce
> request to an arbitrary broker, those requests could reach the batch
> coordinator in different order and lead to rejection of the produce
> requests.
> 
> Jun
> 
> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> 
> > Hi all,
> >
> > We have also thought in a bit more details about transactions and queues,
> > here's the plan.
> >
> > *Transactions*
> >
> > The support for transactions in *classic topics* is based on precise
> > interactions between three actors: clients (mostly producers, but also
> > consumers), brokers (ReplicaManager and other classes), and transaction
> > coordinators. Brokers also run partition leaders with their local state
> > (ProducerStateManager and others).
> >
> > The high level (some details skipped) workflow is the following. When a
> > transactional Produce request is received by the broker:
> > 1. For each partition, the partition leader checks if a non-empty
> > transaction is running for this partition. This is done using its local
> > state derived from the log metadata (ProducerStateManager,
> > VerificationStateEntry, VerificationGuard).
> > 2. The transaction coordinator is informed about all the partitions that
> > aren’t part of the transaction to include them.
> > 3. The partition leaders do additional transactional checks.
> > 4. The partition leaders append the transactional data to their logs and
> > update some of their state (for example, log the fact that the transaction
> > is running for the partition and its first offset).
> >
> > When the transaction is committed or aborted:
> > 1. The producer contacts the transaction coordinator directly with
> > EndTxnRequest.
> > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> > its log and responds to the producer.
> > 3. The transaction coordinator sends WriteTxnMarkersRequest to the leaders
> > of the involved partitions.
> > 4. The partition leaders write the transaction markers to their logs and
> > respond to the coordinator.
> > 5. The coordinator writes the final transaction state COMPLETE_COMMIT or
> > COMPLETE_ABORT.
> >
> > In classic topics, partitions have leaders and lots of important state
> > necessary for supporting this workflow is local. The main challenge in
> > mapping this to Diskless comes from the fact there are no partition
> > leaders, so the corresponding pieces of state need to be globalized in the
> > batch coordinator. We are already doing this to support idempotent produce.
> >
> > The high level workflow for *diskless topics* would look very similar:
> > 1. For each partition, the broker checks if a non-empty transaction is
> > running for this partition. In contrast to classic topics, this is checked
> > against the batch coordinator with a single RPC. Since a transaction could
> > be uniquely identified with producer ID and epoch, the positive result of
> > this check could be cached locally (for the double configured duration of a
> > transaction, for example).
> > 2. The same: The transaction coordinator is informed about all the
> > partitions that aren’t part of the transaction to include them.
> > 3. No transactional checks are done on the broker side.
> > 4. The broker appends the transactional data to the current shared WAL
> > segment. It doesn’t update any transaction-related state for Diskless
> > topics, because it doesn’t have any.
> > 5. The WAL segment is committed to the batch coordinator like in the
> > normal produce flow.
> > 6. The batch coordinator does the final transactional checks of the
> > batches. This procedure would output the same errors like the partition
> > leader in classic topics would do. I.e. some batches could be rejected.
> > This means, there will potentially be garbage in the WAL segment file in
> > case of transactional errors. This is preferable to doing more network
> > round trips, especially considering the WAL segments will be relatively
> > short-living (see the Greg's update above).
> >
> > When the transaction is committed or aborted:
> > 1. The producer contacts the transaction coordinator directly with
> > EndTxnRequest.
> > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> > its log and responds to the producer.
> > 3. *[NEW]* The transaction coordinator informs the batch coordinator that
> > the transaction is finished.
> > 4. *[NEW]* The batch coordinator saves that the transaction is finished
> > and also inserts the control batches in the corresponding logs of the
> > involved Diskless topics. This happens only on the metadata level, no
> > actual control batches are written to any file. They will be dynamically
> > created on Fetch and other read operations. We could technically write
> > these control batches for real, but this would mean extra produce latency,
> > so it's better just to mark them in the batch coordinator and save these
> > milliseconds.
> > 5. The transaction coordinator sends WriteTxnMarkersRequest to the leaders
> > of the involved partitions. – Now only to classic topics now.
> > 6. The partition leaders of classic topics write the transaction markers
> > to their logs and respond to the coordinator.
> > 7. The coordinator writes the final transaction state COMPLETE_COMMIT or
> > COMPLETE_ABORT.
> >
> > Compared to the non-transactional produce flow, we get:
> > 1. An extra network round trip between brokers and the batch coordinator
> > when a new partition appear in the transaction. To mitigate the impact of
> > them:
> >   - The results will be cached.
> >   - The calls for multiple partitions in one Produce request will be
> > grouped.
> >   - The batch coordinator should be optimized for fast response to these
> > RPCs.
> >   - The fact that a single producer normally will communicate with a
> > single broker for the duration of the transaction further reduces the
> > expected number of round trips.
> > 2. An extra round trip between the transaction coordinator and batch
> > coordinator when a transaction is finished.
> >
> > With this proposal, transactions will also be able to span both classic
> > and Diskless topics.
> >
> > *Queues*
> >
> > The share group coordination and management is a side job that doesn't
> > interfere with the topic itself (leadership, replicas, physical storage of
> > records, etc.) and non-queue producers and consumers (Fetch and Produce
> > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > reason why we can't make Diskless topics compatible with share groups the
> > same way as classic topics are. Even on the code level, we don't expect any
> > serious refactoring: the same reading routines are used that are used for
> > fetching (e.g. ReplicaManager.readFromLog).
> >
> >
> > Should the KIPs be modified to include this or it's too
> > implementation-focused?
> >
> > Best regards,
> > Ivan
> >
> > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > Hi all,
> > >
> > > Thank you all for your questions and design input on KIP-1150.
> > >
> > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > summarize
> > > the changes:
> > >
> > > 1. The design prioritizes integrating with the existing KIP-405 Tiered
> > > Storage interfaces, permitting data produced to a Diskless topic to be
> > > moved to tiered storage.
> > > This lowers the scalability requirements for the Batch Coordinator
> > > component, and allows Diskless to compose with Tiered Storage plugin
> > > features such as encryption and alternative data formats.
> > >
> > > 2. Consumer fetches are now served from local segments, making use of the
> > > indexes, page cache, request purgatory, and zero-copy functionality
> > already
> > > built into classic topics.
> > > However, local segments are now considered cache elements, do not need to
> > > be durably stored, and can be built without contacting any other
> > replicas.
> > >
> > > 3. The design has been simplified substantially, by removing the previous
> > > Diskless consume flow, distributed cache component, and "object
> > > compaction/merging" step.
> > >
> > > The design maintains leaderless produces as enabled by the Batch
> > > Coordinator, and the same latency profiles as the earlier design, while
> > > being simpler and integrating better into the existing ecosystem.
> > >
> > > Thanks, and we are eager to hear your feedback on the new design.
> > > Greg Harris
> > >
> > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Hi, Jan,
> > > >
> > > > For me, the main gap of KIP-1150 is the support of all existing client
> > > > APIs. Currently, there is no design for supporting APIs like
> > transactions
> > > > and queues.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > <ja...@kentra.io.invalid> wrote:
> > > >
> > > > > Would it be a good time to ask for the current status of this KIP? I
> > > > > haven't seen much activity here for the past 2 months, the vote got
> > > > vetoed
> > > > > but I think the pending questions have been answered since then.
> > KIP-1183
> > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > >
> > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> > > > > made, with a coordinator-based approach being by far the dominant one
> > > > when
> > > > > it comes to market adoption - but all these are standalone products.
> > > > >
> > > > > I'm a big fan of both approaches, but would hate to see a stall. So
> > the
> > > > > question is: can we get an update?
> > > > >
> > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > questions
> > > > > been answered? If not, is there anything I can do to help? I'm deeply
> > > > > familiar with both architectures and have written about both?
> > > > >
> > > > > Kind regards,
> > > > > Jan
> > > > >
> > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > stanislavkozlovski@apache.org> wrote:
> > > > >
> > > > > > I have some nits - it may be useful to
> > > > > >
> > > > > > a) group all the KIP email threads in the main one (just a bunch of
> > > > links
> > > > > > to everything)
> > > > > > b) create the email threads
> > > > > >
> > > > > > It's a bit hard to track it all - for example, I was searching for
> > a
> > > > > > discuss thread for KIP-1165 for a while; As far as I can tell, it
> > > > doesn't
> > > > > > exist yet.
> > > > > >
> > > > > > Since the KIPs are published (by virtue of having the root KIP be
> > > > > > published, having a DISCUSS thread and links to sub-KIPs where were
> > > > aimed
> > > > > > to move the discussion towards), I think it would be good to create
> > > > > DISCUSS
> > > > > > threads for them all.
> > > > > >
> > > > > > Best,
> > > > > > Stan
> > > > > >
> > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > Hi Kafka Devs!
> > > > > > >
> > > > > > > We want to start a new KIP discussion about introducing a new
> > type of
> > > > > > > topics that would make use of Object Storage as the primary
> > source of
> > > > > > > storage. However, as this KIP is big we decided to split it into
> > > > > multiple
> > > > > > > related KIPs.
> > > > > > > We have the motivational KIP-1150 (
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > )
> > > > > > > that aims to discuss if Apache Kafka should aim to have this
> > type of
> > > > > > > feature at all. This KIP doesn't go onto details on how to
> > implement
> > > > > it.
> > > > > > > This follows the same approach used when we discussed KRaft.
> > > > > > >
> > > > > > > But as we know that it is sometimes really hard to discuss on
> > that
> > > > meta
> > > > > > > level, we also created several sub-kips (linked in KIP-1150) that
> > > > offer
> > > > > > an
> > > > > > > implementation of this feature.
> > > > > > >
> > > > > > > We kindly ask you to use the proper DISCUSS threads for each
> > type of
> > > > > > > concern and keep this one to discuss whether Apache Kafka wants
> > to
> > > > have
> > > > > > > this feature or not.
> > > > > > >
> > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > >
> > > > > > > ------------------
> > > > > > > Josep Prat
> > > > > > > Open Source Engineering Director, Aiven
> > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > Aiven Deutschland GmbH
> > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

I'm very sorry. It seems the mailing list is stripping the attachments.
I'll post the two links:
https://drive.google.com/file/d/1El1Kl2x8JYt3CxdwD0cZ-n6flzZ5s0Gc/view
https://drive.google.com/file/d/1SxfdZIDwimM9OTGYMHCrshFclCkRNqJm/view

Sorry for the noise in the list. I'll do better next time.

- Ivan


On Thu, Oct 2, 2025, at 14:11, Ivan Yurchenko wrote:
> Apologies, it seems the images didn't attach...
> There were only two, I'm attaching them to this message.
> Sorry for the inconvenience!
> 
> - Ivan
> 
> On Thu, Oct 2, 2025, at 14:06, Ivan Yurchenko wrote:
>> Hi dear Kafka community,
>> 
>> In the initial Diskless proposal, we proposed to have a separate component, batch/diskless coordinator, whose role would be to centrally manage the batch and WAL file metadata for diskless topics. This component drew many reasonable comments from the community about how it would support various Kafka features (transactions, queues) and its scalability. While we believe we have good answers to all the expressed concerns, we took a step back and looked at the problem from a different perspective.
>> 
>> We would like to propose an alternative Diskless design *without a centralized coordinator*. We believe this approach has potential and propose to discuss it as it may be more appealing to the community.
>> 
>> Let us explain the idea. Most of the complications with the original Diskless approach come from one necessary architecture change: globalizing the local state of partition leader in the batch coordinator. This causes deviations to the established workflows in various features like produce idempotence and transactions, queues, retention, etc. These deviations need to be carefully considered, designed, and later implemented and tested. In the new approach we want to avoid this by making partition leaders again responsible for managing their partitions, even in diskless topics.
>> 
>> In classic Kafka topics, batch data and metadata are blended together in the one partition log. The crux of the Diskless idea is to decouple them and move data to the remote storage, while keeping metadata somewhere else. Using the central batch coordinator for managing batch metadata is one way, but not the only.
>> 
>> Let’s now think about managing metadata for each user partition independently. Generally partitions are independent and don’t share anything apart from that their data are mixed in WAL files. If we figure out how to commit and later delete WAL files safely, we will achieve the necessary autonomy that allows us to get rid of the central batch coordinator. Instead, *each diskless user partition will be managed by its leader*, as in classic Kafka topics. Also like in classic topics, the leader uses the partition log as the way to persist batch metadata, i.e. the regular batch header + the information about how to find this batch on remote storage. In contrast to classic topics, batch data is in remote storage. 
>> 
>> For clarity, let’s compare the three designs:
>> • Classic topics:
>>    • Data and metadata are co-located in the partition log.
>>    • The partition log content: [Batch header (metadata)|Batch data].
>>    • The partition log is replicated to the followers.
>>    • The replicas and leader have local state built from metadata.
>> • Original Diskless:
>>    • Metadata is in the batch coordinator, data is on remote storage.
>>    • The partition state is global in the batch coordinator.
>> • New Diskless:
>>    • Metadata is in the partition log, data is on remote storage.
>>    • Partition log content: [Batch header (metadata)|Batch coordinates on remote storage].
>>    • The partition log is replicated to the followers.
>>    • The replicas and leader have local state built from metadata.
>> 
>> Let’s consider the produce path. Here’s the reminder of the original Diskless design:
>> 
>> 
>> The new approach could be depicted as the following:
>> 
>> 
>> As you can see, the main difference is that now instead of a single commit request to the batch coordinator, we send multiple parallel commit requests to all the leaders of each partition involved in the WAL file. Each of them will commit its batches independently, without coordinating with other leaders and any other components. Batch data is addressed by the WAL file name, the byte offset and size, which allows partitions to know nothing about other partitions to access their data in shared WAL files.
>> 
>> The number of partitions involved in a single WAL file may be quite large, e.g. a hundred. A hundred network requests to commit one WAL file is very impractical. However, there are ways to reduce this number:
>> 1. Partition leaders are located on brokers. Requests to leaders on one broker could be grouped together into a single physical network request (resembling the normal Produce request that may carry batches for many partitions inside). This will cap the number of network requests to the number of brokers in the cluster.
>> 2. If we craft the cluster metadata to make producers send their requests to the right brokers (with respect to AZs), we may achieve the higher concentration of logical commit requests in physical network requests reducing the number of the latter ones even further, ideally to one.
>> 
>> Obviously, out of multiple commit requests some may fail or time out for a variety of reasons. This is fine. Some producers will receive totally or partially failed responses to their Produce requests, similar to what they would have received when appending to a classic topic fails or times out. If a partition experiences problems, other partitions will not be affected (again, like in classic topics). Of course, the uncommitted data will be garbage in WAL files. But WAL files are short-lived (batches are constantly assembled into segments and offloaded to tiered storage), so this garbage will be eventually deleted.
>> 
>> For safely deleting WAL files we now need to centrally manage them, as this is the only state and logic that spans multiple partitions. On the diagram, you can see another commit request called “Commit file (best effort)” going to the WAL File Manager. This manager will be responsible for the following:
>> 1. Collecting (by requests from brokers) and persisting information about committed WAL files.
>> 2. To handle potential failures in file information delivery, it will be doing prefix scan on the remote storage periodically to find and register unknown files. The period of this scan will be configurable and ideally should be quite long.
>> 3. Checking with the relevant partition leaders (after a grace period) if they still have batches in a particular file.
>> 4. Physically deleting files when they aren’t anymore referred to by any partition.
>> 
>> This new design offers the following advantages:
>> 1. It simplifies the implementation of many Kafka features such as idempotence, transactions, queues, tiered storage, retention. Now we don’t need to abstract away and reuse the code from partition leaders in the batch coordinator. Instead, we will literally use the same code paths in leaders, with little adaptation. Workflows from classic topics mostly remain unchanged.
>> For example, it seems that ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and KafkaApis.handleWriteTxnMarkersRequest used for transaction support on the partition leader side could be used for diskless topics with little adaptation. ProducerStateManager, needed for both idempotent produce and transactions, would be reused.
>> Another example is share groups support, where the share partition leader, being co-located with the partition leader, would execute the same logic for both diskless and classic topics.
>> 2. It returns to the familiar partition-based scaling model, where partitions are independent.
>> 3. It makes the operation and failure patterns closer to the familiar ones from classic topics.
>> 4. It opens a straightforward path to seamless switching the topics modes between diskless and classic.
>> 
>> The rest of the things remain unchanged compared to the previous Diskless design (after all previous discussions). Such things as local segment materialization by replicas, the consume path, tiered storage integration, etc.
>> 
>> If the community finds this design more suitable, we will update the KIP(s) accordingly and continue working on it. Please let us know what you think.
>> 
>> Best regards,
>> Ivan and Diskless team
>> 
>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
>> > Hi Justine,
>> > 
>> > Yes, you're right. We need to track the aborted transactions for in the diskless coordinator for as long as the corresponding offsets are there. With the tiered storage unification Greg mentioned earlier, this will be finite time even for infinite data retention.
>> > 
>> > Best,
>> > Ivan
>> > 
>> > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
>> > > Hey Ivan,
>> > > 
>> > > Thanks for the response. I think most of what you said made sense, but I
>> > > did have some questions about this part:
>> > > 
>> > > > As we understand this, the partition leader in classic topics forgets
>> > > about a transaction once it’s replicated (HWM overpasses it). The
>> > > transaction coordinator acts like the main guardian, allowing partition
>> > > leaders to do this safely. Please correct me if this is wrong. We think
>> > > about relying on this with the batch coordinator and delete the information
>> > > about a transaction once it’s finished (as there’s no replication and HWM
>> > > advances immediately).
>> > > 
>> > > I didn't quite understand this. In classic topics, we have maps for ongoing
>> > > transactions which remove state when the transaction is completed and an
>> > > aborted transactions index which is retained for much longer. Once the
>> > > transaction is completed, the coordinator is no longer involved in
>> > > maintaining this partition side state, and it is subject to compaction etc.
>> > > Looking back at the outline provided above, I didn't see much about the
>> > > fetch path, so maybe that could be expanded a bit further. I saw the
>> > > following in a response:
>> > > > When the broker constructs a fully valid local segment, all the necessary
>> > > control batches will be inserted and indices, including the transaction
>> > > index will be built to serve FetchRequests exactly as they are today.
>> > > 
>> > > Based on this, it seems like we need to retain the information about
>> > > aborted txns for longer.
>> > > 
>> > > Thanks,
>> > > Justine
>> > > 
>> > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>> > > 
>> > > > Hi Justine and all,
>> > > >
>> > > > Thank you for your questions!
>> > > >
>> > > > > JO 1. >Since a transaction could be uniquely identified with producer ID
>> > > > > and epoch, the positive result of this check could be cached locally
>> > > > > Are we saying that only new transaction version 2 transactions can be
>> > > > used
>> > > > > here? If not, we can't uniquely identify transactions with producer id +
>> > > > > epoch
>> > > >
>> > > > You’re right that we (probably unintentionally) focused only on version 2.
>> > > > We can either limit the support to version 2 or consider using some
>> > > > surrogates to support version 1.
>> > > >
>> > > > > JO 2. >The batch coordinator does the final transactional checks of the
>> > > > > batches. This procedure would output the same errors like the partition
>> > > > > leader in classic topics would do.
>> > > > > Can you expand on what these checks are? Would you be checking if the
>> > > > > transaction was still ongoing for example?* *
>> > > >
>> > > > Yes, the producer epoch, that the transaction is ongoing, and of course
>> > > > the normal idempotence checks. What the partition leader in the classic
>> > > > topics does before appending a batch to the local log (e.g. in
>> > > > UnifiedLog.maybeStartTransactionVerification and
>> > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we unfortunately
>> > > > cannot do these checks before appending the data to the WAL segment and
>> > > > uploading it, but we can “tombstone” these batches in the batch coordinator
>> > > > during the final commit.
>> > > >
>> > > > > Is there state about ongoing
>> > > > > transactions in the batch coordinator? I see some other state mentioned
>> > > > in
>> > > > > the End transaction section, but it's not super clear what state is
>> > > > stored
>> > > > > and when it is stored.
>> > > >
>> > > > Right, this should have been more explicit. As the partition leader tracks
>> > > > ongoing transactions for classic topics, the batch coordinator has to as
>> > > > well. So when a transaction starts and ends, the transaction coordinator
>> > > > must inform the batch coordinator about this.
>> > > >
>> > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
>> > > > be
>> > > > > stored in the batch coordinator?
>> > > >
>> > > > Yes. This could be deduced from the committed batches and other
>> > > > information, but for the sake of performance we’d better store it
>> > > > explicitly.
>> > > >
>> > > > > JO 4. Are there any thoughts about how long transactional state is
>> > > > > maintained in the batch coordinator and how it will be cleaned up?
>> > > >
>> > > > As we understand this, the partition leader in classic topics forgets
>> > > > about a transaction once it’s replicated (HWM overpasses it). The
>> > > > transaction coordinator acts like the main guardian, allowing partition
>> > > > leaders to do this safely. Please correct me if this is wrong. We think
>> > > > about relying on this with the batch coordinator and delete the information
>> > > > about a transaction once it’s finished (as there’s no replication and HWM
>> > > > advances immediately).
>> > > >
>> > > > Best,
>> > > > Ivan
>> > > >
>> > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
>> > > > > Hey folks,
>> > > > >
>> > > > > Excited to see some updates related to transactions!
>> > > > >
>> > > > > I had a few questions.
>> > > > >
>> > > > > JO 1. >Since a transaction could be uniquely identified with producer ID
>> > > > > and epoch, the positive result of this check could be cached locally
>> > > > > Are we saying that only new transaction version 2 transactions can be
>> > > > used
>> > > > > here? If not, we can't uniquely identify transactions with producer id +
>> > > > > epoch
>> > > > >
>> > > > > JO 2. >The batch coordinator does the final transactional checks of the
>> > > > > batches. This procedure would output the same errors like the partition
>> > > > > leader in classic topics would do.
>> > > > > Can you expand on what these checks are? Would you be checking if the
>> > > > > transaction was still ongoing for example? Is there state about ongoing
>> > > > > transactions in the batch coordinator? I see some other state mentioned
>> > > > in
>> > > > > the End transaction section, but it's not super clear what state is
>> > > > stored
>> > > > > and when it is stored.
>> > > > >
>> > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
>> > > > be
>> > > > > stored in the batch coordinator?
>> > > > >
>> > > > > JO 4. Are there any thoughts about how long transactional state is
>> > > > > maintained in the batch coordinator and how it will be cleaned up?
>> > > > >
>> > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
>> > > > wrote:
>> > > > >
>> > > > > > Hi, Greg and Ivan,
>> > > > > >
>> > > > > > Thanks for the update. A few comments.
>> > > > > >
>> > > > > > JR 10. "Consumer fetches are now served from local segments, making
>> > > > use of
>> > > > > > the
>> > > > > > indexes, page cache, request purgatory, and zero-copy functionality
>> > > > already
>> > > > > > built into classic topics."
>> > > > > > JR 10.1 Does the broker build the producer state for each partition in
>> > > > > > diskless topics?
>> > > > > > JR 10.2 For transactional data, the consumer fetches need to know
>> > > > aborted
>> > > > > > records. How is that achieved?
>> > > > > >
>> > > > > > JR 11. "The batch coordinator saves that the transaction is finished
>> > > > and
>> > > > > > also inserts the control batches in the corresponding logs of the
>> > > > involved
>> > > > > > Diskless topics. This happens only on the metadata level, no actual
>> > > > control
>> > > > > > batches are written to any file. "
>> > > > > > A fetch response could include multiple transactional batches. How
>> > > > does the
>> > > > > > broker obtain the information about the ending control batch for each
>> > > > > > batch? Does that mean that a fetch response needs to be built by
>> > > > > > stitching record batches and generated control batches together?
>> > > > > >
>> > > > > > JR 12. Queues: Is there still a share partition leader that all
>> > > > consumers
>> > > > > > are routed to?
>> > > > > >
>> > > > > > JR 13. "Should the KIPs be modified to include this or it's too
>> > > > > > implementation-focused?" It would be useful to include enough details
>> > > > to
>> > > > > > understand correctness and performance impact.
>> > > > > >
>> > > > > > HC5. Henry has a valid point. Requests from a given producer contain a
>> > > > > > sequence number, which is ordered. If a producer sends every Produce
>> > > > > > request to an arbitrary broker, those requests could reach the batch
>> > > > > > coordinator in different order and lead to rejection of the produce
>> > > > > > requests.
>> > > > > >
>> > > > > > Jun
>> > > > > >
>> > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > We have also thought in a bit more details about transactions and
>> > > > queues,
>> > > > > > > here's the plan.
>> > > > > > >
>> > > > > > > *Transactions*
>> > > > > > >
>> > > > > > > The support for transactions in *classic topics* is based on precise
>> > > > > > > interactions between three actors: clients (mostly producers, but
>> > > > also
>> > > > > > > consumers), brokers (ReplicaManager and other classes), and
>> > > > transaction
>> > > > > > > coordinators. Brokers also run partition leaders with their local
>> > > > state
>> > > > > > > (ProducerStateManager and others).
>> > > > > > >
>> > > > > > > The high level (some details skipped) workflow is the following.
>> > > > When a
>> > > > > > > transactional Produce request is received by the broker:
>> > > > > > > 1. For each partition, the partition leader checks if a non-empty
>> > > > > > > transaction is running for this partition. This is done using its
>> > > > local
>> > > > > > > state derived from the log metadata (ProducerStateManager,
>> > > > > > > VerificationStateEntry, VerificationGuard).
>> > > > > > > 2. The transaction coordinator is informed about all the partitions
>> > > > that
>> > > > > > > aren’t part of the transaction to include them.
>> > > > > > > 3. The partition leaders do additional transactional checks.
>> > > > > > > 4. The partition leaders append the transactional data to their logs
>> > > > and
>> > > > > > > update some of their state (for example, log the fact that the
>> > > > > > transaction
>> > > > > > > is running for the partition and its first offset).
>> > > > > > >
>> > > > > > > When the transaction is committed or aborted:
>> > > > > > > 1. The producer contacts the transaction coordinator directly with
>> > > > > > > EndTxnRequest.
>> > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
>> > > > PREPARE_ABORT to
>> > > > > > > its log and responds to the producer.
>> > > > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
>> > > > > > leaders
>> > > > > > > of the involved partitions.
>> > > > > > > 4. The partition leaders write the transaction markers to their logs
>> > > > and
>> > > > > > > respond to the coordinator.
>> > > > > > > 5. The coordinator writes the final transaction state
>> > > > COMPLETE_COMMIT or
>> > > > > > > COMPLETE_ABORT.
>> > > > > > >
>> > > > > > > In classic topics, partitions have leaders and lots of important
>> > > > state
>> > > > > > > necessary for supporting this workflow is local. The main challenge
>> > > > in
>> > > > > > > mapping this to Diskless comes from the fact there are no partition
>> > > > > > > leaders, so the corresponding pieces of state need to be globalized
>> > > > in
>> > > > > > the
>> > > > > > > batch coordinator. We are already doing this to support idempotent
>> > > > > > produce.
>> > > > > > >
>> > > > > > > The high level workflow for *diskless topics* would look very
>> > > > similar:
>> > > > > > > 1. For each partition, the broker checks if a non-empty transaction
>> > > > is
>> > > > > > > running for this partition. In contrast to classic topics, this is
>> > > > > > checked
>> > > > > > > against the batch coordinator with a single RPC. Since a transaction
>> > > > > > could
>> > > > > > > be uniquely identified with producer ID and epoch, the positive
>> > > > result of
>> > > > > > > this check could be cached locally (for the double configured
>> > > > duration
>> > > > > > of a
>> > > > > > > transaction, for example).
>> > > > > > > 2. The same: The transaction coordinator is informed about all the
>> > > > > > > partitions that aren’t part of the transaction to include them.
>> > > > > > > 3. No transactional checks are done on the broker side.
>> > > > > > > 4. The broker appends the transactional data to the current shared
>> > > > WAL
>> > > > > > > segment. It doesn’t update any transaction-related state for Diskless
>> > > > > > > topics, because it doesn’t have any.
>> > > > > > > 5. The WAL segment is committed to the batch coordinator like in the
>> > > > > > > normal produce flow.
>> > > > > > > 6. The batch coordinator does the final transactional checks of the
>> > > > > > > batches. This procedure would output the same errors like the
>> > > > partition
>> > > > > > > leader in classic topics would do. I.e. some batches could be
>> > > > rejected.
>> > > > > > > This means, there will potentially be garbage in the WAL segment
>> > > > file in
>> > > > > > > case of transactional errors. This is preferable to doing more
>> > > > network
>> > > > > > > round trips, especially considering the WAL segments will be
>> > > > relatively
>> > > > > > > short-living (see the Greg's update above).
>> > > > > > >
>> > > > > > > When the transaction is committed or aborted:
>> > > > > > > 1. The producer contacts the transaction coordinator directly with
>> > > > > > > EndTxnRequest.
>> > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
>> > > > PREPARE_ABORT to
>> > > > > > > its log and responds to the producer.
>> > > > > > > 3. *[NEW]* The transaction coordinator informs the batch coordinator
>> > > > that
>> > > > > > > the transaction is finished.
>> > > > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
>> > > > finished
>> > > > > > > and also inserts the control batches in the corresponding logs of the
>> > > > > > > involved Diskless topics. This happens only on the metadata level, no
>> > > > > > > actual control batches are written to any file. They will be
>> > > > dynamically
>> > > > > > > created on Fetch and other read operations. We could technically
>> > > > write
>> > > > > > > these control batches for real, but this would mean extra produce
>> > > > > > latency,
>> > > > > > > so it's better just to mark them in the batch coordinator and save
>> > > > these
>> > > > > > > milliseconds.
>> > > > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
>> > > > > > leaders
>> > > > > > > of the involved partitions. – Now only to classic topics now.
>> > > > > > > 6. The partition leaders of classic topics write the transaction
>> > > > markers
>> > > > > > > to their logs and respond to the coordinator.
>> > > > > > > 7. The coordinator writes the final transaction state
>> > > > COMPLETE_COMMIT or
>> > > > > > > COMPLETE_ABORT.
>> > > > > > >
>> > > > > > > Compared to the non-transactional produce flow, we get:
>> > > > > > > 1. An extra network round trip between brokers and the batch
>> > > > coordinator
>> > > > > > > when a new partition appear in the transaction. To mitigate the
>> > > > impact of
>> > > > > > > them:
>> > > > > > >   - The results will be cached.
>> > > > > > >   - The calls for multiple partitions in one Produce request will be
>> > > > > > > grouped.
>> > > > > > >   - The batch coordinator should be optimized for fast response to
>> > > > these
>> > > > > > > RPCs.
>> > > > > > >   - The fact that a single producer normally will communicate with a
>> > > > > > > single broker for the duration of the transaction further reduces the
>> > > > > > > expected number of round trips.
>> > > > > > > 2. An extra round trip between the transaction coordinator and batch
>> > > > > > > coordinator when a transaction is finished.
>> > > > > > >
>> > > > > > > With this proposal, transactions will also be able to span both
>> > > > classic
>> > > > > > > and Diskless topics.
>> > > > > > >
>> > > > > > > *Queues*
>> > > > > > >
>> > > > > > > The share group coordination and management is a side job that
>> > > > doesn't
>> > > > > > > interfere with the topic itself (leadership, replicas, physical
>> > > > storage
>> > > > > > of
>> > > > > > > records, etc.) and non-queue producers and consumers (Fetch and
>> > > > Produce
>> > > > > > > RPCs, consumer group-related RPCs are not affected.) We don't see any
>> > > > > > > reason why we can't make Diskless topics compatible with share
>> > > > groups the
>> > > > > > > same way as classic topics are. Even on the code level, we don't
>> > > > expect
>> > > > > > any
>> > > > > > > serious refactoring: the same reading routines are used that are
>> > > > used for
>> > > > > > > fetching (e.g. ReplicaManager.readFromLog).
>> > > > > > >
>> > > > > > >
>> > > > > > > Should the KIPs be modified to include this or it's too
>> > > > > > > implementation-focused?
>> > > > > > >
>> > > > > > > Best regards,
>> > > > > > > Ivan
>> > > > > > >
>> > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > Thank you all for your questions and design input on KIP-1150.
>> > > > > > > >
>> > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new design. To
>> > > > > > > summarize
>> > > > > > > > the changes:
>> > > > > > > >
>> > > > > > > > 1. The design prioritizes integrating with the existing KIP-405
>> > > > Tiered
>> > > > > > > > Storage interfaces, permitting data produced to a Diskless topic
>> > > > to be
>> > > > > > > > moved to tiered storage.
>> > > > > > > > This lowers the scalability requirements for the Batch Coordinator
>> > > > > > > > component, and allows Diskless to compose with Tiered Storage
>> > > > plugin
>> > > > > > > > features such as encryption and alternative data formats.
>> > > > > > > >
>> > > > > > > > 2. Consumer fetches are now served from local segments, making use
>> > > > of
>> > > > > > the
>> > > > > > > > indexes, page cache, request purgatory, and zero-copy functionality
>> > > > > > > already
>> > > > > > > > built into classic topics.
>> > > > > > > > However, local segments are now considered cache elements, do not
>> > > > need
>> > > > > > to
>> > > > > > > > be durably stored, and can be built without contacting any other
>> > > > > > > replicas.
>> > > > > > > >
>> > > > > > > > 3. The design has been simplified substantially, by removing the
>> > > > > > previous
>> > > > > > > > Diskless consume flow, distributed cache component, and "object
>> > > > > > > > compaction/merging" step.
>> > > > > > > >
>> > > > > > > > The design maintains leaderless produces as enabled by the Batch
>> > > > > > > > Coordinator, and the same latency profiles as the earlier design,
>> > > > while
>> > > > > > > > being simpler and integrating better into the existing ecosystem.
>> > > > > > > >
>> > > > > > > > Thanks, and we are eager to hear your feedback on the new design.
>> > > > > > > > Greg Harris
>> > > > > > > >
>> > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi, Jan,
>> > > > > > > > >
>> > > > > > > > > For me, the main gap of KIP-1150 is the support of all existing
>> > > > > > client
>> > > > > > > > > APIs. Currently, there is no design for supporting APIs like
>> > > > > > > transactions
>> > > > > > > > > and queues.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Jun
>> > > > > > > > >
>> > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
>> > > > > > > > > <ja...@kentra.io.invalid> wrote:
>> > > > > > > > >
>> > > > > > > > > > Would it be a good time to ask for the current status of this
>> > > > KIP?
>> > > > > > I
>> > > > > > > > > > haven't seen much activity here for the past 2 months, the
>> > > > vote got
>> > > > > > > > > vetoed
>> > > > > > > > > > but I think the pending questions have been answered since
>> > > > then.
>> > > > > > > KIP-1183
>> > > > > > > > > > (AutoMQ's proposal) also didn't have any activity since May.
>> > > > > > > > > >
>> > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can
>> > > > be
>> > > > > > > > > > made, with a coordinator-based approach being by far the
>> > > > dominant
>> > > > > > one
>> > > > > > > > > when
>> > > > > > > > > > it comes to market adoption - but all these are standalone
>> > > > > > products.
>> > > > > > > > > >
>> > > > > > > > > > I'm a big fan of both approaches, but would hate to see a
>> > > > stall. So
>> > > > > > > the
>> > > > > > > > > > question is: can we get an update?
>> > > > > > > > > >
>> > > > > > > > > > Maybe it's time to start another vote? Colin McCabe - have your
>> > > > > > > questions
>> > > > > > > > > > been answered? If not, is there anything I can do to help? I'm
>> > > > > > deeply
>> > > > > > > > > > familiar with both architectures and have written about both?
>> > > > > > > > > >
>> > > > > > > > > > Kind regards,
>> > > > > > > > > > Jan
>> > > > > > > > > >
>> > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
>> > > > > > > > > > stanislavkozlovski@apache.org> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > I have some nits - it may be useful to
>> > > > > > > > > > >
>> > > > > > > > > > > a) group all the KIP email threads in the main one (just a
>> > > > bunch
>> > > > > > of
>> > > > > > > > > links
>> > > > > > > > > > > to everything)
>> > > > > > > > > > > b) create the email threads
>> > > > > > > > > > >
>> > > > > > > > > > > It's a bit hard to track it all - for example, I was
>> > > > searching
>> > > > > > for
>> > > > > > > a
>> > > > > > > > > > > discuss thread for KIP-1165 for a while; As far as I can
>> > > > tell, it
>> > > > > > > > > doesn't
>> > > > > > > > > > > exist yet.
>> > > > > > > > > > >
>> > > > > > > > > > > Since the KIPs are published (by virtue of having the root
>> > > > KIP be
>> > > > > > > > > > > published, having a DISCUSS thread and links to sub-KIPs
>> > > > where
>> > > > > > were
>> > > > > > > > > aimed
>> > > > > > > > > > > to move the discussion towards), I think it would be good to
>> > > > > > create
>> > > > > > > > > > DISCUSS
>> > > > > > > > > > > threads for them all.
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Stan
>> > > > > > > > > > >
>> > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
>> > > > > > > > > > > > Hi Kafka Devs!
>> > > > > > > > > > > >
>> > > > > > > > > > > > We want to start a new KIP discussion about introducing a
>> > > > new
>> > > > > > > type of
>> > > > > > > > > > > > topics that would make use of Object Storage as the primary
>> > > > > > > source of
>> > > > > > > > > > > > storage. However, as this KIP is big we decided to split it
>> > > > > > into
>> > > > > > > > > > multiple
>> > > > > > > > > > > > related KIPs.
>> > > > > > > > > > > > We have the motivational KIP-1150 (
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
>> > > > > > > > > > > )
>> > > > > > > > > > > > that aims to discuss if Apache Kafka should aim to have
>> > > > this
>> > > > > > > type of
>> > > > > > > > > > > > feature at all. This KIP doesn't go onto details on how to
>> > > > > > > implement
>> > > > > > > > > > it.
>> > > > > > > > > > > > This follows the same approach used when we discussed
>> > > > KRaft.
>> > > > > > > > > > > >
>> > > > > > > > > > > > But as we know that it is sometimes really hard to discuss
>> > > > on
>> > > > > > > that
>> > > > > > > > > meta
>> > > > > > > > > > > > level, we also created several sub-kips (linked in
>> > > > KIP-1150)
>> > > > > > that
>> > > > > > > > > offer
>> > > > > > > > > > > an
>> > > > > > > > > > > > implementation of this feature.
>> > > > > > > > > > > >
>> > > > > > > > > > > > We kindly ask you to use the proper DISCUSS threads for
>> > > > each
>> > > > > > > type of
>> > > > > > > > > > > > concern and keep this one to discuss whether Apache Kafka
>> > > > wants
>> > > > > > > to
>> > > > > > > > > have
>> > > > > > > > > > > > this feature or not.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks in advance on behalf of all the authors of this KIP.
>> > > > > > > > > > > >
>> > > > > > > > > > > > ------------------
>> > > > > > > > > > > > Josep Prat
>> > > > > > > > > > > > Open Source Engineering Director, Aiven
>> > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
>> > > > > > > > > > > > Aiven Deutschland GmbH
>> > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
>> > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
>> > > > > > > > > > > > Anna Richardson, Kenneth Chen
>> > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > 
>> > 
>> 
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Apologies, it seems the images didn't attach...
There were only two, I'm attaching them to this message.
Sorry for the inconvenience!

- Ivan

On Thu, Oct 2, 2025, at 14:06, Ivan Yurchenko wrote:
> Hi dear Kafka community,
> 
> In the initial Diskless proposal, we proposed to have a separate component, batch/diskless coordinator, whose role would be to centrally manage the batch and WAL file metadata for diskless topics. This component drew many reasonable comments from the community about how it would support various Kafka features (transactions, queues) and its scalability. While we believe we have good answers to all the expressed concerns, we took a step back and looked at the problem from a different perspective.
> 
> We would like to propose an alternative Diskless design *without a centralized coordinator*. We believe this approach has potential and propose to discuss it as it may be more appealing to the community.
> 
> Let us explain the idea. Most of the complications with the original Diskless approach come from one necessary architecture change: globalizing the local state of partition leader in the batch coordinator. This causes deviations to the established workflows in various features like produce idempotence and transactions, queues, retention, etc. These deviations need to be carefully considered, designed, and later implemented and tested. In the new approach we want to avoid this by making partition leaders again responsible for managing their partitions, even in diskless topics.
> 
> In classic Kafka topics, batch data and metadata are blended together in the one partition log. The crux of the Diskless idea is to decouple them and move data to the remote storage, while keeping metadata somewhere else. Using the central batch coordinator for managing batch metadata is one way, but not the only.
> 
> Let’s now think about managing metadata for each user partition independently. Generally partitions are independent and don’t share anything apart from that their data are mixed in WAL files. If we figure out how to commit and later delete WAL files safely, we will achieve the necessary autonomy that allows us to get rid of the central batch coordinator. Instead, *each diskless user partition will be managed by its leader*, as in classic Kafka topics. Also like in classic topics, the leader uses the partition log as the way to persist batch metadata, i.e. the regular batch header + the information about how to find this batch on remote storage. In contrast to classic topics, batch data is in remote storage. 
> 
> For clarity, let’s compare the three designs:
> • Classic topics:
>    • Data and metadata are co-located in the partition log.
>    • The partition log content: [Batch header (metadata)|Batch data].
>    • The partition log is replicated to the followers.
>    • The replicas and leader have local state built from metadata.
> • Original Diskless:
>    • Metadata is in the batch coordinator, data is on remote storage.
>    • The partition state is global in the batch coordinator.
> • New Diskless:
>    • Metadata is in the partition log, data is on remote storage.
>    • Partition log content: [Batch header (metadata)|Batch coordinates on remote storage].
>    • The partition log is replicated to the followers.
>    • The replicas and leader have local state built from metadata.
> 
> Let’s consider the produce path. Here’s the reminder of the original Diskless design:
> 
> 
> The new approach could be depicted as the following:
> 
> 
> As you can see, the main difference is that now instead of a single commit request to the batch coordinator, we send multiple parallel commit requests to all the leaders of each partition involved in the WAL file. Each of them will commit its batches independently, without coordinating with other leaders and any other components. Batch data is addressed by the WAL file name, the byte offset and size, which allows partitions to know nothing about other partitions to access their data in shared WAL files.
> 
> The number of partitions involved in a single WAL file may be quite large, e.g. a hundred. A hundred network requests to commit one WAL file is very impractical. However, there are ways to reduce this number:
> 1. Partition leaders are located on brokers. Requests to leaders on one broker could be grouped together into a single physical network request (resembling the normal Produce request that may carry batches for many partitions inside). This will cap the number of network requests to the number of brokers in the cluster.
> 2. If we craft the cluster metadata to make producers send their requests to the right brokers (with respect to AZs), we may achieve the higher concentration of logical commit requests in physical network requests reducing the number of the latter ones even further, ideally to one.
> 
> Obviously, out of multiple commit requests some may fail or time out for a variety of reasons. This is fine. Some producers will receive totally or partially failed responses to their Produce requests, similar to what they would have received when appending to a classic topic fails or times out. If a partition experiences problems, other partitions will not be affected (again, like in classic topics). Of course, the uncommitted data will be garbage in WAL files. But WAL files are short-lived (batches are constantly assembled into segments and offloaded to tiered storage), so this garbage will be eventually deleted.
> 
> For safely deleting WAL files we now need to centrally manage them, as this is the only state and logic that spans multiple partitions. On the diagram, you can see another commit request called “Commit file (best effort)” going to the WAL File Manager. This manager will be responsible for the following:
> 1. Collecting (by requests from brokers) and persisting information about committed WAL files.
> 2. To handle potential failures in file information delivery, it will be doing prefix scan on the remote storage periodically to find and register unknown files. The period of this scan will be configurable and ideally should be quite long.
> 3. Checking with the relevant partition leaders (after a grace period) if they still have batches in a particular file.
> 4. Physically deleting files when they aren’t anymore referred to by any partition.
> 
> This new design offers the following advantages:
> 1. It simplifies the implementation of many Kafka features such as idempotence, transactions, queues, tiered storage, retention. Now we don’t need to abstract away and reuse the code from partition leaders in the batch coordinator. Instead, we will literally use the same code paths in leaders, with little adaptation. Workflows from classic topics mostly remain unchanged.
> For example, it seems that ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and KafkaApis.handleWriteTxnMarkersRequest used for transaction support on the partition leader side could be used for diskless topics with little adaptation. ProducerStateManager, needed for both idempotent produce and transactions, would be reused.
> Another example is share groups support, where the share partition leader, being co-located with the partition leader, would execute the same logic for both diskless and classic topics.
> 2. It returns to the familiar partition-based scaling model, where partitions are independent.
> 3. It makes the operation and failure patterns closer to the familiar ones from classic topics.
> 4. It opens a straightforward path to seamless switching the topics modes between diskless and classic.
> 
> The rest of the things remain unchanged compared to the previous Diskless design (after all previous discussions). Such things as local segment materialization by replicas, the consume path, tiered storage integration, etc.
> 
> If the community finds this design more suitable, we will update the KIP(s) accordingly and continue working on it. Please let us know what you think.
> 
> Best regards,
> Ivan and Diskless team
> 
> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > Hi Justine,
> > 
> > Yes, you're right. We need to track the aborted transactions for in the diskless coordinator for as long as the corresponding offsets are there. With the tiered storage unification Greg mentioned earlier, this will be finite time even for infinite data retention.
> > 
> > Best,
> > Ivan
> > 
> > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > Hey Ivan,
> > > 
> > > Thanks for the response. I think most of what you said made sense, but I
> > > did have some questions about this part:
> > > 
> > > > As we understand this, the partition leader in classic topics forgets
> > > about a transaction once it’s replicated (HWM overpasses it). The
> > > transaction coordinator acts like the main guardian, allowing partition
> > > leaders to do this safely. Please correct me if this is wrong. We think
> > > about relying on this with the batch coordinator and delete the information
> > > about a transaction once it’s finished (as there’s no replication and HWM
> > > advances immediately).
> > > 
> > > I didn't quite understand this. In classic topics, we have maps for ongoing
> > > transactions which remove state when the transaction is completed and an
> > > aborted transactions index which is retained for much longer. Once the
> > > transaction is completed, the coordinator is no longer involved in
> > > maintaining this partition side state, and it is subject to compaction etc.
> > > Looking back at the outline provided above, I didn't see much about the
> > > fetch path, so maybe that could be expanded a bit further. I saw the
> > > following in a response:
> > > > When the broker constructs a fully valid local segment, all the necessary
> > > control batches will be inserted and indices, including the transaction
> > > index will be built to serve FetchRequests exactly as they are today.
> > > 
> > > Based on this, it seems like we need to retain the information about
> > > aborted txns for longer.
> > > 
> > > Thanks,
> > > Justine
> > > 
> > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > 
> > > > Hi Justine and all,
> > > >
> > > > Thank you for your questions!
> > > >
> > > > > JO 1. >Since a transaction could be uniquely identified with producer ID
> > > > > and epoch, the positive result of this check could be cached locally
> > > > > Are we saying that only new transaction version 2 transactions can be
> > > > used
> > > > > here? If not, we can't uniquely identify transactions with producer id +
> > > > > epoch
> > > >
> > > > You’re right that we (probably unintentionally) focused only on version 2.
> > > > We can either limit the support to version 2 or consider using some
> > > > surrogates to support version 1.
> > > >
> > > > > JO 2. >The batch coordinator does the final transactional checks of the
> > > > > batches. This procedure would output the same errors like the partition
> > > > > leader in classic topics would do.
> > > > > Can you expand on what these checks are? Would you be checking if the
> > > > > transaction was still ongoing for example?* *
> > > >
> > > > Yes, the producer epoch, that the transaction is ongoing, and of course
> > > > the normal idempotence checks. What the partition leader in the classic
> > > > topics does before appending a batch to the local log (e.g. in
> > > > UnifiedLog.maybeStartTransactionVerification and
> > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we unfortunately
> > > > cannot do these checks before appending the data to the WAL segment and
> > > > uploading it, but we can “tombstone” these batches in the batch coordinator
> > > > during the final commit.
> > > >
> > > > > Is there state about ongoing
> > > > > transactions in the batch coordinator? I see some other state mentioned
> > > > in
> > > > > the End transaction section, but it's not super clear what state is
> > > > stored
> > > > > and when it is stored.
> > > >
> > > > Right, this should have been more explicit. As the partition leader tracks
> > > > ongoing transactions for classic topics, the batch coordinator has to as
> > > > well. So when a transaction starts and ends, the transaction coordinator
> > > > must inform the batch coordinator about this.
> > > >
> > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> > > > be
> > > > > stored in the batch coordinator?
> > > >
> > > > Yes. This could be deduced from the committed batches and other
> > > > information, but for the sake of performance we’d better store it
> > > > explicitly.
> > > >
> > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > >
> > > > As we understand this, the partition leader in classic topics forgets
> > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > transaction coordinator acts like the main guardian, allowing partition
> > > > leaders to do this safely. Please correct me if this is wrong. We think
> > > > about relying on this with the batch coordinator and delete the information
> > > > about a transaction once it’s finished (as there’s no replication and HWM
> > > > advances immediately).
> > > >
> > > > Best,
> > > > Ivan
> > > >
> > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > Hey folks,
> > > > >
> > > > > Excited to see some updates related to transactions!
> > > > >
> > > > > I had a few questions.
> > > > >
> > > > > JO 1. >Since a transaction could be uniquely identified with producer ID
> > > > > and epoch, the positive result of this check could be cached locally
> > > > > Are we saying that only new transaction version 2 transactions can be
> > > > used
> > > > > here? If not, we can't uniquely identify transactions with producer id +
> > > > > epoch
> > > > >
> > > > > JO 2. >The batch coordinator does the final transactional checks of the
> > > > > batches. This procedure would output the same errors like the partition
> > > > > leader in classic topics would do.
> > > > > Can you expand on what these checks are? Would you be checking if the
> > > > > transaction was still ongoing for example? Is there state about ongoing
> > > > > transactions in the batch coordinator? I see some other state mentioned
> > > > in
> > > > > the End transaction section, but it's not super clear what state is
> > > > stored
> > > > > and when it is stored.
> > > > >
> > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> > > > be
> > > > > stored in the batch coordinator?
> > > > >
> > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > >
> > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi, Greg and Ivan,
> > > > > >
> > > > > > Thanks for the update. A few comments.
> > > > > >
> > > > > > JR 10. "Consumer fetches are now served from local segments, making
> > > > use of
> > > > > > the
> > > > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > > already
> > > > > > built into classic topics."
> > > > > > JR 10.1 Does the broker build the producer state for each partition in
> > > > > > diskless topics?
> > > > > > JR 10.2 For transactional data, the consumer fetches need to know
> > > > aborted
> > > > > > records. How is that achieved?
> > > > > >
> > > > > > JR 11. "The batch coordinator saves that the transaction is finished
> > > > and
> > > > > > also inserts the control batches in the corresponding logs of the
> > > > involved
> > > > > > Diskless topics. This happens only on the metadata level, no actual
> > > > control
> > > > > > batches are written to any file. "
> > > > > > A fetch response could include multiple transactional batches. How
> > > > does the
> > > > > > broker obtain the information about the ending control batch for each
> > > > > > batch? Does that mean that a fetch response needs to be built by
> > > > > > stitching record batches and generated control batches together?
> > > > > >
> > > > > > JR 12. Queues: Is there still a share partition leader that all
> > > > consumers
> > > > > > are routed to?
> > > > > >
> > > > > > JR 13. "Should the KIPs be modified to include this or it's too
> > > > > > implementation-focused?" It would be useful to include enough details
> > > > to
> > > > > > understand correctness and performance impact.
> > > > > >
> > > > > > HC5. Henry has a valid point. Requests from a given producer contain a
> > > > > > sequence number, which is ordered. If a producer sends every Produce
> > > > > > request to an arbitrary broker, those requests could reach the batch
> > > > > > coordinator in different order and lead to rejection of the produce
> > > > > > requests.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have also thought in a bit more details about transactions and
> > > > queues,
> > > > > > > here's the plan.
> > > > > > >
> > > > > > > *Transactions*
> > > > > > >
> > > > > > > The support for transactions in *classic topics* is based on precise
> > > > > > > interactions between three actors: clients (mostly producers, but
> > > > also
> > > > > > > consumers), brokers (ReplicaManager and other classes), and
> > > > transaction
> > > > > > > coordinators. Brokers also run partition leaders with their local
> > > > state
> > > > > > > (ProducerStateManager and others).
> > > > > > >
> > > > > > > The high level (some details skipped) workflow is the following.
> > > > When a
> > > > > > > transactional Produce request is received by the broker:
> > > > > > > 1. For each partition, the partition leader checks if a non-empty
> > > > > > > transaction is running for this partition. This is done using its
> > > > local
> > > > > > > state derived from the log metadata (ProducerStateManager,
> > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > 2. The transaction coordinator is informed about all the partitions
> > > > that
> > > > > > > aren’t part of the transaction to include them.
> > > > > > > 3. The partition leaders do additional transactional checks.
> > > > > > > 4. The partition leaders append the transactional data to their logs
> > > > and
> > > > > > > update some of their state (for example, log the fact that the
> > > > > > transaction
> > > > > > > is running for the partition and its first offset).
> > > > > > >
> > > > > > > When the transaction is committed or aborted:
> > > > > > > 1. The producer contacts the transaction coordinator directly with
> > > > > > > EndTxnRequest.
> > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > PREPARE_ABORT to
> > > > > > > its log and responds to the producer.
> > > > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > > > > leaders
> > > > > > > of the involved partitions.
> > > > > > > 4. The partition leaders write the transaction markers to their logs
> > > > and
> > > > > > > respond to the coordinator.
> > > > > > > 5. The coordinator writes the final transaction state
> > > > COMPLETE_COMMIT or
> > > > > > > COMPLETE_ABORT.
> > > > > > >
> > > > > > > In classic topics, partitions have leaders and lots of important
> > > > state
> > > > > > > necessary for supporting this workflow is local. The main challenge
> > > > in
> > > > > > > mapping this to Diskless comes from the fact there are no partition
> > > > > > > leaders, so the corresponding pieces of state need to be globalized
> > > > in
> > > > > > the
> > > > > > > batch coordinator. We are already doing this to support idempotent
> > > > > > produce.
> > > > > > >
> > > > > > > The high level workflow for *diskless topics* would look very
> > > > similar:
> > > > > > > 1. For each partition, the broker checks if a non-empty transaction
> > > > is
> > > > > > > running for this partition. In contrast to classic topics, this is
> > > > > > checked
> > > > > > > against the batch coordinator with a single RPC. Since a transaction
> > > > > > could
> > > > > > > be uniquely identified with producer ID and epoch, the positive
> > > > result of
> > > > > > > this check could be cached locally (for the double configured
> > > > duration
> > > > > > of a
> > > > > > > transaction, for example).
> > > > > > > 2. The same: The transaction coordinator is informed about all the
> > > > > > > partitions that aren’t part of the transaction to include them.
> > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > 4. The broker appends the transactional data to the current shared
> > > > WAL
> > > > > > > segment. It doesn’t update any transaction-related state for Diskless
> > > > > > > topics, because it doesn’t have any.
> > > > > > > 5. The WAL segment is committed to the batch coordinator like in the
> > > > > > > normal produce flow.
> > > > > > > 6. The batch coordinator does the final transactional checks of the
> > > > > > > batches. This procedure would output the same errors like the
> > > > partition
> > > > > > > leader in classic topics would do. I.e. some batches could be
> > > > rejected.
> > > > > > > This means, there will potentially be garbage in the WAL segment
> > > > file in
> > > > > > > case of transactional errors. This is preferable to doing more
> > > > network
> > > > > > > round trips, especially considering the WAL segments will be
> > > > relatively
> > > > > > > short-living (see the Greg's update above).
> > > > > > >
> > > > > > > When the transaction is committed or aborted:
> > > > > > > 1. The producer contacts the transaction coordinator directly with
> > > > > > > EndTxnRequest.
> > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > PREPARE_ABORT to
> > > > > > > its log and responds to the producer.
> > > > > > > 3. *[NEW]* The transaction coordinator informs the batch coordinator
> > > > that
> > > > > > > the transaction is finished.
> > > > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> > > > finished
> > > > > > > and also inserts the control batches in the corresponding logs of the
> > > > > > > involved Diskless topics. This happens only on the metadata level, no
> > > > > > > actual control batches are written to any file. They will be
> > > > dynamically
> > > > > > > created on Fetch and other read operations. We could technically
> > > > write
> > > > > > > these control batches for real, but this would mean extra produce
> > > > > > latency,
> > > > > > > so it's better just to mark them in the batch coordinator and save
> > > > these
> > > > > > > milliseconds.
> > > > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > > > > leaders
> > > > > > > of the involved partitions. – Now only to classic topics now.
> > > > > > > 6. The partition leaders of classic topics write the transaction
> > > > markers
> > > > > > > to their logs and respond to the coordinator.
> > > > > > > 7. The coordinator writes the final transaction state
> > > > COMPLETE_COMMIT or
> > > > > > > COMPLETE_ABORT.
> > > > > > >
> > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > 1. An extra network round trip between brokers and the batch
> > > > coordinator
> > > > > > > when a new partition appear in the transaction. To mitigate the
> > > > impact of
> > > > > > > them:
> > > > > > >   - The results will be cached.
> > > > > > >   - The calls for multiple partitions in one Produce request will be
> > > > > > > grouped.
> > > > > > >   - The batch coordinator should be optimized for fast response to
> > > > these
> > > > > > > RPCs.
> > > > > > >   - The fact that a single producer normally will communicate with a
> > > > > > > single broker for the duration of the transaction further reduces the
> > > > > > > expected number of round trips.
> > > > > > > 2. An extra round trip between the transaction coordinator and batch
> > > > > > > coordinator when a transaction is finished.
> > > > > > >
> > > > > > > With this proposal, transactions will also be able to span both
> > > > classic
> > > > > > > and Diskless topics.
> > > > > > >
> > > > > > > *Queues*
> > > > > > >
> > > > > > > The share group coordination and management is a side job that
> > > > doesn't
> > > > > > > interfere with the topic itself (leadership, replicas, physical
> > > > storage
> > > > > > of
> > > > > > > records, etc.) and non-queue producers and consumers (Fetch and
> > > > Produce
> > > > > > > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > > > > > > reason why we can't make Diskless topics compatible with share
> > > > groups the
> > > > > > > same way as classic topics are. Even on the code level, we don't
> > > > expect
> > > > > > any
> > > > > > > serious refactoring: the same reading routines are used that are
> > > > used for
> > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > >
> > > > > > >
> > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > implementation-focused?
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Thank you all for your questions and design input on KIP-1150.
> > > > > > > >
> > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > > > > > > summarize
> > > > > > > > the changes:
> > > > > > > >
> > > > > > > > 1. The design prioritizes integrating with the existing KIP-405
> > > > Tiered
> > > > > > > > Storage interfaces, permitting data produced to a Diskless topic
> > > > to be
> > > > > > > > moved to tiered storage.
> > > > > > > > This lowers the scalability requirements for the Batch Coordinator
> > > > > > > > component, and allows Diskless to compose with Tiered Storage
> > > > plugin
> > > > > > > > features such as encryption and alternative data formats.
> > > > > > > >
> > > > > > > > 2. Consumer fetches are now served from local segments, making use
> > > > of
> > > > > > the
> > > > > > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > > > > > already
> > > > > > > > built into classic topics.
> > > > > > > > However, local segments are now considered cache elements, do not
> > > > need
> > > > > > to
> > > > > > > > be durably stored, and can be built without contacting any other
> > > > > > > replicas.
> > > > > > > >
> > > > > > > > 3. The design has been simplified substantially, by removing the
> > > > > > previous
> > > > > > > > Diskless consume flow, distributed cache component, and "object
> > > > > > > > compaction/merging" step.
> > > > > > > >
> > > > > > > > The design maintains leaderless produces as enabled by the Batch
> > > > > > > > Coordinator, and the same latency profiles as the earlier design,
> > > > while
> > > > > > > > being simpler and integrating better into the existing ecosystem.
> > > > > > > >
> > > > > > > > Thanks, and we are eager to hear your feedback on the new design.
> > > > > > > > Greg Harris
> > > > > > > >
> > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Jan,
> > > > > > > > >
> > > > > > > > > For me, the main gap of KIP-1150 is the support of all existing
> > > > > > client
> > > > > > > > > APIs. Currently, there is no design for supporting APIs like
> > > > > > > transactions
> > > > > > > > > and queues.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Would it be a good time to ask for the current status of this
> > > > KIP?
> > > > > > I
> > > > > > > > > > haven't seen much activity here for the past 2 months, the
> > > > vote got
> > > > > > > > > vetoed
> > > > > > > > > > but I think the pending questions have been answered since
> > > > then.
> > > > > > > KIP-1183
> > > > > > > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > > > > > > >
> > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can
> > > > be
> > > > > > > > > > made, with a coordinator-based approach being by far the
> > > > dominant
> > > > > > one
> > > > > > > > > when
> > > > > > > > > > it comes to market adoption - but all these are standalone
> > > > > > products.
> > > > > > > > > >
> > > > > > > > > > I'm a big fan of both approaches, but would hate to see a
> > > > stall. So
> > > > > > > the
> > > > > > > > > > question is: can we get an update?
> > > > > > > > > >
> > > > > > > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > > > > > > questions
> > > > > > > > > > been answered? If not, is there anything I can do to help? I'm
> > > > > > deeply
> > > > > > > > > > familiar with both architectures and have written about both?
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > > Jan
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > >
> > > > > > > > > > > a) group all the KIP email threads in the main one (just a
> > > > bunch
> > > > > > of
> > > > > > > > > links
> > > > > > > > > > > to everything)
> > > > > > > > > > > b) create the email threads
> > > > > > > > > > >
> > > > > > > > > > > It's a bit hard to track it all - for example, I was
> > > > searching
> > > > > > for
> > > > > > > a
> > > > > > > > > > > discuss thread for KIP-1165 for a while; As far as I can
> > > > tell, it
> > > > > > > > > doesn't
> > > > > > > > > > > exist yet.
> > > > > > > > > > >
> > > > > > > > > > > Since the KIPs are published (by virtue of having the root
> > > > KIP be
> > > > > > > > > > > published, having a DISCUSS thread and links to sub-KIPs
> > > > where
> > > > > > were
> > > > > > > > > aimed
> > > > > > > > > > > to move the discussion towards), I think it would be good to
> > > > > > create
> > > > > > > > > > DISCUSS
> > > > > > > > > > > threads for them all.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Stan
> > > > > > > > > > >
> > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > >
> > > > > > > > > > > > We want to start a new KIP discussion about introducing a
> > > > new
> > > > > > > type of
> > > > > > > > > > > > topics that would make use of Object Storage as the primary
> > > > > > > source of
> > > > > > > > > > > > storage. However, as this KIP is big we decided to split it
> > > > > > into
> > > > > > > > > > multiple
> > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > )
> > > > > > > > > > > > that aims to discuss if Apache Kafka should aim to have
> > > > this
> > > > > > > type of
> > > > > > > > > > > > feature at all. This KIP doesn't go onto details on how to
> > > > > > > implement
> > > > > > > > > > it.
> > > > > > > > > > > > This follows the same approach used when we discussed
> > > > KRaft.
> > > > > > > > > > > >
> > > > > > > > > > > > But as we know that it is sometimes really hard to discuss
> > > > on
> > > > > > > that
> > > > > > > > > meta
> > > > > > > > > > > > level, we also created several sub-kips (linked in
> > > > KIP-1150)
> > > > > > that
> > > > > > > > > offer
> > > > > > > > > > > an
> > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > >
> > > > > > > > > > > > We kindly ask you to use the proper DISCUSS threads for
> > > > each
> > > > > > > type of
> > > > > > > > > > > > concern and keep this one to discuss whether Apache Kafka
> > > > wants
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > > this feature or not.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > ------------------
> > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > 
> > 
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jack Vanlightly <va...@apache.org>]

Hi all,

I feel we are at a kind of fork in the road at this point as to the
direction we want to take with this KIP. I think it's worth taking a step
back for a moment to evaluate what we really want for Apache Kafka in terms
of properties such as elasticity, operational complexity balanced by cost
of implementation and ongoing code maintenance.

We have three major revisions:
 rev1. The original version of the KIP. Stateless producer and consumer
serving, Batch Coordinators for sequencing and batch coordinate storage.
 rev2. Use tiered storage for long term data management, with local log
segment caching entering the write-path to long term storage. Producer load
still leaderless but consumption side is more stateful/sticky and
requiring disks.
 rev3. No more batch coordinators. Replace with topic partitions to do per
TP sequencing/metadata storage (enables code reuse for txns) and a WAL File
Manager to handle WAL cleanup. Plus a few new non-trivial hurdles to
overcome.

I would propose to the community that we evaluate two broad approaches,
both of which more of less map onto KIP-1150 (in one of its forms):
A) Do we want a more cloud native design, with better separation between
stateless and stateful components, but more costly in engineering effort
and maintenance (due to duplication of txns and stuff like that). The north
star being stateless and diskless producer/consumer serving nodes (like the
'proxy' role in WarpStream), then batch coordinators acting as the stateful
coordination component.
B. Or do we want to prioritize reduced long-term code maintenance and
upfront implementation effort, by reusing existing code for txns, using
tiered storage for long term data management (instead of S3 topic specific
object compaction), by sacrificing on elasticity, placing disks in the
(async) write-path for S3 topics.

Preferring (A) means a design more similar to the first version of this
KIP. If we prefer (B) then that aligns more with revisions 2/3.

I'd like to hear from other community members on what kind of solution they
think best equips Kafka for the next decade.

Thanks
Jack Vanlightly

On Thu, Oct 16, 2025 at 6:01 PM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi Henry,
>
> Indeed, there will be some additional files and descriptors, but as I
> mentioned in my previous message, we don't think there will be a lot more
> compared to the classic Kafka partitions as baseline. We will not cache
> every batch or other small amount in a separate file on disk. There will
> normal Kafka segments (likely in a separate directory as you said) to
> append these batches sequentially. Only recent data will be cached this
> way, so this cache will not grow (neither in disk usage nor in the number
> of file descriptors) proportionally to the total partition size.
>
> You're right that this caching/log materialization part is indeed a little
> underspecified at the moment. We're sorry for this, but as an excuse, we're
> trying to fix the most fundamental parts of the design first, such as the
> approach to the batch coordinator. Local log materialization is mostly
> independent of the coordinator design and may be discussed down the road.
>
> Best,
> Ivan
>
> On Thu, Oct 16, 2025, at 01:42, Henry Haiying Cai wrote:
> > Ivan,
> > For the answer on how many data/meta-data partitions the diskless KIP
> needs to create (hence the number of open file descriptors), there seems to
> be a missing gap.
> > In the current Kafka design, for each topic partition, there is a
> corresponding log directory and a sequence of log segment files on the
> broker disk for persistence.  For example,
> /mnt/kafka/partitions/topic0-0/00000.log for topic 0 partition 0 (I used
> /mnt/kafka/partitions as the root log directory name for illustration
> purpose since that is the directory name in our system).  After user
> enabled topic0 to be diskless, the content in the log data file now
> contains the pointers to WAL log segment file in S3 (the metadata) so you
> are re-using the file descriptors in /mnt/kafka/partitions, but what about
> the real data content for topic0-0?  I think the KIP is proposing uploading
> those data as part of the WAL log segment onto S3, but there also needs to
> be a local cached copy of data content for topic0-0 on both the leader
> broker as well as the follower brokers for consumers to read from.
> > The KIP did not mention how that cache is built or organized or whether
> it is disk backed.  I remember in the earlier replies you mentioned you are
> going to reuse Kafka's existing page cache system for the data content
> cache, but if you are reusing Kafka's page cache, the data you downloaded
> from S3 or received from produceRequest for topic0-0 will needs to be
> stored in /mnt/kafka/partitions/topic0-0 (or maybe a new directory such as
> /mnt/kafka/partitions/data-topic0-0 to avoid the name conflict), but if you
> do this mapping for 100 original data partitions you will end up with 100
> more new partition directories and file descriptors on the disk.
> > Also for the followers to build that cache, there will be temporary file
> and file descriptors being created for the S3 download, and those will be
> new file descriptors being created constantly for each data segment you are
> downloading from S3.
> >     On Tuesday, October 14, 2025 at 11:03:46 AM PDT, Ivan Yurchenko <
> ivan@ivanyu.me> wrote:
> >
> > Hi Jun,
> >
> > Thank you for your message. I’m sorry that I failed to clearly explain
> the idea. Let me try to fix this.
> >
> > > Does each partition now have a metadata partition and a separate data
> > > partition? If so, I am concerned that it essentially doubles the
> number of
> > > partitions, which impacts the number of open file descriptors and the
> > > required IOPS, and so on. It also seems wasteful to have a separate
> > > partition just to store the metadata. It's as if we are creating an
> > > internal topic with an unbounded number of partitions.
> >
> > No. There will be only one physical partition per diskless partition.
> Let me explain this with an example. Let’s say we have a diskless partition
> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce some
> batches to this partition. The content of the segment file will be
> something like this (for each batch):
> >
> > BaseOffset: 00000000000000000000 (like in classic)
> > Length: 123456 (like in classic)
> > PartitionLeaderEpoch: like in classic
> > Magic: like in classic
> > CRC: like in classic
> > Attributes: like in classic
> > LastOffsetDelta: like in classic
> > BaseTimestamp: like in classic
> > MaxTimestamp: like in classic
> > ProducerId: like in classic
> > ProducerEpoch: like in classic
> > BaseSequence: like in classic
> > RecordsCount: like in classic
> > Records:
> >   path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> 123456
> >
> > It looks very much like classic log entries. The only difference is that
> instead of writing real Records, we write the reference to the WAL file
> with the batch data (I guess we need only the name and the byte offset,
> because the byte length is the standard field above). Otherwise, it’s a
> normal Kafka log with the leader and replicas.
> >
> > So we have as many partitions for diskless as for classic. As of open
> file descriptors, let’s proceed to the following:
> >
> > > Are the metadata and
> > > the data for the same partition always collocated on the same broker?
> If
> > > so, how do we enforce that when replicas are reassigned?
> >
> > The source of truth for the data is still in WAL files on object
> storage. The source of truth for the metadata is in segment files on the
> brokers in the replica set. Two new mechanisms are planned, both
> independent of this new proposal, but I want to present them to give the
> idea that only a limited amount of data files will be operated locally:
> > - We want to assemble batches into segment files and offload them to
> tiered storage in order to prevent the unbounded growth of batch metadata.
> For this, we need to open only  a few file descriptors (for the segment
> file itself + the necessary indexes) before the segment is fully written
> and handed over to RemoteLogManager.
> > - We want to assemble local segment files for caching purposes as well,
> i.e. to speed up fetching. This will not materialize the full content of
> the log, but only the hot set according to some policy (or configurable
> policies), i.e. the number of segments and file descriptors will also be
> limited.
> >
> > > The number of RPCs in the produce path is significantly higher. For
> > > example, if a produce request has 100 partitions, in a cluster with 100
> > > brokers, each produce request could generate 100 more RPC requests.
> This
> > > will significantly increase the request rate.
> >
> > This is a valid concern that we considered, but this issue can be
> mitigated. I’ll try to explain the approach.
> > The situation with a single broker is trivial: all the commit requests
> go from the broker to itself.
> > Let’s scale this to a multi-broker cluster, but located in the single
> rack (AZ). Any broker can accept Produce requests for diskless partitions,
> but we can tell producers (through metadata) to always send Produce
> requests to leaders. For example, broker 0 hosts the leader replicas for
> diskless partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> requests for these partitions in various combinations, but only for them.
> >
> >                     Broker 0
> >               +-----------------+
> >               |    t1-0        |
> >               |    t2-1 <--------------------+
> >               |    t3-0        |            |
> > produce      | +-------------+ |            |
> > requests    | |  diskless  | |            |
> > --------------->|  produce  +--------------+
> > for these    | | WAL buffer  | |    commit requests
> > partitions  | +-------------+ |    for these partitions
> >               |                |
> >               +-----------------+
> >
> > The same applies for other brokers in this cluster. Effectively, each
> broker will commit only to itself, which effectively means 1 commit request
> per WAL buffer (this may be 0 physical network calls, if we wish, just a
> local function call).
> >
> > Now let’s scale this to multiple racks (AZs). Obviously, we cannot
> always send Produce requests to the designated leaders of diskless
> partitions: this would mean inter-AZ network traffic, which we would like
> to avoid. To avoid it, we say that every broker has a “diskless produce
> representative” in every AZ. If we continue our example: when a Produce
> request for t1-0, t2-1, or t3-0 comes from a producer in AZ 0, it lands on
> broker 0 (in the broker’s AZ the representative is the broker itself).
> However, if it comes from AZ 1, it lands on broker 1; in AZ 2, it’s broker
> 2.
> >
> >   |produce requests        |produce requests        |produce requests
> >   |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0
> >   |from AZ 0                |from AZ 1              |from AZ 2
> >   v                        v                        v
> > Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > +---------------+      +---------------+      +---------------+
> > |    t1-0      |      |              |      |              |
> > |    t2-1      |      |              |      |              |
> > |    t3-0      |      |              |      |              |
> > +---------------+      +--------+------+      +--------+------+
> >     ^    ^                    |                      |
> >     |    +--------------------+                      |
> >     |    commit requests for these partitions        |
> >     |                                                |
> >     +-------------------------------------------------+
> >           commit requests for these partitions
> >
> > All the partitions that broker 0 is the leader of will be “represented”
> by brokers 1 and 2 in their AZs.
> >
> > Of course, this relationship goes both ways between AZs (not necessarily
> between the same brokers). It means that provided the cluster is balanced
> by the number of brokers per AZ, each broker will represent (number_of_azs
> - 1) other brokers. This will result in the situation that for the majority
> of commits, each broker will do up to (number_of_azs - 1) network commit
> requests (plus one local). Cloud regions tend to have 3 AZs, very rarely
> more. That means, brokers will be doing up to 2 network commit requests per
> WAL file.
> >
> > There are the following exceptions:
> > 1. Broker count imbalance between AZs. For example, when we have 2 AZs
> and one has three brokers and another AZ has one. This one broker will do
> between 1 and 3 commit requests per WAL file. This is not an extreme
> amplification. Such an imbalance is not healthy in most practical setups
> and should be avoided anyway.
> > 2. Leadership changes and metadata propagation period. When the
> partition t3-0 is relocated from broker 0 to some broker 3, the producers
> will not know this immediately (unless we want to be strict and respond
> with NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come together
> in a WAL buffer on broker 2, it will have to send two commit requests: to
> broker 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
> situation is not permanent and as producers update the cluster metadata, it
> will be resolved.
> >
> > This all could be built with the metadata crafting mechanism only (which
> is anyway needed for Diskless in one way or another to direct producers and
> consumers where we need to avoid inter-AZ traffic), just with the right
> policy for it (for example, some deterministic hash-based formula). I.e. no
> explicit support for “produce representative” or anything like this is
> needed on the cluster level, in KRaft, etc.
> >
> > > The same WAL file metadata is now duplicated into two places, partition
> > > leader and WAL File Manager. Which one is the source of truth, and how
> do
> > > we maintain consistency between the two places?
> >
> > We do only two operations on WAL files that span multiple diskless
> partitions: committing and deleting. Commits can be done independently as
> described above. But deletes are different, because when a file is deleted,
> this affects all the partitions that still have alive batches in this file
> (if any).
> >
> > The WAL file manager is a necessary point of coordination to delete WAL
> files safely. We can say it is the source of truth about files themselves,
> while the partition leaders and their logs hold the truth about whether a
> particular file contains live batches of this particular partition.
> >
> > The file manager will do this important task: be able to say for sure
> that a file does not contain any live batch of any existing partition. For
> this, it will have to periodically check against the partition leaders.
> Considering that batch deletion is irreversible, when we declare a file
> “empty”, this is guaranteed to be and stay so.
> >
> > The file manager has to know about files being committed to start track
> them and periodically check if they are empty. We can consider various ways
> to achieve this:
> > 1. As was proposed in my previous message: best effort commit by brokers
> + periodic prefix scans of object storage to detect files that went below
> the radar due to network issue or the file manager temporary
> unavailability. We’re speaking about listing the file names only and
> opening only previously unknown files in order to find the partitions
> involved with them.
> > 2. Only do scans without explicit commit, i.e. fill the list of files
> fully asynchronously and in the background. This may be not ideal due to
> costs and performance of scanning tons of files. However, the number of
> live WAL files should be limited due to tiered storage offloading + we can
> optimize this if we give files some global soft order in their names.
> >
> > > I am not sure how this design simplifies the implementation. The
> existing
> > > producer/replication code can't be simply reused. Adjusting both the
> write
> > > path in the leader and the replication path in the follower to
> understand
> > > batch-header only data is quite intrusive to the existing logic.
> >
> > It is true that we’ll have to change LocalLog and UnifiedLog in order to
> support these changes. However, it seems that idempotence, transactions,
> queues, tiered storage will have to be changed less than with the original
> design. This is because the partition leader state would remain in the same
> place (on brokers) and existing workflows that involve it would have to be
> changed less compared to the situation where we globalize the partition
> leader state in the batch coordinator. I admit this is hard to make
> convincing without both real implementations to hand :)
> >
> > > I am also
> > > not sure how this enables seamless switching the topic modes between
> > > diskless and classic. Could you provide more details on those?
> >
> > Let’s consider the scenario of turning a classic topic into diskless.
> The user sets diskless.enabled=true, the leader receives this metadata
> update and does the following:
> > 1. Stop accepting normal append writes.
> > 2. Close the current active segment.
> > 3. Start a new segment that will be written in the diskless format (i.e.
> without data).
> > 4. Start accepting diskless commits.
> >
> > Since it’s the same log, the followers will know about that switch
> consistently. They will finish replicating the classic segments and start
> replicating the diskless ones. They will always know where each batch is
> located (either inside a classic segment or referenced by a diskless one).
> Switching back should be similar.
> >
> > Doing this with the coordinator is possible, but has some caveats. The
> leader must do the following:
> > 1. Stop accepting normal append writes.
> > 2. Close the current active segment.
> > 3. Write a special control segment to persist and replicate the fact
> that from offset N the partition is now in the diskless mode.
> > 4. Inform the coordinator about the first offset N of the “diskless era”.
> > 5. Inform the controller quorum that the transition has finished and
> that brokers now can process diskless writes for this partition.
> > This could fail at some points, so this will probably require some
> explicit state machine with replication either in the partition log or in
> KRaft.
> >
> > It seems that the coordinator-less approach makes this simpler because
> the “coordinator” for the partition and the partition leader are the same
> and they store the partition metadata in the same log, too. While in the
> coordinator approach we have to perform some kind of a distributed commit
> to handover metadata management from the classic partition leader to the
> batch coordinator.
> >
> > I hope these explanations help to clarify the idea. Please let me know
> if I should go deeper anywhere.
> >
> > Best,
> > Ivan and the Diskless team
> >
> > On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > Hi, Ivan,
> > >
> > > Thanks for the update.
> > >
> > > I am not sure that I fully understand the new design, but it seems less
> > > clean than before.
> > >
> > > Does each partition now have a metadata partition and a separate data
> > > partition? If so, I am concerned that it essentially doubles the
> number of
> > > partitions, which impacts the number of open file descriptors and the
> > > required IOPS, and so on. It also seems wasteful to have a separate
> > > partition just to store the metadata. It's as if we are creating an
> > > internal topic with an unbounded number of partitions. Are the
> metadata and
> > > the data for the same partition always collocated on the same broker?
> If
> > > so, how do we enforce that when replicas are reassigned?
> > >
> > > The number of RPCs in the produce path is significantly higher. For
> > > example, if a produce request has 100 partitions, in a cluster with 100
> > > brokers, each produce request could generate 100 more RPC requests.
> This
> > > will significantly increase the request rate.
> > >
> > > The same WAL file metadata is now duplicated into two places, partition
> > > leader and WAL File Manager. Which one is the source of truth, and how
> do
> > > we maintain consistency between the two places?
> > >
> > > I am not sure how this design simplifies the implementation. The
> existing
> > > producer/replication code can't be simply reused. Adjusting both the
> write
> > > path in the leader and the replication path in the follower to
> understand
> > > batch-header only data is quite intrusive to the existing logic. I am
> also
> > > not sure how this enables seamless switching the topic modes between
> > > diskless and classic. Could you provide more details on those?
> > >
> > > Jun
> > >
> > > On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > >
> > > > Hi dear Kafka community,
> > > >
> > > > In the initial Diskless proposal, we proposed to have a separate
> > > > component, batch/diskless coordinator, whose role would be to
> centrally
> > > > manage the batch and WAL file metadata for diskless topics. This
> component
> > > > drew many reasonable comments from the community about how it would
> support
> > > > various Kafka features (transactions, queues) and its scalability.
> While we
> > > > believe we have good answers to all the expressed concerns, we took
> a step
> > > > back and looked at the problem from a different perspective.
> > > >
> > > > We would like to propose an alternative Diskless design *without a
> > > > centralized coordinator*. We believe this approach has potential and
> > > > propose to discuss it as it may be more appealing to the community.
> > > >
> > > > Let us explain the idea. Most of the complications with the original
> > > > Diskless approach come from one necessary architecture change:
> globalizing
> > > > the local state of partition leader in the batch coordinator. This
> causes
> > > > deviations to the established workflows in various features like
> produce
> > > > idempotence and transactions, queues, retention, etc. These
> deviations need
> > > > to be carefully considered, designed, and later implemented and
> tested. In
> > > > the new approach we want to avoid this by making partition leaders
> again
> > > > responsible for managing their partitions, even in diskless topics.
> > > >
> > > > In classic Kafka topics, batch data and metadata are blended
> together in
> > > > the one partition log. The crux of the Diskless idea is to decouple
> them
> > > > and move data to the remote storage, while keeping metadata
> somewhere else.
> > > > Using the central batch coordinator for managing batch metadata is
> one way,
> > > > but not the only.
> > > >
> > > > Let’s now think about managing metadata for each user partition
> > > > independently. Generally partitions are independent and don’t share
> > > > anything apart from that their data are mixed in WAL files. If we
> figure
> > > > out how to commit and later delete WAL files safely, we will achieve
> the
> > > > necessary autonomy that allows us to get rid of the central batch
> > > > coordinator. Instead, *each diskless user partition will be managed
> by its
> > > > leader*, as in classic Kafka topics. Also like in classic topics, the
> > > > leader uses the partition log as the way to persist batch metadata,
> i.e.
> > > > the regular batch header + the information about how to find this
> batch on
> > > > remote storage. In contrast to classic topics, batch data is in
> remote
> > > > storage.
> > > >
> > > > For clarity, let’s compare the three designs:
> > > >  • Classic topics:
> > > >    • Data and metadata are co-located in the partition log.
> > > >    • The partition log content: [Batch header (metadata)|Batch data].
> > > >    • The partition log is replicated to the followers.
> > > >    • The replicas and leader have local state built from metadata.
> > > >  • Original Diskless:
> > > >    • Metadata is in the batch coordinator, data is on remote storage.
> > > >    • The partition state is global in the batch coordinator.
> > > >  • New Diskless:
> > > >    • Metadata is in the partition log, data is on remote storage.
> > > >    • Partition log content: [Batch header (metadata)|Batch
> coordinates on
> > > > remote storage].
> > > >    • The partition log is replicated to the followers.
> > > >    • The replicas and leader have local state built from metadata.
> > > >
> > > > Let’s consider the produce path. Here’s the reminder of the original
> > > > Diskless design:
> > > >
> > > >
> > > > The new approach could be depicted as the following:
> > > >
> > > >
> > > > As you can see, the main difference is that now instead of a single
> commit
> > > > request to the batch coordinator, we send multiple parallel commit
> requests
> > > > to all the leaders of each partition involved in the WAL file. Each
> of them
> > > > will commit its batches independently, without coordinating with
> other
> > > > leaders and any other components. Batch data is addressed by the WAL
> file
> > > > name, the byte offset and size, which allows partitions to know
> nothing
> > > > about other partitions to access their data in shared WAL files.
> > > >
> > > > The number of partitions involved in a single WAL file may be quite
> large,
> > > > e.g. a hundred. A hundred network requests to commit one WAL file is
> very
> > > > impractical. However, there are ways to reduce this number:
> > > >  1. Partition leaders are located on brokers. Requests to leaders on
> one
> > > > broker could be grouped together into a single physical network
> request
> > > > (resembling the normal Produce request that may carry batches for
> many
> > > > partitions inside). This will cap the number of network requests to
> the
> > > > number of brokers in the cluster.
> > > >  2. If we craft the cluster metadata to make producers send their
> requests
> > > > to the right brokers (with respect to AZs), we may achieve the higher
> > > > concentration of logical commit requests in physical network requests
> > > > reducing the number of the latter ones even further, ideally to one.
> > > >
> > > > Obviously, out of multiple commit requests some may fail or time out
> for a
> > > > variety of reasons. This is fine. Some producers will receive
> totally or
> > > > partially failed responses to their Produce requests, similar to
> what they
> > > > would have received when appending to a classic topic fails or times
> out.
> > > > If a partition experiences problems, other partitions will not be
> affected
> > > > (again, like in classic topics). Of course, the uncommitted data
> will be
> > > > garbage in WAL files. But WAL files are short-lived (batches are
> constantly
> > > > assembled into segments and offloaded to tiered storage), so this
> garbage
> > > > will be eventually deleted.
> > > >
> > > > For safely deleting WAL files we now need to centrally manage them,
> as
> > > > this is the only state and logic that spans multiple partitions. On
> the
> > > > diagram, you can see another commit request called “Commit file (best
> > > > effort)” going to the WAL File Manager. This manager will be
> responsible
> > > > for the following:
> > > >  1. Collecting (by requests from brokers) and persisting information
> about
> > > > committed WAL files.
> > > >  2. To handle potential failures in file information delivery, it
> will be
> > > > doing prefix scan on the remote storage periodically to find and
> register
> > > > unknown files. The period of this scan will be configurable and
> ideally
> > > > should be quite long.
> > > >  3. Checking with the relevant partition leaders (after a grace
> period) if
> > > > they still have batches in a particular file.
> > > >  4. Physically deleting files when they aren’t anymore referred to
> by any
> > > > partition.
> > > >
> > > > This new design offers the following advantages:
> > > >  1. It simplifies the implementation of many Kafka features such as
> > > > idempotence, transactions, queues, tiered storage, retention. Now we
> don’t
> > > > need to abstract away and reuse the code from partition leaders in
> the
> > > > batch coordinator. Instead, we will literally use the same code
> paths in
> > > > leaders, with little adaptation. Workflows from classic topics mostly
> > > > remain unchanged.
> > > > For example, it seems that
> > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > > > KafkaApis.handleWriteTxnMarkersRequest used for transaction support
> on the
> > > > partition leader side could be used for diskless topics with little
> > > > adaptation. ProducerStateManager, needed for both idempotent produce
> and
> > > > transactions, would be reused.
> > > > Another example is share groups support, where the share partition
> leader,
> > > > being co-located with the partition leader, would execute the same
> logic
> > > > for both diskless and classic topics.
> > > >  2. It returns to the familiar partition-based scaling model, where
> > > > partitions are independent.
> > > >  3. It makes the operation and failure patterns closer to the
> familiar
> > > > ones from classic topics.
> > > >  4. It opens a straightforward path to seamless switching the topics
> modes
> > > > between diskless and classic.
> > > >
> > > > The rest of the things remain unchanged compared to the previous
> Diskless
> > > > design (after all previous discussions). Such things as local segment
> > > > materialization by replicas, the consume path, tiered storage
> integration,
> > > > etc.
> > > >
> > > > If the community finds this design more suitable, we will update the
> > > > KIP(s) accordingly and continue working on it. Please let us know
> what you
> > > > think.
> > > >
> > > > Best regards,
> > > > Ivan and Diskless team
> > > >
> > > > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > Hi Justine,
> > > > >
> > > > > Yes, you're right. We need to track the aborted transactions for
> in the
> > > > diskless coordinator for as long as the corresponding offsets are
> there.
> > > > With the tiered storage unification Greg mentioned earlier, this
> will be
> > > > finite time even for infinite data retention.
> > > > >
> > > > > Best,
> > > > > Ivan
> > > > >
> > > > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > > Hey Ivan,
> > > > > >
> > > > > > Thanks for the response. I think most of what you said made
> sense, but
> > > > I
> > > > > > did have some questions about this part:
> > > > > >
> > > > > > > As we understand this, the partition leader in classic topics
> forgets
> > > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > > transaction coordinator acts like the main guardian, allowing
> partition
> > > > > > leaders to do this safely. Please correct me if this is wrong.
> We think
> > > > > > about relying on this with the batch coordinator and delete the
> > > > information
> > > > > > about a transaction once it’s finished (as there’s no
> replication and
> > > > HWM
> > > > > > advances immediately).
> > > > > >
> > > > > > I didn't quite understand this. In classic topics, we have maps
> for
> > > > ongoing
> > > > > > transactions which remove state when the transaction is
> completed and
> > > > an
> > > > > > aborted transactions index which is retained for much longer.
> Once the
> > > > > > transaction is completed, the coordinator is no longer involved
> in
> > > > > > maintaining this partition side state, and it is subject to
> compaction
> > > > etc.
> > > > > > Looking back at the outline provided above, I didn't see much
> about the
> > > > > > fetch path, so maybe that could be expanded a bit further. I saw
> the
> > > > > > following in a response:
> > > > > > > When the broker constructs a fully valid local segment, all the
> > > > necessary
> > > > > > control batches will be inserted and indices, including the
> transaction
> > > > > > index will be built to serve FetchRequests exactly as they are
> today.
> > > > > >
> > > > > > Based on this, it seems like we need to retain the information
> about
> > > > > > aborted txns for longer.
> > > > > >
> > > > > > Thanks,
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> > > > > >
> > > > > > > Hi Justine and all,
> > > > > > >
> > > > > > > Thank you for your questions!
> > > > > > >
> > > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > > producer ID
> > > > > > > > and epoch, the positive result of this check could be cached
> > > > locally
> > > > > > > > Are we saying that only new transaction version 2
> transactions can
> > > > be
> > > > > > > used
> > > > > > > > here? If not, we can't uniquely identify transactions with
> > > > producer id +
> > > > > > > > epoch
> > > > > > >
> > > > > > > You’re right that we (probably unintentionally) focused only on
> > > > version 2.
> > > > > > > We can either limit the support to version 2 or consider using
> some
> > > > > > > surrogates to support version 1.
> > > > > > >
> > > > > > > > JO 2. >The batch coordinator does the final transactional
> checks
> > > > of the
> > > > > > > > batches. This procedure would output the same errors like the
> > > > partition
> > > > > > > > leader in classic topics would do.
> > > > > > > > Can you expand on what these checks are? Would you be
> checking if
> > > > the
> > > > > > > > transaction was still ongoing for example?* *
> > > > > > >
> > > > > > > Yes, the producer epoch, that the transaction is ongoing, and
> of
> > > > course
> > > > > > > the normal idempotence checks. What the partition leader in the
> > > > classic
> > > > > > > topics does before appending a batch to the local log (e.g. in
> > > > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> > > > unfortunately
> > > > > > > cannot do these checks before appending the data to the WAL
> segment
> > > > and
> > > > > > > uploading it, but we can “tombstone” these batches in the batch
> > > > coordinator
> > > > > > > during the final commit.
> > > > > > >
> > > > > > > > Is there state about ongoing
> > > > > > > > transactions in the batch coordinator? I see some other state
> > > > mentioned
> > > > > > > in
> > > > > > > > the End transaction section, but it's not super clear what
> state is
> > > > > > > stored
> > > > > > > > and when it is stored.
> > > > > > >
> > > > > > > Right, this should have been more explicit. As the partition
> leader
> > > > tracks
> > > > > > > ongoing transactions for classic topics, the batch coordinator
> has
> > > > to as
> > > > > > > well. So when a transaction starts and ends, the transaction
> > > > coordinator
> > > > > > > must inform the batch coordinator about this.
> > > > > > >
> > > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> that
> > > > would
> > > > > > > be
> > > > > > > > stored in the batch coordinator?
> > > > > > >
> > > > > > > Yes. This could be deduced from the committed batches and other
> > > > > > > information, but for the sake of performance we’d better store
> it
> > > > > > > explicitly.
> > > > > > >
> > > > > > > > JO 4. Are there any thoughts about how long transactional
> state is
> > > > > > > > maintained in the batch coordinator and how it will be
> cleaned up?
> > > > > > >
> > > > > > > As we understand this, the partition leader in classic topics
> forgets
> > > > > > > about a transaction once it’s replicated (HWM overpasses it).
> The
> > > > > > > transaction coordinator acts like the main guardian, allowing
> > > > partition
> > > > > > > leaders to do this safely. Please correct me if this is wrong.
> We
> > > > think
> > > > > > > about relying on this with the batch coordinator and delete the
> > > > information
> > > > > > > about a transaction once it’s finished (as there’s no
> replication
> > > > and HWM
> > > > > > > advances immediately).
> > > > > > >
> > > > > > > Best,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > > > Hey folks,
> > > > > > > >
> > > > > > > > Excited to see some updates related to transactions!
> > > > > > > >
> > > > > > > > I had a few questions.
> > > > > > > >
> > > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > > producer ID
> > > > > > > > and epoch, the positive result of this check could be cached
> > > > locally
> > > > > > > > Are we saying that only new transaction version 2
> transactions can
> > > > be
> > > > > > > used
> > > > > > > > here? If not, we can't uniquely identify transactions with
> > > > producer id +
> > > > > > > > epoch
> > > > > > > >
> > > > > > > > JO 2. >The batch coordinator does the final transactional
> checks
> > > > of the
> > > > > > > > batches. This procedure would output the same errors like the
> > > > partition
> > > > > > > > leader in classic topics would do.
> > > > > > > > Can you expand on what these checks are? Would you be
> checking if
> > > > the
> > > > > > > > transaction was still ongoing for example? Is there state
> about
> > > > ongoing
> > > > > > > > transactions in the batch coordinator? I see some other state
> > > > mentioned
> > > > > > > in
> > > > > > > > the End transaction section, but it's not super clear what
> state is
> > > > > > > stored
> > > > > > > > and when it is stored.
> > > > > > > >
> > > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> that
> > > > would
> > > > > > > be
> > > > > > > > stored in the batch coordinator?
> > > > > > > >
> > > > > > > > JO 4. Are there any thoughts about how long transactional
> state is
> > > > > > > > maintained in the batch coordinator and how it will be
> cleaned up?
> > > > > > > >
> > > > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> <ju...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Greg and Ivan,
> > > > > > > > >
> > > > > > > > > Thanks for the update. A few comments.
> > > > > > > > >
> > > > > > > > > JR 10. "Consumer fetches are now served from local
> segments,
> > > > making
> > > > > > > use of
> > > > > > > > > the
> > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > functionality
> > > > > > > already
> > > > > > > > > built into classic topics."
> > > > > > > > > JR 10.1 Does the broker build the producer state for each
> > > > partition in
> > > > > > > > > diskless topics?
> > > > > > > > > JR 10.2 For transactional data, the consumer fetches need
> to know
> > > > > > > aborted
> > > > > > > > > records. How is that achieved?
> > > > > > > > >
> > > > > > > > > JR 11. "The batch coordinator saves that the transaction is
> > > > finished
> > > > > > > and
> > > > > > > > > also inserts the control batches in the corresponding logs
> of the
> > > > > > > involved
> > > > > > > > > Diskless topics. This happens only on the metadata level,
> no
> > > > actual
> > > > > > > control
> > > > > > > > > batches are written to any file. "
> > > > > > > > > A fetch response could include multiple transactional
> batches.
> > > > How
> > > > > > > does the
> > > > > > > > > broker obtain the information about the ending control
> batch for
> > > > each
> > > > > > > > > batch? Does that mean that a fetch response needs to be
> built by
> > > > > > > > > stitching record batches and generated control batches
> together?
> > > > > > > > >
> > > > > > > > > JR 12. Queues: Is there still a share partition leader
> that all
> > > > > > > consumers
> > > > > > > > > are routed to?
> > > > > > > > >
> > > > > > > > > JR 13. "Should the KIPs be modified to include this or
> it's too
> > > > > > > > > implementation-focused?" It would be useful to include
> enough
> > > > details
> > > > > > > to
> > > > > > > > > understand correctness and performance impact.
> > > > > > > > >
> > > > > > > > > HC5. Henry has a valid point. Requests from a given
> producer
> > > > contain a
> > > > > > > > > sequence number, which is ordered. If a producer sends
> every
> > > > Produce
> > > > > > > > > request to an arbitrary broker, those requests could reach
> the
> > > > batch
> > > > > > > > > coordinator in different order and lead to rejection of the
> > > > produce
> > > > > > > > > requests.
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> ivan@ivanyu.me>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > We have also thought in a bit more details about
> transactions
> > > > and
> > > > > > > queues,
> > > > > > > > > > here's the plan.
> > > > > > > > > >
> > > > > > > > > > *Transactions*
> > > > > > > > > >
> > > > > > > > > > The support for transactions in *classic topics* is
> based on
> > > > precise
> > > > > > > > > > interactions between three actors: clients (mostly
> producers,
> > > > but
> > > > > > > also
> > > > > > > > > > consumers), brokers (ReplicaManager and other classes),
> and
> > > > > > > transaction
> > > > > > > > > > coordinators. Brokers also run partition leaders with
> their
> > > > local
> > > > > > > state
> > > > > > > > > > (ProducerStateManager and others).
> > > > > > > > > >
> > > > > > > > > > The high level (some details skipped) workflow is the
> > > > following.
> > > > > > > When a
> > > > > > > > > > transactional Produce request is received by the broker:
> > > > > > > > > > 1. For each partition, the partition leader checks if a
> > > > non-empty
> > > > > > > > > > transaction is running for this partition. This is done
> using
> > > > its
> > > > > > > local
> > > > > > > > > > state derived from the log metadata
> (ProducerStateManager,
> > > > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > > > 2. The transaction coordinator is informed about all the
> > > > partitions
> > > > > > > that
> > > > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > > > 3. The partition leaders do additional transactional
> checks.
> > > > > > > > > > 4. The partition leaders append the transactional data to
> > > > their logs
> > > > > > > and
> > > > > > > > > > update some of their state (for example, log the fact
> that the
> > > > > > > > > transaction
> > > > > > > > > > is running for the partition and its first offset).
> > > > > > > > > >
> > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > 1. The producer contacts the transaction coordinator
> directly
> > > > with
> > > > > > > > > > EndTxnRequest.
> > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > > PREPARE_ABORT to
> > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > 3. The transaction coordinator sends
> WriteTxnMarkersRequest to
> > > > the
> > > > > > > > > leaders
> > > > > > > > > > of the involved partitions.
> > > > > > > > > > 4. The partition leaders write the transaction markers to
> > > > their logs
> > > > > > > and
> > > > > > > > > > respond to the coordinator.
> > > > > > > > > > 5. The coordinator writes the final transaction state
> > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > >
> > > > > > > > > > In classic topics, partitions have leaders and lots of
> > > > important
> > > > > > > state
> > > > > > > > > > necessary for supporting this workflow is local. The main
> > > > challenge
> > > > > > > in
> > > > > > > > > > mapping this to Diskless comes from the fact there are no
> > > > partition
> > > > > > > > > > leaders, so the corresponding pieces of state need to be
> > > > globalized
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > batch coordinator. We are already doing this to support
> > > > idempotent
> > > > > > > > > produce.
> > > > > > > > > >
> > > > > > > > > > The high level workflow for *diskless topics* would look
> very
> > > > > > > similar:
> > > > > > > > > > 1. For each partition, the broker checks if a non-empty
> > > > transaction
> > > > > > > is
> > > > > > > > > > running for this partition. In contrast to classic
> topics,
> > > > this is
> > > > > > > > > checked
> > > > > > > > > > against the batch coordinator with a single RPC. Since a
> > > > transaction
> > > > > > > > > could
> > > > > > > > > > be uniquely identified with producer ID and epoch, the
> positive
> > > > > > > result of
> > > > > > > > > > this check could be cached locally (for the double
> configured
> > > > > > > duration
> > > > > > > > > of a
> > > > > > > > > > transaction, for example).
> > > > > > > > > > 2. The same: The transaction coordinator is informed
> about all
> > > > the
> > > > > > > > > > partitions that aren’t part of the transaction to
> include them.
> > > > > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > > > > 4. The broker appends the transactional data to the
> current
> > > > shared
> > > > > > > WAL
> > > > > > > > > > segment. It doesn’t update any transaction-related state
> for
> > > > Diskless
> > > > > > > > > > topics, because it doesn’t have any.
> > > > > > > > > > 5. The WAL segment is committed to the batch coordinator
> like
> > > > in the
> > > > > > > > > > normal produce flow.
> > > > > > > > > > 6. The batch coordinator does the final transactional
> checks
> > > > of the
> > > > > > > > > > batches. This procedure would output the same errors
> like the
> > > > > > > partition
> > > > > > > > > > leader in classic topics would do. I.e. some batches
> could be
> > > > > > > rejected.
> > > > > > > > > > This means, there will potentially be garbage in the WAL
> > > > segment
> > > > > > > file in
> > > > > > > > > > case of transactional errors. This is preferable to
> doing more
> > > > > > > network
> > > > > > > > > > round trips, especially considering the WAL segments
> will be
> > > > > > > relatively
> > > > > > > > > > short-living (see the Greg's update above).
> > > > > > > > > >
> > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > 1. The producer contacts the transaction coordinator
> directly
> > > > with
> > > > > > > > > > EndTxnRequest.
> > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > > PREPARE_ABORT to
> > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> > > > coordinator
> > > > > > > that
> > > > > > > > > > the transaction is finished.
> > > > > > > > > > 4. *[NEW]* The batch coordinator saves that the
> transaction is
> > > > > > > finished
> > > > > > > > > > and also inserts the control batches in the
> corresponding logs
> > > > of the
> > > > > > > > > > involved Diskless topics. This happens only on the
> metadata
> > > > level, no
> > > > > > > > > > actual control batches are written to any file. They
> will be
> > > > > > > dynamically
> > > > > > > > > > created on Fetch and other read operations. We could
> > > > technically
> > > > > > > write
> > > > > > > > > > these control batches for real, but this would mean extra
> > > > produce
> > > > > > > > > latency,
> > > > > > > > > > so it's better just to mark them in the batch
> coordinator and
> > > > save
> > > > > > > these
> > > > > > > > > > milliseconds.
> > > > > > > > > > 5. The transaction coordinator sends
> WriteTxnMarkersRequest to
> > > > the
> > > > > > > > > leaders
> > > > > > > > > > of the involved partitions. – Now only to classic topics
> now.
> > > > > > > > > > 6. The partition leaders of classic topics write the
> > > > transaction
> > > > > > > markers
> > > > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > > > 7. The coordinator writes the final transaction state
> > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > >
> > > > > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > > > > 1. An extra network round trip between brokers and the
> batch
> > > > > > > coordinator
> > > > > > > > > > when a new partition appear in the transaction. To
> mitigate the
> > > > > > > impact of
> > > > > > > > > > them:
> > > > > > > > > >  - The results will be cached.
> > > > > > > > > >  - The calls for multiple partitions in one Produce
> request
> > > > will be
> > > > > > > > > > grouped.
> > > > > > > > > >  - The batch coordinator should be optimized for fast
> > > > response to
> > > > > > > these
> > > > > > > > > > RPCs.
> > > > > > > > > >  - The fact that a single producer normally will
> communicate
> > > > with a
> > > > > > > > > > single broker for the duration of the transaction further
> > > > reduces the
> > > > > > > > > > expected number of round trips.
> > > > > > > > > > 2. An extra round trip between the transaction
> coordinator and
> > > > batch
> > > > > > > > > > coordinator when a transaction is finished.
> > > > > > > > > >
> > > > > > > > > > With this proposal, transactions will also be able to
> span both
> > > > > > > classic
> > > > > > > > > > and Diskless topics.
> > > > > > > > > >
> > > > > > > > > > *Queues*
> > > > > > > > > >
> > > > > > > > > > The share group coordination and management is a side
> job that
> > > > > > > doesn't
> > > > > > > > > > interfere with the topic itself (leadership, replicas,
> physical
> > > > > > > storage
> > > > > > > > > of
> > > > > > > > > > records, etc.) and non-queue producers and consumers
> (Fetch and
> > > > > > > Produce
> > > > > > > > > > RPCs, consumer group-related RPCs are not affected.) We
> don't
> > > > see any
> > > > > > > > > > reason why we can't make Diskless topics compatible with
> share
> > > > > > > groups the
> > > > > > > > > > same way as classic topics are. Even on the code level,
> we
> > > > don't
> > > > > > > expect
> > > > > > > > > any
> > > > > > > > > > serious refactoring: the same reading routines are used
> that
> > > > are
> > > > > > > used for
> > > > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > > > > implementation-focused?
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Ivan
> > > > > > > > > >
> > > > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Thank you all for your questions and design input on
> > > > KIP-1150.
> > > > > > > > > > >
> > > > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> > > > design. To
> > > > > > > > > > summarize
> > > > > > > > > > > the changes:
> > > > > > > > > > >
> > > > > > > > > > > 1. The design prioritizes integrating with the existing
> > > > KIP-405
> > > > > > > Tiered
> > > > > > > > > > > Storage interfaces, permitting data produced to a
> Diskless
> > > > topic
> > > > > > > to be
> > > > > > > > > > > moved to tiered storage.
> > > > > > > > > > > This lowers the scalability requirements for the Batch
> > > > Coordinator
> > > > > > > > > > > component, and allows Diskless to compose with Tiered
> Storage
> > > > > > > plugin
> > > > > > > > > > > features such as encryption and alternative data
> formats.
> > > > > > > > > > >
> > > > > > > > > > > 2. Consumer fetches are now served from local segments,
> > > > making use
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > functionality
> > > > > > > > > > already
> > > > > > > > > > > built into classic topics.
> > > > > > > > > > > However, local segments are now considered cache
> elements,
> > > > do not
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > be durably stored, and can be built without contacting
> any
> > > > other
> > > > > > > > > > replicas.
> > > > > > > > > > >
> > > > > > > > > > > 3. The design has been simplified substantially, by
> removing
> > > > the
> > > > > > > > > previous
> > > > > > > > > > > Diskless consume flow, distributed cache component, and
> > > > "object
> > > > > > > > > > > compaction/merging" step.
> > > > > > > > > > >
> > > > > > > > > > > The design maintains leaderless produces as enabled by
> the
> > > > Batch
> > > > > > > > > > > Coordinator, and the same latency profiles as the
> earlier
> > > > design,
> > > > > > > while
> > > > > > > > > > > being simpler and integrating better into the existing
> > > > ecosystem.
> > > > > > > > > > >
> > > > > > > > > > > Thanks, and we are eager to hear your feedback on the
> new
> > > > design.
> > > > > > > > > > > Greg Harris
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > <ju...@confluent.io.invalid>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Jan,
> > > > > > > > > > > >
> > > > > > > > > > > > For me, the main gap of KIP-1150 is the support of
> all
> > > > existing
> > > > > > > > > client
> > > > > > > > > > > > APIs. Currently, there is no design for supporting
> APIs
> > > > like
> > > > > > > > > > transactions
> > > > > > > > > > > > and queues.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Would it be a good time to ask for the current
> status of
> > > > this
> > > > > > > KIP?
> > > > > > > > > I
> > > > > > > > > > > > > haven't seen much activity here for the past 2
> months,
> > > > the
> > > > > > > vote got
> > > > > > > > > > > > vetoed
> > > > > > > > > > > > > but I think the pending questions have been
> answered
> > > > since
> > > > > > > then.
> > > > > > > > > > KIP-1183
> > > > > > > > > > > > > (AutoMQ's proposal) also didn't have any activity
> since
> > > > May.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real
> choices
> > > > that can
> > > > > > > be
> > > > > > > > > > > > > made, with a coordinator-based approach being by
> far the
> > > > > > > dominant
> > > > > > > > > one
> > > > > > > > > > > > when
> > > > > > > > > > > > > it comes to market adoption - but all these are
> > > > standalone
> > > > > > > > > products.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm a big fan of both approaches, but would hate
> to see a
> > > > > > > stall. So
> > > > > > > > > > the
> > > > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Maybe it's time to start another vote? Colin
> McCabe -
> > > > have your
> > > > > > > > > > questions
> > > > > > > > > > > > > been answered? If not, is there anything I can do
> to
> > > > help? I'm
> > > > > > > > > deeply
> > > > > > > > > > > > > familiar with both architectures and have written
> about
> > > > both?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > Jan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> Kozlovski <
> > > > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > a) group all the KIP email threads in the main
> one
> > > > (just a
> > > > > > > bunch
> > > > > > > > > of
> > > > > > > > > > > > links
> > > > > > > > > > > > > > to everything)
> > > > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It's a bit hard to track it all - for example, I
> was
> > > > > > > searching
> > > > > > > > > for
> > > > > > > > > > a
> > > > > > > > > > > > > > discuss thread for KIP-1165 for a while; As far
> as I
> > > > can
> > > > > > > tell, it
> > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > exist yet.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Since the KIPs are published (by virtue of
> having the
> > > > root
> > > > > > > KIP be
> > > > > > > > > > > > > > published, having a DISCUSS thread and links to
> > > > sub-KIPs
> > > > > > > where
> > > > > > > > > were
> > > > > > > > > > > > aimed
> > > > > > > > > > > > > > to move the discussion towards), I think it
> would be
> > > > good to
> > > > > > > > > create
> > > > > > > > > > > > > DISCUSS
> > > > > > > > > > > > > > threads for them all.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Stan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We want to start a new KIP discussion about
> > > > introducing a
> > > > > > > new
> > > > > > > > > > type of
> > > > > > > > > > > > > > > topics that would make use of Object Storage
> as the
> > > > primary
> > > > > > > > > > source of
> > > > > > > > > > > > > > > storage. However, as this KIP is big we
> decided to
> > > > split it
> > > > > > > > > into
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > that aims to discuss if Apache Kafka should
> aim to
> > > > have
> > > > > > > this
> > > > > > > > > > type of
> > > > > > > > > > > > > > > feature at all. This KIP doesn't go onto
> details on
> > > > how to
> > > > > > > > > > implement
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > This follows the same approach used when we
> discussed
> > > > > > > KRaft.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > But as we know that it is sometimes really
> hard to
> > > > discuss
> > > > > > > on
> > > > > > > > > > that
> > > > > > > > > > > > meta
> > > > > > > > > > > > > > > level, we also created several sub-kips
> (linked in
> > > > > > > KIP-1150)
> > > > > > > > > that
> > > > > > > > > > > > offer
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS
> threads
> > > > for
> > > > > > > each
> > > > > > > > > > type of
> > > > > > > > > > > > > > > concern and keep this one to discuss whether
> Apache
> > > > Kafka
> > > > > > > wants
> > > > > > > > > > to
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks in advance on behalf of all the authors
> of
> > > > this KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > > > josep.prat@aiven.io  |  +491715557497 |
> aiven.io
> > > > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu
> Valtonen,
> > > > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Henry,

Indeed, there will be some additional files and descriptors, but as I mentioned in my previous message, we don't think there will be a lot more compared to the classic Kafka partitions as baseline. We will not cache every batch or other small amount in a separate file on disk. There will normal Kafka segments (likely in a separate directory as you said) to append these batches sequentially. Only recent data will be cached this way, so this cache will not grow (neither in disk usage nor in the number of file descriptors) proportionally to the total partition size. 

You're right that this caching/log materialization part is indeed a little underspecified at the moment. We're sorry for this, but as an excuse, we're trying to fix the most fundamental parts of the design first, such as the approach to the batch coordinator. Local log materialization is mostly independent of the coordinator design and may be discussed down the road.

Best,
Ivan

On Thu, Oct 16, 2025, at 01:42, Henry Haiying Cai wrote:
> Ivan,
> For the answer on how many data/meta-data partitions the diskless KIP needs to create (hence the number of open file descriptors), there seems to be a missing gap.
> In the current Kafka design, for each topic partition, there is a corresponding log directory and a sequence of log segment files on the broker disk for persistence.  For example, /mnt/kafka/partitions/topic0-0/00000.log for topic 0 partition 0 (I used /mnt/kafka/partitions as the root log directory name for illustration purpose since that is the directory name in our system).  After user enabled topic0 to be diskless, the content in the log data file now contains the pointers to WAL log segment file in S3 (the metadata) so you are re-using the file descriptors in /mnt/kafka/partitions, but what about the real data content for topic0-0?  I think the KIP is proposing uploading those data as part of the WAL log segment onto S3, but there also needs to be a local cached copy of data content for topic0-0 on both the leader broker as well as the follower brokers for consumers to read from.
> The KIP did not mention how that cache is built or organized or whether it is disk backed.  I remember in the earlier replies you mentioned you are going to reuse Kafka's existing page cache system for the data content cache, but if you are reusing Kafka's page cache, the data you downloaded from S3 or received from produceRequest for topic0-0 will needs to be stored in /mnt/kafka/partitions/topic0-0 (or maybe a new directory such as /mnt/kafka/partitions/data-topic0-0 to avoid the name conflict), but if you do this mapping for 100 original data partitions you will end up with 100 more new partition directories and file descriptors on the disk.
> Also for the followers to build that cache, there will be temporary file and file descriptors being created for the S3 download, and those will be new file descriptors being created constantly for each data segment you are downloading from S3.
>     On Tuesday, October 14, 2025 at 11:03:46 AM PDT, Ivan Yurchenko <iv...@ivanyu.me> wrote:  
>  
> Hi Jun,
> 
> Thank you for your message. I’m sorry that I failed to clearly explain the idea. Let me try to fix this.
> 
> > Does each partition now have a metadata partition and a separate data
> > partition? If so, I am concerned that it essentially doubles the number of
> > partitions, which impacts the number of open file descriptors and the
> > required IOPS, and so on. It also seems wasteful to have a separate
> > partition just to store the metadata. It's as if we are creating an
> > internal topic with an unbounded number of partitions.
> 
> No. There will be only one physical partition per diskless partition. Let me explain this with an example. Let’s say we have a diskless partition topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce some batches to this partition. The content of the segment file will be something like this (for each batch):
> 
> BaseOffset: 00000000000000000000 (like in classic)
> Length: 123456 (like in classic)
> PartitionLeaderEpoch: like in classic
> Magic: like in classic
> CRC: like in classic
> Attributes: like in classic
> LastOffsetDelta: like in classic
> BaseTimestamp: like in classic
> MaxTimestamp: like in classic
> ProducerId: like in classic
> ProducerEpoch: like in classic
> BaseSequence: like in classic
> RecordsCount: like in classic
> Records:
>   path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset 123456
> 
> It looks very much like classic log entries. The only difference is that instead of writing real Records, we write the reference to the WAL file with the batch data (I guess we need only the name and the byte offset, because the byte length is the standard field above). Otherwise, it’s a normal Kafka log with the leader and replicas. 
> 
> So we have as many partitions for diskless as for classic. As of open file descriptors, let’s proceed to the following:
> 
> > Are the metadata and
> > the data for the same partition always collocated on the same broker? If
> > so, how do we enforce that when replicas are reassigned?
> 
> The source of truth for the data is still in WAL files on object storage. The source of truth for the metadata is in segment files on the brokers in the replica set. Two new mechanisms are planned, both independent of this new proposal, but I want to present them to give the idea that only a limited amount of data files will be operated locally:
> - We want to assemble batches into segment files and offload them to tiered storage in order to prevent the unbounded growth of batch metadata. For this, we need to open only  a few file descriptors (for the segment file itself + the necessary indexes) before the segment is fully written and handed over to RemoteLogManager.
> - We want to assemble local segment files for caching purposes as well, i.e. to speed up fetching. This will not materialize the full content of the log, but only the hot set according to some policy (or configurable policies), i.e. the number of segments and file descriptors will also be limited.
> 
> > The number of RPCs in the produce path is significantly higher. For
> > example, if a produce request has 100 partitions, in a cluster with 100
> > brokers, each produce request could generate 100 more RPC requests. This
> > will significantly increase the request rate.
> 
> This is a valid concern that we considered, but this issue can be mitigated. I’ll try to explain the approach.
> The situation with a single broker is trivial: all the commit requests go from the broker to itself. 
> Let’s scale this to a multi-broker cluster, but located in the single rack (AZ). Any broker can accept Produce requests for diskless partitions, but we can tell producers (through metadata) to always send Produce requests to leaders. For example, broker 0 hosts the leader replicas for diskless partitions t1-0, t2-1, t3-0. It will receive diskless Produce requests for these partitions in various combinations, but only for them.
> 
>                     Broker 0
>               +-----------------+
>               |    t1-0        |
>               |    t2-1 <--------------------+
>               |    t3-0        |            |
> produce      | +-------------+ |            |
> requests    | |  diskless  | |            |
> --------------->|  produce  +--------------+
> for these    | | WAL buffer  | |    commit requests
> partitions  | +-------------+ |    for these partitions
>               |                |
>               +-----------------+
> 
> The same applies for other brokers in this cluster. Effectively, each broker will commit only to itself, which effectively means 1 commit request per WAL buffer (this may be 0 physical network calls, if we wish, just a local function call).
> 
> Now let’s scale this to multiple racks (AZs). Obviously, we cannot always send Produce requests to the designated leaders of diskless partitions: this would mean inter-AZ network traffic, which we would like to avoid. To avoid it, we say that every broker has a “diskless produce representative” in every AZ. If we continue our example: when a Produce request for t1-0, t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0 (in the broker’s AZ the representative is the broker itself). However, if it comes from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2. 
> 
>   |produce requests        |produce requests        |produce requests
>   |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0
>   |from AZ 0                |from AZ 1              |from AZ 2
>   v                        v                        v
> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> +---------------+      +---------------+      +---------------+
> |    t1-0      |      |              |      |              |
> |    t2-1      |      |              |      |              |
> |    t3-0      |      |              |      |              |
> +---------------+      +--------+------+      +--------+------+
>     ^    ^                    |                      |
>     |    +--------------------+                      |
>     |    commit requests for these partitions        |
>     |                                                |
>     +-------------------------------------------------+
>           commit requests for these partitions
> 
> All the partitions that broker 0 is the leader of will be “represented” by brokers 1 and 2 in their AZs.
> 
> Of course, this relationship goes both ways between AZs (not necessarily between the same brokers). It means that provided the cluster is balanced by the number of brokers per AZ, each broker will represent (number_of_azs - 1) other brokers. This will result in the situation that for the majority of commits, each broker will do up to (number_of_azs - 1) network commit requests (plus one local). Cloud regions tend to have 3 AZs, very rarely more. That means, brokers will be doing up to 2 network commit requests per WAL file.
> 
> There are the following exceptions:
> 1. Broker count imbalance between AZs. For example, when we have 2 AZs and one has three brokers and another AZ has one. This one broker will do between 1 and 3 commit requests per WAL file. This is not an extreme amplification. Such an imbalance is not healthy in most practical setups and should be avoided anyway.
> 2. Leadership changes and metadata propagation period. When the partition t3-0 is relocated from broker 0 to some broker 3, the producers will not know this immediately (unless we want to be strict and respond with NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come together in a WAL buffer on broker 2, it will have to send two commit requests: to broker 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This situation is not permanent and as producers update the cluster metadata, it will be resolved.
> 
> This all could be built with the metadata crafting mechanism only (which is anyway needed for Diskless in one way or another to direct producers and consumers where we need to avoid inter-AZ traffic), just with the right policy for it (for example, some deterministic hash-based formula). I.e. no explicit support for “produce representative” or anything like this is needed on the cluster level, in KRaft, etc.
> 
> > The same WAL file metadata is now duplicated into two places, partition
> > leader and WAL File Manager. Which one is the source of truth, and how do
> > we maintain consistency between the two places?
> 
> We do only two operations on WAL files that span multiple diskless partitions: committing and deleting. Commits can be done independently as described above. But deletes are different, because when a file is deleted, this affects all the partitions that still have alive batches in this file (if any).
> 
> The WAL file manager is a necessary point of coordination to delete WAL files safely. We can say it is the source of truth about files themselves, while the partition leaders and their logs hold the truth about whether a particular file contains live batches of this particular partition.
> 
> The file manager will do this important task: be able to say for sure that a file does not contain any live batch of any existing partition. For this, it will have to periodically check against the partition leaders. Considering that batch deletion is irreversible, when we declare a file “empty”, this is guaranteed to be and stay so.
> 
> The file manager has to know about files being committed to start track them and periodically check if they are empty. We can consider various ways to achieve this:
> 1. As was proposed in my previous message: best effort commit by brokers + periodic prefix scans of object storage to detect files that went below the radar due to network issue or the file manager temporary unavailability. We’re speaking about listing the file names only and opening only previously unknown files in order to find the partitions involved with them.
> 2. Only do scans without explicit commit, i.e. fill the list of files fully asynchronously and in the background. This may be not ideal due to costs and performance of scanning tons of files. However, the number of live WAL files should be limited due to tiered storage offloading + we can optimize this if we give files some global soft order in their names.
> 
> > I am not sure how this design simplifies the implementation. The existing
> > producer/replication code can't be simply reused. Adjusting both the write
> > path in the leader and the replication path in the follower to understand
> > batch-header only data is quite intrusive to the existing logic.
> 
> It is true that we’ll have to change LocalLog and UnifiedLog in order to support these changes. However, it seems that idempotence, transactions, queues, tiered storage will have to be changed less than with the original design. This is because the partition leader state would remain in the same place (on brokers) and existing workflows that involve it would have to be changed less compared to the situation where we globalize the partition leader state in the batch coordinator. I admit this is hard to make convincing without both real implementations to hand :) 
> 
> > I am also
> > not sure how this enables seamless switching the topic modes between
> > diskless and classic. Could you provide more details on those?
> 
> Let’s consider the scenario of turning a classic topic into diskless. The user sets diskless.enabled=true, the leader receives this metadata update and does the following:
> 1. Stop accepting normal append writes.
> 2. Close the current active segment.
> 3. Start a new segment that will be written in the diskless format (i.e. without data).
> 4. Start accepting diskless commits.
> 
> Since it’s the same log, the followers will know about that switch consistently. They will finish replicating the classic segments and start replicating the diskless ones. They will always know where each batch is located (either inside a classic segment or referenced by a diskless one). Switching back should be similar.
> 
> Doing this with the coordinator is possible, but has some caveats. The leader must do the following:
> 1. Stop accepting normal append writes.
> 2. Close the current active segment.
> 3. Write a special control segment to persist and replicate the fact that from offset N the partition is now in the diskless mode.
> 4. Inform the coordinator about the first offset N of the “diskless era”.
> 5. Inform the controller quorum that the transition has finished and that brokers now can process diskless writes for this partition.
> This could fail at some points, so this will probably require some explicit state machine with replication either in the partition log or in KRaft.
> 
> It seems that the coordinator-less approach makes this simpler because the “coordinator” for the partition and the partition leader are the same and they store the partition metadata in the same log, too. While in the coordinator approach we have to perform some kind of a distributed commit to handover metadata management from the classic partition leader to the batch coordinator.
> 
> I hope these explanations help to clarify the idea. Please let me know if I should go deeper anywhere.
> 
> Best,
> Ivan and the Diskless team
> 
> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > Hi, Ivan,
> > 
> > Thanks for the update.
> > 
> > I am not sure that I fully understand the new design, but it seems less
> > clean than before.
> > 
> > Does each partition now have a metadata partition and a separate data
> > partition? If so, I am concerned that it essentially doubles the number of
> > partitions, which impacts the number of open file descriptors and the
> > required IOPS, and so on. It also seems wasteful to have a separate
> > partition just to store the metadata. It's as if we are creating an
> > internal topic with an unbounded number of partitions. Are the metadata and
> > the data for the same partition always collocated on the same broker? If
> > so, how do we enforce that when replicas are reassigned?
> > 
> > The number of RPCs in the produce path is significantly higher. For
> > example, if a produce request has 100 partitions, in a cluster with 100
> > brokers, each produce request could generate 100 more RPC requests. This
> > will significantly increase the request rate.
> > 
> > The same WAL file metadata is now duplicated into two places, partition
> > leader and WAL File Manager. Which one is the source of truth, and how do
> > we maintain consistency between the two places?
> > 
> > I am not sure how this design simplifies the implementation. The existing
> > producer/replication code can't be simply reused. Adjusting both the write
> > path in the leader and the replication path in the follower to understand
> > batch-header only data is quite intrusive to the existing logic. I am also
> > not sure how this enables seamless switching the topic modes between
> > diskless and classic. Could you provide more details on those?
> > 
> > Jun
> > 
> > On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > 
> > > Hi dear Kafka community,
> > >
> > > In the initial Diskless proposal, we proposed to have a separate
> > > component, batch/diskless coordinator, whose role would be to centrally
> > > manage the batch and WAL file metadata for diskless topics. This component
> > > drew many reasonable comments from the community about how it would support
> > > various Kafka features (transactions, queues) and its scalability. While we
> > > believe we have good answers to all the expressed concerns, we took a step
> > > back and looked at the problem from a different perspective.
> > >
> > > We would like to propose an alternative Diskless design *without a
> > > centralized coordinator*. We believe this approach has potential and
> > > propose to discuss it as it may be more appealing to the community.
> > >
> > > Let us explain the idea. Most of the complications with the original
> > > Diskless approach come from one necessary architecture change: globalizing
> > > the local state of partition leader in the batch coordinator. This causes
> > > deviations to the established workflows in various features like produce
> > > idempotence and transactions, queues, retention, etc. These deviations need
> > > to be carefully considered, designed, and later implemented and tested. In
> > > the new approach we want to avoid this by making partition leaders again
> > > responsible for managing their partitions, even in diskless topics.
> > >
> > > In classic Kafka topics, batch data and metadata are blended together in
> > > the one partition log. The crux of the Diskless idea is to decouple them
> > > and move data to the remote storage, while keeping metadata somewhere else.
> > > Using the central batch coordinator for managing batch metadata is one way,
> > > but not the only.
> > >
> > > Let’s now think about managing metadata for each user partition
> > > independently. Generally partitions are independent and don’t share
> > > anything apart from that their data are mixed in WAL files. If we figure
> > > out how to commit and later delete WAL files safely, we will achieve the
> > > necessary autonomy that allows us to get rid of the central batch
> > > coordinator. Instead, *each diskless user partition will be managed by its
> > > leader*, as in classic Kafka topics. Also like in classic topics, the
> > > leader uses the partition log as the way to persist batch metadata, i.e.
> > > the regular batch header + the information about how to find this batch on
> > > remote storage. In contrast to classic topics, batch data is in remote
> > > storage.
> > >
> > > For clarity, let’s compare the three designs:
> > >  • Classic topics:
> > >    • Data and metadata are co-located in the partition log.
> > >    • The partition log content: [Batch header (metadata)|Batch data].
> > >    • The partition log is replicated to the followers.
> > >    • The replicas and leader have local state built from metadata.
> > >  • Original Diskless:
> > >    • Metadata is in the batch coordinator, data is on remote storage.
> > >    • The partition state is global in the batch coordinator.
> > >  • New Diskless:
> > >    • Metadata is in the partition log, data is on remote storage.
> > >    • Partition log content: [Batch header (metadata)|Batch coordinates on
> > > remote storage].
> > >    • The partition log is replicated to the followers.
> > >    • The replicas and leader have local state built from metadata.
> > >
> > > Let’s consider the produce path. Here’s the reminder of the original
> > > Diskless design:
> > >
> > >
> > > The new approach could be depicted as the following:
> > >
> > >
> > > As you can see, the main difference is that now instead of a single commit
> > > request to the batch coordinator, we send multiple parallel commit requests
> > > to all the leaders of each partition involved in the WAL file. Each of them
> > > will commit its batches independently, without coordinating with other
> > > leaders and any other components. Batch data is addressed by the WAL file
> > > name, the byte offset and size, which allows partitions to know nothing
> > > about other partitions to access their data in shared WAL files.
> > >
> > > The number of partitions involved in a single WAL file may be quite large,
> > > e.g. a hundred. A hundred network requests to commit one WAL file is very
> > > impractical. However, there are ways to reduce this number:
> > >  1. Partition leaders are located on brokers. Requests to leaders on one
> > > broker could be grouped together into a single physical network request
> > > (resembling the normal Produce request that may carry batches for many
> > > partitions inside). This will cap the number of network requests to the
> > > number of brokers in the cluster.
> > >  2. If we craft the cluster metadata to make producers send their requests
> > > to the right brokers (with respect to AZs), we may achieve the higher
> > > concentration of logical commit requests in physical network requests
> > > reducing the number of the latter ones even further, ideally to one.
> > >
> > > Obviously, out of multiple commit requests some may fail or time out for a
> > > variety of reasons. This is fine. Some producers will receive totally or
> > > partially failed responses to their Produce requests, similar to what they
> > > would have received when appending to a classic topic fails or times out.
> > > If a partition experiences problems, other partitions will not be affected
> > > (again, like in classic topics). Of course, the uncommitted data will be
> > > garbage in WAL files. But WAL files are short-lived (batches are constantly
> > > assembled into segments and offloaded to tiered storage), so this garbage
> > > will be eventually deleted.
> > >
> > > For safely deleting WAL files we now need to centrally manage them, as
> > > this is the only state and logic that spans multiple partitions. On the
> > > diagram, you can see another commit request called “Commit file (best
> > > effort)” going to the WAL File Manager. This manager will be responsible
> > > for the following:
> > >  1. Collecting (by requests from brokers) and persisting information about
> > > committed WAL files.
> > >  2. To handle potential failures in file information delivery, it will be
> > > doing prefix scan on the remote storage periodically to find and register
> > > unknown files. The period of this scan will be configurable and ideally
> > > should be quite long.
> > >  3. Checking with the relevant partition leaders (after a grace period) if
> > > they still have batches in a particular file.
> > >  4. Physically deleting files when they aren’t anymore referred to by any
> > > partition.
> > >
> > > This new design offers the following advantages:
> > >  1. It simplifies the implementation of many Kafka features such as
> > > idempotence, transactions, queues, tiered storage, retention. Now we don’t
> > > need to abstract away and reuse the code from partition leaders in the
> > > batch coordinator. Instead, we will literally use the same code paths in
> > > leaders, with little adaptation. Workflows from classic topics mostly
> > > remain unchanged.
> > > For example, it seems that
> > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > > KafkaApis.handleWriteTxnMarkersRequest used for transaction support on the
> > > partition leader side could be used for diskless topics with little
> > > adaptation. ProducerStateManager, needed for both idempotent produce and
> > > transactions, would be reused.
> > > Another example is share groups support, where the share partition leader,
> > > being co-located with the partition leader, would execute the same logic
> > > for both diskless and classic topics.
> > >  2. It returns to the familiar partition-based scaling model, where
> > > partitions are independent.
> > >  3. It makes the operation and failure patterns closer to the familiar
> > > ones from classic topics.
> > >  4. It opens a straightforward path to seamless switching the topics modes
> > > between diskless and classic.
> > >
> > > The rest of the things remain unchanged compared to the previous Diskless
> > > design (after all previous discussions). Such things as local segment
> > > materialization by replicas, the consume path, tiered storage integration,
> > > etc.
> > >
> > > If the community finds this design more suitable, we will update the
> > > KIP(s) accordingly and continue working on it. Please let us know what you
> > > think.
> > >
> > > Best regards,
> > > Ivan and Diskless team
> > >
> > > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > Hi Justine,
> > > >
> > > > Yes, you're right. We need to track the aborted transactions for in the
> > > diskless coordinator for as long as the corresponding offsets are there.
> > > With the tiered storage unification Greg mentioned earlier, this will be
> > > finite time even for infinite data retention.
> > > >
> > > > Best,
> > > > Ivan
> > > >
> > > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > Hey Ivan,
> > > > >
> > > > > Thanks for the response. I think most of what you said made sense, but
> > > I
> > > > > did have some questions about this part:
> > > > >
> > > > > > As we understand this, the partition leader in classic topics forgets
> > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > transaction coordinator acts like the main guardian, allowing partition
> > > > > leaders to do this safely. Please correct me if this is wrong. We think
> > > > > about relying on this with the batch coordinator and delete the
> > > information
> > > > > about a transaction once it’s finished (as there’s no replication and
> > > HWM
> > > > > advances immediately).
> > > > >
> > > > > I didn't quite understand this. In classic topics, we have maps for
> > > ongoing
> > > > > transactions which remove state when the transaction is completed and
> > > an
> > > > > aborted transactions index which is retained for much longer. Once the
> > > > > transaction is completed, the coordinator is no longer involved in
> > > > > maintaining this partition side state, and it is subject to compaction
> > > etc.
> > > > > Looking back at the outline provided above, I didn't see much about the
> > > > > fetch path, so maybe that could be expanded a bit further. I saw the
> > > > > following in a response:
> > > > > > When the broker constructs a fully valid local segment, all the
> > > necessary
> > > > > control batches will be inserted and indices, including the transaction
> > > > > index will be built to serve FetchRequests exactly as they are today.
> > > > >
> > > > > Based on this, it seems like we need to retain the information about
> > > > > aborted txns for longer.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > > >
> > > > > > Hi Justine and all,
> > > > > >
> > > > > > Thank you for your questions!
> > > > > >
> > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > producer ID
> > > > > > > and epoch, the positive result of this check could be cached
> > > locally
> > > > > > > Are we saying that only new transaction version 2 transactions can
> > > be
> > > > > > used
> > > > > > > here? If not, we can't uniquely identify transactions with
> > > producer id +
> > > > > > > epoch
> > > > > >
> > > > > > You’re right that we (probably unintentionally) focused only on
> > > version 2.
> > > > > > We can either limit the support to version 2 or consider using some
> > > > > > surrogates to support version 1.
> > > > > >
> > > > > > > JO 2. >The batch coordinator does the final transactional checks
> > > of the
> > > > > > > batches. This procedure would output the same errors like the
> > > partition
> > > > > > > leader in classic topics would do.
> > > > > > > Can you expand on what these checks are? Would you be checking if
> > > the
> > > > > > > transaction was still ongoing for example?* *
> > > > > >
> > > > > > Yes, the producer epoch, that the transaction is ongoing, and of
> > > course
> > > > > > the normal idempotence checks. What the partition leader in the
> > > classic
> > > > > > topics does before appending a batch to the local log (e.g. in
> > > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> > > unfortunately
> > > > > > cannot do these checks before appending the data to the WAL segment
> > > and
> > > > > > uploading it, but we can “tombstone” these batches in the batch
> > > coordinator
> > > > > > during the final commit.
> > > > > >
> > > > > > > Is there state about ongoing
> > > > > > > transactions in the batch coordinator? I see some other state
> > > mentioned
> > > > > > in
> > > > > > > the End transaction section, but it's not super clear what state is
> > > > > > stored
> > > > > > > and when it is stored.
> > > > > >
> > > > > > Right, this should have been more explicit. As the partition leader
> > > tracks
> > > > > > ongoing transactions for classic topics, the batch coordinator has
> > > to as
> > > > > > well. So when a transaction starts and ends, the transaction
> > > coordinator
> > > > > > must inform the batch coordinator about this.
> > > > > >
> > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> > > would
> > > > > > be
> > > > > > > stored in the batch coordinator?
> > > > > >
> > > > > > Yes. This could be deduced from the committed batches and other
> > > > > > information, but for the sake of performance we’d better store it
> > > > > > explicitly.
> > > > > >
> > > > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > > >
> > > > > > As we understand this, the partition leader in classic topics forgets
> > > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > > transaction coordinator acts like the main guardian, allowing
> > > partition
> > > > > > leaders to do this safely. Please correct me if this is wrong. We
> > > think
> > > > > > about relying on this with the batch coordinator and delete the
> > > information
> > > > > > about a transaction once it’s finished (as there’s no replication
> > > and HWM
> > > > > > advances immediately).
> > > > > >
> > > > > > Best,
> > > > > > Ivan
> > > > > >
> > > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > > Hey folks,
> > > > > > >
> > > > > > > Excited to see some updates related to transactions!
> > > > > > >
> > > > > > > I had a few questions.
> > > > > > >
> > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > producer ID
> > > > > > > and epoch, the positive result of this check could be cached
> > > locally
> > > > > > > Are we saying that only new transaction version 2 transactions can
> > > be
> > > > > > used
> > > > > > > here? If not, we can't uniquely identify transactions with
> > > producer id +
> > > > > > > epoch
> > > > > > >
> > > > > > > JO 2. >The batch coordinator does the final transactional checks
> > > of the
> > > > > > > batches. This procedure would output the same errors like the
> > > partition
> > > > > > > leader in classic topics would do.
> > > > > > > Can you expand on what these checks are? Would you be checking if
> > > the
> > > > > > > transaction was still ongoing for example? Is there state about
> > > ongoing
> > > > > > > transactions in the batch coordinator? I see some other state
> > > mentioned
> > > > > > in
> > > > > > > the End transaction section, but it's not super clear what state is
> > > > > > stored
> > > > > > > and when it is stored.
> > > > > > >
> > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> > > would
> > > > > > be
> > > > > > > stored in the batch coordinator?
> > > > > > >
> > > > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > > > >
> > > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Greg and Ivan,
> > > > > > > >
> > > > > > > > Thanks for the update. A few comments.
> > > > > > > >
> > > > > > > > JR 10. "Consumer fetches are now served from local segments,
> > > making
> > > > > > use of
> > > > > > > > the
> > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > functionality
> > > > > > already
> > > > > > > > built into classic topics."
> > > > > > > > JR 10.1 Does the broker build the producer state for each
> > > partition in
> > > > > > > > diskless topics?
> > > > > > > > JR 10.2 For transactional data, the consumer fetches need to know
> > > > > > aborted
> > > > > > > > records. How is that achieved?
> > > > > > > >
> > > > > > > > JR 11. "The batch coordinator saves that the transaction is
> > > finished
> > > > > > and
> > > > > > > > also inserts the control batches in the corresponding logs of the
> > > > > > involved
> > > > > > > > Diskless topics. This happens only on the metadata level, no
> > > actual
> > > > > > control
> > > > > > > > batches are written to any file. "
> > > > > > > > A fetch response could include multiple transactional batches.
> > > How
> > > > > > does the
> > > > > > > > broker obtain the information about the ending control batch for
> > > each
> > > > > > > > batch? Does that mean that a fetch response needs to be built by
> > > > > > > > stitching record batches and generated control batches together?
> > > > > > > >
> > > > > > > > JR 12. Queues: Is there still a share partition leader that all
> > > > > > consumers
> > > > > > > > are routed to?
> > > > > > > >
> > > > > > > > JR 13. "Should the KIPs be modified to include this or it's too
> > > > > > > > implementation-focused?" It would be useful to include enough
> > > details
> > > > > > to
> > > > > > > > understand correctness and performance impact.
> > > > > > > >
> > > > > > > > HC5. Henry has a valid point. Requests from a given producer
> > > contain a
> > > > > > > > sequence number, which is ordered. If a producer sends every
> > > Produce
> > > > > > > > request to an arbitrary broker, those requests could reach the
> > > batch
> > > > > > > > coordinator in different order and lead to rejection of the
> > > produce
> > > > > > > > requests.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > We have also thought in a bit more details about transactions
> > > and
> > > > > > queues,
> > > > > > > > > here's the plan.
> > > > > > > > >
> > > > > > > > > *Transactions*
> > > > > > > > >
> > > > > > > > > The support for transactions in *classic topics* is based on
> > > precise
> > > > > > > > > interactions between three actors: clients (mostly producers,
> > > but
> > > > > > also
> > > > > > > > > consumers), brokers (ReplicaManager and other classes), and
> > > > > > transaction
> > > > > > > > > coordinators. Brokers also run partition leaders with their
> > > local
> > > > > > state
> > > > > > > > > (ProducerStateManager and others).
> > > > > > > > >
> > > > > > > > > The high level (some details skipped) workflow is the
> > > following.
> > > > > > When a
> > > > > > > > > transactional Produce request is received by the broker:
> > > > > > > > > 1. For each partition, the partition leader checks if a
> > > non-empty
> > > > > > > > > transaction is running for this partition. This is done using
> > > its
> > > > > > local
> > > > > > > > > state derived from the log metadata (ProducerStateManager,
> > > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > > 2. The transaction coordinator is informed about all the
> > > partitions
> > > > > > that
> > > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > > 3. The partition leaders do additional transactional checks.
> > > > > > > > > 4. The partition leaders append the transactional data to
> > > their logs
> > > > > > and
> > > > > > > > > update some of their state (for example, log the fact that the
> > > > > > > > transaction
> > > > > > > > > is running for the partition and its first offset).
> > > > > > > > >
> > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > 1. The producer contacts the transaction coordinator directly
> > > with
> > > > > > > > > EndTxnRequest.
> > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > PREPARE_ABORT to
> > > > > > > > > its log and responds to the producer.
> > > > > > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to
> > > the
> > > > > > > > leaders
> > > > > > > > > of the involved partitions.
> > > > > > > > > 4. The partition leaders write the transaction markers to
> > > their logs
> > > > > > and
> > > > > > > > > respond to the coordinator.
> > > > > > > > > 5. The coordinator writes the final transaction state
> > > > > > COMPLETE_COMMIT or
> > > > > > > > > COMPLETE_ABORT.
> > > > > > > > >
> > > > > > > > > In classic topics, partitions have leaders and lots of
> > > important
> > > > > > state
> > > > > > > > > necessary for supporting this workflow is local. The main
> > > challenge
> > > > > > in
> > > > > > > > > mapping this to Diskless comes from the fact there are no
> > > partition
> > > > > > > > > leaders, so the corresponding pieces of state need to be
> > > globalized
> > > > > > in
> > > > > > > > the
> > > > > > > > > batch coordinator. We are already doing this to support
> > > idempotent
> > > > > > > > produce.
> > > > > > > > >
> > > > > > > > > The high level workflow for *diskless topics* would look very
> > > > > > similar:
> > > > > > > > > 1. For each partition, the broker checks if a non-empty
> > > transaction
> > > > > > is
> > > > > > > > > running for this partition. In contrast to classic topics,
> > > this is
> > > > > > > > checked
> > > > > > > > > against the batch coordinator with a single RPC. Since a
> > > transaction
> > > > > > > > could
> > > > > > > > > be uniquely identified with producer ID and epoch, the positive
> > > > > > result of
> > > > > > > > > this check could be cached locally (for the double configured
> > > > > > duration
> > > > > > > > of a
> > > > > > > > > transaction, for example).
> > > > > > > > > 2. The same: The transaction coordinator is informed about all
> > > the
> > > > > > > > > partitions that aren’t part of the transaction to include them.
> > > > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > > > 4. The broker appends the transactional data to the current
> > > shared
> > > > > > WAL
> > > > > > > > > segment. It doesn’t update any transaction-related state for
> > > Diskless
> > > > > > > > > topics, because it doesn’t have any.
> > > > > > > > > 5. The WAL segment is committed to the batch coordinator like
> > > in the
> > > > > > > > > normal produce flow.
> > > > > > > > > 6. The batch coordinator does the final transactional checks
> > > of the
> > > > > > > > > batches. This procedure would output the same errors like the
> > > > > > partition
> > > > > > > > > leader in classic topics would do. I.e. some batches could be
> > > > > > rejected.
> > > > > > > > > This means, there will potentially be garbage in the WAL
> > > segment
> > > > > > file in
> > > > > > > > > case of transactional errors. This is preferable to doing more
> > > > > > network
> > > > > > > > > round trips, especially considering the WAL segments will be
> > > > > > relatively
> > > > > > > > > short-living (see the Greg's update above).
> > > > > > > > >
> > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > 1. The producer contacts the transaction coordinator directly
> > > with
> > > > > > > > > EndTxnRequest.
> > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > PREPARE_ABORT to
> > > > > > > > > its log and responds to the producer.
> > > > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> > > coordinator
> > > > > > that
> > > > > > > > > the transaction is finished.
> > > > > > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> > > > > > finished
> > > > > > > > > and also inserts the control batches in the corresponding logs
> > > of the
> > > > > > > > > involved Diskless topics. This happens only on the metadata
> > > level, no
> > > > > > > > > actual control batches are written to any file. They will be
> > > > > > dynamically
> > > > > > > > > created on Fetch and other read operations. We could
> > > technically
> > > > > > write
> > > > > > > > > these control batches for real, but this would mean extra
> > > produce
> > > > > > > > latency,
> > > > > > > > > so it's better just to mark them in the batch coordinator and
> > > save
> > > > > > these
> > > > > > > > > milliseconds.
> > > > > > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to
> > > the
> > > > > > > > leaders
> > > > > > > > > of the involved partitions. – Now only to classic topics now.
> > > > > > > > > 6. The partition leaders of classic topics write the
> > > transaction
> > > > > > markers
> > > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > > 7. The coordinator writes the final transaction state
> > > > > > COMPLETE_COMMIT or
> > > > > > > > > COMPLETE_ABORT.
> > > > > > > > >
> > > > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > > > 1. An extra network round trip between brokers and the batch
> > > > > > coordinator
> > > > > > > > > when a new partition appear in the transaction. To mitigate the
> > > > > > impact of
> > > > > > > > > them:
> > > > > > > > >  - The results will be cached.
> > > > > > > > >  - The calls for multiple partitions in one Produce request
> > > will be
> > > > > > > > > grouped.
> > > > > > > > >  - The batch coordinator should be optimized for fast
> > > response to
> > > > > > these
> > > > > > > > > RPCs.
> > > > > > > > >  - The fact that a single producer normally will communicate
> > > with a
> > > > > > > > > single broker for the duration of the transaction further
> > > reduces the
> > > > > > > > > expected number of round trips.
> > > > > > > > > 2. An extra round trip between the transaction coordinator and
> > > batch
> > > > > > > > > coordinator when a transaction is finished.
> > > > > > > > >
> > > > > > > > > With this proposal, transactions will also be able to span both
> > > > > > classic
> > > > > > > > > and Diskless topics.
> > > > > > > > >
> > > > > > > > > *Queues*
> > > > > > > > >
> > > > > > > > > The share group coordination and management is a side job that
> > > > > > doesn't
> > > > > > > > > interfere with the topic itself (leadership, replicas, physical
> > > > > > storage
> > > > > > > > of
> > > > > > > > > records, etc.) and non-queue producers and consumers (Fetch and
> > > > > > Produce
> > > > > > > > > RPCs, consumer group-related RPCs are not affected.) We don't
> > > see any
> > > > > > > > > reason why we can't make Diskless topics compatible with share
> > > > > > groups the
> > > > > > > > > same way as classic topics are. Even on the code level, we
> > > don't
> > > > > > expect
> > > > > > > > any
> > > > > > > > > serious refactoring: the same reading routines are used that
> > > are
> > > > > > used for
> > > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > > > implementation-focused?
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Ivan
> > > > > > > > >
> > > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Thank you all for your questions and design input on
> > > KIP-1150.
> > > > > > > > > >
> > > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> > > design. To
> > > > > > > > > summarize
> > > > > > > > > > the changes:
> > > > > > > > > >
> > > > > > > > > > 1. The design prioritizes integrating with the existing
> > > KIP-405
> > > > > > Tiered
> > > > > > > > > > Storage interfaces, permitting data produced to a Diskless
> > > topic
> > > > > > to be
> > > > > > > > > > moved to tiered storage.
> > > > > > > > > > This lowers the scalability requirements for the Batch
> > > Coordinator
> > > > > > > > > > component, and allows Diskless to compose with Tiered Storage
> > > > > > plugin
> > > > > > > > > > features such as encryption and alternative data formats.
> > > > > > > > > >
> > > > > > > > > > 2. Consumer fetches are now served from local segments,
> > > making use
> > > > > > of
> > > > > > > > the
> > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > functionality
> > > > > > > > > already
> > > > > > > > > > built into classic topics.
> > > > > > > > > > However, local segments are now considered cache elements,
> > > do not
> > > > > > need
> > > > > > > > to
> > > > > > > > > > be durably stored, and can be built without contacting any
> > > other
> > > > > > > > > replicas.
> > > > > > > > > >
> > > > > > > > > > 3. The design has been simplified substantially, by removing
> > > the
> > > > > > > > previous
> > > > > > > > > > Diskless consume flow, distributed cache component, and
> > > "object
> > > > > > > > > > compaction/merging" step.
> > > > > > > > > >
> > > > > > > > > > The design maintains leaderless produces as enabled by the
> > > Batch
> > > > > > > > > > Coordinator, and the same latency profiles as the earlier
> > > design,
> > > > > > while
> > > > > > > > > > being simpler and integrating better into the existing
> > > ecosystem.
> > > > > > > > > >
> > > > > > > > > > Thanks, and we are eager to hear your feedback on the new
> > > design.
> > > > > > > > > > Greg Harris
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > <ju...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Jan,
> > > > > > > > > > >
> > > > > > > > > > > For me, the main gap of KIP-1150 is the support of all
> > > existing
> > > > > > > > client
> > > > > > > > > > > APIs. Currently, there is no design for supporting APIs
> > > like
> > > > > > > > > transactions
> > > > > > > > > > > and queues.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Would it be a good time to ask for the current status of
> > > this
> > > > > > KIP?
> > > > > > > > I
> > > > > > > > > > > > haven't seen much activity here for the past 2 months,
> > > the
> > > > > > vote got
> > > > > > > > > > > vetoed
> > > > > > > > > > > > but I think the pending questions have been answered
> > > since
> > > > > > then.
> > > > > > > > > KIP-1183
> > > > > > > > > > > > (AutoMQ's proposal) also didn't have any activity since
> > > May.
> > > > > > > > > > > >
> > > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices
> > > that can
> > > > > > be
> > > > > > > > > > > > made, with a coordinator-based approach being by far the
> > > > > > dominant
> > > > > > > > one
> > > > > > > > > > > when
> > > > > > > > > > > > it comes to market adoption - but all these are
> > > standalone
> > > > > > > > products.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm a big fan of both approaches, but would hate to see a
> > > > > > stall. So
> > > > > > > > > the
> > > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > > >
> > > > > > > > > > > > Maybe it's time to start another vote? Colin McCabe -
> > > have your
> > > > > > > > > questions
> > > > > > > > > > > > been answered? If not, is there anything I can do to
> > > help? I'm
> > > > > > > > deeply
> > > > > > > > > > > > familiar with both architectures and have written about
> > > both?
> > > > > > > > > > > >
> > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > Jan
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > > >
> > > > > > > > > > > > > a) group all the KIP email threads in the main one
> > > (just a
> > > > > > bunch
> > > > > > > > of
> > > > > > > > > > > links
> > > > > > > > > > > > > to everything)
> > > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's a bit hard to track it all - for example, I was
> > > > > > searching
> > > > > > > > for
> > > > > > > > > a
> > > > > > > > > > > > > discuss thread for KIP-1165 for a while; As far as I
> > > can
> > > > > > tell, it
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > exist yet.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Since the KIPs are published (by virtue of having the
> > > root
> > > > > > KIP be
> > > > > > > > > > > > > published, having a DISCUSS thread and links to
> > > sub-KIPs
> > > > > > where
> > > > > > > > were
> > > > > > > > > > > aimed
> > > > > > > > > > > > > to move the discussion towards), I think it would be
> > > good to
> > > > > > > > create
> > > > > > > > > > > > DISCUSS
> > > > > > > > > > > > > threads for them all.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Stan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We want to start a new KIP discussion about
> > > introducing a
> > > > > > new
> > > > > > > > > type of
> > > > > > > > > > > > > > topics that would make use of Object Storage as the
> > > primary
> > > > > > > > > source of
> > > > > > > > > > > > > > storage. However, as this KIP is big we decided to
> > > split it
> > > > > > > > into
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > that aims to discuss if Apache Kafka should aim to
> > > have
> > > > > > this
> > > > > > > > > type of
> > > > > > > > > > > > > > feature at all. This KIP doesn't go onto details on
> > > how to
> > > > > > > > > implement
> > > > > > > > > > > > it.
> > > > > > > > > > > > > > This follows the same approach used when we discussed
> > > > > > KRaft.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > But as we know that it is sometimes really hard to
> > > discuss
> > > > > > on
> > > > > > > > > that
> > > > > > > > > > > meta
> > > > > > > > > > > > > > level, we also created several sub-kips (linked in
> > > > > > KIP-1150)
> > > > > > > > that
> > > > > > > > > > > offer
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS threads
> > > for
> > > > > > each
> > > > > > > > > type of
> > > > > > > > > > > > > > concern and keep this one to discuss whether Apache
> > > Kafka
> > > > > > wants
> > > > > > > > > to
> > > > > > > > > > > have
> > > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks in advance on behalf of all the authors of
> > > this KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > > josep.prat@aiven.io  |  +491715557497 | aiven.io
> > > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > 
>   

Re: [DISCUSS] KIP-1150 Diskless Topics

[Henry Haiying Cai <ha...@yahoo.com.INVALID>]

 Ivan,
For the answer on how many data/meta-data partitions the diskless KIP needs to create (hence the number of open file descriptors), there seems to be a missing gap.
In the current Kafka design, for each topic partition, there is a corresponding log directory and a sequence of log segment files on the broker disk for persistence.  For example, /mnt/kafka/partitions/topic0-0/00000.log for topic 0 partition 0 (I used /mnt/kafka/partitions as the root log directory name for illustration purpose since that is the directory name in our system).  After user enabled topic0 to be diskless, the content in the log data file now contains the pointers to WAL log segment file in S3 (the metadata) so you are re-using the file descriptors in /mnt/kafka/partitions, but what about the real data content for topic0-0?  I think the KIP is proposing uploading those data as part of the WAL log segment onto S3, but there also needs to be a local cached copy of data content for topic0-0 on both the leader broker as well as the follower brokers for consumers to read from.
The KIP did not mention how that cache is built or organized or whether it is disk backed.  I remember in the earlier replies you mentioned you are going to reuse Kafka's existing page cache system for the data content cache, but if you are reusing Kafka's page cache, the data you downloaded from S3 or received from produceRequest for topic0-0 will needs to be stored in /mnt/kafka/partitions/topic0-0 (or maybe a new directory such as /mnt/kafka/partitions/data-topic0-0 to avoid the name conflict), but if you do this mapping for 100 original data partitions you will end up with 100 more new partition directories and file descriptors on the disk.
Also for the followers to build that cache, there will be temporary file and file descriptors being created for the S3 download, and those will be new file descriptors being created constantly for each data segment you are downloading from S3.
    On Tuesday, October 14, 2025 at 11:03:46 AM PDT, Ivan Yurchenko <iv...@ivanyu.me> wrote:  
 
 Hi Jun,

Thank you for your message. I’m sorry that I failed to clearly explain the idea. Let me try to fix this.

> Does each partition now have a metadata partition and a separate data
> partition? If so, I am concerned that it essentially doubles the number of
> partitions, which impacts the number of open file descriptors and the
> required IOPS, and so on. It also seems wasteful to have a separate
> partition just to store the metadata. It's as if we are creating an
> internal topic with an unbounded number of partitions.

No. There will be only one physical partition per diskless partition. Let me explain this with an example. Let’s say we have a diskless partition topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce some batches to this partition. The content of the segment file will be something like this (for each batch):

BaseOffset: 00000000000000000000 (like in classic)
Length: 123456 (like in classic)
PartitionLeaderEpoch: like in classic
Magic: like in classic
CRC: like in classic
Attributes: like in classic
LastOffsetDelta: like in classic
BaseTimestamp: like in classic
MaxTimestamp: like in classic
ProducerId: like in classic
ProducerEpoch: like in classic
BaseSequence: like in classic
RecordsCount: like in classic
Records:
  path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset 123456

It looks very much like classic log entries. The only difference is that instead of writing real Records, we write the reference to the WAL file with the batch data (I guess we need only the name and the byte offset, because the byte length is the standard field above). Otherwise, it’s a normal Kafka log with the leader and replicas. 

So we have as many partitions for diskless as for classic. As of open file descriptors, let’s proceed to the following:

> Are the metadata and
> the data for the same partition always collocated on the same broker? If
> so, how do we enforce that when replicas are reassigned?

The source of truth for the data is still in WAL files on object storage. The source of truth for the metadata is in segment files on the brokers in the replica set. Two new mechanisms are planned, both independent of this new proposal, but I want to present them to give the idea that only a limited amount of data files will be operated locally:
- We want to assemble batches into segment files and offload them to tiered storage in order to prevent the unbounded growth of batch metadata. For this, we need to open only  a few file descriptors (for the segment file itself + the necessary indexes) before the segment is fully written and handed over to RemoteLogManager.
- We want to assemble local segment files for caching purposes as well, i.e. to speed up fetching. This will not materialize the full content of the log, but only the hot set according to some policy (or configurable policies), i.e. the number of segments and file descriptors will also be limited.

> The number of RPCs in the produce path is significantly higher. For
> example, if a produce request has 100 partitions, in a cluster with 100
> brokers, each produce request could generate 100 more RPC requests. This
> will significantly increase the request rate.

This is a valid concern that we considered, but this issue can be mitigated. I’ll try to explain the approach.
The situation with a single broker is trivial: all the commit requests go from the broker to itself. 
Let’s scale this to a multi-broker cluster, but located in the single rack (AZ). Any broker can accept Produce requests for diskless partitions, but we can tell producers (through metadata) to always send Produce requests to leaders. For example, broker 0 hosts the leader replicas for diskless partitions t1-0, t2-1, t3-0. It will receive diskless Produce requests for these partitions in various combinations, but only for them.

                    Broker 0
              +-----------------+
              |    t1-0        |
              |    t2-1 <--------------------+
              |    t3-0        |            |
 produce      | +-------------+ |            |
 requests    | |  diskless  | |            |
--------------->|  produce  +--------------+
 for these    | | WAL buffer  | |    commit requests
 partitions  | +-------------+ |    for these partitions
              |                |
              +-----------------+

The same applies for other brokers in this cluster. Effectively, each broker will commit only to itself, which effectively means 1 commit request per WAL buffer (this may be 0 physical network calls, if we wish, just a local function call).

Now let’s scale this to multiple racks (AZs). Obviously, we cannot always send Produce requests to the designated leaders of diskless partitions: this would mean inter-AZ network traffic, which we would like to avoid. To avoid it, we say that every broker has a “diskless produce representative” in every AZ. If we continue our example: when a Produce request for t1-0, t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0 (in the broker’s AZ the representative is the broker itself). However, if it comes from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2. 

  |produce requests        |produce requests        |produce requests
  |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0
  |from AZ 0                |from AZ 1              |from AZ 2
  v                        v                        v
 Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
+---------------+      +---------------+      +---------------+
|    t1-0      |      |              |      |              |
|    t2-1      |      |              |      |              |
|    t3-0      |      |              |      |              |
+---------------+      +--------+------+      +--------+------+
    ^    ^                    |                      |
    |    +--------------------+                      |
    |    commit requests for these partitions        |
    |                                                |
    +-------------------------------------------------+
          commit requests for these partitions

All the partitions that broker 0 is the leader of will be “represented” by brokers 1 and 2 in their AZs.

Of course, this relationship goes both ways between AZs (not necessarily between the same brokers). It means that provided the cluster is balanced by the number of brokers per AZ, each broker will represent (number_of_azs - 1) other brokers. This will result in the situation that for the majority of commits, each broker will do up to (number_of_azs - 1) network commit requests (plus one local). Cloud regions tend to have 3 AZs, very rarely more. That means, brokers will be doing up to 2 network commit requests per WAL file.

There are the following exceptions:
1. Broker count imbalance between AZs. For example, when we have 2 AZs and one has three brokers and another AZ has one. This one broker will do between 1 and 3 commit requests per WAL file. This is not an extreme amplification. Such an imbalance is not healthy in most practical setups and should be avoided anyway.
2. Leadership changes and metadata propagation period. When the partition t3-0 is relocated from broker 0 to some broker 3, the producers will not know this immediately (unless we want to be strict and respond with NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come together in a WAL buffer on broker 2, it will have to send two commit requests: to broker 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This situation is not permanent and as producers update the cluster metadata, it will be resolved.

This all could be built with the metadata crafting mechanism only (which is anyway needed for Diskless in one way or another to direct producers and consumers where we need to avoid inter-AZ traffic), just with the right policy for it (for example, some deterministic hash-based formula). I.e. no explicit support for “produce representative” or anything like this is needed on the cluster level, in KRaft, etc.

> The same WAL file metadata is now duplicated into two places, partition
> leader and WAL File Manager. Which one is the source of truth, and how do
> we maintain consistency between the two places?

We do only two operations on WAL files that span multiple diskless partitions: committing and deleting. Commits can be done independently as described above. But deletes are different, because when a file is deleted, this affects all the partitions that still have alive batches in this file (if any).

The WAL file manager is a necessary point of coordination to delete WAL files safely. We can say it is the source of truth about files themselves, while the partition leaders and their logs hold the truth about whether a particular file contains live batches of this particular partition.

The file manager will do this important task: be able to say for sure that a file does not contain any live batch of any existing partition. For this, it will have to periodically check against the partition leaders. Considering that batch deletion is irreversible, when we declare a file “empty”, this is guaranteed to be and stay so.

The file manager has to know about files being committed to start track them and periodically check if they are empty. We can consider various ways to achieve this:
1. As was proposed in my previous message: best effort commit by brokers + periodic prefix scans of object storage to detect files that went below the radar due to network issue or the file manager temporary unavailability. We’re speaking about listing the file names only and opening only previously unknown files in order to find the partitions involved with them.
2. Only do scans without explicit commit, i.e. fill the list of files fully asynchronously and in the background. This may be not ideal due to costs and performance of scanning tons of files. However, the number of live WAL files should be limited due to tiered storage offloading + we can optimize this if we give files some global soft order in their names.

> I am not sure how this design simplifies the implementation. The existing
> producer/replication code can't be simply reused. Adjusting both the write
> path in the leader and the replication path in the follower to understand
> batch-header only data is quite intrusive to the existing logic.

It is true that we’ll have to change LocalLog and UnifiedLog in order to support these changes. However, it seems that idempotence, transactions, queues, tiered storage will have to be changed less than with the original design. This is because the partition leader state would remain in the same place (on brokers) and existing workflows that involve it would have to be changed less compared to the situation where we globalize the partition leader state in the batch coordinator. I admit this is hard to make convincing without both real implementations to hand :) 

> I am also
> not sure how this enables seamless switching the topic modes between
> diskless and classic. Could you provide more details on those?

Let’s consider the scenario of turning a classic topic into diskless. The user sets diskless.enabled=true, the leader receives this metadata update and does the following:
1. Stop accepting normal append writes.
2. Close the current active segment.
3. Start a new segment that will be written in the diskless format (i.e. without data).
4. Start accepting diskless commits.

Since it’s the same log, the followers will know about that switch consistently. They will finish replicating the classic segments and start replicating the diskless ones. They will always know where each batch is located (either inside a classic segment or referenced by a diskless one). Switching back should be similar.

Doing this with the coordinator is possible, but has some caveats. The leader must do the following:
1. Stop accepting normal append writes.
2. Close the current active segment.
3. Write a special control segment to persist and replicate the fact that from offset N the partition is now in the diskless mode.
4. Inform the coordinator about the first offset N of the “diskless era”.
5. Inform the controller quorum that the transition has finished and that brokers now can process diskless writes for this partition.
This could fail at some points, so this will probably require some explicit state machine with replication either in the partition log or in KRaft.

It seems that the coordinator-less approach makes this simpler because the “coordinator” for the partition and the partition leader are the same and they store the partition metadata in the same log, too. While in the coordinator approach we have to perform some kind of a distributed commit to handover metadata management from the classic partition leader to the batch coordinator.

I hope these explanations help to clarify the idea. Please let me know if I should go deeper anywhere.

Best,
Ivan and the Diskless team

On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> Hi, Ivan,
> 
> Thanks for the update.
> 
> I am not sure that I fully understand the new design, but it seems less
> clean than before.
> 
> Does each partition now have a metadata partition and a separate data
> partition? If so, I am concerned that it essentially doubles the number of
> partitions, which impacts the number of open file descriptors and the
> required IOPS, and so on. It also seems wasteful to have a separate
> partition just to store the metadata. It's as if we are creating an
> internal topic with an unbounded number of partitions. Are the metadata and
> the data for the same partition always collocated on the same broker? If
> so, how do we enforce that when replicas are reassigned?
> 
> The number of RPCs in the produce path is significantly higher. For
> example, if a produce request has 100 partitions, in a cluster with 100
> brokers, each produce request could generate 100 more RPC requests. This
> will significantly increase the request rate.
> 
> The same WAL file metadata is now duplicated into two places, partition
> leader and WAL File Manager. Which one is the source of truth, and how do
> we maintain consistency between the two places?
> 
> I am not sure how this design simplifies the implementation. The existing
> producer/replication code can't be simply reused. Adjusting both the write
> path in the leader and the replication path in the follower to understand
> batch-header only data is quite intrusive to the existing logic. I am also
> not sure how this enables seamless switching the topic modes between
> diskless and classic. Could you provide more details on those?
> 
> Jun
> 
> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> 
> > Hi dear Kafka community,
> >
> > In the initial Diskless proposal, we proposed to have a separate
> > component, batch/diskless coordinator, whose role would be to centrally
> > manage the batch and WAL file metadata for diskless topics. This component
> > drew many reasonable comments from the community about how it would support
> > various Kafka features (transactions, queues) and its scalability. While we
> > believe we have good answers to all the expressed concerns, we took a step
> > back and looked at the problem from a different perspective.
> >
> > We would like to propose an alternative Diskless design *without a
> > centralized coordinator*. We believe this approach has potential and
> > propose to discuss it as it may be more appealing to the community.
> >
> > Let us explain the idea. Most of the complications with the original
> > Diskless approach come from one necessary architecture change: globalizing
> > the local state of partition leader in the batch coordinator. This causes
> > deviations to the established workflows in various features like produce
> > idempotence and transactions, queues, retention, etc. These deviations need
> > to be carefully considered, designed, and later implemented and tested. In
> > the new approach we want to avoid this by making partition leaders again
> > responsible for managing their partitions, even in diskless topics.
> >
> > In classic Kafka topics, batch data and metadata are blended together in
> > the one partition log. The crux of the Diskless idea is to decouple them
> > and move data to the remote storage, while keeping metadata somewhere else.
> > Using the central batch coordinator for managing batch metadata is one way,
> > but not the only.
> >
> > Let’s now think about managing metadata for each user partition
> > independently. Generally partitions are independent and don’t share
> > anything apart from that their data are mixed in WAL files. If we figure
> > out how to commit and later delete WAL files safely, we will achieve the
> > necessary autonomy that allows us to get rid of the central batch
> > coordinator. Instead, *each diskless user partition will be managed by its
> > leader*, as in classic Kafka topics. Also like in classic topics, the
> > leader uses the partition log as the way to persist batch metadata, i.e.
> > the regular batch header + the information about how to find this batch on
> > remote storage. In contrast to classic topics, batch data is in remote
> > storage.
> >
> > For clarity, let’s compare the three designs:
> >  • Classic topics:
> >    • Data and metadata are co-located in the partition log.
> >    • The partition log content: [Batch header (metadata)|Batch data].
> >    • The partition log is replicated to the followers.
> >    • The replicas and leader have local state built from metadata.
> >  • Original Diskless:
> >    • Metadata is in the batch coordinator, data is on remote storage.
> >    • The partition state is global in the batch coordinator.
> >  • New Diskless:
> >    • Metadata is in the partition log, data is on remote storage.
> >    • Partition log content: [Batch header (metadata)|Batch coordinates on
> > remote storage].
> >    • The partition log is replicated to the followers.
> >    • The replicas and leader have local state built from metadata.
> >
> > Let’s consider the produce path. Here’s the reminder of the original
> > Diskless design:
> >
> >
> > The new approach could be depicted as the following:
> >
> >
> > As you can see, the main difference is that now instead of a single commit
> > request to the batch coordinator, we send multiple parallel commit requests
> > to all the leaders of each partition involved in the WAL file. Each of them
> > will commit its batches independently, without coordinating with other
> > leaders and any other components. Batch data is addressed by the WAL file
> > name, the byte offset and size, which allows partitions to know nothing
> > about other partitions to access their data in shared WAL files.
> >
> > The number of partitions involved in a single WAL file may be quite large,
> > e.g. a hundred. A hundred network requests to commit one WAL file is very
> > impractical. However, there are ways to reduce this number:
> >  1. Partition leaders are located on brokers. Requests to leaders on one
> > broker could be grouped together into a single physical network request
> > (resembling the normal Produce request that may carry batches for many
> > partitions inside). This will cap the number of network requests to the
> > number of brokers in the cluster.
> >  2. If we craft the cluster metadata to make producers send their requests
> > to the right brokers (with respect to AZs), we may achieve the higher
> > concentration of logical commit requests in physical network requests
> > reducing the number of the latter ones even further, ideally to one.
> >
> > Obviously, out of multiple commit requests some may fail or time out for a
> > variety of reasons. This is fine. Some producers will receive totally or
> > partially failed responses to their Produce requests, similar to what they
> > would have received when appending to a classic topic fails or times out.
> > If a partition experiences problems, other partitions will not be affected
> > (again, like in classic topics). Of course, the uncommitted data will be
> > garbage in WAL files. But WAL files are short-lived (batches are constantly
> > assembled into segments and offloaded to tiered storage), so this garbage
> > will be eventually deleted.
> >
> > For safely deleting WAL files we now need to centrally manage them, as
> > this is the only state and logic that spans multiple partitions. On the
> > diagram, you can see another commit request called “Commit file (best
> > effort)” going to the WAL File Manager. This manager will be responsible
> > for the following:
> >  1. Collecting (by requests from brokers) and persisting information about
> > committed WAL files.
> >  2. To handle potential failures in file information delivery, it will be
> > doing prefix scan on the remote storage periodically to find and register
> > unknown files. The period of this scan will be configurable and ideally
> > should be quite long.
> >  3. Checking with the relevant partition leaders (after a grace period) if
> > they still have batches in a particular file.
> >  4. Physically deleting files when they aren’t anymore referred to by any
> > partition.
> >
> > This new design offers the following advantages:
> >  1. It simplifies the implementation of many Kafka features such as
> > idempotence, transactions, queues, tiered storage, retention. Now we don’t
> > need to abstract away and reuse the code from partition leaders in the
> > batch coordinator. Instead, we will literally use the same code paths in
> > leaders, with little adaptation. Workflows from classic topics mostly
> > remain unchanged.
> > For example, it seems that
> > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > KafkaApis.handleWriteTxnMarkersRequest used for transaction support on the
> > partition leader side could be used for diskless topics with little
> > adaptation. ProducerStateManager, needed for both idempotent produce and
> > transactions, would be reused.
> > Another example is share groups support, where the share partition leader,
> > being co-located with the partition leader, would execute the same logic
> > for both diskless and classic topics.
> >  2. It returns to the familiar partition-based scaling model, where
> > partitions are independent.
> >  3. It makes the operation and failure patterns closer to the familiar
> > ones from classic topics.
> >  4. It opens a straightforward path to seamless switching the topics modes
> > between diskless and classic.
> >
> > The rest of the things remain unchanged compared to the previous Diskless
> > design (after all previous discussions). Such things as local segment
> > materialization by replicas, the consume path, tiered storage integration,
> > etc.
> >
> > If the community finds this design more suitable, we will update the
> > KIP(s) accordingly and continue working on it. Please let us know what you
> > think.
> >
> > Best regards,
> > Ivan and Diskless team
> >
> > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > Hi Justine,
> > >
> > > Yes, you're right. We need to track the aborted transactions for in the
> > diskless coordinator for as long as the corresponding offsets are there.
> > With the tiered storage unification Greg mentioned earlier, this will be
> > finite time even for infinite data retention.
> > >
> > > Best,
> > > Ivan
> > >
> > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > Hey Ivan,
> > > >
> > > > Thanks for the response. I think most of what you said made sense, but
> > I
> > > > did have some questions about this part:
> > > >
> > > > > As we understand this, the partition leader in classic topics forgets
> > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > transaction coordinator acts like the main guardian, allowing partition
> > > > leaders to do this safely. Please correct me if this is wrong. We think
> > > > about relying on this with the batch coordinator and delete the
> > information
> > > > about a transaction once it’s finished (as there’s no replication and
> > HWM
> > > > advances immediately).
> > > >
> > > > I didn't quite understand this. In classic topics, we have maps for
> > ongoing
> > > > transactions which remove state when the transaction is completed and
> > an
> > > > aborted transactions index which is retained for much longer. Once the
> > > > transaction is completed, the coordinator is no longer involved in
> > > > maintaining this partition side state, and it is subject to compaction
> > etc.
> > > > Looking back at the outline provided above, I didn't see much about the
> > > > fetch path, so maybe that could be expanded a bit further. I saw the
> > > > following in a response:
> > > > > When the broker constructs a fully valid local segment, all the
> > necessary
> > > > control batches will be inserted and indices, including the transaction
> > > > index will be built to serve FetchRequests exactly as they are today.
> > > >
> > > > Based on this, it seems like we need to retain the information about
> > > > aborted txns for longer.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > >
> > > > > Hi Justine and all,
> > > > >
> > > > > Thank you for your questions!
> > > > >
> > > > > > JO 1. >Since a transaction could be uniquely identified with
> > producer ID
> > > > > > and epoch, the positive result of this check could be cached
> > locally
> > > > > > Are we saying that only new transaction version 2 transactions can
> > be
> > > > > used
> > > > > > here? If not, we can't uniquely identify transactions with
> > producer id +
> > > > > > epoch
> > > > >
> > > > > You’re right that we (probably unintentionally) focused only on
> > version 2.
> > > > > We can either limit the support to version 2 or consider using some
> > > > > surrogates to support version 1.
> > > > >
> > > > > > JO 2. >The batch coordinator does the final transactional checks
> > of the
> > > > > > batches. This procedure would output the same errors like the
> > partition
> > > > > > leader in classic topics would do.
> > > > > > Can you expand on what these checks are? Would you be checking if
> > the
> > > > > > transaction was still ongoing for example?* *
> > > > >
> > > > > Yes, the producer epoch, that the transaction is ongoing, and of
> > course
> > > > > the normal idempotence checks. What the partition leader in the
> > classic
> > > > > topics does before appending a batch to the local log (e.g. in
> > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> > unfortunately
> > > > > cannot do these checks before appending the data to the WAL segment
> > and
> > > > > uploading it, but we can “tombstone” these batches in the batch
> > coordinator
> > > > > during the final commit.
> > > > >
> > > > > > Is there state about ongoing
> > > > > > transactions in the batch coordinator? I see some other state
> > mentioned
> > > > > in
> > > > > > the End transaction section, but it's not super clear what state is
> > > > > stored
> > > > > > and when it is stored.
> > > > >
> > > > > Right, this should have been more explicit. As the partition leader
> > tracks
> > > > > ongoing transactions for classic topics, the batch coordinator has
> > to as
> > > > > well. So when a transaction starts and ends, the transaction
> > coordinator
> > > > > must inform the batch coordinator about this.
> > > > >
> > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> > would
> > > > > be
> > > > > > stored in the batch coordinator?
> > > > >
> > > > > Yes. This could be deduced from the committed batches and other
> > > > > information, but for the sake of performance we’d better store it
> > > > > explicitly.
> > > > >
> > > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > >
> > > > > As we understand this, the partition leader in classic topics forgets
> > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > transaction coordinator acts like the main guardian, allowing
> > partition
> > > > > leaders to do this safely. Please correct me if this is wrong. We
> > think
> > > > > about relying on this with the batch coordinator and delete the
> > information
> > > > > about a transaction once it’s finished (as there’s no replication
> > and HWM
> > > > > advances immediately).
> > > > >
> > > > > Best,
> > > > > Ivan
> > > > >
> > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > Hey folks,
> > > > > >
> > > > > > Excited to see some updates related to transactions!
> > > > > >
> > > > > > I had a few questions.
> > > > > >
> > > > > > JO 1. >Since a transaction could be uniquely identified with
> > producer ID
> > > > > > and epoch, the positive result of this check could be cached
> > locally
> > > > > > Are we saying that only new transaction version 2 transactions can
> > be
> > > > > used
> > > > > > here? If not, we can't uniquely identify transactions with
> > producer id +
> > > > > > epoch
> > > > > >
> > > > > > JO 2. >The batch coordinator does the final transactional checks
> > of the
> > > > > > batches. This procedure would output the same errors like the
> > partition
> > > > > > leader in classic topics would do.
> > > > > > Can you expand on what these checks are? Would you be checking if
> > the
> > > > > > transaction was still ongoing for example? Is there state about
> > ongoing
> > > > > > transactions in the batch coordinator? I see some other state
> > mentioned
> > > > > in
> > > > > > the End transaction section, but it's not super clear what state is
> > > > > stored
> > > > > > and when it is stored.
> > > > > >
> > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> > would
> > > > > be
> > > > > > stored in the batch coordinator?
> > > > > >
> > > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > > >
> > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Greg and Ivan,
> > > > > > >
> > > > > > > Thanks for the update. A few comments.
> > > > > > >
> > > > > > > JR 10. "Consumer fetches are now served from local segments,
> > making
> > > > > use of
> > > > > > > the
> > > > > > > indexes, page cache, request purgatory, and zero-copy
> > functionality
> > > > > already
> > > > > > > built into classic topics."
> > > > > > > JR 10.1 Does the broker build the producer state for each
> > partition in
> > > > > > > diskless topics?
> > > > > > > JR 10.2 For transactional data, the consumer fetches need to know
> > > > > aborted
> > > > > > > records. How is that achieved?
> > > > > > >
> > > > > > > JR 11. "The batch coordinator saves that the transaction is
> > finished
> > > > > and
> > > > > > > also inserts the control batches in the corresponding logs of the
> > > > > involved
> > > > > > > Diskless topics. This happens only on the metadata level, no
> > actual
> > > > > control
> > > > > > > batches are written to any file. "
> > > > > > > A fetch response could include multiple transactional batches.
> > How
> > > > > does the
> > > > > > > broker obtain the information about the ending control batch for
> > each
> > > > > > > batch? Does that mean that a fetch response needs to be built by
> > > > > > > stitching record batches and generated control batches together?
> > > > > > >
> > > > > > > JR 12. Queues: Is there still a share partition leader that all
> > > > > consumers
> > > > > > > are routed to?
> > > > > > >
> > > > > > > JR 13. "Should the KIPs be modified to include this or it's too
> > > > > > > implementation-focused?" It would be useful to include enough
> > details
> > > > > to
> > > > > > > understand correctness and performance impact.
> > > > > > >
> > > > > > > HC5. Henry has a valid point. Requests from a given producer
> > contain a
> > > > > > > sequence number, which is ordered. If a producer sends every
> > Produce
> > > > > > > request to an arbitrary broker, those requests could reach the
> > batch
> > > > > > > coordinator in different order and lead to rejection of the
> > produce
> > > > > > > requests.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > We have also thought in a bit more details about transactions
> > and
> > > > > queues,
> > > > > > > > here's the plan.
> > > > > > > >
> > > > > > > > *Transactions*
> > > > > > > >
> > > > > > > > The support for transactions in *classic topics* is based on
> > precise
> > > > > > > > interactions between three actors: clients (mostly producers,
> > but
> > > > > also
> > > > > > > > consumers), brokers (ReplicaManager and other classes), and
> > > > > transaction
> > > > > > > > coordinators. Brokers also run partition leaders with their
> > local
> > > > > state
> > > > > > > > (ProducerStateManager and others).
> > > > > > > >
> > > > > > > > The high level (some details skipped) workflow is the
> > following.
> > > > > When a
> > > > > > > > transactional Produce request is received by the broker:
> > > > > > > > 1. For each partition, the partition leader checks if a
> > non-empty
> > > > > > > > transaction is running for this partition. This is done using
> > its
> > > > > local
> > > > > > > > state derived from the log metadata (ProducerStateManager,
> > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > 2. The transaction coordinator is informed about all the
> > partitions
> > > > > that
> > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > 3. The partition leaders do additional transactional checks.
> > > > > > > > 4. The partition leaders append the transactional data to
> > their logs
> > > > > and
> > > > > > > > update some of their state (for example, log the fact that the
> > > > > > > transaction
> > > > > > > > is running for the partition and its first offset).
> > > > > > > >
> > > > > > > > When the transaction is committed or aborted:
> > > > > > > > 1. The producer contacts the transaction coordinator directly
> > with
> > > > > > > > EndTxnRequest.
> > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > PREPARE_ABORT to
> > > > > > > > its log and responds to the producer.
> > > > > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to
> > the
> > > > > > > leaders
> > > > > > > > of the involved partitions.
> > > > > > > > 4. The partition leaders write the transaction markers to
> > their logs
> > > > > and
> > > > > > > > respond to the coordinator.
> > > > > > > > 5. The coordinator writes the final transaction state
> > > > > COMPLETE_COMMIT or
> > > > > > > > COMPLETE_ABORT.
> > > > > > > >
> > > > > > > > In classic topics, partitions have leaders and lots of
> > important
> > > > > state
> > > > > > > > necessary for supporting this workflow is local. The main
> > challenge
> > > > > in
> > > > > > > > mapping this to Diskless comes from the fact there are no
> > partition
> > > > > > > > leaders, so the corresponding pieces of state need to be
> > globalized
> > > > > in
> > > > > > > the
> > > > > > > > batch coordinator. We are already doing this to support
> > idempotent
> > > > > > > produce.
> > > > > > > >
> > > > > > > > The high level workflow for *diskless topics* would look very
> > > > > similar:
> > > > > > > > 1. For each partition, the broker checks if a non-empty
> > transaction
> > > > > is
> > > > > > > > running for this partition. In contrast to classic topics,
> > this is
> > > > > > > checked
> > > > > > > > against the batch coordinator with a single RPC. Since a
> > transaction
> > > > > > > could
> > > > > > > > be uniquely identified with producer ID and epoch, the positive
> > > > > result of
> > > > > > > > this check could be cached locally (for the double configured
> > > > > duration
> > > > > > > of a
> > > > > > > > transaction, for example).
> > > > > > > > 2. The same: The transaction coordinator is informed about all
> > the
> > > > > > > > partitions that aren’t part of the transaction to include them.
> > > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > > 4. The broker appends the transactional data to the current
> > shared
> > > > > WAL
> > > > > > > > segment. It doesn’t update any transaction-related state for
> > Diskless
> > > > > > > > topics, because it doesn’t have any.
> > > > > > > > 5. The WAL segment is committed to the batch coordinator like
> > in the
> > > > > > > > normal produce flow.
> > > > > > > > 6. The batch coordinator does the final transactional checks
> > of the
> > > > > > > > batches. This procedure would output the same errors like the
> > > > > partition
> > > > > > > > leader in classic topics would do. I.e. some batches could be
> > > > > rejected.
> > > > > > > > This means, there will potentially be garbage in the WAL
> > segment
> > > > > file in
> > > > > > > > case of transactional errors. This is preferable to doing more
> > > > > network
> > > > > > > > round trips, especially considering the WAL segments will be
> > > > > relatively
> > > > > > > > short-living (see the Greg's update above).
> > > > > > > >
> > > > > > > > When the transaction is committed or aborted:
> > > > > > > > 1. The producer contacts the transaction coordinator directly
> > with
> > > > > > > > EndTxnRequest.
> > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > PREPARE_ABORT to
> > > > > > > > its log and responds to the producer.
> > > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> > coordinator
> > > > > that
> > > > > > > > the transaction is finished.
> > > > > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> > > > > finished
> > > > > > > > and also inserts the control batches in the corresponding logs
> > of the
> > > > > > > > involved Diskless topics. This happens only on the metadata
> > level, no
> > > > > > > > actual control batches are written to any file. They will be
> > > > > dynamically
> > > > > > > > created on Fetch and other read operations. We could
> > technically
> > > > > write
> > > > > > > > these control batches for real, but this would mean extra
> > produce
> > > > > > > latency,
> > > > > > > > so it's better just to mark them in the batch coordinator and
> > save
> > > > > these
> > > > > > > > milliseconds.
> > > > > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to
> > the
> > > > > > > leaders
> > > > > > > > of the involved partitions. – Now only to classic topics now.
> > > > > > > > 6. The partition leaders of classic topics write the
> > transaction
> > > > > markers
> > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > 7. The coordinator writes the final transaction state
> > > > > COMPLETE_COMMIT or
> > > > > > > > COMPLETE_ABORT.
> > > > > > > >
> > > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > > 1. An extra network round trip between brokers and the batch
> > > > > coordinator
> > > > > > > > when a new partition appear in the transaction. To mitigate the
> > > > > impact of
> > > > > > > > them:
> > > > > > > >  - The results will be cached.
> > > > > > > >  - The calls for multiple partitions in one Produce request
> > will be
> > > > > > > > grouped.
> > > > > > > >  - The batch coordinator should be optimized for fast
> > response to
> > > > > these
> > > > > > > > RPCs.
> > > > > > > >  - The fact that a single producer normally will communicate
> > with a
> > > > > > > > single broker for the duration of the transaction further
> > reduces the
> > > > > > > > expected number of round trips.
> > > > > > > > 2. An extra round trip between the transaction coordinator and
> > batch
> > > > > > > > coordinator when a transaction is finished.
> > > > > > > >
> > > > > > > > With this proposal, transactions will also be able to span both
> > > > > classic
> > > > > > > > and Diskless topics.
> > > > > > > >
> > > > > > > > *Queues*
> > > > > > > >
> > > > > > > > The share group coordination and management is a side job that
> > > > > doesn't
> > > > > > > > interfere with the topic itself (leadership, replicas, physical
> > > > > storage
> > > > > > > of
> > > > > > > > records, etc.) and non-queue producers and consumers (Fetch and
> > > > > Produce
> > > > > > > > RPCs, consumer group-related RPCs are not affected.) We don't
> > see any
> > > > > > > > reason why we can't make Diskless topics compatible with share
> > > > > groups the
> > > > > > > > same way as classic topics are. Even on the code level, we
> > don't
> > > > > expect
> > > > > > > any
> > > > > > > > serious refactoring: the same reading routines are used that
> > are
> > > > > used for
> > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > >
> > > > > > > >
> > > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > > implementation-focused?
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Ivan
> > > > > > > >
> > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Thank you all for your questions and design input on
> > KIP-1150.
> > > > > > > > >
> > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> > design. To
> > > > > > > > summarize
> > > > > > > > > the changes:
> > > > > > > > >
> > > > > > > > > 1. The design prioritizes integrating with the existing
> > KIP-405
> > > > > Tiered
> > > > > > > > > Storage interfaces, permitting data produced to a Diskless
> > topic
> > > > > to be
> > > > > > > > > moved to tiered storage.
> > > > > > > > > This lowers the scalability requirements for the Batch
> > Coordinator
> > > > > > > > > component, and allows Diskless to compose with Tiered Storage
> > > > > plugin
> > > > > > > > > features such as encryption and alternative data formats.
> > > > > > > > >
> > > > > > > > > 2. Consumer fetches are now served from local segments,
> > making use
> > > > > of
> > > > > > > the
> > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > functionality
> > > > > > > > already
> > > > > > > > > built into classic topics.
> > > > > > > > > However, local segments are now considered cache elements,
> > do not
> > > > > need
> > > > > > > to
> > > > > > > > > be durably stored, and can be built without contacting any
> > other
> > > > > > > > replicas.
> > > > > > > > >
> > > > > > > > > 3. The design has been simplified substantially, by removing
> > the
> > > > > > > previous
> > > > > > > > > Diskless consume flow, distributed cache component, and
> > "object
> > > > > > > > > compaction/merging" step.
> > > > > > > > >
> > > > > > > > > The design maintains leaderless produces as enabled by the
> > Batch
> > > > > > > > > Coordinator, and the same latency profiles as the earlier
> > design,
> > > > > while
> > > > > > > > > being simpler and integrating better into the existing
> > ecosystem.
> > > > > > > > >
> > > > > > > > > Thanks, and we are eager to hear your feedback on the new
> > design.
> > > > > > > > > Greg Harris
> > > > > > > > >
> > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > <ju...@confluent.io.invalid>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Jan,
> > > > > > > > > >
> > > > > > > > > > For me, the main gap of KIP-1150 is the support of all
> > existing
> > > > > > > client
> > > > > > > > > > APIs. Currently, there is no design for supporting APIs
> > like
> > > > > > > > transactions
> > > > > > > > > > and queues.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > > Would it be a good time to ask for the current status of
> > this
> > > > > KIP?
> > > > > > > I
> > > > > > > > > > > haven't seen much activity here for the past 2 months,
> > the
> > > > > vote got
> > > > > > > > > > vetoed
> > > > > > > > > > > but I think the pending questions have been answered
> > since
> > > > > then.
> > > > > > > > KIP-1183
> > > > > > > > > > > (AutoMQ's proposal) also didn't have any activity since
> > May.
> > > > > > > > > > >
> > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices
> > that can
> > > > > be
> > > > > > > > > > > made, with a coordinator-based approach being by far the
> > > > > dominant
> > > > > > > one
> > > > > > > > > > when
> > > > > > > > > > > it comes to market adoption - but all these are
> > standalone
> > > > > > > products.
> > > > > > > > > > >
> > > > > > > > > > > I'm a big fan of both approaches, but would hate to see a
> > > > > stall. So
> > > > > > > > the
> > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > >
> > > > > > > > > > > Maybe it's time to start another vote? Colin McCabe -
> > have your
> > > > > > > > questions
> > > > > > > > > > > been answered? If not, is there anything I can do to
> > help? I'm
> > > > > > > deeply
> > > > > > > > > > > familiar with both architectures and have written about
> > both?
> > > > > > > > > > >
> > > > > > > > > > > Kind regards,
> > > > > > > > > > > Jan
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > >
> > > > > > > > > > > > a) group all the KIP email threads in the main one
> > (just a
> > > > > bunch
> > > > > > > of
> > > > > > > > > > links
> > > > > > > > > > > > to everything)
> > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > >
> > > > > > > > > > > > It's a bit hard to track it all - for example, I was
> > > > > searching
> > > > > > > for
> > > > > > > > a
> > > > > > > > > > > > discuss thread for KIP-1165 for a while; As far as I
> > can
> > > > > tell, it
> > > > > > > > > > doesn't
> > > > > > > > > > > > exist yet.
> > > > > > > > > > > >
> > > > > > > > > > > > Since the KIPs are published (by virtue of having the
> > root
> > > > > KIP be
> > > > > > > > > > > > published, having a DISCUSS thread and links to
> > sub-KIPs
> > > > > where
> > > > > > > were
> > > > > > > > > > aimed
> > > > > > > > > > > > to move the discussion towards), I think it would be
> > good to
> > > > > > > create
> > > > > > > > > > > DISCUSS
> > > > > > > > > > > > threads for them all.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Stan
> > > > > > > > > > > >
> > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > >
> > > > > > > > > > > > > We want to start a new KIP discussion about
> > introducing a
> > > > > new
> > > > > > > > type of
> > > > > > > > > > > > > topics that would make use of Object Storage as the
> > primary
> > > > > > > > source of
> > > > > > > > > > > > > storage. However, as this KIP is big we decided to
> > split it
> > > > > > > into
> > > > > > > > > > > multiple
> > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > )
> > > > > > > > > > > > > that aims to discuss if Apache Kafka should aim to
> > have
> > > > > this
> > > > > > > > type of
> > > > > > > > > > > > > feature at all. This KIP doesn't go onto details on
> > how to
> > > > > > > > implement
> > > > > > > > > > > it.
> > > > > > > > > > > > > This follows the same approach used when we discussed
> > > > > KRaft.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But as we know that it is sometimes really hard to
> > discuss
> > > > > on
> > > > > > > > that
> > > > > > > > > > meta
> > > > > > > > > > > > > level, we also created several sub-kips (linked in
> > > > > KIP-1150)
> > > > > > > that
> > > > > > > > > > offer
> > > > > > > > > > > > an
> > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS threads
> > for
> > > > > each
> > > > > > > > type of
> > > > > > > > > > > > > concern and keep this one to discuss whether Apache
> > Kafka
> > > > > wants
> > > > > > > > to
> > > > > > > > > > have
> > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance on behalf of all the authors of
> > this KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > josep.prat@aiven.io  |  +491715557497 | aiven.io
> > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 
  

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Andrew,

I want to apologize: somehow we forgot to transfer the section on cluster metadata from our draft document to the wiki. Now it's added [1]

I hope it may answer AS1 and AS2. Effectively the idea is the following:
1. If the MetadataRequest is of the new version and contains RackId, no lying: we tell the real leader, the replicas, the preferred produce brokers and IsDiskless flag to inform the client and let it decide where to send it's Produce, Fetch, and ShareFetch requests. If it's a share group consumer, it'll send ShareFetch requests to the leader. If it's a normal consumer, it'll send Fetch requests to preferred replica.
2. If the MetadataRequest is older, we also tell the truth. Older clients will continue to work, but won't be able to benefit from inter-rack traffic elimination.
2.1. However, as an workaround, we can encode the rack information in the client ID. In this case we "lie" to the consumer by replacing the leader with one of the replicas from the same rack if available.

Best,
Ivan

[1] https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=350783976&selectedPageVersions=17&selectedPageVersions=16

On Thu, Jan 22, 2026, at 11:58, Andrew Schofield wrote:
> Hi Ivan and other KIP authors,
> A few questions about the metadata for clients.
> 
> AS1: I’m slightly amused by the “lying” for metadata responses and I’m not really sure it’s lying at
> all. I think including the rack ID in the MetadataRequest is a good signal that the client wants
> a rack-aligned response. For a share consumer, we could just omit the rack ID from the metadata
> request and then get the real leader. That might be enough. I wonder whether we should
> have the intent of the sender for the MetadataRequest in there too, such as whether it’s a producer.
> 
> On a side note, we do send the rack ID in ShareGroupHeartbeat to allow for a rack-aware
> assignor, for which the KIP is on my list to create.
> 
> AS2: Why do we need the “IsDiskless” in the MetadataResponse? For one thing, I would
> prefer a more logical name about what the flag signifies in the response. If it indicates the
> response is not the actual leader but the broker can handle produce/consume, maybe the
> flag should have a name which reflects that instead. My worry is that years in the future
> when we want to set the flag for a new kind of non-diskless megatopics, the name is going
> to appear very unfashionable.
> 
> Thanks,
> Andrew
> 
> > On 22 Jan 2026, at 09:12, Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > Hi Robin,
> >
> > Your proposal actually resonates with what we proposed [1] a little while ago (nicknamed revision 3), so we exercised similar ideas. We abandoned them in favor of the original coordinator-based approach for the reasons described in the mailing list and summarized in the rejected alternatives of KIP-1163 [2].
> >
> > If you consider your proposal the alternative to KIP-1163 under the umbrella of KIP-1150, we propose to file a formal KIP so we can have a more visible community discussion of the merits of both approach. What do you think?
> >
> > Best,
> > Ivan
> >
> > [1] https://lists.apache.org/thread/w03qffwllynkx5ygvcd48vb3vjvsf8k7
> > [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=350783976#KIP1163:DisklessCore-Coordinator-lessapproaches
> >
> > On Wed, Jan 21, 2026, at 09:24, Robin Han wrote:
> >> Hi Greg,
> >>
> >> Thanks for the reply
> >>
> >>> Based on your description, the GET operations on the leader and followers
> >>> must complete before offsets are assigned, adding Produce request latency
> >>> that is not present in the KIP-1163 design.
> >>
> >> Yes, the ProduceRequestLatency is 1 * PUT(In Broker N) + 1 GET(In Broker M)
> >> + 1 GET(Follower Synchronization).
> >> - Compared with KIP-1163: Produce Latency has two more GET time costs;
> >> Consume E2E Latency has one more GET time cost.
> >> - Compared with KIP-1163: There is no need for coordination by a
> >> centralized Coordinator, thus eliminating the delay caused by Coordinator
> >> confirmation and avoiding the impact of Coordinator single-point failure on
> >> the availability of the entire cluster.
> >>
> >> In the design of Diskless Topic, I suspect that adding the time cost of two
> >> more GET operations will not bring about an essential change (with P50
> >> increasing from 500ms to 700ms).
> >>
> >>> KIP-1163 also assumes that object storage is more reliable than disk
> >>> storage, which is the case in the hyperscaler clouds. The parts of your
> >>> design that make it compatible with low-reliability object storage become
> >>> counterproductive when Diskless is used with a high-reliability object
> >>> storage, such as re-introducing direct replication and cross-az costs.
> >>
> >> It might be that my description is not accurate enough. The design is also
> >> strongly dependent on object storage. Regarding the part about object
> >> management and recycling, what I want to emphasize more is that the current
> >> design does not require complex object metadata management, and objects are
> >> just a temporary state.
> >>
> >>> We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> >>> in favor of a complete solution.
> >>
> >> I understand that the main reasons why "KIP-1176 was rejected" are that it
> >> did not solve the cross-AZ traffic costs for Produce, and that implementing
> >> Active Segment Tiered Storage added a lot of complexity to metadata
> >> management.
> >> The design I mentioned can not only solve all cross-AZ traffic costs
> >> including sending and replication (the core goal of KIP-1150), but also
> >> does not add complex management of Object metadata, nor does it require
> >> large-scale modifications to Apache Kafka.
> >> It could be a good design that can be adopted by KIP-1150.
> >>
> >> Greg Harris via dev <de...@kafka.apache.org> 于2026年1月21日周三 03:20写道：
> >>
> >>> Hi Robin,
> >>>
> >>> Thanks for your design sketch.
> >>>
> >>> Based on your description, the GET operations on the leader and followers
> >>> must complete before offsets are assigned, adding Produce request latency
> >>> that is not present in the KIP-1163 design.
> >>>
> >>> KIP-1163 also assumes that object storage is more reliable than disk
> >>> storage, which is the case in the hyperscaler clouds. The parts of your
> >>> design that make it compatible with low-reliability object storage become
> >>> counterproductive when Diskless is used with a high-reliability object
> >>> storage, such as re-introducing direct replication and cross-az costs.
> >>>
> >>> We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> >>> in favor of a complete solution. The folks who championed that proposal are
> >>> now working with us on KIP-1163, and are currently designing Compaction in
> >>> Tiered Storage to make Diskless a complete solution for compacted topics as
> >>> well. Similarly, we would appreciate your contributions to the design of
> >>> KIP-1163, and your vote on the motivational KIP-1150.
> >>>
> >>> Thanks!
> >>>
> >>> On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >>>
> >>>> Hi Jun,
> >>>>
> >>>> Here are our answers to your questions and comments:
> >>>>
> >>>>> JR10. "Even in the case where network traffic is not accounted; we
> >>>> believe
> >>>>> that the operational benefits of Diskless topics are still appealing to
> >>>>> Kafka users."
> >>>>> Could you list other benefits such as better scalability and
> >>> durability?
> >>>>
> >>>> Thank you, good point, added this.
> >>>>
> >>>>> JR11. "AWS charges 0.02$ per GiB"
> >>>>> The object store is not free and has a different cost model based on
> >>>>> requests. It would be useful summarize the tradeoff on cost and
> >>> latency,
> >>>>> and when diskless topics are cost effective.
> >>>>
> >>>> Yes, there is a per-request cost for object storage which is not directly
> >>>> comparable to inter-zone transfer costs. We have found that the design in
> >>>> KIP-1163 amortizes those request costs such that they are insignificant
> >>>> compared to the $0.02/gb cost which is called out in KIP-1150.
> >>>> We currently have very precise cost models that were put together for an
> >>>> earlier iteration of the Diskless topics. These are somewhat stale and
> >>>> inaccurate for the latest design, so we will work in the background to
> >>> post
> >>>> concrete numbers in KIP-1163 before voting on that proposal. Would this
> >>> be
> >>>> OK?
> >>>>
> >>>>> JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> >>>> been
> >>>>> updated in the last 6 months. The metadata change in KIP-1181 is now
> >>>>> included in KIP-1163.
> >>>>
> >>>> Thanks for calling out. We have updated KIP-1164 with the latest design.
> >>>> It's missing public interfaces, but we think it may be a bit too early to
> >>>> define them as some rounds of discussion is ahead, with potential to
> >>> change
> >>>> the design. But otherwise the KIP is good for the first review.
> >>>> We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
> >>>>
> >>>> Best,
> >>>> Ivan
> >>>>
> >>>> On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> >>>>> Hi, Anatolii,
> >>>>>
> >>>>> Thanks for the updated KIP. Looks good to me overall. A few more
> >>>> comments.
> >>>>>
> >>>>> JR10. "Even in the case where network traffic is not accounted; we
> >>>> believe
> >>>>> that the operational benefits of Diskless topics are still appealing to
> >>>>> Kafka users."
> >>>>> Could you list other benefits such as better scalability and
> >>> durability?
> >>>>>
> >>>>> JR11. "AWS charges 0.02$ per GiB"
> >>>>> The object store is not free and has a different cost model based on
> >>>>> requests. It would be useful summarize the tradeoff on cost and
> >>> latency,
> >>>>> and when diskless topics are cost effective.
> >>>>>
> >>>>> JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> >>>> been
> >>>>> updated in the last 6 months. The metadata change in KIP-1181 is now
> >>>>> included in KIP-1163.
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> >>>> dev@kafka.apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> Thank you for the extensive feedback.
> >>>>>>
> >>>>>> We have substantially updated the KIP to address the points raised.
> >>>> Given
> >>>>>> the scope of these changes, we have started a new VOTE thread to
> >>>> restart
> >>>>>> the voting process cleanly.
> >>>>>>
> >>>>>> You can find the new thread here:
> >>>>>> https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> >>>>>>
> >>>>>> Best,
> >>>>>> Anatolii
> >>>>>>
> >>>>>> On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> >>>>>> dev@kafka.apache.org> wrote:
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> We (the team at Slack) have been following the recent discussion
> >>>>>> regarding
> >>>>>>> the scope and timeline of KIP-1150. We agree with the community
> >>>> sentiment
> >>>>>>> that Diskless Topics represents the right long-term architecture
> >>> for
> >>>>>> Kafka
> >>>>>>> in the cloud, but we also recognize the valid concerns raised
> >>>> regarding
> >>>>>> the
> >>>>>>> engineering resources required to deliver such an ambitious change.
> >>>>>>>
> >>>>>>> To help address these concerns and accelerate the timeline, we are
> >>>> happy
> >>>>>> to
> >>>>>>> announce that we are partnering with the KIP-1150 authors to
> >>>> co-develop
> >>>>>>> this feature.
> >>>>>>>
> >>>>>>> We previously proposed KIP-1176: Tiered Storage for Active Log
> >>>> Segments
> >>>>>> to
> >>>>>>> solve similar problems. However, rather than fragmenting the
> >>>> community's
> >>>>>>> efforts across competing designs, we have decided to withdraw
> >>>> KIP-1176
> >>>>>> and
> >>>>>>> consolidate our engineering resources behind KIP-1150.
> >>>>>>>
> >>>>>>> To start, we plan to take ownership of Compaction for Tiered
> >>> Storage.
> >>>>>> This
> >>>>>>> has long been a missing feature in KIP-405, and it becomes critical
> >>>> in a
> >>>>>>> Diskless architecture where long-lived data is obligated to be
> >>>> tiered. By
> >>>>>>> driving this related prerequisite feature, we hope to allow for
> >>>> faster
> >>>>>>> delivery of KIP-1150.
> >>>>>>>
> >>>>>>> We are excited to collaborate on this to ensure a robust and timely
> >>>>>>> delivery for the community.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Tom & Henry
> >>>>>>>
> >>>>>>> On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
> >>> wrote:
> >>>>>>>
> >>>>>>>> Hi Greg,
> >>>>>>>>
> >>>>>>>> Thanks for sharing the meeting notes.
> >>>>>>>> I agree we should keep polishing the contents of 1150 & high
> >>> level
> >>>>>> design
> >>>>>>>> in 1163 to prepare for a vote.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>> Luke
> >>>>>>>>
> >>>>>>>> On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> >>>>>> <greg.harris@aiven.io.invalid
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi all,
> >>>>>>>>>
> >>>>>>>>> There was a video call between myself, Ivan Yurchenko, Jun Rao,
> >>>> and
> >>>>>>>> Andrew
> >>>>>>>>> Schofield pertaining to KIP-1150. Here are the notes from that
> >>>>>> meeting:
> >>>>>>>>>
> >>>>>>>>> Ivan: What is the future state of Kafka in this area, in 5
> >>> years?
> >>>>>>>>> Jun: Do we want something more cloud native? Yes, started with
> >>>> Tiered
> >>>>>>>>> Storage. If there’s a better way, we should explore it. In the
> >>>> long
> >>>>>>> term
> >>>>>>>>> this will be useful
> >>>>>>>>> Because Kafka is used so widely, we need to make sure
> >>> everything
> >>>> we
> >>>>>> add
> >>>>>>>> is
> >>>>>>>>> for the long term and for everyone, not just for a single
> >>>> company.
> >>>>>>>>> When we add TS, it doesn’t just solve Uber’s use-case. We want
> >>>>>>> something
> >>>>>>>>> that’s high quality/lasts/maintainable, and can work with all
> >>>>>> existing
> >>>>>>>>> capabilities.
> >>>>>>>>> If both 1150 and 1176 proceed at the same time, it’s confusing.
> >>>> They
> >>>>>>>>> overlap, but Diskless is more ambitious.
> >>>>>>>>> If both KIPs are being seriously worked on, then we don’t
> >>> really
> >>>> need
> >>>>>>>> both,
> >>>>>>>>> because Diskless clearly is better. Having multiple will
> >>> confuse
> >>>>>>> people.
> >>>>>>>> It
> >>>>>>>>> will duplicate some of the effort.
> >>>>>>>>> If we want diskless ultimately, what is the short term
> >>> strategy,
> >>>> to
> >>>>>> get
> >>>>>>>>> some early wins first?
> >>>>>>>>> Ivan: Andrew, do you want a more revolutionary approach?
> >>>>>>>>> Andrew: Eventually the architecture will change substantially,
> >>>> it may
> >>>>>>> not
> >>>>>>>>> be necessary to put all of that bill onto Diskless at once.
> >>>>>>>>> Greg: We all agree on having a high quality feature merged
> >>>> upstream,
> >>>>>>> and
> >>>>>>>>> supporting all APIs
> >>>>>>>>> Jun: We should try and keep things simple, but there is some
> >>>> minimum
> >>>>>>>>> complexity needed.
> >>>>>>>>> When doing the short term changes (1176), it doesn’t really
> >>>> progress
> >>>>>> in
> >>>>>>>>> changing to a more modern architecture.
> >>>>>>>>> Greg: Was TS+Compaction the only feature miss we’ve had so far?
> >>>>>>>>> Jun: The danger of only applying changes to some part of the
> >>>> API, you
> >>>>>>> set
> >>>>>>>>> the precedence that you only have to implement part of the API.
> >>>>>>>> Supporting
> >>>>>>>>> the full API set should be a minimum requirement.
> >>>>>>>>> Andrew: When we started Kraft, how much did we know the design?
> >>>>>>>>> Jun: For Kraft we didn’t really know much about the migration,
> >>>> but
> >>>>>> the
> >>>>>>>>> high-level was clear.
> >>>>>>>>> Greg: Is 1150 votable in its current state?
> >>>>>>>>> Jun: 1150 should promise to support all APIs. It doesn’t have
> >>> to
> >>>> have
> >>>>>>> all
> >>>>>>>>> the details/apis/etc. KIP-500 didn’t have it.
> >>>>>>>>> We do need some high-level design enough to give confidence
> >>> that
> >>>> the
> >>>>>>>>> promise is able to be fulfilled.
> >>>>>>>>> Greg: Is the draft version in 1163 enough detail or is more
> >>>> needed?
> >>>>>>>>> Jun: We need to agree on the core design, such as leaderless
> >>>> etc. And
> >>>>>>> how
> >>>>>>>>> the APIs will be supported.
> >>>>>>>>> Greg: Okay we can include these things, and provide a sketch of
> >>>> how
> >>>>>> the
> >>>>>>>>> other leader-based features operate.
> >>>>>>>>> Jun: Yeah if at a high level the sketch appears to work, we can
> >>>>>> approve
> >>>>>>>>> that functionality.
> >>>>>>>>> Are you committed to doing the more involved and big project?
> >>>>>>>>> Greg: Yes, we’re committed to the 1163 design and can’t really
> >>>> accept
> >>>>>>>> 1176.
> >>>>>>>>> Jun: TS was slow because of Uber resourcing problems
> >>>>>>>>> Greg: We’ll push internally for resources, and use the
> >>> community
> >>>>>>>> sentiment
> >>>>>>>>> to motivate Aiven.
> >>>>>>>>> How far into the future should we look? What sort of scale?
> >>>>>>>>> Jun: As long as there’s a path forward, and we’re not closing
> >>> off
> >>>>>>> future
> >>>>>>>>> improvements, we can figure out how to handle a larger scale
> >>>> when it
> >>>>>>>>> arises.
> >>>>>>>>> Greg: Random replica placement is very harmful, can we
> >>> recommend
> >>>>>> users
> >>>>>>> to
> >>>>>>>>> use an external tool like CruiseControl?
> >>>>>>>>> Jun: Not everyone uses CruiseControl, we would probably need
> >>> some
> >>>>>>>> solution
> >>>>>>>>> for this out of the box
> >>>>>>>>> Ivan: Should the Batch Coordinator be pluggable?
> >>>>>>>>> Jun: Out-of-box experience should be good, good to allow other
> >>>>>>>>> implementations
> >>>>>>>>> Greg: But it could hurt Kafka feature/upgrade velocity when we
> >>>> wait
> >>>>>> for
> >>>>>>>>> plugin providers to implement it
> >>>>>>>>> Ivan: We imagined that maybe cloud hyperscalers could implement
> >>>> it
> >>>>>> with
> >>>>>>>>> e.g. dynamodb
> >>>>>>>>> Greg: Could we bake more details of the different providers
> >>> into
> >>>>>> Kafka,
> >>>>>>>> or
> >>>>>>>>> does it still make sense for it to be pluggable?
> >>>>>>>>> Jun: Make it whatever is easiest to roll out and add new
> >>> clients
> >>>>>>>>> Andrew: What happens next? Do you want to get KIP-1150 voted?
> >>>>>>>>> Ivan: The vote is already open, we’re not too pressed for time.
> >>>> We’ll
> >>>>>>> go
> >>>>>>>>> improve the 1163 design and communication.
> >>>>>>>>> Is 1176 a competing design? Someone will ask.
> >>>>>>>>> Jun: If we are seriously working on something more ambitious,
> >>>> yeah we
> >>>>>>>>> shouldn’t do the stop-gap solution.
> >>>>>>>>> It’s diverting review resources. If we can get the short term
> >>>> thing
> >>>>>> in
> >>>>>>>> 1yr
> >>>>>>>>> but Diskless solution is 2y it makes sense to go for Diskless.
> >>> If
> >>>>>> it’s
> >>>>>>>> 5yr,
> >>>>>>>>> that’s different and maybe the stop-gap solution is needed.
> >>>>>>>>> Greg: I’m biased but I believe we’re in the 1yr/2yr case.
> >>> Should
> >>>> we
> >>>>>>>>> explicitly exclude 1176?
> >>>>>>>>> Andrew: Put your arms around the feature set you actually want,
> >>>> and
> >>>>>> use
> >>>>>>>>> that to rule out 1176.
> >>>>>>>>> Probably don’t need -1 votes, most likely KIPs just don’t
> >>> receive
> >>>>>>> votes.
> >>>>>>>>> Ivan: Should we have sync meetings like tiered storage did?
> >>>>>>>>> Jun: Satish posted meeting notes regularly, we should do the
> >>>> same.
> >>>>>>>>>
> >>>>>>>>> To summarize, we will be polishing the contents of 1150 & high
> >>>> level
> >>>>>>>> design
> >>>>>>>>> in 1163 to prepare for a vote.
> >>>>>>>>> We believe that the community should select the feature set of
> >>>> 1150
> >>>>>> to
> >>>>>>>>> fully eliminate producer cross-zone costs, and make the
> >>>> investment
> >>>>>> in a
> >>>>>>>>> high quality Diskless Topics implementation rather than in
> >>>> stop-gap
> >>>>>>>>> solutions.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Greg
> >>>>>>>>>
> >>>>>>>>> On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> This may be a tangent, but we needed to offload storage off
> >>> of
> >>>>>> Kafka
> >>>>>>>> into
> >>>>>>>>>> S3. We are keeping Kafka not as a source of truth, but as a
> >>>> mostly
> >>>>>>>>>> ephemeral broker that can come and go as it pleases. Be that
> >>>>>> scaling
> >>>>>>> or
> >>>>>>>>>> outage. Disks can be destroyed and recreated at will, we
> >>> still
> >>>>>> retain
> >>>>>>>>> data
> >>>>>>>>>> and use broker for just that, brokering messages. Not only
> >>>> that, we
> >>>>>>>>> reduced
> >>>>>>>>>> the requirement on the actual Kafka resources by reducing the
> >>>> size
> >>>>>>> of a
> >>>>>>>>>> payload via a claim check pattern. Maybe this is an
> >>>> anti–pattern,
> >>>>>> but
> >>>>>>>> it
> >>>>>>>>> is
> >>>>>>>>>> super fast and highly cost efficient. We reworked
> >>>> ProducerRequest
> >>>>>> to
> >>>>>>>>> allow
> >>>>>>>>>> plugins. We added a custom http plugin that submits every
> >>>> request
> >>>>>>> via a
> >>>>>>>>>> persisted connection to a microservice. Microservice stores
> >>> the
> >>>>>>> payload
> >>>>>>>>> and
> >>>>>>>>>> returns a tiny json metadata object,a claim check, that can
> >>> be
> >>>> used
> >>>>>>> to
> >>>>>>>>> find
> >>>>>>>>>> the actual data. Think of it as zipping the payload. This
> >>> claim
> >>>>>> check
> >>>>>>>>>> metadata traverses the pipelines with consumers using the
> >>> urls
> >>>> in
> >>>>>>>>> metadata
> >>>>>>>>>> to pull what they need. Think unzipping. This allowed us to
> >>>> also
> >>>>>> pull
> >>>>>>>>> ONLY
> >>>>>>>>>> the data that we need in graphql like manner. So if you have
> >>> a
> >>>> 100K
> >>>>>>>> json
> >>>>>>>>>> payload and you need only a subsection, you can pull that by
> >>>>>>> jmespath.
> >>>>>>>>> When
> >>>>>>>>>> you have multiple consumer groups yanking down huge payloads
> >>>> it is
> >>>>>>>>>> cumbersome on the broker. When you have the same consumer
> >>>> groups
> >>>>>>>> yanking
> >>>>>>>>>> down a claim check, and then going out of band directly to
> >>> the
> >>>>>> source
> >>>>>>>> of
> >>>>>>>>>> truth, the broker has some breathing room. Obviously our
> >>>>>> microservice
> >>>>>>>>> does
> >>>>>>>>>> not go directly to the cloud storage, as that would be too
> >>>> slow. It
> >>>>>>>>> stores
> >>>>>>>>>> the payload in high speed memory cache and returns asap. That
> >>>>>> memory
> >>>>>>> is
> >>>>>>>>>> eventually persisted into S3. The retrieval goest against the
> >>>> cache
> >>>>>>>>> first,
> >>>>>>>>>> then against the S3. Overall a rather cheappy and zippy
> >>>> solution. I
> >>>>>>>> tried
> >>>>>>>>>> proposing the KIP for this, but there was no excitement.
> >>> Check
> >>>> this
> >>>>>>>> out:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 7, 2025, at 5:49 PM, Jun Rao
> >>> <jun@confluent.io.INVALID
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, Andrew,
> >>>>>>>>>>>
> >>>>>>>>>>> If we want to focus only on reducing cross-zone replication
> >>>>>> costs,
> >>>>>>>>> there
> >>>>>>>>>> is
> >>>>>>>>>>> an alternative design in the KIP-1176 discussion thread
> >>> that
> >>>>>> seems
> >>>>>>>>>> simpler
> >>>>>>>>>>> than the proposal here. I am copying the outline of that
> >>>> approach
> >>>>>>>>> below.
> >>>>>>>>>>>
> >>>>>>>>>>> 1. A new leader is elected.
> >>>>>>>>>>> 2. Leader maintains a first tiered offset, which is
> >>>> initialized
> >>>>>> to
> >>>>>>>> log
> >>>>>>>>>> end
> >>>>>>>>>>> offset.
> >>>>>>>>>>> 3. Leader writes produced data from the client to local
> >>> log.
> >>>>>>>>>>> 4. Leader uploads produced data from all local logs as a
> >>>> combined
> >>>>>>>>> object
> >>>>>>>>>>> 5. Leader stores the metadata for the combined object in
> >>>> memory.
> >>>>>>>>>>> 6. If a follower fetch request has an offset >= first
> >>> tiered
> >>>>>>> offset,
> >>>>>>>>> the
> >>>>>>>>>>> metadata for the corresponding combined object is returned.
> >>>>>>>> Otherwise,
> >>>>>>>>>> the
> >>>>>>>>>>> local data is returned.
> >>>>>>>>>>> 7. Leader periodically advances first tiered offset.
> >>>>>>>>>>>
> >>>>>>>>>>> It's still a bit unnatural, but it could work.
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, Ivan,
> >>>>>>>>>>>
> >>>>>>>>>>> Are you still committed to proceeding with the original
> >>>> design of
> >>>>>>>>>> KIP-1150?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> >>>>>>>>>> andrew_schofield@live.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>> I’ve been following KIP-1150 and friends for a while. I’m
> >>>> going
> >>>>>> to
> >>>>>>>>> jump
> >>>>>>>>>>>> into the discussions too.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Looking back at Jack Vanlightly’s message, I am not quite
> >>> so
> >>>>>>>> convinced
> >>>>>>>>>>>> that it’s a kind of fork in the road. The primary aim of
> >>> the
> >>>>>>> effort
> >>>>>>>> is
> >>>>>>>>>> to
> >>>>>>>>>>>> reduce cross-zone replication costs so Apache Kafka is not
> >>>>>>>>> prohibitively
> >>>>>>>>>>>> expensive to use on cloud storage. I think it would be
> >>>> entirely
> >>>>>>>> viable
> >>>>>>>>>> to
> >>>>>>>>>>>> prioritise code reuse for an initial implementation of
> >>>> diskless
> >>>>>>>>> topics,
> >>>>>>>>>> and
> >>>>>>>>>>>> we could still have a more cloud-native design in the
> >>>> future.
> >>>>>> It’s
> >>>>>>>>> hard
> >>>>>>>>>> to
> >>>>>>>>>>>> predict what the community will prioritise in the future.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Of the three major revisions, I’m in the rev3 camp. We can
> >>>>>> support
> >>>>>>>>>>>> leaderless produce requests, first writing WAL segments
> >>> into
> >>>>>>> object
> >>>>>>>>>>>> storage, and then using the regular partition leaders to
> >>>>>> sequence
> >>>>>>>> the
> >>>>>>>>>>>> records. The active log segment for a diskless topic will
> >>>>>>> initially
> >>>>>>>>>> contain
> >>>>>>>>>>>> batch coordinates rather than record batches. The batch
> >>>>>>> coordinates
> >>>>>>>>> can
> >>>>>>>>>> be
> >>>>>>>>>>>> resolved from WAL segments for consumers, and also in
> >>> order
> >>>> to
> >>>>>>>> prepare
> >>>>>>>>>> log
> >>>>>>>>>>>> segments for uploading to tiered storage. Jun is probably
> >>>>>> correct
> >>>>>>>> that
> >>>>>>>>>> we
> >>>>>>>>>>>> need a more frequent object merging process than tiered
> >>>> storage
> >>>>>>>>>> provides.
> >>>>>>>>>>>> This is just the transition from write-optimised WAL
> >>>> segments to
> >>>>>>>>>>>> read-optimised tiered segments, and all of the object
> >>>>>>> storage-based
> >>>>>>>>>>>> implementations of Kafka that I’m aware of do this
> >>>>>> rearrangement.
> >>>>>>>> But
> >>>>>>>>>>>> perhaps this more frequent object merging is a pre-GA
> >>>>>> improvement,
> >>>>>>>>>> rather
> >>>>>>>>>>>> than a strict requirement for an initial implementation
> >>> for
> >>>>>> early
> >>>>>>>>> access
> >>>>>>>>>>>> use.
> >>>>>>>>>>>>
> >>>>>>>>>>>> For zone-aligned share consumers, the share group assignor
> >>>> is
> >>>>>>>> intended
> >>>>>>>>>> to
> >>>>>>>>>>>> be rack-aware. Consumers should be assigned to partitions
> >>>> with
> >>>>>>>> leaders
> >>>>>>>>>> in
> >>>>>>>>>>>> their zone. The simple assignor is not rack-aware, but it
> >>>> easily
> >>>>>>>> could
> >>>>>>>>>> be
> >>>>>>>>>>>> or we could have a rack-aware assignor.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Andrew
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 24 Oct 2025, at 23:14, Jun Rao
> >>> <jun@confluent.io.INVALID
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Ivan,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> "As I understand, you’re speaking about locally
> >>>> materialized
> >>>>>>>>> segments.
> >>>>>>>>>>>> They
> >>>>>>>>>>>>> will indeed consume some IOPS. See them as a cache that
> >>>> could
> >>>>>>>> always
> >>>>>>>>> be
> >>>>>>>>>>>>> restored from the remote storage. While it’s not ideal,
> >>>> it's
> >>>>>>> still
> >>>>>>>> OK
> >>>>>>>>>> to
> >>>>>>>>>>>>> lose data in them due to a machine crash, for example.
> >>>> Because
> >>>>>> of
> >>>>>>>>> this,
> >>>>>>>>>>>> we
> >>>>>>>>>>>>> can avoid explicit flushing on local materialized
> >>> segments
> >>>> at
> >>>>>> all
> >>>>>>>> and
> >>>>>>>>>> let
> >>>>>>>>>>>>> the file system and page cache figure out when to flush
> >>>>>>> optimally.
> >>>>>>>>> This
> >>>>>>>>>>>>> would not eliminate the extra IOPS, but should reduce it
> >>>>>>>>> dramatically,
> >>>>>>>>>>>>> depending on throughput for each partition. We, of
> >>> course,
> >>>>>>> continue
> >>>>>>>>>>>>> flushing the metadata segments as before."
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If we have a mix of classic and diskless topics on the
> >>> same
> >>>>>>> broker,
> >>>>>>>>>> it's
> >>>>>>>>>>>>> important that the classic topics' data is flushed to
> >>> disk
> >>>> as
> >>>>>>>> quickly
> >>>>>>>>>> as
> >>>>>>>>>>>>> possible. To achieve this, users typically set
> >>>>>>>> dirty_expire_centisecs
> >>>>>>>>>> in
> >>>>>>>>>>>>> the kernel based on the number of available disk IOPS.
> >>>> Once you
> >>>>>>> set
> >>>>>>>>>> this
> >>>>>>>>>>>>> number, it applies to all dirty files, including the
> >>> cached
> >>>>>> data
> >>>>>>> in
> >>>>>>>>>>>>> diskless topics. So, if there are more files actively
> >>>>>>> accumulating
> >>>>>>>>>> data,
> >>>>>>>>>>>>> the flush frequency and therefore RPO is reduced for
> >>>> classic
> >>>>>>>> topics.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> "We should have mentioned this explicitly, but this step,
> >>>> in
> >>>>>>> fact,
> >>>>>>>>>>>> remains
> >>>>>>>>>>>>> in the form of segments offloading to tiered storage.
> >>> When
> >>>> we
> >>>>>>>>> assemble
> >>>>>>>>>> a
> >>>>>>>>>>>>> segment and hand it over to RemoteLogManager, we’re
> >>>> effectively
> >>>>>>>> doing
> >>>>>>>>>>>>> metadata compaction: replacing a big number of pieces of
> >>>>>> metadata
> >>>>>>>>> about
> >>>>>>>>>>>>> individual batches with a single record in
> >>>>>>> __remote_log_metadata."
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The object merging in tier storage typically only kicks
> >>> in
> >>>>>> after
> >>>>>>> a
> >>>>>>>>> few
> >>>>>>>>>>>>> hours. The impact is (1) the amount of accumulated
> >>>> metadata is
> >>>>>>>> still
> >>>>>>>>>>>> quite
> >>>>>>>>>>>>> large; (2) there are many small objects, leading to poor
> >>>> read
> >>>>>>>>>>>> performance.
> >>>>>>>>>>>>> I think we need a more frequent object merging process
> >>> than
> >>>>>> tier
> >>>>>>>>>> storage
> >>>>>>>>>>>>> provides.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> >>>>>> ivan@ivanyu.me>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello Jack, Jun, Luke, and all!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thank you for your messages.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let me first address some of Jun’s comments.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> First, it degrades the durability.
> >>>>>>>>>>>>>>> For each partition, now there are two files being
> >>>> actively
> >>>>>>>> written
> >>>>>>>>>> at a
> >>>>>>>>>>>>>>> given point of time, one for the data and another for
> >>> the
> >>>>>>>> metadata.
> >>>>>>>>>>>>>>> Flushing each file requires a separate IO. If the disk
> >>>> has 1K
> >>>>>>>> IOPS
> >>>>>>>>>> and
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>> have 5K partitions in a broker, currently we can afford
> >>>> to
> >>>>>>> flush
> >>>>>>>>> each
> >>>>>>>>>>>>>>> partition every 5 seconds, achieving an RPO of 5
> >>>> seconds. If
> >>>>>> we
> >>>>>>>>>> double
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> number of files per partition, we can only flush each
> >>>>>> partition
> >>>>>>>>> every
> >>>>>>>>>>>> 10
> >>>>>>>>>>>>>>> seconds, which makes RPO twice as bad.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As I understand, you’re speaking about locally
> >>>> materialized
> >>>>>>>>> segments.
> >>>>>>>>>>>> They
> >>>>>>>>>>>>>> will indeed consume some IOPS. See them as a cache that
> >>>> could
> >>>>>>>> always
> >>>>>>>>>> be
> >>>>>>>>>>>>>> restored from the remote storage. While it’s not ideal,
> >>>> it's
> >>>>>>> still
> >>>>>>>>> OK
> >>>>>>>>>> to
> >>>>>>>>>>>>>> lose data in them due to a machine crash, for example.
> >>>> Because
> >>>>>>> of
> >>>>>>>>>> this,
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>> can avoid explicit flushing on local materialized
> >>>> segments at
> >>>>>>> all
> >>>>>>>>> and
> >>>>>>>>>>>> let
> >>>>>>>>>>>>>> the file system and page cache figure out when to flush
> >>>>>>> optimally.
> >>>>>>>>>> This
> >>>>>>>>>>>>>> would not eliminate the extra IOPS, but should reduce it
> >>>>>>>>> dramatically,
> >>>>>>>>>>>>>> depending on throughput for each partition. We, of
> >>> course,
> >>>>>>>> continue
> >>>>>>>>>>>>>> flushing the metadata segments as before.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It’s worth making a note on caching. I think nobody will
> >>>>>>> disagree
> >>>>>>>>> that
> >>>>>>>>>>>>>> doing direct reads from remote storage every time a
> >>> batch
> >>>> is
> >>>>>>>>> requested
> >>>>>>>>>>>> by a
> >>>>>>>>>>>>>> consumer will not be practical neither from the
> >>>> performance
> >>>>>> nor
> >>>>>>>> from
> >>>>>>>>>> the
> >>>>>>>>>>>>>> economy point of view. We need a way to keep the number
> >>>> of GET
> >>>>>>>>>> requests
> >>>>>>>>>>>>>> down. There are multiple options, for example:
> >>>>>>>>>>>>>> 1. Rack-aware distributed in-memory caching.
> >>>>>>>>>>>>>> 2. Local in-memory caching. Comes with less network
> >>>> chattiness
> >>>>>>> and
> >>>>>>>>>>>> works
> >>>>>>>>>>>>>> well if we have more or less stable brokers to consume
> >>>> from.
> >>>>>>>>>>>>>> 3. Materialization of diskless logs on local disk. Way
> >>>> lower
> >>>>>>>> impact
> >>>>>>>>> on
> >>>>>>>>>>>>>> RAM and also requires stable brokers for consumption
> >>>> (using
> >>>>>> just
> >>>>>>>>>>>> assigned
> >>>>>>>>>>>>>> replicas will probably work well).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Materialization is one of possible options, but we can
> >>>> choose
> >>>>>>>>> another
> >>>>>>>>>>>> one.
> >>>>>>>>>>>>>> However, we will have this dilemma regardless of whether
> >>>> we
> >>>>>> have
> >>>>>>>> an
> >>>>>>>>>>>>>> explicit coordinator or we go “coordinator-less”.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Second, if we ever need this
> >>>>>>>>>>>>>>> metadata somewhere else, say in the WAL file manager,
> >>> the
> >>>>>>>> consumer
> >>>>>>>>>>>> needs
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> subscribe to every partition in the cluster, which is
> >>>>>>>> inefficient.
> >>>>>>>>>> The
> >>>>>>>>>>>>>>> actual benefit of this approach is also questionable.
> >>> On
> >>>> the
> >>>>>>>>> surface,
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>>> might seem that we could reduce the number of lines
> >>> that
> >>>> need
> >>>>>>> to
> >>>>>>>> be
> >>>>>>>>>>>>>> changed
> >>>>>>>>>>>>>>> for this KIP. However, the changes are quite intrusive
> >>>> to the
> >>>>>>>>> classic
> >>>>>>>>>>>>>>> partition's code path and will probably make the code
> >>>> base
> >>>>>>> harder
> >>>>>>>>> to
> >>>>>>>>>>>>>>> maintain in the long run. I like the original approach
> >>>> based
> >>>>>> on
> >>>>>>>> the
> >>>>>>>>>>>> batch
> >>>>>>>>>>>>>>> coordinator much better than this one. We could
> >>> probably
> >>>>>>> refactor
> >>>>>>>>> the
> >>>>>>>>>>>>>>> producer state code so that it could be reused in the
> >>>> batch
> >>>>>>>>>>>> coordinator.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It’s hard to disagree with this. The explicit
> >>> coordinator
> >>>> is
> >>>>>>> more
> >>>>>>>> a
> >>>>>>>>>> side
> >>>>>>>>>>>>>> thing, while coordinator-less approach is more about
> >>>> extending
> >>>>>>>>>>>>>> ReplicaManager, UnifiedLog and others substantially.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for addressing the concerns on the number of
> >>> RPCs
> >>>> in
> >>>>>> the
> >>>>>>>>>> produce
> >>>>>>>>>>>>>>> path. I agree that with the metadata crafting
> >>> mechanism,
> >>>> we
> >>>>>>> could
> >>>>>>>>>>>>>> mitigate
> >>>>>>>>>>>>>>> the PRC problem. However, since we now require the
> >>>> metadata
> >>>>>> to
> >>>>>>> be
> >>>>>>>>>>>>>>> collocated with the data on the same set of brokers,
> >>> it's
> >>>>>> weird
> >>>>>>>>> that
> >>>>>>>>>>>> they
> >>>>>>>>>>>>>>> are now managed by different mechanisms. The data
> >>>> assignment
> >>>>>>> now
> >>>>>>>>> uses
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> metadata crafting mechanism, but the metadata is stored
> >>>> in
> >>>>>> the
> >>>>>>>>>> classic
> >>>>>>>>>>>>>>> partition using its own assignment strategy. It will be
> >>>>>>>> complicated
> >>>>>>>>>> to
> >>>>>>>>>>>>>> keep
> >>>>>>>>>>>>>>> them collocated.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I would like to note that the metadata crafting is
> >>> needed
> >>>> only
> >>>>>>> to
> >>>>>>>>> tell
> >>>>>>>>>>>>>> producers which brokers they should send Produce
> >>> requests
> >>>> to,
> >>>>>>> but
> >>>>>>>>> data
> >>>>>>>>>>>> (as
> >>>>>>>>>>>>>> in “locally materialized log”) is located on partition
> >>>>>> replicas,
> >>>>>>>>> i.e.
> >>>>>>>>>>>>>> automatically co-located with metadata.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As a side note, it would probably be better that instead
> >>>> of
> >>>>>>>>> implicitly
> >>>>>>>>>>>>>> crafting partition metadata, we extend the metadata
> >>>> protocol
> >>>>>> so
> >>>>>>>> that
> >>>>>>>>>> for
> >>>>>>>>>>>>>> diskless partitions we return not only the leader and
> >>>>>> replicas,
> >>>>>>>> but
> >>>>>>>>>> also
> >>>>>>>>>>>>>> some “recommended produce brokers”, selected for optimal
> >>>>>>>> performance
> >>>>>>>>>> and
> >>>>>>>>>>>>>> costs. Producers will pick ones in their racks.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am also concerned about the removal of the object
> >>>>>>>>>> compaction/merging
> >>>>>>>>>>>>>>> step.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We should have mentioned this explicitly, but this step,
> >>>> in
> >>>>>>> fact,
> >>>>>>>>>>>> remains
> >>>>>>>>>>>>>> in the form of segments offloading to tiered storage.
> >>>> When we
> >>>>>>>>>> assemble a
> >>>>>>>>>>>>>> segment and hand it over to RemoteLogManager, we’re
> >>>>>> effectively
> >>>>>>>>> doing
> >>>>>>>>>>>>>> metadata compaction: replacing a big number of pieces of
> >>>>>>> metadata
> >>>>>>>>>> about
> >>>>>>>>>>>>>> individual batches with a single record in
> >>>>>>> __remote_log_metadata.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We could create a Diskless-specific merging mechanism
> >>>> instead
> >>>>>> if
> >>>>>>>>>> needed.
> >>>>>>>>>>>>>> It’s rather easy with the explicit coordinator approach.
> >>>> With
> >>>>>>> the
> >>>>>>>>>>>>>> coordinator-less approach, this would probably be a bit
> >>>> more
> >>>>>>>> tricky
> >>>>>>>>>>>>>> (rewriting the tail of the log by the leader +
> >>> replicating
> >>>>>> this
> >>>>>>>>> change
> >>>>>>>>>>>>>> reliably).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I see a tendency toward primarily optimizing for the
> >>>> fewest
> >>>>>>> code
> >>>>>>>>>>>> changes
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the KIP. Instead, our primary goal should be a clean
> >>>> design
> >>>>>>> that
> >>>>>>>>> can
> >>>>>>>>>>>> last
> >>>>>>>>>>>>>>> for the long term.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes, totally agree.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Luke,
> >>>>>>>>>>>>>>> I'm wondering if the complexity of designing txn and
> >>>> queue is
> >>>>>>>>> because
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>> leaderless cluster, do you think it will be simpler if
> >>> we
> >>>>>> only
> >>>>>>>>> focus
> >>>>>>>>>> on
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> "diskless" design to handle object compaction/merging
> >>>> to/from
> >>>>>>> the
> >>>>>>>>>>>> remote
> >>>>>>>>>>>>>>> storage to save the cross-AZ cost?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> After some evolution of the original proposal,
> >>> leaderless
> >>>> is
> >>>>>> now
> >>>>>>>>>>>> limited.
> >>>>>>>>>>>>>> We only need to be able to accept Produce requests on
> >>> more
> >>>>>> than
> >>>>>>>> one
> >>>>>>>>>>>> broker
> >>>>>>>>>>>>>> to eliminate the cross-AZ costs for producers. Do I get
> >>> it
> >>>>>> right
> >>>>>>>>> that
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>> propose to get rid of this? Or do I misunderstand?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let’s now look at this problem from a higher level, as
> >>>> Jack
> >>>>>>>>> proposed.
> >>>>>>>>>> As
> >>>>>>>>>>>>>> it was said, the big choice we need to make is whether
> >>> we
> >>>> 1)
> >>>>>>>> create
> >>>>>>>>> an
> >>>>>>>>>>>>>> explicit batch coordinator; or 2) go for the
> >>>> coordinator-less
> >>>>>>>>>> approach,
> >>>>>>>>>>>>>> where each diskless partition is managed by its leader
> >>> as
> >>>> in
> >>>>>>>> classic
> >>>>>>>>>>>> topics.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If we try to compare the two approaches:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Pluggability:
> >>>>>>>>>>>>>> - Explicit coordinator: Possible. For example, some
> >>>> setups may
> >>>>>>>>> benefit
> >>>>>>>>>>>>>> from batch metadata being stored in a cloud database
> >>>> (such as
> >>>>>>> AWS
> >>>>>>>>>>>> DynamoDB
> >>>>>>>>>>>>>> or GCP Spanner).
> >>>>>>>>>>>>>> - Coordinator-less: Impossible.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Scalability and fault tolerance:
> >>>>>>>>>>>>>> - Explicit coordinator: Depends on the implementation
> >>> and
> >>>> it’s
> >>>>>>>> also
> >>>>>>>>>>>>>> necessary to actively work for it.
> >>>>>>>>>>>>>> - Coordinator-less: Closer to classic Kafka topics.
> >>>> Scaling is
> >>>>>>>> done
> >>>>>>>>> by
> >>>>>>>>>>>>>> partition placement, partitions could fail
> >>> independently.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Separation of concerns:
> >>>>>>>>>>>>>> - Explicit coordinator: Very good. Diskless remains more
> >>>>>>>> independent
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>> classic topics in terms of code and workflows. For
> >>>> example,
> >>>>>> the
> >>>>>>>>>>>>>> above-mentioned non-tiered storage metadata compaction
> >>>>>> mechanism
> >>>>>>>>> could
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>> relatively simply implemented with it. As a flip side of
> >>>> this,
> >>>>>>>> some
> >>>>>>>>>>>>>> workflows (e.g. transactions) will have to be adapted.
> >>>>>>>>>>>>>> - Coordinator-less: Less so. It leans to the opposite:
> >>>>>> bringing
> >>>>>>>>>> diskless
> >>>>>>>>>>>>>> closer to classic topics. Some code paths and workflows
> >>>> could
> >>>>>> be
> >>>>>>>>> more
> >>>>>>>>>>>>>> straightforwardly reused, but they will inevitably have
> >>>> to be
> >>>>>>>>> adapted
> >>>>>>>>>> to
> >>>>>>>>>>>>>> accommodate both topic types as also discussed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cloud-nativeness. This is a vague concept, also related
> >>>> to the
> >>>>>>>>>> previous,
> >>>>>>>>>>>>>> but let’s try:
> >>>>>>>>>>>>>> - Explicit coordinator: Storing and processing metadata
> >>>>>>> separately
> >>>>>>>>>> makes
> >>>>>>>>>>>>>> it easier for brokers to take different roles, be purely
> >>>>>>> stateless
> >>>>>>>>> if
> >>>>>>>>>>>>>> needed, etc.
> >>>>>>>>>>>>>> - Coordinator-less: Less so. Something could be achieved
> >>>> with
> >>>>>>>>> creative
> >>>>>>>>>>>>>> partition placement, but not much.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Both seem to have their pros and cons. However,
> >>> answering
> >>>>>> Jack’s
> >>>>>>>>>>>> question,
> >>>>>>>>>>>>>> the explicit coordinator approach may indeed lead to a
> >>>> more
> >>>>>>>> flexible
> >>>>>>>>>>>> design.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The purpose of this deviation in the discussion was to
> >>>>>> receive a
> >>>>>>>>>>>>>> preliminary community evaluation of the coordinator-less
> >>>>>>> approach
> >>>>>>>>>>>> without
> >>>>>>>>>>>>>> taking on the task of writing a separate KIP and fitting
> >>>> it in
> >>>>>>> the
> >>>>>>>>>>>> system
> >>>>>>>>>>>>>> of KIP-1150 and its children. We’re open to stopping it
> >>>> and
> >>>>>>>> getting
> >>>>>>>>>>>> back to
> >>>>>>>>>>>>>> working out the coordinator design if the community
> >>>> doesn’t
> >>>>>>> favor
> >>>>>>>>> the
> >>>>>>>>>>>>>> proposed approach.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Ivan and Diskless team
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> >>>>>>>>>>>>>>> Hi Ivan,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As Jun pointed out, the updated design seems to have
> >>> some
> >>>>>>>>>> shortcomings
> >>>>>>>>>>>>>>> although it simplifies the implementation.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'm wondering if the complexity of designing txn and
> >>>> queue is
> >>>>>>>>> because
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>> leaderless cluster, do you think it will be simpler if
> >>> we
> >>>>>> only
> >>>>>>>>> focus
> >>>>>>>>>> on
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> "diskless" design to handle object compaction/merging
> >>>> to/from
> >>>>>>> the
> >>>>>>>>>>>> remote
> >>>>>>>>>>>>>>> storage to save the cross-AZ cost?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> >>>>>>> <jun@confluent.io.invalid
> >>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi, Ivan,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the explanation.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> "we write the reference to the WAL file with the batch
> >>>> data"
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I understand the approach now, but I think it is a
> >>> hacky
> >>>>>> one.
> >>>>>>>>> There
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> multiple short comings with this design. First, it
> >>>> degrades
> >>>>>>> the
> >>>>>>>>>>>>>> durability.
> >>>>>>>>>>>>>>>> For each partition, now there are two files being
> >>>> actively
> >>>>>>>> written
> >>>>>>>>>> at
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> given point of time, one for the data and another for
> >>>> the
> >>>>>>>>> metadata.
> >>>>>>>>>>>>>>>> Flushing each file requires a separate IO. If the disk
> >>>> has
> >>>>>> 1K
> >>>>>>>> IOPS
> >>>>>>>>>> and
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> have 5K partitions in a broker, currently we can
> >>> afford
> >>>> to
> >>>>>>> flush
> >>>>>>>>>> each
> >>>>>>>>>>>>>>>> partition every 5 seconds, achieving an RPO of 5
> >>>> seconds. If
> >>>>>>> we
> >>>>>>>>>> double
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> number of files per partition, we can only flush each
> >>>>>>> partition
> >>>>>>>>>> every
> >>>>>>>>>>>>>> 10
> >>>>>>>>>>>>>>>> seconds, which makes RPO twice as bad. Second, if we
> >>>> ever
> >>>>>> need
> >>>>>>>>> this
> >>>>>>>>>>>>>>>> metadata somewhere else, say in the WAL file manager,
> >>>> the
> >>>>>>>> consumer
> >>>>>>>>>>>>>> needs to
> >>>>>>>>>>>>>>>> subscribe to every partition in the cluster, which is
> >>>>>>>> inefficient.
> >>>>>>>>>> The
> >>>>>>>>>>>>>>>> actual benefit of this approach is also questionable.
> >>>> On the
> >>>>>>>>>> surface,
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> might seem that we could reduce the number of lines
> >>> that
> >>>>>> need
> >>>>>>> to
> >>>>>>>>> be
> >>>>>>>>>>>>>> changed
> >>>>>>>>>>>>>>>> for this KIP. However, the changes are quite intrusive
> >>>> to
> >>>>>> the
> >>>>>>>>>> classic
> >>>>>>>>>>>>>>>> partition's code path and will probably make the code
> >>>> base
> >>>>>>>> harder
> >>>>>>>>> to
> >>>>>>>>>>>>>>>> maintain in the long run. I like the original approach
> >>>> based
> >>>>>>> on
> >>>>>>>>> the
> >>>>>>>>>>>>>> batch
> >>>>>>>>>>>>>>>> coordinator much better than this one. We could
> >>> probably
> >>>>>>>> refactor
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> producer state code so that it could be reused in the
> >>>> batch
> >>>>>>>>>>>>>> coordinator.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for addressing the concerns on the number of
> >>>> RPCs in
> >>>>>>> the
> >>>>>>>>>>>> produce
> >>>>>>>>>>>>>>>> path. I agree that with the metadata crafting
> >>>> mechanism, we
> >>>>>>>> could
> >>>>>>>>>>>>>> mitigate
> >>>>>>>>>>>>>>>> the PRC problem. However, since we now require the
> >>>> metadata
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>>>>>>>> collocated with the data on the same set of brokers,
> >>>> it's
> >>>>>>> weird
> >>>>>>>>> that
> >>>>>>>>>>>>>> they
> >>>>>>>>>>>>>>>> are now managed by different mechanisms. The data
> >>>> assignment
> >>>>>>> now
> >>>>>>>>>> uses
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> metadata crafting mechanism, but the metadata is
> >>> stored
> >>>> in
> >>>>>> the
> >>>>>>>>>> classic
> >>>>>>>>>>>>>>>> partition using its own assignment strategy. It will
> >>> be
> >>>>>>>>> complicated
> >>>>>>>>>> to
> >>>>>>>>>>>>>> keep
> >>>>>>>>>>>>>>>> them collocated.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I am also concerned about the removal of the object
> >>>>>>>>>> compaction/merging
> >>>>>>>>>>>>>>>> step. My first concern is on the amount of metadata
> >>> that
> >>>>>> need
> >>>>>>> to
> >>>>>>>>> be
> >>>>>>>>>>>>>> kept.
> >>>>>>>>>>>>>>>> Without object compcation, the metadata generated in
> >>> the
> >>>>>>> produce
> >>>>>>>>>> path
> >>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>> only be deleted after remote tiering kicks in. Let's
> >>>> say for
> >>>>>>>> every
> >>>>>>>>>>>>>> 250ms we
> >>>>>>>>>>>>>>>> produce 100 byte of metadata per partition. Let's say
> >>>>>> remoting
> >>>>>>>>>> tiering
> >>>>>>>>>>>>>>>> kicks in after 5 hours. In a cluster with 100K
> >>>> partitions,
> >>>>>> we
> >>>>>>>> need
> >>>>>>>>>> to
> >>>>>>>>>>>>>> keep
> >>>>>>>>>>>>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> >>>> metadata,
> >>>>>>>> quite
> >>>>>>>>>>>>>>>> signficant. A second concern is on performance. Every
> >>>> time
> >>>>>> we
> >>>>>>>> need
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>> rebuild the caching data, we need to read a bunch of
> >>>> small
> >>>>>>>> objects
> >>>>>>>>>>>>>> from S3,
> >>>>>>>>>>>>>>>> slowing down the building process. If a consumer
> >>>> happens to
> >>>>>>> need
> >>>>>>>>>> such
> >>>>>>>>>>>>>> data,
> >>>>>>>>>>>>>>>> it could slow down the application.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I see a tendency toward primarily optimizing for the
> >>>> fewest
> >>>>>>> code
> >>>>>>>>>>>>>> changes in
> >>>>>>>>>>>>>>>> the KIP. Instead, our primary goal should be a clean
> >>>> design
> >>>>>>> that
> >>>>>>>>> can
> >>>>>>>>>>>>>> last
> >>>>>>>>>>>>>>>> for the long term.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> >>>>>>> ivan@ivanyu.me
> >>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thank you for your message. I’m sorry that I failed
> >>> to
> >>>>>>> clearly
> >>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> idea. Let me try to fix this.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Does each partition now have a metadata partition
> >>> and
> >>>> a
> >>>>>>>> separate
> >>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>> partition? If so, I am concerned that it essentially
> >>>>>> doubles
> >>>>>>>> the
> >>>>>>>>>>>>>> number
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> partitions, which impacts the number of open file
> >>>>>>> descriptors
> >>>>>>>>> and
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> required IOPS, and so on. It also seems wasteful to
> >>>> have a
> >>>>>>>>>> separate
> >>>>>>>>>>>>>>>>>> partition just to store the metadata. It's as if we
> >>>> are
> >>>>>>>> creating
> >>>>>>>>>> an
> >>>>>>>>>>>>>>>>>> internal topic with an unbounded number of
> >>> partitions.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> No. There will be only one physical partition per
> >>>> diskless
> >>>>>>>>>>>>>> partition. Let
> >>>>>>>>>>>>>>>>> me explain this with an example. Let’s say we have a
> >>>>>> diskless
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> >>>> leader. We
> >>>>>>>>> produce
> >>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>>> batches to this partition. The content of the segment
> >>>> file
> >>>>>>> will
> >>>>>>>>> be
> >>>>>>>>>>>>>>>>> something like this (for each batch):
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> >>>>>>>>>>>>>>>>> Length: 123456 (like in classic)
> >>>>>>>>>>>>>>>>> PartitionLeaderEpoch: like in classic
> >>>>>>>>>>>>>>>>> Magic: like in classic
> >>>>>>>>>>>>>>>>> CRC: like in classic
> >>>>>>>>>>>>>>>>> Attributes: like in classic
> >>>>>>>>>>>>>>>>> LastOffsetDelta: like in classic
> >>>>>>>>>>>>>>>>> BaseTimestamp: like in classic
> >>>>>>>>>>>>>>>>> MaxTimestamp: like in classic
> >>>>>>>>>>>>>>>>> ProducerId: like in classic
> >>>>>>>>>>>>>>>>> ProducerEpoch: like in classic
> >>>>>>>>>>>>>>>>> BaseSequence: like in classic
> >>>>>>>>>>>>>>>>> RecordsCount: like in classic
> >>>>>>>>>>>>>>>>> Records:
> >>>>>>>>>>>>>>>>>
> >>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> >>>>>> byte
> >>>>>>>>> offset
> >>>>>>>>>>>>>>>>> 123456
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It looks very much like classic log entries. The only
> >>>>>>>> difference
> >>>>>>>>> is
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> instead of writing real Records, we write the
> >>>> reference to
> >>>>>>> the
> >>>>>>>>> WAL
> >>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>> with the batch data (I guess we need only the name
> >>> and
> >>>> the
> >>>>>>> byte
> >>>>>>>>>>>>>> offset,
> >>>>>>>>>>>>>>>>> because the byte length is the standard field above).
> >>>>>>>> Otherwise,
> >>>>>>>>>>>>>> it’s a
> >>>>>>>>>>>>>>>>> normal Kafka log with the leader and replicas.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So we have as many partitions for diskless as for
> >>>> classic.
> >>>>>> As
> >>>>>>>> of
> >>>>>>>>>> open
> >>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>> descriptors, let’s proceed to the following:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Are the metadata and
> >>>>>>>>>>>>>>>>>> the data for the same partition always collocated on
> >>>> the
> >>>>>>> same
> >>>>>>>>>>>>>> broker?
> >>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>> so, how do we enforce that when replicas are
> >>>> reassigned?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The source of truth for the data is still in WAL
> >>> files
> >>>> on
> >>>>>>>> object
> >>>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>>>>> The source of truth for the metadata is in segment
> >>>> files on
> >>>>>>> the
> >>>>>>>>>>>>>> brokers
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the replica set. Two new mechanisms are planned, both
> >>>>>>>> independent
> >>>>>>>>>> of
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> new proposal, but I want to present them to give the
> >>>> idea
> >>>>>>> that
> >>>>>>>>>> only a
> >>>>>>>>>>>>>>>>> limited amount of data files will be operated
> >>> locally:
> >>>>>>>>>>>>>>>>> - We want to assemble batches into segment files and
> >>>>>> offload
> >>>>>>>> them
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>> tiered storage in order to prevent the unbounded
> >>>> growth of
> >>>>>>>> batch
> >>>>>>>>>>>>>>>> metadata.
> >>>>>>>>>>>>>>>>> For this, we need to open only  a few file
> >>> descriptors
> >>>> (for
> >>>>>>> the
> >>>>>>>>>>>>>> segment
> >>>>>>>>>>>>>>>>> file itself + the necessary indexes) before the
> >>>> segment is
> >>>>>>>> fully
> >>>>>>>>>>>>>> written
> >>>>>>>>>>>>>>>>> and handed over to RemoteLogManager.
> >>>>>>>>>>>>>>>>> - We want to assemble local segment files for caching
> >>>>>>> purposes
> >>>>>>>> as
> >>>>>>>>>>>>>> well,
> >>>>>>>>>>>>>>>>> i.e. to speed up fetching. This will not materialize
> >>>> the
> >>>>>> full
> >>>>>>>>>>>>>> content of
> >>>>>>>>>>>>>>>>> the log, but only the hot set according to some
> >>> policy
> >>>> (or
> >>>>>>>>>>>>>> configurable
> >>>>>>>>>>>>>>>>> policies), i.e. the number of segments and file
> >>>> descriptors
> >>>>>>>> will
> >>>>>>>>>>>>>> also be
> >>>>>>>>>>>>>>>>> limited.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The number of RPCs in the produce path is
> >>>> significantly
> >>>>>>>> higher.
> >>>>>>>>>> For
> >>>>>>>>>>>>>>>>>> example, if a produce request has 100 partitions,
> >>> in a
> >>>>>>> cluster
> >>>>>>>>>>>>>> with 100
> >>>>>>>>>>>>>>>>>> brokers, each produce request could generate 100
> >>> more
> >>>> RPC
> >>>>>>>>>> requests.
> >>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>> will significantly increase the request rate.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> This is a valid concern that we considered, but this
> >>>> issue
> >>>>>>> can
> >>>>>>>> be
> >>>>>>>>>>>>>>>>> mitigated. I’ll try to explain the approach.
> >>>>>>>>>>>>>>>>> The situation with a single broker is trivial: all
> >>> the
> >>>>>> commit
> >>>>>>>>>>>>>> requests go
> >>>>>>>>>>>>>>>>> from the broker to itself.
> >>>>>>>>>>>>>>>>> Let’s scale this to a multi-broker cluster, but
> >>>> located in
> >>>>>>> the
> >>>>>>>>>> single
> >>>>>>>>>>>>>>>> rack
> >>>>>>>>>>>>>>>>> (AZ). Any broker can accept Produce requests for
> >>>> diskless
> >>>>>>>>>>>>>> partitions, but
> >>>>>>>>>>>>>>>>> we can tell producers (through metadata) to always
> >>> send
> >>>>>>> Produce
> >>>>>>>>>>>>>> requests
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> leaders. For example, broker 0 hosts the leader
> >>>> replicas
> >>>>>> for
> >>>>>>>>>> diskless
> >>>>>>>>>>>>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> >>>>>> Produce
> >>>>>>>>>>>>>> requests
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> these partitions in various combinations, but only
> >>> for
> >>>>>> them.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>                  Broker 0
> >>>>>>>>>>>>>>>>>            +-----------------+
> >>>>>>>>>>>>>>>>>            |    t1-0         |
> >>>>>>>>>>>>>>>>>            |    t2-1 <--------------------+
> >>>>>>>>>>>>>>>>>            |    t3-0         |            |
> >>>>>>>>>>>>>>>>> produce      | +-------------+ |            |
> >>>>>>>>>>>>>>>>> requests     | |  diskless   | |            |
> >>>>>>>>>>>>>>>>> --------------->|   produce   +--------------+
> >>>>>>>>>>>>>>>>> for these    | | WAL buffer  | |    commit requests
> >>>>>>>>>>>>>>>>> partitions   | +-------------+ |    for these
> >>>> partitions
> >>>>>>>>>>>>>>>>>            |                 |
> >>>>>>>>>>>>>>>>>            +-----------------+
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The same applies for other brokers in this cluster.
> >>>>>>>> Effectively,
> >>>>>>>>>> each
> >>>>>>>>>>>>>>>>> broker will commit only to itself, which effectively
> >>>> means
> >>>>>> 1
> >>>>>>>>> commit
> >>>>>>>>>>>>>>>> request
> >>>>>>>>>>>>>>>>> per WAL buffer (this may be 0 physical network calls,
> >>>> if we
> >>>>>>>> wish,
> >>>>>>>>>>>>>> just a
> >>>>>>>>>>>>>>>>> local function call).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Now let’s scale this to multiple racks (AZs).
> >>>> Obviously, we
> >>>>>>>>> cannot
> >>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>>>> send Produce requests to the designated leaders of
> >>>> diskless
> >>>>>>>>>>>>>> partitions:
> >>>>>>>>>>>>>>>>> this would mean inter-AZ network traffic, which we
> >>>> would
> >>>>>> like
> >>>>>>>> to
> >>>>>>>>>>>>>> avoid.
> >>>>>>>>>>>>>>>> To
> >>>>>>>>>>>>>>>>> avoid it, we say that every broker has a “diskless
> >>>> produce
> >>>>>>>>>>>>>>>> representative”
> >>>>>>>>>>>>>>>>> in every AZ. If we continue our example: when a
> >>> Produce
> >>>>>>> request
> >>>>>>>>> for
> >>>>>>>>>>>>>> t1-0,
> >>>>>>>>>>>>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands
> >>>> on
> >>>>>>>> broker 0
> >>>>>>>>>>>>>> (in the
> >>>>>>>>>>>>>>>>> broker’s AZ the representative is the broker itself).
> >>>>>>> However,
> >>>>>>>> if
> >>>>>>>>>> it
> >>>>>>>>>>>>>>>> comes
> >>>>>>>>>>>>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker
> >>>> 2.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> |produce requests         |produce requests
> >>>> |produce
> >>>>>>>>>>>>>> requests
> >>>>>>>>>>>>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
> >>> |for
> >>>>>> t1-0,
> >>>>>>>>> t2-1,
> >>>>>>>>>>>>>>>> t3-0
> >>>>>>>>>>>>>>>>> |from AZ 0                |from AZ 1
> >>>> |from
> >>>>>> AZ 2
> >>>>>>>>>>>>>>>>> v                         v                        v
> >>>>>>>>>>>>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker
> >>> 2
> >>>> (AZ
> >>>>>> 2)
> >>>>>>>>>>>>>>>>> +---------------+      +---------------+
> >>>>>>> +---------------+
> >>>>>>>>>>>>>>>>> |     t1-0      |      |               |      |
> >>>>>>> |
> >>>>>>>>>>>>>>>>> |     t2-1      |      |               |      |
> >>>>>>> |
> >>>>>>>>>>>>>>>>> |     t3-0      |      |               |      |
> >>>>>>> |
> >>>>>>>>>>>>>>>>> +---------------+      +--------+------+
> >>>>>>> +--------+------+
> >>>>>>>>>>>>>>>>>   ^     ^                    |
> >>> |
> >>>>>>>>>>>>>>>>>   |     +--------------------+
> >>> |
> >>>>>>>>>>>>>>>>>   |     commit requests for these partitions
> >>> |
> >>>>>>>>>>>>>>>>>   |
> >>> |
> >>>>>>>>>>>>>>>>>
> >>> +-------------------------------------------------+
> >>>>>>>>>>>>>>>>>         commit requests for these partitions
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> All the partitions that broker 0 is the leader of
> >>> will
> >>>> be
> >>>>>>>>>>>>>> “represented”
> >>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>> brokers 1 and 2 in their AZs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Of course, this relationship goes both ways between
> >>> AZs
> >>>>>> (not
> >>>>>>>>>>>>>> necessarily
> >>>>>>>>>>>>>>>>> between the same brokers). It means that provided the
> >>>>>> cluster
> >>>>>>>> is
> >>>>>>>>>>>>>> balanced
> >>>>>>>>>>>>>>>>> by the number of brokers per AZ, each broker will
> >>>> represent
> >>>>>>>>>>>>>>>> (number_of_azs
> >>>>>>>>>>>>>>>>> - 1) other brokers. This will result in the situation
> >>>> that
> >>>>>>> for
> >>>>>>>>> the
> >>>>>>>>>>>>>>>> majority
> >>>>>>>>>>>>>>>>> of commits, each broker will do up to (number_of_azs
> >>> -
> >>>> 1)
> >>>>>>>> network
> >>>>>>>>>>>>>> commit
> >>>>>>>>>>>>>>>>> requests (plus one local). Cloud regions tend to
> >>> have 3
> >>>>>> AZs,
> >>>>>>>> very
> >>>>>>>>>>>>>> rarely
> >>>>>>>>>>>>>>>>> more. That means, brokers will be doing up to 2
> >>> network
> >>>>>>> commit
> >>>>>>>>>>>>>> requests
> >>>>>>>>>>>>>>>> per
> >>>>>>>>>>>>>>>>> WAL file.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> There are the following exceptions:
> >>>>>>>>>>>>>>>>> 1. Broker count imbalance between AZs. For example,
> >>>> when we
> >>>>>>>> have
> >>>>>>>>> 2
> >>>>>>>>>>>>>> AZs
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> one has three brokers and another AZ has one. This
> >>> one
> >>>>>> broker
> >>>>>>>>> will
> >>>>>>>>>> do
> >>>>>>>>>>>>>>>>> between 1 and 3 commit requests per WAL file. This is
> >>>> not
> >>>>>> an
> >>>>>>>>>> extreme
> >>>>>>>>>>>>>>>>> amplification. Such an imbalance is not healthy in
> >>> most
> >>>>>>>> practical
> >>>>>>>>>>>>>> setups
> >>>>>>>>>>>>>>>>> and should be avoided anyway.
> >>>>>>>>>>>>>>>>> 2. Leadership changes and metadata propagation
> >>> period.
> >>>> When
> >>>>>>> the
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> >>>>>>> producers
> >>>>>>>>>> will
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> know this immediately (unless we want to be strict
> >>> and
> >>>>>>> respond
> >>>>>>>>> with
> >>>>>>>>>>>>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> >>>> will
> >>>>>> come
> >>>>>>>>>>>>>> together
> >>>>>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>>>> WAL buffer on broker 2, it will have to send two
> >>> commit
> >>>>>>>> requests:
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>> broker
> >>>>>>>>>>>>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
> >>>> t3-0.
> >>>>>>> This
> >>>>>>>>>>>>>> situation
> >>>>>>>>>>>>>>>>> is not permanent and as producers update the cluster
> >>>>>>> metadata,
> >>>>>>>> it
> >>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>> resolved.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> This all could be built with the metadata crafting
> >>>>>> mechanism
> >>>>>>>> only
> >>>>>>>>>>>>>> (which
> >>>>>>>>>>>>>>>>> is anyway needed for Diskless in one way or another
> >>> to
> >>>>>> direct
> >>>>>>>>>>>>>> producers
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> consumers where we need to avoid inter-AZ traffic),
> >>>> just
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>>>>>>> right
> >>>>>>>>>>>>>>>>> policy for it (for example, some deterministic
> >>>> hash-based
> >>>>>>>>> formula).
> >>>>>>>>>>>>>> I.e.
> >>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>>> explicit support for “produce representative” or
> >>>> anything
> >>>>>>> like
> >>>>>>>>> this
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> needed on the cluster level, in KRaft, etc.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The same WAL file metadata is now duplicated into
> >>> two
> >>>>>>> places,
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>> leader and WAL File Manager. Which one is the source
> >>>> of
> >>>>>>> truth,
> >>>>>>>>> and
> >>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>>>> we maintain consistency between the two places?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> We do only two operations on WAL files that span
> >>>> multiple
> >>>>>>>>> diskless
> >>>>>>>>>>>>>>>>> partitions: committing and deleting. Commits can be
> >>>> done
> >>>>>>>>>>>>>> independently as
> >>>>>>>>>>>>>>>>> described above. But deletes are different, because
> >>>> when a
> >>>>>>> file
> >>>>>>>>> is
> >>>>>>>>>>>>>>>> deleted,
> >>>>>>>>>>>>>>>>> this affects all the partitions that still have alive
> >>>>>> batches
> >>>>>>>> in
> >>>>>>>>>> this
> >>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>> (if any).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The WAL file manager is a necessary point of
> >>>> coordination
> >>>>>> to
> >>>>>>>>> delete
> >>>>>>>>>>>>>> WAL
> >>>>>>>>>>>>>>>>> files safely. We can say it is the source of truth
> >>>> about
> >>>>>>> files
> >>>>>>>>>>>>>>>> themselves,
> >>>>>>>>>>>>>>>>> while the partition leaders and their logs hold the
> >>>> truth
> >>>>>>> about
> >>>>>>>>>>>>>> whether a
> >>>>>>>>>>>>>>>>> particular file contains live batches of this
> >>>> particular
> >>>>>>>>> partition.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The file manager will do this important task: be able
> >>>> to
> >>>>>> say
> >>>>>>>> for
> >>>>>>>>>> sure
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> a file does not contain any live batch of any
> >>> existing
> >>>>>>>> partition.
> >>>>>>>>>> For
> >>>>>>>>>>>>>>>> this,
> >>>>>>>>>>>>>>>>> it will have to periodically check against the
> >>>> partition
> >>>>>>>> leaders.
> >>>>>>>>>>>>>>>>> Considering that batch deletion is irreversible, when
> >>>> we
> >>>>>>>> declare
> >>>>>>>>> a
> >>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>> “empty”, this is guaranteed to be and stay so.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The file manager has to know about files being
> >>>> committed to
> >>>>>>>> start
> >>>>>>>>>>>>>> track
> >>>>>>>>>>>>>>>>> them and periodically check if they are empty. We can
> >>>>>>> consider
> >>>>>>>>>>>>>> various
> >>>>>>>>>>>>>>>> ways
> >>>>>>>>>>>>>>>>> to achieve this:
> >>>>>>>>>>>>>>>>> 1. As was proposed in my previous message: best
> >>> effort
> >>>>>> commit
> >>>>>>>> by
> >>>>>>>>>>>>>> brokers
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> periodic prefix scans of object storage to detect
> >>> files
> >>>>>> that
> >>>>>>>> went
> >>>>>>>>>>>>>> below
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> radar due to network issue or the file manager
> >>>> temporary
> >>>>>>>>>>>>>> unavailability.
> >>>>>>>>>>>>>>>>> We’re speaking about listing the file names only and
> >>>>>> opening
> >>>>>>>> only
> >>>>>>>>>>>>>>>>> previously unknown files in order to find the
> >>>> partitions
> >>>>>>>> involved
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>> them.
> >>>>>>>>>>>>>>>>> 2. Only do scans without explicit commit, i.e. fill
> >>> the
> >>>>>> list
> >>>>>>> of
> >>>>>>>>>> files
> >>>>>>>>>>>>>>>>> fully asynchronously and in the background. This may
> >>>> be not
> >>>>>>>> ideal
> >>>>>>>>>>>>>> due to
> >>>>>>>>>>>>>>>>> costs and performance of scanning tons of files.
> >>>> However,
> >>>>>> the
> >>>>>>>>>> number
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> live WAL files should be limited due to tiered
> >>> storage
> >>>>>>>>> offloading +
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>> optimize this if we give files some global soft order
> >>>> in
> >>>>>>> their
> >>>>>>>>>> names.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I am not sure how this design simplifies the
> >>>>>> implementation.
> >>>>>>>> The
> >>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>> producer/replication code can't be simply reused.
> >>>>>> Adjusting
> >>>>>>>> both
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> write
> >>>>>>>>>>>>>>>>>> path in the leader and the replication path in the
> >>>>>> follower
> >>>>>>> to
> >>>>>>>>>>>>>>>> understand
> >>>>>>>>>>>>>>>>>> batch-header only data is quite intrusive to the
> >>>> existing
> >>>>>>>> logic.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It is true that we’ll have to change LocalLog and
> >>>>>> UnifiedLog
> >>>>>>> in
> >>>>>>>>>>>>>> order to
> >>>>>>>>>>>>>>>>> support these changes. However, it seems that
> >>>> idempotence,
> >>>>>>>>>>>>>> transactions,
> >>>>>>>>>>>>>>>>> queues, tiered storage will have to be changed less
> >>>> than
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>> design. This is because the partition leader state
> >>>> would
> >>>>>>> remain
> >>>>>>>>> in
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>> place (on brokers) and existing workflows that
> >>> involve
> >>>> it
> >>>>>>> would
> >>>>>>>>>> have
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> changed less compared to the situation where we
> >>>> globalize
> >>>>>> the
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> leader state in the batch coordinator. I admit this
> >>> is
> >>>> hard
> >>>>>>> to
> >>>>>>>>> make
> >>>>>>>>>>>>>>>>> convincing without both real implementations to hand
> >>> :)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I am also
> >>>>>>>>>>>>>>>>>> not sure how this enables seamless switching the
> >>> topic
> >>>>>> modes
> >>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>> diskless and classic. Could you provide more details
> >>>> on
> >>>>>>> those?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Let’s consider the scenario of turning a classic
> >>> topic
> >>>> into
> >>>>>>>>>>>>>> diskless. The
> >>>>>>>>>>>>>>>>> user sets diskless.enabled=true, the leader receives
> >>>> this
> >>>>>>>>> metadata
> >>>>>>>>>>>>>> update
> >>>>>>>>>>>>>>>>> and does the following:
> >>>>>>>>>>>>>>>>> 1. Stop accepting normal append writes.
> >>>>>>>>>>>>>>>>> 2. Close the current active segment.
> >>>>>>>>>>>>>>>>> 3. Start a new segment that will be written in the
> >>>> diskless
> >>>>>>>>> format
> >>>>>>>>>>>>>> (i.e.
> >>>>>>>>>>>>>>>>> without data).
> >>>>>>>>>>>>>>>>> 4. Start accepting diskless commits.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Since it’s the same log, the followers will know
> >>> about
> >>>> that
> >>>>>>>>> switch
> >>>>>>>>>>>>>>>>> consistently. They will finish replicating the
> >>> classic
> >>>>>>> segments
> >>>>>>>>> and
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>> replicating the diskless ones. They will always know
> >>>> where
> >>>>>>> each
> >>>>>>>>>>>>>> batch is
> >>>>>>>>>>>>>>>>> located (either inside a classic segment or
> >>> referenced
> >>>> by a
> >>>>>>>>>> diskless
> >>>>>>>>>>>>>>>> one).
> >>>>>>>>>>>>>>>>> Switching back should be similar.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Doing this with the coordinator is possible, but has
> >>>> some
> >>>>>>>>> caveats.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>> leader must do the following:
> >>>>>>>>>>>>>>>>> 1. Stop accepting normal append writes.
> >>>>>>>>>>>>>>>>> 2. Close the current active segment.
> >>>>>>>>>>>>>>>>> 3. Write a special control segment to persist and
> >>>> replicate
> >>>>>>> the
> >>>>>>>>>> fact
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> from offset N the partition is now in the diskless
> >>>> mode.
> >>>>>>>>>>>>>>>>> 4. Inform the coordinator about the first offset N of
> >>>> the
> >>>>>>>>> “diskless
> >>>>>>>>>>>>>> era”.
> >>>>>>>>>>>>>>>>> 5. Inform the controller quorum that the transition
> >>> has
> >>>>>>>> finished
> >>>>>>>>>> and
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> brokers now can process diskless writes for this
> >>>> partition.
> >>>>>>>>>>>>>>>>> This could fail at some points, so this will probably
> >>>>>> require
> >>>>>>>>> some
> >>>>>>>>>>>>>>>>> explicit state machine with replication either in the
> >>>>>>> partition
> >>>>>>>>> log
> >>>>>>>>>>>>>> or in
> >>>>>>>>>>>>>>>>> KRaft.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It seems that the coordinator-less approach makes
> >>> this
> >>>>>>> simpler
> >>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> “coordinator” for the partition and the partition
> >>>> leader
> >>>>>> are
> >>>>>>>> the
> >>>>>>>>>>>>>> same and
> >>>>>>>>>>>>>>>>> they store the partition metadata in the same log,
> >>> too.
> >>>>>> While
> >>>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> coordinator approach we have to perform some kind of
> >>> a
> >>>>>>>>> distributed
> >>>>>>>>>>>>>> commit
> >>>>>>>>>>>>>>>>> to handover metadata management from the classic
> >>>> partition
> >>>>>>>> leader
> >>>>>>>>>> to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> batch coordinator.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I hope these explanations help to clarify the idea.
> >>>> Please
> >>>>>>> let
> >>>>>>>> me
> >>>>>>>>>>>>>> know if
> >>>>>>>>>>>>>>>>> I should go deeper anywhere.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>> Ivan and the Diskless team
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> >>>>>>>>>>>>>>>>>> Hi, Ivan,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the update.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I am not sure that I fully understand the new
> >>> design,
> >>>> but
> >>>>>> it
> >>>>>>>>> seems
> >>>>>>>>>>>>>> less
> >>>>>>>>>>>>>>>>>> clean than before.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Does each partition now have a metadata partition
> >>> and
> >>>> a
> >>>>>>>> separate
> >>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>> partition? If so, I am concerned that it essentially
> >>>>>> doubles
> >>>>>>>> the
> >>>>>>>>>>>>>> number
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> partitions, which impacts the number of open file
> >>>>>>> descriptors
> >>>>>>>>> and
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> required IOPS, and so on. It also seems wasteful to
> >>>> have a
> >>>>>>>>>> separate
> >>>>>>>>>>>>>>>>>> partition just to store the metadata. It's as if we
> >>>> are
> >>>>>>>> creating
> >>>>>>>>>> an
> >>>>>>>>>>>>>>>>>> internal topic with an unbounded number of
> >>>> partitions. Are
> >>>>>>> the
> >>>>>>>>>>>>>> metadata
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> the data for the same partition always collocated on
> >>>> the
> >>>>>>> same
> >>>>>>>>>>>>>> broker?
> >>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>> so, how do we enforce that when replicas are
> >>>> reassigned?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The number of RPCs in the produce path is
> >>>> significantly
> >>>>>>>> higher.
> >>>>>>>>>> For
> >>>>>>>>>>>>>>>>>> example, if a produce request has 100 partitions,
> >>> in a
> >>>>>>> cluster
> >>>>>>>>>>>>>> with 100
> >>>>>>>>>>>>>>>>>> brokers, each produce request could generate 100
> >>> more
> >>>> RPC
> >>>>>>>>>> requests.
> >>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>> will significantly increase the request rate.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The same WAL file metadata is now duplicated into
> >>> two
> >>>>>>> places,
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>> leader and WAL File Manager. Which one is the source
> >>>> of
> >>>>>>> truth,
> >>>>>>>>> and
> >>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>>>> we maintain consistency between the two places?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I am not sure how this design simplifies the
> >>>>>> implementation.
> >>>>>>>> The
> >>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>> producer/replication code can't be simply reused.
> >>>>>> Adjusting
> >>>>>>>> both
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> write
> >>>>>>>>>>>>>>>>>> path in the leader and the replication path in the
> >>>>>> follower
> >>>>>>> to
> >>>>>>>>>>>>>>>> understand
> >>>>>>>>>>>>>>>>>> batch-header only data is quite intrusive to the
> >>>> existing
> >>>>>>>>> logic. I
> >>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>> not sure how this enables seamless switching the
> >>> topic
> >>>>>> modes
> >>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>> diskless and classic. Could you provide more details
> >>>> on
> >>>>>>> those?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> >>>>>>> ivan@ivanyu.me
> >>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi dear Kafka community,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> In the initial Diskless proposal, we proposed to
> >>>> have a
> >>>>>>>>> separate
> >>>>>>>>>>>>>>>>>>> component, batch/diskless coordinator, whose role
> >>>> would
> >>>>>> be
> >>>>>>> to
> >>>>>>>>>>>>>>>> centrally
> >>>>>>>>>>>>>>>>>>> manage the batch and WAL file metadata for diskless
> >>>>>> topics.
> >>>>>>>>> This
> >>>>>>>>>>>>>>>>> component
> >>>>>>>>>>>>>>>>>>> drew many reasonable comments from the community
> >>>> about
> >>>>>> how
> >>>>>>> it
> >>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>> various Kafka features (transactions, queues) and
> >>> its
> >>>>>>>>>>>>>> scalability.
> >>>>>>>>>>>>>>>>> While we
> >>>>>>>>>>>>>>>>>>> believe we have good answers to all the expressed
> >>>>>> concerns,
> >>>>>>>> we
> >>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>> step
> >>>>>>>>>>>>>>>>>>> back and looked at the problem from a different
> >>>>>>> perspective.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We would like to propose an alternative Diskless
> >>>> design
> >>>>>>>>> *without
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> centralized coordinator*. We believe this approach
> >>>> has
> >>>>>>>>> potential
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> propose to discuss it as it may be more appealing
> >>> to
> >>>> the
> >>>>>>>>>>>>>> community.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Let us explain the idea. Most of the complications
> >>>> with
> >>>>>> the
> >>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>> Diskless approach come from one necessary
> >>>> architecture
> >>>>>>>> change:
> >>>>>>>>>>>>>>>>> globalizing
> >>>>>>>>>>>>>>>>>>> the local state of partition leader in the batch
> >>>>>>> coordinator.
> >>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>> causes
> >>>>>>>>>>>>>>>>>>> deviations to the established workflows in various
> >>>>>> features
> >>>>>>>>> like
> >>>>>>>>>>>>>>>>> produce
> >>>>>>>>>>>>>>>>>>> idempotence and transactions, queues, retention,
> >>> etc.
> >>>>>> These
> >>>>>>>>>>>>>>>> deviations
> >>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>> to be carefully considered, designed, and later
> >>>>>> implemented
> >>>>>>>> and
> >>>>>>>>>>>>>>>>> tested. In
> >>>>>>>>>>>>>>>>>>> the new approach we want to avoid this by making
> >>>>>> partition
> >>>>>>>>>>>>>> leaders
> >>>>>>>>>>>>>>>>> again
> >>>>>>>>>>>>>>>>>>> responsible for managing their partitions, even in
> >>>>>> diskless
> >>>>>>>>>>>>>> topics.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> In classic Kafka topics, batch data and metadata
> >>> are
> >>>>>>> blended
> >>>>>>>>>>>>>> together
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> the one partition log. The crux of the Diskless
> >>> idea
> >>>> is
> >>>>>> to
> >>>>>>>>>>>>>> decouple
> >>>>>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>>>> and move data to the remote storage, while keeping
> >>>>>> metadata
> >>>>>>>>>>>>>> somewhere
> >>>>>>>>>>>>>>>>> else.
> >>>>>>>>>>>>>>>>>>> Using the central batch coordinator for managing
> >>>> batch
> >>>>>>>> metadata
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>>>>> but not the only.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Let’s now think about managing metadata for each
> >>> user
> >>>>>>>> partition
> >>>>>>>>>>>>>>>>>>> independently. Generally partitions are independent
> >>>> and
> >>>>>>> don’t
> >>>>>>>>>>>>>> share
> >>>>>>>>>>>>>>>>>>> anything apart from that their data are mixed in
> >>> WAL
> >>>>>> files.
> >>>>>>>> If
> >>>>>>>>> we
> >>>>>>>>>>>>>>>>> figure
> >>>>>>>>>>>>>>>>>>> out how to commit and later delete WAL files
> >>> safely,
> >>>> we
> >>>>>>> will
> >>>>>>>>>>>>>> achieve
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> necessary autonomy that allows us to get rid of the
> >>>>>> central
> >>>>>>>>> batch
> >>>>>>>>>>>>>>>>>>> coordinator. Instead, *each diskless user partition
> >>>> will
> >>>>>> be
> >>>>>>>>>>>>>> managed
> >>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>> its
> >>>>>>>>>>>>>>>>>>> leader*, as in classic Kafka topics. Also like in
> >>>> classic
> >>>>>>>>>>>>>> topics, the
> >>>>>>>>>>>>>>>>>>> leader uses the partition log as the way to persist
> >>>> batch
> >>>>>>>>>>>>>> metadata,
> >>>>>>>>>>>>>>>>> i.e.
> >>>>>>>>>>>>>>>>>>> the regular batch header + the information about
> >>> how
> >>>> to
> >>>>>>> find
> >>>>>>>>> this
> >>>>>>>>>>>>>>>>> batch on
> >>>>>>>>>>>>>>>>>>> remote storage. In contrast to classic topics,
> >>> batch
> >>>> data
> >>>>>>> is
> >>>>>>>> in
> >>>>>>>>>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For clarity, let’s compare the three designs:
> >>>>>>>>>>>>>>>>>>> • Classic topics:
> >>>>>>>>>>>>>>>>>>> • Data and metadata are co-located in the
> >>> partition
> >>>> log.
> >>>>>>>>>>>>>>>>>>> • The partition log content: [Batch header
> >>>>>>> (metadata)|Batch
> >>>>>>>>>>>>>> data].
> >>>>>>>>>>>>>>>>>>> • The partition log is replicated to the
> >>> followers.
> >>>>>>>>>>>>>>>>>>> • The replicas and leader have local state built
> >>>> from
> >>>>>>>>>>>>>> metadata.
> >>>>>>>>>>>>>>>>>>> • Original Diskless:
> >>>>>>>>>>>>>>>>>>> • Metadata is in the batch coordinator, data is on
> >>>>>> remote
> >>>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>>>>>>> • The partition state is global in the batch
> >>>>>> coordinator.
> >>>>>>>>>>>>>>>>>>> • New Diskless:
> >>>>>>>>>>>>>>>>>>> • Metadata is in the partition log, data is on
> >>>> remote
> >>>>>>>> storage.
> >>>>>>>>>>>>>>>>>>> • Partition log content: [Batch header
> >>>> (metadata)|Batch
> >>>>>>>>>>>>>>>> coordinates
> >>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>> remote storage].
> >>>>>>>>>>>>>>>>>>> • The partition log is replicated to the
> >>> followers.
> >>>>>>>>>>>>>>>>>>> • The replicas and leader have local state built
> >>>> from
> >>>>>>>>>>>>>> metadata.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Let’s consider the produce path. Here’s the
> >>> reminder
> >>>> of
> >>>>>> the
> >>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>> Diskless design:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The new approach could be depicted as the
> >>> following:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> As you can see, the main difference is that now
> >>>> instead
> >>>>>> of
> >>>>>>> a
> >>>>>>>>>>>>>> single
> >>>>>>>>>>>>>>>>> commit
> >>>>>>>>>>>>>>>>>>> request to the batch coordinator, we send multiple
> >>>>>> parallel
> >>>>>>>>>>>>>> commit
> >>>>>>>>>>>>>>>>> requests
> >>>>>>>>>>>>>>>>>>> to all the leaders of each partition involved in
> >>> the
> >>>> WAL
> >>>>>>>> file.
> >>>>>>>>>>>>>> Each
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>>>> will commit its batches independently, without
> >>>>>> coordinating
> >>>>>>>>> with
> >>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>> leaders and any other components. Batch data is
> >>>> addressed
> >>>>>>> by
> >>>>>>>>> the
> >>>>>>>>>>>>>> WAL
> >>>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>>>> name, the byte offset and size, which allows
> >>>> partitions
> >>>>>> to
> >>>>>>>> know
> >>>>>>>>>>>>>>>> nothing
> >>>>>>>>>>>>>>>>>>> about other partitions to access their data in
> >>>> shared WAL
> >>>>>>>>> files.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The number of partitions involved in a single WAL
> >>>> file
> >>>>>> may
> >>>>>>> be
> >>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>>> large,
> >>>>>>>>>>>>>>>>>>> e.g. a hundred. A hundred network requests to
> >>> commit
> >>>> one
> >>>>>>> WAL
> >>>>>>>>>>>>>> file is
> >>>>>>>>>>>>>>>>> very
> >>>>>>>>>>>>>>>>>>> impractical. However, there are ways to reduce this
> >>>>>> number:
> >>>>>>>>>>>>>>>>>>> 1. Partition leaders are located on brokers.
> >>>> Requests to
> >>>>>>>>>>>>>> leaders on
> >>>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>>> broker could be grouped together into a single
> >>>> physical
> >>>>>>>> network
> >>>>>>>>>>>>>>>> request
> >>>>>>>>>>>>>>>>>>> (resembling the normal Produce request that may
> >>> carry
> >>>>>>> batches
> >>>>>>>>> for
> >>>>>>>>>>>>>>>> many
> >>>>>>>>>>>>>>>>>>> partitions inside). This will cap the number of
> >>>> network
> >>>>>>>>> requests
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> number of brokers in the cluster.
> >>>>>>>>>>>>>>>>>>> 2. If we craft the cluster metadata to make
> >>> producers
> >>>>>> send
> >>>>>>>>> their
> >>>>>>>>>>>>>>>>> requests
> >>>>>>>>>>>>>>>>>>> to the right brokers (with respect to AZs), we may
> >>>>>> achieve
> >>>>>>>> the
> >>>>>>>>>>>>>> higher
> >>>>>>>>>>>>>>>>>>> concentration of logical commit requests in
> >>> physical
> >>>>>>> network
> >>>>>>>>>>>>>> requests
> >>>>>>>>>>>>>>>>>>> reducing the number of the latter ones even
> >>> further,
> >>>>>>> ideally
> >>>>>>>> to
> >>>>>>>>>>>>>> one.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Obviously, out of multiple commit requests some may
> >>>> fail
> >>>>>> or
> >>>>>>>>> time
> >>>>>>>>>>>>>> out
> >>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>> variety of reasons. This is fine. Some producers
> >>> will
> >>>>>>> receive
> >>>>>>>>>>>>>> totally
> >>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> partially failed responses to their Produce
> >>> requests,
> >>>>>>> similar
> >>>>>>>>> to
> >>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>> they
> >>>>>>>>>>>>>>>>>>> would have received when appending to a classic
> >>> topic
> >>>>>> fails
> >>>>>>>> or
> >>>>>>>>>>>>>> times
> >>>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>> If a partition experiences problems, other
> >>> partitions
> >>>>>> will
> >>>>>>>> not
> >>>>>>>>> be
> >>>>>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>>>> (again, like in classic topics). Of course, the
> >>>>>> uncommitted
> >>>>>>>>> data
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> garbage in WAL files. But WAL files are short-lived
> >>>>>>> (batches
> >>>>>>>>> are
> >>>>>>>>>>>>>>>>> constantly
> >>>>>>>>>>>>>>>>>>> assembled into segments and offloaded to tiered
> >>>> storage),
> >>>>>>> so
> >>>>>>>>> this
> >>>>>>>>>>>>>>>>> garbage
> >>>>>>>>>>>>>>>>>>> will be eventually deleted.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For safely deleting WAL files we now need to
> >>>> centrally
> >>>>>>> manage
> >>>>>>>>>>>>>> them,
> >>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>> this is the only state and logic that spans
> >>> multiple
> >>>>>>>>> partitions.
> >>>>>>>>>>>>>> On
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> diagram, you can see another commit request called
> >>>>>> “Commit
> >>>>>>>> file
> >>>>>>>>>>>>>> (best
> >>>>>>>>>>>>>>>>>>> effort)” going to the WAL File Manager. This
> >>> manager
> >>>> will
> >>>>>>> be
> >>>>>>>>>>>>>>>>> responsible
> >>>>>>>>>>>>>>>>>>> for the following:
> >>>>>>>>>>>>>>>>>>> 1. Collecting (by requests from brokers) and
> >>>> persisting
> >>>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>> committed WAL files.
> >>>>>>>>>>>>>>>>>>> 2. To handle potential failures in file information
> >>>>>>> delivery,
> >>>>>>>>> it
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> doing prefix scan on the remote storage
> >>> periodically
> >>>> to
> >>>>>>> find
> >>>>>>>>> and
> >>>>>>>>>>>>>>>>> register
> >>>>>>>>>>>>>>>>>>> unknown files. The period of this scan will be
> >>>>>> configurable
> >>>>>>>> and
> >>>>>>>>>>>>>>>> ideally
> >>>>>>>>>>>>>>>>>>> should be quite long.
> >>>>>>>>>>>>>>>>>>> 3. Checking with the relevant partition leaders
> >>>> (after a
> >>>>>>>> grace
> >>>>>>>>>>>>>>>>> period) if
> >>>>>>>>>>>>>>>>>>> they still have batches in a particular file.
> >>>>>>>>>>>>>>>>>>> 4. Physically deleting files when they aren’t
> >>> anymore
> >>>>>>>> referred
> >>>>>>>>>>>>>> to by
> >>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>> partition.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This new design offers the following advantages:
> >>>>>>>>>>>>>>>>>>> 1. It simplifies the implementation of many Kafka
> >>>>>> features
> >>>>>>>> such
> >>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>> idempotence, transactions, queues, tiered storage,
> >>>>>>> retention.
> >>>>>>>>>>>>>> Now we
> >>>>>>>>>>>>>>>>> don’t
> >>>>>>>>>>>>>>>>>>> need to abstract away and reuse the code from
> >>>> partition
> >>>>>>>> leaders
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> batch coordinator. Instead, we will literally use
> >>> the
> >>>>>> same
> >>>>>>>> code
> >>>>>>>>>>>>>> paths
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> leaders, with little adaptation. Workflows from
> >>>> classic
> >>>>>>>> topics
> >>>>>>>>>>>>>> mostly
> >>>>>>>>>>>>>>>>>>> remain unchanged.
> >>>>>>>>>>>>>>>>>>> For example, it seems that
> >>>>>>>>>>>>>>>>>>>
> >>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> >>>>>>>> and
> >>>>>>>>>>>>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> >>>>>> transaction
> >>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> partition leader side could be used for diskless
> >>>> topics
> >>>>>>> with
> >>>>>>>>>>>>>> little
> >>>>>>>>>>>>>>>>>>> adaptation. ProducerStateManager, needed for both
> >>>>>>> idempotent
> >>>>>>>>>>>>>> produce
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> transactions, would be reused.
> >>>>>>>>>>>>>>>>>>> Another example is share groups support, where the
> >>>> share
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> leader,
> >>>>>>>>>>>>>>>>>>> being co-located with the partition leader, would
> >>>> execute
> >>>>>>> the
> >>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>> for both diskless and classic topics.
> >>>>>>>>>>>>>>>>>>> 2. It returns to the familiar partition-based
> >>> scaling
> >>>>>>> model,
> >>>>>>>>>>>>>> where
> >>>>>>>>>>>>>>>>>>> partitions are independent.
> >>>>>>>>>>>>>>>>>>> 3. It makes the operation and failure patterns
> >>>> closer to
> >>>>>>> the
> >>>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>> ones from classic topics.
> >>>>>>>>>>>>>>>>>>> 4. It opens a straightforward path to seamless
> >>>> switching
> >>>>>>> the
> >>>>>>>>>>>>>> topics
> >>>>>>>>>>>>>>>>> modes
> >>>>>>>>>>>>>>>>>>> between diskless and classic.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The rest of the things remain unchanged compared to
> >>>> the
> >>>>>>>>> previous
> >>>>>>>>>>>>>>>>> Diskless
> >>>>>>>>>>>>>>>>>>> design (after all previous discussions). Such
> >>> things
> >>>> as
> >>>>>>> local
> >>>>>>>>>>>>>> segment
> >>>>>>>>>>>>>>>>>>> materialization by replicas, the consume path,
> >>> tiered
> >>>>>>> storage
> >>>>>>>>>>>>>>>>> integration,
> >>>>>>>>>>>>>>>>>>> etc.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If the community finds this design more suitable,
> >>> we
> >>>> will
> >>>>>>>>> update
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> KIP(s) accordingly and continue working on it.
> >>>> Please let
> >>>>>>> us
> >>>>>>>>> know
> >>>>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>> think.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>> Ivan and Diskless team
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
> >>> wrote:
> >>>>>>>>>>>>>>>>>>>> Hi Justine,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Yes, you're right. We need to track the aborted
> >>>>>>> transactions
> >>>>>>>>>>>>>> for in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> diskless coordinator for as long as the
> >>> corresponding
> >>>>>>> offsets
> >>>>>>>>> are
> >>>>>>>>>>>>>>>>> there.
> >>>>>>>>>>>>>>>>>>> With the tiered storage unification Greg mentioned
> >>>>>> earlier,
> >>>>>>>>> this
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> finite time even for infinite data retention.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> Ivan
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> >>>> wrote:
> >>>>>>>>>>>>>>>>>>>>> Hey Ivan,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the response. I think most of what you
> >>>> said
> >>>>>>> made
> >>>>>>>>>>>>>>>> sense,
> >>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>> did have some questions about this part:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> As we understand this, the partition leader in
> >>>> classic
> >>>>>>>>>>>>>> topics
> >>>>>>>>>>>>>>>>> forgets
> >>>>>>>>>>>>>>>>>>>>> about a transaction once it’s replicated (HWM
> >>>>>> overpasses
> >>>>>>>>>>>>>> it). The
> >>>>>>>>>>>>>>>>>>>>> transaction coordinator acts like the main
> >>>> guardian,
> >>>>>>>> allowing
> >>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>> leaders to do this safely. Please correct me if
> >>>> this is
> >>>>>>>>>>>>>> wrong. We
> >>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>> about relying on this with the batch coordinator
> >>>> and
> >>>>>>> delete
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>>>>>> about a transaction once it’s finished (as
> >>> there’s
> >>>> no
> >>>>>>>>>>>>>> replication
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> HWM
> >>>>>>>>>>>>>>>>>>>>> advances immediately).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I didn't quite understand this. In classic
> >>> topics,
> >>>> we
> >>>>>>> have
> >>>>>>>>>>>>>> maps
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> ongoing
> >>>>>>>>>>>>>>>>>>>>> transactions which remove state when the
> >>>> transaction is
> >>>>>>>>>>>>>> completed
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>> aborted transactions index which is retained for
> >>>> much
> >>>>>>>> longer.
> >>>>>>>>>>>>>>>> Once
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> transaction is completed, the coordinator is no
> >>>> longer
> >>>>>>>>>>>>>> involved
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> maintaining this partition side state, and it is
> >>>>>> subject
> >>>>>>> to
> >>>>>>>>>>>>>>>>> compaction
> >>>>>>>>>>>>>>>>>>> etc.
> >>>>>>>>>>>>>>>>>>>>> Looking back at the outline provided above, I
> >>>> didn't
> >>>>>> see
> >>>>>>>> much
> >>>>>>>>>>>>>>>>> about the
> >>>>>>>>>>>>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> >>>>>>> further.
> >>>>>>>> I
> >>>>>>>>>>>>>> saw
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> following in a response:
> >>>>>>>>>>>>>>>>>>>>>> When the broker constructs a fully valid local
> >>>>>> segment,
> >>>>>>>>>>>>>> all the
> >>>>>>>>>>>>>>>>>>> necessary
> >>>>>>>>>>>>>>>>>>>>> control batches will be inserted and indices,
> >>>> including
> >>>>>>> the
> >>>>>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>>>>>>>> index will be built to serve FetchRequests
> >>> exactly
> >>>> as
> >>>>>>> they
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>> today.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Based on this, it seems like we need to retain
> >>> the
> >>>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>> aborted txns for longer.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> Justine
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> >>>>>>>>>>>>>> ivan@ivanyu.me>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Justine and all,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thank you for your questions!
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> >>>>>> identified
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>> producer ID
> >>>>>>>>>>>>>>>>>>>>>>> and epoch, the positive result of this check
> >>>> could be
> >>>>>>>>>>>>>> cached
> >>>>>>>>>>>>>>>>>>> locally
> >>>>>>>>>>>>>>>>>>>>>>> Are we saying that only new transaction
> >>> version 2
> >>>>>>>>>>>>>>>> transactions
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>> here? If not, we can't uniquely identify
> >>>> transactions
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>> producer id +
> >>>>>>>>>>>>>>>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> You’re right that we (probably unintentionally)
> >>>>>> focused
> >>>>>>>>>>>>>> only on
> >>>>>>>>>>>>>>>>>>> version 2.
> >>>>>>>>>>>>>>>>>>>>>> We can either limit the support to version 2 or
> >>>>>> consider
> >>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>>>>>>>> surrogates to support version 1.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> >>>>>>> transactional
> >>>>>>>>>>>>>>>>> checks
> >>>>>>>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>>>>>> batches. This procedure would output the same
> >>>> errors
> >>>>>>>>>>>>>> like the
> >>>>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>>>> leader in classic topics would do.
> >>>>>>>>>>>>>>>>>>>>>>> Can you expand on what these checks are? Would
> >>>> you be
> >>>>>>>>>>>>>>>> checking
> >>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> transaction was still ongoing for example?* *
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> >>>>>>> ongoing,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> course
> >>>>>>>>>>>>>>>>>>>>>> the normal idempotence checks. What the
> >>> partition
> >>>>>> leader
> >>>>>>>>>>>>>> in the
> >>>>>>>>>>>>>>>>>>> classic
> >>>>>>>>>>>>>>>>>>>>>> topics does before appending a batch to the
> >>> local
> >>>> log
> >>>>>>>>>>>>>> (e.g. in
> >>>>>>>>>>>>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> >>>>>>>>>>>>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> >>>>>>> Diskless,
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> unfortunately
> >>>>>>>>>>>>>>>>>>>>>> cannot do these checks before appending the data
> >>>> to
> >>>>>> the
> >>>>>>>> WAL
> >>>>>>>>>>>>>>>>> segment
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> uploading it, but we can “tombstone” these
> >>>> batches in
> >>>>>>> the
> >>>>>>>>>>>>>> batch
> >>>>>>>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>>>>>>> during the final commit.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Is there state about ongoing
> >>>>>>>>>>>>>>>>>>>>>>> transactions in the batch coordinator? I see
> >>> some
> >>>>>> other
> >>>>>>>>>>>>>> state
> >>>>>>>>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> the End transaction section, but it's not super
> >>>> clear
> >>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>> state is
> >>>>>>>>>>>>>>>>>>>>>> stored
> >>>>>>>>>>>>>>>>>>>>>>> and when it is stored.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Right, this should have been more explicit. As
> >>> the
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> leader
> >>>>>>>>>>>>>>>>>>> tracks
> >>>>>>>>>>>>>>>>>>>>>> ongoing transactions for classic topics, the
> >>> batch
> >>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>> to as
> >>>>>>>>>>>>>>>>>>>>>> well. So when a transaction starts and ends, the
> >>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>>>>>>> must inform the batch coordinator about this.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> >>>> LSO --
> >>>>>>>>>>>>>> perhaps
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>> stored in the batch coordinator?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Yes. This could be deduced from the committed
> >>>> batches
> >>>>>>> and
> >>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>> information, but for the sake of performance
> >>> we’d
> >>>>>> better
> >>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> explicitly.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> >>>>>>> transactional
> >>>>>>>>>>>>>>>>> state is
> >>>>>>>>>>>>>>>>>>>>>>> maintained in the batch coordinator and how it
> >>>> will
> >>>>>> be
> >>>>>>>>>>>>>>>> cleaned
> >>>>>>>>>>>>>>>>> up?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> As we understand this, the partition leader in
> >>>> classic
> >>>>>>>>>>>>>> topics
> >>>>>>>>>>>>>>>>> forgets
> >>>>>>>>>>>>>>>>>>>>>> about a transaction once it’s replicated (HWM
> >>>>>> overpasses
> >>>>>>>>>>>>>> it).
> >>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>> transaction coordinator acts like the main
> >>>> guardian,
> >>>>>>>>>>>>>> allowing
> >>>>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>>> leaders to do this safely. Please correct me if
> >>>> this
> >>>>>> is
> >>>>>>>>>>>>>> wrong.
> >>>>>>>>>>>>>>>> We
> >>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>> about relying on this with the batch coordinator
> >>>> and
> >>>>>>>>>>>>>> delete the
> >>>>>>>>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>>>>>>> about a transaction once it’s finished (as
> >>>> there’s no
> >>>>>>>>>>>>>>>> replication
> >>>>>>>>>>>>>>>>>>> and HWM
> >>>>>>>>>>>>>>>>>>>>>> advances immediately).
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>> Ivan
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> >>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Hey folks,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Excited to see some updates related to
> >>>> transactions!
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I had a few questions.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> >>>>>> identified
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>> producer ID
> >>>>>>>>>>>>>>>>>>>>>>> and epoch, the positive result of this check
> >>>> could be
> >>>>>>>>>>>>>> cached
> >>>>>>>>>>>>>>>>>>> locally
> >>>>>>>>>>>>>>>>>>>>>>> Are we saying that only new transaction
> >>> version 2
> >>>>>>>>>>>>>>>> transactions
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>> here? If not, we can't uniquely identify
> >>>> transactions
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>> producer id +
> >>>>>>>>>>>>>>>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> >>>>>>> transactional
> >>>>>>>>>>>>>>>>> checks
> >>>>>>>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>>>>>> batches. This procedure would output the same
> >>>> errors
> >>>>>>>>>>>>>> like the
> >>>>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>>>> leader in classic topics would do.
> >>>>>>>>>>>>>>>>>>>>>>> Can you expand on what these checks are? Would
> >>>> you be
> >>>>>>>>>>>>>>>> checking
> >>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> transaction was still ongoing for example? Is
> >>>> there
> >>>>>>> state
> >>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>> ongoing
> >>>>>>>>>>>>>>>>>>>>>>> transactions in the batch coordinator? I see
> >>> some
> >>>>>> other
> >>>>>>>>>>>>>> state
> >>>>>>>>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> the End transaction section, but it's not super
> >>>> clear
> >>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>> state is
> >>>>>>>>>>>>>>>>>>>>>> stored
> >>>>>>>>>>>>>>>>>>>>>>> and when it is stored.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> >>>> LSO --
> >>>>>>>>>>>>>> perhaps
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>> stored in the batch coordinator?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> >>>>>>> transactional
> >>>>>>>>>>>>>>>>> state is
> >>>>>>>>>>>>>>>>>>>>>>> maintained in the batch coordinator and how it
> >>>> will
> >>>>>> be
> >>>>>>>>>>>>>>>> cleaned
> >>>>>>>>>>>>>>>>> up?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> >>>>>>>>>>>>>>>>> <ju...@confluent.io.invalid>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi, Greg and Ivan,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the update. A few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> >>>> local
> >>>>>>>>>>>>>>>> segments,
> >>>>>>>>>>>>>>>>>>> making
> >>>>>>>>>>>>>>>>>>>>>> use of
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> >>>>>> zero-copy
> >>>>>>>>>>>>>>>>>>> functionality
> >>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>> built into classic topics."
> >>>>>>>>>>>>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> >>>> state for
> >>>>>>>>>>>>>> each
> >>>>>>>>>>>>>>>>>>> partition in
> >>>>>>>>>>>>>>>>>>>>>>>> diskless topics?
> >>>>>>>>>>>>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> >>>> fetches
> >>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> know
> >>>>>>>>>>>>>>>>>>>>>> aborted
> >>>>>>>>>>>>>>>>>>>>>>>> records. How is that achieved?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> >>>>>>>>>>>>>> transaction is
> >>>>>>>>>>>>>>>>>>> finished
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> also inserts the control batches in the
> >>>>>> corresponding
> >>>>>>>>>>>>>> logs
> >>>>>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>>>>> involved
> >>>>>>>>>>>>>>>>>>>>>>>> Diskless topics. This happens only on the
> >>>> metadata
> >>>>>>>>>>>>>> level,
> >>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>>>>> actual
> >>>>>>>>>>>>>>>>>>>>>> control
> >>>>>>>>>>>>>>>>>>>>>>>> batches are written to any file. "
> >>>>>>>>>>>>>>>>>>>>>>>> A fetch response could include multiple
> >>>>>> transactional
> >>>>>>>>>>>>>>>>> batches.
> >>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>> does the
> >>>>>>>>>>>>>>>>>>>>>>>> broker obtain the information about the ending
> >>>>>> control
> >>>>>>>>>>>>>>>> batch
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> each
> >>>>>>>>>>>>>>>>>>>>>>>> batch? Does that mean that a fetch response
> >>>> needs to
> >>>>>>> be
> >>>>>>>>>>>>>>>>> built by
> >>>>>>>>>>>>>>>>>>>>>>>> stitching record batches and generated control
> >>>>>> batches
> >>>>>>>>>>>>>>>>> together?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> JR 12. Queues: Is there still a share
> >>> partition
> >>>>>> leader
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>> consumers
> >>>>>>>>>>>>>>>>>>>>>>>> are routed to?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
> >>>> this
> >>>>>> or
> >>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>> too
> >>>>>>>>>>>>>>>>>>>>>>>> implementation-focused?" It would be useful to
> >>>>>> include
> >>>>>>>>>>>>>>>> enough
> >>>>>>>>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> understand correctness and performance impact.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
> >>>> given
> >>>>>>>>>>>>>>>> producer
> >>>>>>>>>>>>>>>>>>> contain a
> >>>>>>>>>>>>>>>>>>>>>>>> sequence number, which is ordered. If a
> >>> producer
> >>>>>> sends
> >>>>>>>>>>>>>>>> every
> >>>>>>>>>>>>>>>>>>> Produce
> >>>>>>>>>>>>>>>>>>>>>>>> request to an arbitrary broker, those requests
> >>>> could
> >>>>>>>>>>>>>> reach
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> batch
> >>>>>>>>>>>>>>>>>>>>>>>> coordinator in different order and lead to
> >>>> rejection
> >>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>> produce
> >>>>>>>>>>>>>>>>>>>>>>>> requests.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
> >>> Yurchenko <
> >>>>>>>>>>>>>>>>> ivan@ivanyu.me>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> We have also thought in a bit more details
> >>>> about
> >>>>>>>>>>>>>>>>> transactions
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> queues,
> >>>>>>>>>>>>>>>>>>>>>>>>> here's the plan.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> *Transactions*
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The support for transactions in *classic
> >>>> topics* is
> >>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>> precise
> >>>>>>>>>>>>>>>>>>>>>>>>> interactions between three actors: clients
> >>>> (mostly
> >>>>>>>>>>>>>>>>> producers,
> >>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> >>>>>>>>>>>>>> classes),
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>>>>>>>>>>>> coordinators. Brokers also run partition
> >>>> leaders
> >>>>>> with
> >>>>>>>>>>>>>>>> their
> >>>>>>>>>>>>>>>>>>> local
> >>>>>>>>>>>>>>>>>>>>>> state
> >>>>>>>>>>>>>>>>>>>>>>>>> (ProducerStateManager and others).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The high level (some details skipped)
> >>> workflow
> >>>> is
> >>>>>> the
> >>>>>>>>>>>>>>>>>>> following.
> >>>>>>>>>>>>>>>>>>>>>> When a
> >>>>>>>>>>>>>>>>>>>>>>>>> transactional Produce request is received by
> >>>> the
> >>>>>>>>>>>>>> broker:
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. For each partition, the partition leader
> >>>> checks
> >>>>>>>>>>>>>> if a
> >>>>>>>>>>>>>>>>>>> non-empty
> >>>>>>>>>>>>>>>>>>>>>>>>> transaction is running for this partition.
> >>>> This is
> >>>>>>>>>>>>>> done
> >>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>> its
> >>>>>>>>>>>>>>>>>>>>>> local
> >>>>>>>>>>>>>>>>>>>>>>>>> state derived from the log metadata
> >>>>>>>>>>>>>>>> (ProducerStateManager,
> >>>>>>>>>>>>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> >>>> about
> >>>>>> all
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> aren’t part of the transaction to include
> >>> them.
> >>>>>>>>>>>>>>>>>>>>>>>>> 3. The partition leaders do additional
> >>>>>> transactional
> >>>>>>>>>>>>>>>>> checks.
> >>>>>>>>>>>>>>>>>>>>>>>>> 4. The partition leaders append the
> >>>> transactional
> >>>>>>>>>>>>>> data to
> >>>>>>>>>>>>>>>>>>> their logs
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> update some of their state (for example, log
> >>>> the
> >>>>>> fact
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>>>>>>>>>>>> is running for the partition and its first
> >>>> offset).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> When the transaction is committed or aborted:
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. The producer contacts the transaction
> >>>>>> coordinator
> >>>>>>>>>>>>>>>>> directly
> >>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> EndTxnRequest.
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. The transaction coordinator writes
> >>>>>> PREPARE_COMMIT
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>> PREPARE_ABORT to
> >>>>>>>>>>>>>>>>>>>>>>>>> its log and responds to the producer.
> >>>>>>>>>>>>>>>>>>>>>>>>> 3. The transaction coordinator sends
> >>>>>>>>>>>>>>>>> WriteTxnMarkersRequest to
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> leaders
> >>>>>>>>>>>>>>>>>>>>>>>>> of the involved partitions.
> >>>>>>>>>>>>>>>>>>>>>>>>> 4. The partition leaders write the
> >>> transaction
> >>>>>>>>>>>>>> markers to
> >>>>>>>>>>>>>>>>>>> their logs
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> respond to the coordinator.
> >>>>>>>>>>>>>>>>>>>>>>>>> 5. The coordinator writes the final
> >>> transaction
> >>>>>> state
> >>>>>>>>>>>>>>>>>>>>>> COMPLETE_COMMIT or
> >>>>>>>>>>>>>>>>>>>>>>>>> COMPLETE_ABORT.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> In classic topics, partitions have leaders
> >>> and
> >>>> lots
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> important
> >>>>>>>>>>>>>>>>>>>>>> state
> >>>>>>>>>>>>>>>>>>>>>>>>> necessary for supporting this workflow is
> >>>> local.
> >>>>>> The
> >>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>> challenge
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
> >>>> there
> >>>>>>>>>>>>>> are no
> >>>>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
> >>>> need
> >>>>>>>>>>>>>> to be
> >>>>>>>>>>>>>>>>>>> globalized
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> batch coordinator. We are already doing this
> >>> to
> >>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>> idempotent
> >>>>>>>>>>>>>>>>>>>>>>>> produce.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The high level workflow for *diskless topics*
> >>>> would
> >>>>>>>>>>>>>> look
> >>>>>>>>>>>>>>>>> very
> >>>>>>>>>>>>>>>>>>>>>> similar:
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> >>>>>>>>>>>>>> non-empty
> >>>>>>>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>> running for this partition. In contrast to
> >>>> classic
> >>>>>>>>>>>>>>>> topics,
> >>>>>>>>>>>>>>>>>>> this is
> >>>>>>>>>>>>>>>>>>>>>>>> checked
> >>>>>>>>>>>>>>>>>>>>>>>>> against the batch coordinator with a single
> >>>> RPC.
> >>>>>>>>>>>>>> Since a
> >>>>>>>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>>>>> be uniquely identified with producer ID and
> >>>> epoch,
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> positive
> >>>>>>>>>>>>>>>>>>>>>> result of
> >>>>>>>>>>>>>>>>>>>>>>>>> this check could be cached locally (for the
> >>>> double
> >>>>>>>>>>>>>>>>> configured
> >>>>>>>>>>>>>>>>>>>>>> duration
> >>>>>>>>>>>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>>>>> transaction, for example).
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> >>>>>> informed
> >>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> partitions that aren’t part of the
> >>> transaction
> >>>> to
> >>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>> them.
> >>>>>>>>>>>>>>>>>>>>>>>>> 3. No transactional checks are done on the
> >>>> broker
> >>>>>>>>>>>>>> side.
> >>>>>>>>>>>>>>>>>>>>>>>>> 4. The broker appends the transactional data
> >>>> to the
> >>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>> shared
> >>>>>>>>>>>>>>>>>>>>>> WAL
> >>>>>>>>>>>>>>>>>>>>>>>>> segment. It doesn’t update any
> >>>> transaction-related
> >>>>>>>>>>>>>> state
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> Diskless
> >>>>>>>>>>>>>>>>>>>>>>>>> topics, because it doesn’t have any.
> >>>>>>>>>>>>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> >>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>> in the
> >>>>>>>>>>>>>>>>>>>>>>>>> normal produce flow.
> >>>>>>>>>>>>>>>>>>>>>>>>> 6. The batch coordinator does the final
> >>>>>> transactional
> >>>>>>>>>>>>>>>>> checks
> >>>>>>>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>>>>>>>> batches. This procedure would output the same
> >>>>>> errors
> >>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> >>>>>> batches
> >>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> rejected.
> >>>>>>>>>>>>>>>>>>>>>>>>> This means, there will potentially be garbage
> >>>> in
> >>>>>> the
> >>>>>>>>>>>>>> WAL
> >>>>>>>>>>>>>>>>>>> segment
> >>>>>>>>>>>>>>>>>>>>>> file in
> >>>>>>>>>>>>>>>>>>>>>>>>> case of transactional errors. This is
> >>>> preferable to
> >>>>>>>>>>>>>> doing
> >>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>> round trips, especially considering the WAL
> >>>>>> segments
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> relatively
> >>>>>>>>>>>>>>>>>>>>>>>>> short-living (see the Greg's update above).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> When the transaction is committed or aborted:
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. The producer contacts the transaction
> >>>>>> coordinator
> >>>>>>>>>>>>>>>>> directly
> >>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> EndTxnRequest.
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. The transaction coordinator writes
> >>>>>> PREPARE_COMMIT
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>> PREPARE_ABORT to
> >>>>>>>>>>>>>>>>>>>>>>>>> its log and responds to the producer.
> >>>>>>>>>>>>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
> >>> informs
> >>>> the
> >>>>>>>>>>>>>> batch
> >>>>>>>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> the transaction is finished.
> >>>>>>>>>>>>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
> >>> the
> >>>>>>>>>>>>>>>>> transaction is
> >>>>>>>>>>>>>>>>>>>>>> finished
> >>>>>>>>>>>>>>>>>>>>>>>>> and also inserts the control batches in the
> >>>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>> logs
> >>>>>>>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>>>>>>>> involved Diskless topics. This happens only
> >>> on
> >>>> the
> >>>>>>>>>>>>>>>> metadata
> >>>>>>>>>>>>>>>>>>> level, no
> >>>>>>>>>>>>>>>>>>>>>>>>> actual control batches are written to any
> >>> file.
> >>>>>> They
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> dynamically
> >>>>>>>>>>>>>>>>>>>>>>>>> created on Fetch and other read operations.
> >>> We
> >>>>>> could
> >>>>>>>>>>>>>>>>>>> technically
> >>>>>>>>>>>>>>>>>>>>>> write
> >>>>>>>>>>>>>>>>>>>>>>>>> these control batches for real, but this
> >>> would
> >>>> mean
> >>>>>>>>>>>>>> extra
> >>>>>>>>>>>>>>>>>>> produce
> >>>>>>>>>>>>>>>>>>>>>>>> latency,
> >>>>>>>>>>>>>>>>>>>>>>>>> so it's better just to mark them in the batch
> >>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> save
> >>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds.
> >>>>>>>>>>>>>>>>>>>>>>>>> 5. The transaction coordinator sends
> >>>>>>>>>>>>>>>>> WriteTxnMarkersRequest to
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> leaders
> >>>>>>>>>>>>>>>>>>>>>>>>> of the involved partitions. – Now only to
> >>>> classic
> >>>>>>>>>>>>>> topics
> >>>>>>>>>>>>>>>>> now.
> >>>>>>>>>>>>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> >>>> write
> >>>>>> the
> >>>>>>>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>>>>>>>>> markers
> >>>>>>>>>>>>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> >>>>>>>>>>>>>>>>>>>>>>>>> 7. The coordinator writes the final
> >>> transaction
> >>>>>> state
> >>>>>>>>>>>>>>>>>>>>>> COMPLETE_COMMIT or
> >>>>>>>>>>>>>>>>>>>>>>>>> COMPLETE_ABORT.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Compared to the non-transactional produce
> >>>> flow, we
> >>>>>>>>>>>>>> get:
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. An extra network round trip between
> >>> brokers
> >>>> and
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> batch
> >>>>>>>>>>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>>>>>>>>>> when a new partition appear in the
> >>>> transaction. To
> >>>>>>>>>>>>>>>>> mitigate the
> >>>>>>>>>>>>>>>>>>>>>> impact of
> >>>>>>>>>>>>>>>>>>>>>>>>> them:
> >>>>>>>>>>>>>>>>>>>>>>>>> - The results will be cached.
> >>>>>>>>>>>>>>>>>>>>>>>>> - The calls for multiple partitions in one
> >>>> Produce
> >>>>>>>>>>>>>>>>> request
> >>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>>>> grouped.
> >>>>>>>>>>>>>>>>>>>>>>>>> - The batch coordinator should be optimized
> >>> for
> >>>>>>>>>>>>>> fast
> >>>>>>>>>>>>>>>>>>> response to
> >>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>> RPCs.
> >>>>>>>>>>>>>>>>>>>>>>>>> - The fact that a single producer normally
> >>> will
> >>>>>>>>>>>>>>>>> communicate
> >>>>>>>>>>>>>>>>>>> with a
> >>>>>>>>>>>>>>>>>>>>>>>>> single broker for the duration of the
> >>>> transaction
> >>>>>>>>>>>>>> further
> >>>>>>>>>>>>>>>>>>> reduces the
> >>>>>>>>>>>>>>>>>>>>>>>>> expected number of round trips.
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. An extra round trip between the
> >>> transaction
> >>>>>>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> batch
> >>>>>>>>>>>>>>>>>>>>>>>>> coordinator when a transaction is finished.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> With this proposal, transactions will also be
> >>>> able
> >>>>>> to
> >>>>>>>>>>>>>>>> span
> >>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>> classic
> >>>>>>>>>>>>>>>>>>>>>>>>> and Diskless topics.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> *Queues*
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The share group coordination and management
> >>> is
> >>>> a
> >>>>>>>>>>>>>> side job
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> >>>>>>>>>>>>>> replicas,
> >>>>>>>>>>>>>>>>> physical
> >>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> >>>>>> consumers
> >>>>>>>>>>>>>>>>> (Fetch and
> >>>>>>>>>>>>>>>>>>>>>> Produce
> >>>>>>>>>>>>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> >>>>>> affected.)
> >>>>>>>>>>>>>> We
> >>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>> see any
> >>>>>>>>>>>>>>>>>>>>>>>>> reason why we can't make Diskless topics
> >>>> compatible
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> share
> >>>>>>>>>>>>>>>>>>>>>> groups the
> >>>>>>>>>>>>>>>>>>>>>>>>> same way as classic topics are. Even on the
> >>>> code
> >>>>>>>>>>>>>> level,
> >>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>> expect
> >>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>> serious refactoring: the same reading
> >>> routines
> >>>> are
> >>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>> used for
> >>>>>>>>>>>>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Should the KIPs be modified to include this
> >>> or
> >>>> it's
> >>>>>>>>>>>>>> too
> >>>>>>>>>>>>>>>>>>>>>>>>> implementation-focused?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>> Ivan
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> >>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thank you all for your questions and design
> >>>> input
> >>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>> KIP-1150.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> >>>> with a
> >>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> design. To
> >>>>>>>>>>>>>>>>>>>>>>>>> summarize
> >>>>>>>>>>>>>>>>>>>>>>>>>> the changes:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
> >>> the
> >>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>> KIP-405
> >>>>>>>>>>>>>>>>>>>>>> Tiered
> >>>>>>>>>>>>>>>>>>>>>>>>>> Storage interfaces, permitting data produced
> >>>> to a
> >>>>>>>>>>>>>>>>> Diskless
> >>>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>>> to be
> >>>>>>>>>>>>>>>>>>>>>>>>>> moved to tiered storage.
> >>>>>>>>>>>>>>>>>>>>>>>>>> This lowers the scalability requirements for
> >>>> the
> >>>>>>>>>>>>>> Batch
> >>>>>>>>>>>>>>>>>>> Coordinator
> >>>>>>>>>>>>>>>>>>>>>>>>>> component, and allows Diskless to compose
> >>> with
> >>>>>>>>>>>>>> Tiered
> >>>>>>>>>>>>>>>>> Storage
> >>>>>>>>>>>>>>>>>>>>>> plugin
> >>>>>>>>>>>>>>>>>>>>>>>>>> features such as encryption and alternative
> >>>> data
> >>>>>>>>>>>>>>>> formats.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
> >>> local
> >>>>>>>>>>>>>> segments,
> >>>>>>>>>>>>>>>>>>> making use
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> >>>>>>>>>>>>>> zero-copy
> >>>>>>>>>>>>>>>>>>> functionality
> >>>>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>>>> built into classic topics.
> >>>>>>>>>>>>>>>>>>>>>>>>>> However, local segments are now considered
> >>>> cache
> >>>>>>>>>>>>>>>>> elements,
> >>>>>>>>>>>>>>>>>>> do not
> >>>>>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> be durably stored, and can be built without
> >>>>>>>>>>>>>> contacting
> >>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>> replicas.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 3. The design has been simplified
> >>>> substantially,
> >>>>>> by
> >>>>>>>>>>>>>>>>> removing
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> >>>>>>>>>>>>>> component, and
> >>>>>>>>>>>>>>>>>>> "object
> >>>>>>>>>>>>>>>>>>>>>>>>>> compaction/merging" step.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> The design maintains leaderless produces as
> >>>>>>>>>>>>>> enabled by
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> Batch
> >>>>>>>>>>>>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
> >>> as
> >>>> the
> >>>>>>>>>>>>>>>> earlier
> >>>>>>>>>>>>>>>>>>> design,
> >>>>>>>>>>>>>>>>>>>>>> while
> >>>>>>>>>>>>>>>>>>>>>>>>>> being simpler and integrating better into
> >>> the
> >>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>> ecosystem.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> >>>> feedback on
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> design.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Greg Harris
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> >>>>>>>>>>>>>>>>>>> <ju...@confluent.io.invalid>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jan,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> >>>> support
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>>> client
> >>>>>>>>>>>>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> >>>>>>>>>>>>>> supporting
> >>>>>>>>>>>>>>>> APIs
> >>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>>> transactions
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and queues.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> >>>> Siekierski
> >>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> >>>> current
> >>>>>>>>>>>>>>>>> status of
> >>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>> KIP?
> >>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> >>>> past 2
> >>>>>>>>>>>>>>>>> months,
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> vote got
> >>>>>>>>>>>>>>>>>>>>>>>>>>> vetoed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> but I think the pending questions have
> >>> been
> >>>>>>>>>>>>>>>> answered
> >>>>>>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>>>>> then.
> >>>>>>>>>>>>>>>>>>>>>>>>> KIP-1183
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> >>>>>>>>>>>>>> activity
> >>>>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>> May.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> >>>> real
> >>>>>>>>>>>>>>>> choices
> >>>>>>>>>>>>>>>>>>> that can
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> >>>> being
> >>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> dominant
> >>>>>>>>>>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> it comes to market adoption - but all
> >>> these
> >>>> are
> >>>>>>>>>>>>>>>>>>> standalone
> >>>>>>>>>>>>>>>>>>>>>>>> products.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
> >>> would
> >>>>>>>>>>>>>> hate to
> >>>>>>>>>>>>>>>>> see a
> >>>>>>>>>>>>>>>>>>>>>> stall. So
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> question is: can we get an update?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
> >>> Colin
> >>>>>>>>>>>>>>>> McCabe -
> >>>>>>>>>>>>>>>>>>> have your
> >>>>>>>>>>>>>>>>>>>>>>>>> questions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> been answered? If not, is there anything I
> >>>> can
> >>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> help? I'm
> >>>>>>>>>>>>>>>>>>>>>>>> deeply
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar with both architectures and have
> >>>>>>>>>>>>>> written
> >>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>> both?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jan
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> >>>>>>>>>>>>>>>> Kozlovski
> >>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> >>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>>> (just a
> >>>>>>>>>>>>>>>>>>>>>> bunch
> >>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> links
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to everything)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) create the email threads
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> >>>>>>>>>>>>>> example, I
> >>>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>>>>>> searching
> >>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while;
> >>> As
> >>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>> tell, it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> exist yet.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
> >>> of
> >>>>>>>>>>>>>> having
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> root
> >>>>>>>>>>>>>>>>>>>>>> KIP be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> >>>> links
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> sub-KIPs
> >>>>>>>>>>>>>>>>>>>>>> where
> >>>>>>>>>>>>>>>>>>>>>>>> were
> >>>>>>>>>>>>>>>>>>>>>>>>>>> aimed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to move the discussion towards), I think
> >>> it
> >>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> good to
> >>>>>>>>>>>>>>>>>>>>>>>> create
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> DISCUSS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> threads for them all.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stan
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> >>>> about
> >>>>>>>>>>>>>>>>>>> introducing a
> >>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> >>>>>>>>>>>>>> Storage as
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> primary
> >>>>>>>>>>>>>>>>>>>>>>>>> source of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> >>>>>>>>>>>>>> decided
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> split it
> >>>>>>>>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> related KIPs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> >>>>>>>>>>>>>> should aim
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> >>>>>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>> how to
> >>>>>>>>>>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This follows the same approach used when
> >>>> we
> >>>>>>>>>>>>>>>>> discussed
> >>>>>>>>>>>>>>>>>>>>>> KRaft.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
> >>> really
> >>>>>>>>>>>>>> hard
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> discuss
> >>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> meta
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> >>>>>>>>>>>>>> (linked
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> KIP-1150)
> >>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> offer
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation of this feature.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> >>>> DISCUSS
> >>>>>>>>>>>>>>>>> threads
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> each
> >>>>>>>>>>>>>>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> >>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>> Apache
> >>>>>>>>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>>>>>>>>> wants
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this feature or not.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> >>>>>>>>>>>>>> authors
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Josep Prat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
> >>> +491715557497 |
> >>>>>>>>>>>>>>>>> aiven.io
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> >>>>>>>>>>>>>>>>> Valtonen,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Anatolii Popov
> >>>>>> Senior Software Developer, *Aiven OY*
> >>>>>> m: +358505126242
> >>>>>> w: aiven.io  e: anatolii.popov@aiven.io
> >>>>>> <https://www.facebook.com/aivencloud>
> >>>>>> <https://www.linkedin.com/company/aiven/>   <
> >>>> https://twitter.com/aiven_io>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> 
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Andrew Schofield <an...@live.com>]

Hi Ivan and other KIP authors,
A few questions about the metadata for clients.

AS1: I’m slightly amused by the “lying” for metadata responses and I’m not really sure it’s lying at
all. I think including the rack ID in the MetadataRequest is a good signal that the client wants
a rack-aligned response. For a share consumer, we could just omit the rack ID from the metadata
request and then get the real leader. That might be enough. I wonder whether we should
have the intent of the sender for the MetadataRequest in there too, such as whether it’s a producer.

On a side note, we do send the rack ID in ShareGroupHeartbeat to allow for a rack-aware
assignor, for which the KIP is on my list to create.

AS2: Why do we need the “IsDiskless” in the MetadataResponse? For one thing, I would
prefer a more logical name about what the flag signifies in the response. If it indicates the
response is not the actual leader but the broker can handle produce/consume, maybe the
flag should have a name which reflects that instead. My worry is that years in the future
when we want to set the flag for a new kind of non-diskless megatopics, the name is going
to appear very unfashionable.

Thanks,
Andrew

> On 22 Jan 2026, at 09:12, Ivan Yurchenko <iv...@ivanyu.me> wrote:
>
> Hi Robin,
>
> Your proposal actually resonates with what we proposed [1] a little while ago (nicknamed revision 3), so we exercised similar ideas. We abandoned them in favor of the original coordinator-based approach for the reasons described in the mailing list and summarized in the rejected alternatives of KIP-1163 [2].
>
> If you consider your proposal the alternative to KIP-1163 under the umbrella of KIP-1150, we propose to file a formal KIP so we can have a more visible community discussion of the merits of both approach. What do you think?
>
> Best,
> Ivan
>
> [1] https://lists.apache.org/thread/w03qffwllynkx5ygvcd48vb3vjvsf8k7
> [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=350783976#KIP1163:DisklessCore-Coordinator-lessapproaches
>
> On Wed, Jan 21, 2026, at 09:24, Robin Han wrote:
>> Hi Greg,
>>
>> Thanks for the reply
>>
>>> Based on your description, the GET operations on the leader and followers
>>> must complete before offsets are assigned, adding Produce request latency
>>> that is not present in the KIP-1163 design.
>>
>> Yes, the ProduceRequestLatency is 1 * PUT(In Broker N) + 1 GET(In Broker M)
>> + 1 GET(Follower Synchronization).
>> - Compared with KIP-1163: Produce Latency has two more GET time costs;
>> Consume E2E Latency has one more GET time cost.
>> - Compared with KIP-1163: There is no need for coordination by a
>> centralized Coordinator, thus eliminating the delay caused by Coordinator
>> confirmation and avoiding the impact of Coordinator single-point failure on
>> the availability of the entire cluster.
>>
>> In the design of Diskless Topic, I suspect that adding the time cost of two
>> more GET operations will not bring about an essential change (with P50
>> increasing from 500ms to 700ms).
>>
>>> KIP-1163 also assumes that object storage is more reliable than disk
>>> storage, which is the case in the hyperscaler clouds. The parts of your
>>> design that make it compatible with low-reliability object storage become
>>> counterproductive when Diskless is used with a high-reliability object
>>> storage, such as re-introducing direct replication and cross-az costs.
>>
>> It might be that my description is not accurate enough. The design is also
>> strongly dependent on object storage. Regarding the part about object
>> management and recycling, what I want to emphasize more is that the current
>> design does not require complex object metadata management, and objects are
>> just a temporary state.
>>
>>> We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
>>> in favor of a complete solution.
>>
>> I understand that the main reasons why "KIP-1176 was rejected" are that it
>> did not solve the cross-AZ traffic costs for Produce, and that implementing
>> Active Segment Tiered Storage added a lot of complexity to metadata
>> management.
>> The design I mentioned can not only solve all cross-AZ traffic costs
>> including sending and replication (the core goal of KIP-1150), but also
>> does not add complex management of Object metadata, nor does it require
>> large-scale modifications to Apache Kafka.
>> It could be a good design that can be adopted by KIP-1150.
>>
>> Greg Harris via dev <de...@kafka.apache.org> 于2026年1月21日周三 03:20写道：
>>
>>> Hi Robin,
>>>
>>> Thanks for your design sketch.
>>>
>>> Based on your description, the GET operations on the leader and followers
>>> must complete before offsets are assigned, adding Produce request latency
>>> that is not present in the KIP-1163 design.
>>>
>>> KIP-1163 also assumes that object storage is more reliable than disk
>>> storage, which is the case in the hyperscaler clouds. The parts of your
>>> design that make it compatible with low-reliability object storage become
>>> counterproductive when Diskless is used with a high-reliability object
>>> storage, such as re-introducing direct replication and cross-az costs.
>>>
>>> We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
>>> in favor of a complete solution. The folks who championed that proposal are
>>> now working with us on KIP-1163, and are currently designing Compaction in
>>> Tiered Storage to make Diskless a complete solution for compacted topics as
>>> well. Similarly, we would appreciate your contributions to the design of
>>> KIP-1163, and your vote on the motivational KIP-1150.
>>>
>>> Thanks!
>>>
>>> On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>>>
>>>> Hi Jun,
>>>>
>>>> Here are our answers to your questions and comments:
>>>>
>>>>> JR10. "Even in the case where network traffic is not accounted; we
>>>> believe
>>>>> that the operational benefits of Diskless topics are still appealing to
>>>>> Kafka users."
>>>>> Could you list other benefits such as better scalability and
>>> durability?
>>>>
>>>> Thank you, good point, added this.
>>>>
>>>>> JR11. "AWS charges 0.02$ per GiB"
>>>>> The object store is not free and has a different cost model based on
>>>>> requests. It would be useful summarize the tradeoff on cost and
>>> latency,
>>>>> and when diskless topics are cost effective.
>>>>
>>>> Yes, there is a per-request cost for object storage which is not directly
>>>> comparable to inter-zone transfer costs. We have found that the design in
>>>> KIP-1163 amortizes those request costs such that they are insignificant
>>>> compared to the $0.02/gb cost which is called out in KIP-1150.
>>>> We currently have very precise cost models that were put together for an
>>>> earlier iteration of the Diskless topics. These are somewhat stale and
>>>> inaccurate for the latest design, so we will work in the background to
>>> post
>>>> concrete numbers in KIP-1163 before voting on that proposal. Would this
>>> be
>>>> OK?
>>>>
>>>>> JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
>>>> been
>>>>> updated in the last 6 months. The metadata change in KIP-1181 is now
>>>>> included in KIP-1163.
>>>>
>>>> Thanks for calling out. We have updated KIP-1164 with the latest design.
>>>> It's missing public interfaces, but we think it may be a bit too early to
>>>> define them as some rounds of discussion is ahead, with potential to
>>> change
>>>> the design. But otherwise the KIP is good for the first review.
>>>> We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
>>>>
>>>> Best,
>>>> Ivan
>>>>
>>>> On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
>>>>> Hi, Anatolii,
>>>>>
>>>>> Thanks for the updated KIP. Looks good to me overall. A few more
>>>> comments.
>>>>>
>>>>> JR10. "Even in the case where network traffic is not accounted; we
>>>> believe
>>>>> that the operational benefits of Diskless topics are still appealing to
>>>>> Kafka users."
>>>>> Could you list other benefits such as better scalability and
>>> durability?
>>>>>
>>>>> JR11. "AWS charges 0.02$ per GiB"
>>>>> The object store is not free and has a different cost model based on
>>>>> requests. It would be useful summarize the tradeoff on cost and
>>> latency,
>>>>> and when diskless topics are cost effective.
>>>>>
>>>>> JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
>>>> been
>>>>> updated in the last 6 months. The metadata change in KIP-1181 is now
>>>>> included in KIP-1163.
>>>>>
>>>>> Jun
>>>>>
>>>>> On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
>>>> dev@kafka.apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Thank you for the extensive feedback.
>>>>>>
>>>>>> We have substantially updated the KIP to address the points raised.
>>>> Given
>>>>>> the scope of these changes, we have started a new VOTE thread to
>>>> restart
>>>>>> the voting process cleanly.
>>>>>>
>>>>>> You can find the new thread here:
>>>>>> https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
>>>>>>
>>>>>> Best,
>>>>>> Anatolii
>>>>>>
>>>>>> On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
>>>>>> dev@kafka.apache.org> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We (the team at Slack) have been following the recent discussion
>>>>>> regarding
>>>>>>> the scope and timeline of KIP-1150. We agree with the community
>>>> sentiment
>>>>>>> that Diskless Topics represents the right long-term architecture
>>> for
>>>>>> Kafka
>>>>>>> in the cloud, but we also recognize the valid concerns raised
>>>> regarding
>>>>>> the
>>>>>>> engineering resources required to deliver such an ambitious change.
>>>>>>>
>>>>>>> To help address these concerns and accelerate the timeline, we are
>>>> happy
>>>>>> to
>>>>>>> announce that we are partnering with the KIP-1150 authors to
>>>> co-develop
>>>>>>> this feature.
>>>>>>>
>>>>>>> We previously proposed KIP-1176: Tiered Storage for Active Log
>>>> Segments
>>>>>> to
>>>>>>> solve similar problems. However, rather than fragmenting the
>>>> community's
>>>>>>> efforts across competing designs, we have decided to withdraw
>>>> KIP-1176
>>>>>> and
>>>>>>> consolidate our engineering resources behind KIP-1150.
>>>>>>>
>>>>>>> To start, we plan to take ownership of Compaction for Tiered
>>> Storage.
>>>>>> This
>>>>>>> has long been a missing feature in KIP-405, and it becomes critical
>>>> in a
>>>>>>> Diskless architecture where long-lived data is obligated to be
>>>> tiered. By
>>>>>>> driving this related prerequisite feature, we hope to allow for
>>>> faster
>>>>>>> delivery of KIP-1150.
>>>>>>>
>>>>>>> We are excited to collaborate on this to ensure a robust and timely
>>>>>>> delivery for the community.
>>>>>>>
>>>>>>> Best,
>>>>>>> Tom & Henry
>>>>>>>
>>>>>>> On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
>>> wrote:
>>>>>>>
>>>>>>>> Hi Greg,
>>>>>>>>
>>>>>>>> Thanks for sharing the meeting notes.
>>>>>>>> I agree we should keep polishing the contents of 1150 & high
>>> level
>>>>>> design
>>>>>>>> in 1163 to prepare for a vote.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>> Luke
>>>>>>>>
>>>>>>>> On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
>>>>>> <greg.harris@aiven.io.invalid
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> There was a video call between myself, Ivan Yurchenko, Jun Rao,
>>>> and
>>>>>>>> Andrew
>>>>>>>>> Schofield pertaining to KIP-1150. Here are the notes from that
>>>>>> meeting:
>>>>>>>>>
>>>>>>>>> Ivan: What is the future state of Kafka in this area, in 5
>>> years?
>>>>>>>>> Jun: Do we want something more cloud native? Yes, started with
>>>> Tiered
>>>>>>>>> Storage. If there’s a better way, we should explore it. In the
>>>> long
>>>>>>> term
>>>>>>>>> this will be useful
>>>>>>>>> Because Kafka is used so widely, we need to make sure
>>> everything
>>>> we
>>>>>> add
>>>>>>>> is
>>>>>>>>> for the long term and for everyone, not just for a single
>>>> company.
>>>>>>>>> When we add TS, it doesn’t just solve Uber’s use-case. We want
>>>>>>> something
>>>>>>>>> that’s high quality/lasts/maintainable, and can work with all
>>>>>> existing
>>>>>>>>> capabilities.
>>>>>>>>> If both 1150 and 1176 proceed at the same time, it’s confusing.
>>>> They
>>>>>>>>> overlap, but Diskless is more ambitious.
>>>>>>>>> If both KIPs are being seriously worked on, then we don’t
>>> really
>>>> need
>>>>>>>> both,
>>>>>>>>> because Diskless clearly is better. Having multiple will
>>> confuse
>>>>>>> people.
>>>>>>>> It
>>>>>>>>> will duplicate some of the effort.
>>>>>>>>> If we want diskless ultimately, what is the short term
>>> strategy,
>>>> to
>>>>>> get
>>>>>>>>> some early wins first?
>>>>>>>>> Ivan: Andrew, do you want a more revolutionary approach?
>>>>>>>>> Andrew: Eventually the architecture will change substantially,
>>>> it may
>>>>>>> not
>>>>>>>>> be necessary to put all of that bill onto Diskless at once.
>>>>>>>>> Greg: We all agree on having a high quality feature merged
>>>> upstream,
>>>>>>> and
>>>>>>>>> supporting all APIs
>>>>>>>>> Jun: We should try and keep things simple, but there is some
>>>> minimum
>>>>>>>>> complexity needed.
>>>>>>>>> When doing the short term changes (1176), it doesn’t really
>>>> progress
>>>>>> in
>>>>>>>>> changing to a more modern architecture.
>>>>>>>>> Greg: Was TS+Compaction the only feature miss we’ve had so far?
>>>>>>>>> Jun: The danger of only applying changes to some part of the
>>>> API, you
>>>>>>> set
>>>>>>>>> the precedence that you only have to implement part of the API.
>>>>>>>> Supporting
>>>>>>>>> the full API set should be a minimum requirement.
>>>>>>>>> Andrew: When we started Kraft, how much did we know the design?
>>>>>>>>> Jun: For Kraft we didn’t really know much about the migration,
>>>> but
>>>>>> the
>>>>>>>>> high-level was clear.
>>>>>>>>> Greg: Is 1150 votable in its current state?
>>>>>>>>> Jun: 1150 should promise to support all APIs. It doesn’t have
>>> to
>>>> have
>>>>>>> all
>>>>>>>>> the details/apis/etc. KIP-500 didn’t have it.
>>>>>>>>> We do need some high-level design enough to give confidence
>>> that
>>>> the
>>>>>>>>> promise is able to be fulfilled.
>>>>>>>>> Greg: Is the draft version in 1163 enough detail or is more
>>>> needed?
>>>>>>>>> Jun: We need to agree on the core design, such as leaderless
>>>> etc. And
>>>>>>> how
>>>>>>>>> the APIs will be supported.
>>>>>>>>> Greg: Okay we can include these things, and provide a sketch of
>>>> how
>>>>>> the
>>>>>>>>> other leader-based features operate.
>>>>>>>>> Jun: Yeah if at a high level the sketch appears to work, we can
>>>>>> approve
>>>>>>>>> that functionality.
>>>>>>>>> Are you committed to doing the more involved and big project?
>>>>>>>>> Greg: Yes, we’re committed to the 1163 design and can’t really
>>>> accept
>>>>>>>> 1176.
>>>>>>>>> Jun: TS was slow because of Uber resourcing problems
>>>>>>>>> Greg: We’ll push internally for resources, and use the
>>> community
>>>>>>>> sentiment
>>>>>>>>> to motivate Aiven.
>>>>>>>>> How far into the future should we look? What sort of scale?
>>>>>>>>> Jun: As long as there’s a path forward, and we’re not closing
>>> off
>>>>>>> future
>>>>>>>>> improvements, we can figure out how to handle a larger scale
>>>> when it
>>>>>>>>> arises.
>>>>>>>>> Greg: Random replica placement is very harmful, can we
>>> recommend
>>>>>> users
>>>>>>> to
>>>>>>>>> use an external tool like CruiseControl?
>>>>>>>>> Jun: Not everyone uses CruiseControl, we would probably need
>>> some
>>>>>>>> solution
>>>>>>>>> for this out of the box
>>>>>>>>> Ivan: Should the Batch Coordinator be pluggable?
>>>>>>>>> Jun: Out-of-box experience should be good, good to allow other
>>>>>>>>> implementations
>>>>>>>>> Greg: But it could hurt Kafka feature/upgrade velocity when we
>>>> wait
>>>>>> for
>>>>>>>>> plugin providers to implement it
>>>>>>>>> Ivan: We imagined that maybe cloud hyperscalers could implement
>>>> it
>>>>>> with
>>>>>>>>> e.g. dynamodb
>>>>>>>>> Greg: Could we bake more details of the different providers
>>> into
>>>>>> Kafka,
>>>>>>>> or
>>>>>>>>> does it still make sense for it to be pluggable?
>>>>>>>>> Jun: Make it whatever is easiest to roll out and add new
>>> clients
>>>>>>>>> Andrew: What happens next? Do you want to get KIP-1150 voted?
>>>>>>>>> Ivan: The vote is already open, we’re not too pressed for time.
>>>> We’ll
>>>>>>> go
>>>>>>>>> improve the 1163 design and communication.
>>>>>>>>> Is 1176 a competing design? Someone will ask.
>>>>>>>>> Jun: If we are seriously working on something more ambitious,
>>>> yeah we
>>>>>>>>> shouldn’t do the stop-gap solution.
>>>>>>>>> It’s diverting review resources. If we can get the short term
>>>> thing
>>>>>> in
>>>>>>>> 1yr
>>>>>>>>> but Diskless solution is 2y it makes sense to go for Diskless.
>>> If
>>>>>> it’s
>>>>>>>> 5yr,
>>>>>>>>> that’s different and maybe the stop-gap solution is needed.
>>>>>>>>> Greg: I’m biased but I believe we’re in the 1yr/2yr case.
>>> Should
>>>> we
>>>>>>>>> explicitly exclude 1176?
>>>>>>>>> Andrew: Put your arms around the feature set you actually want,
>>>> and
>>>>>> use
>>>>>>>>> that to rule out 1176.
>>>>>>>>> Probably don’t need -1 votes, most likely KIPs just don’t
>>> receive
>>>>>>> votes.
>>>>>>>>> Ivan: Should we have sync meetings like tiered storage did?
>>>>>>>>> Jun: Satish posted meeting notes regularly, we should do the
>>>> same.
>>>>>>>>>
>>>>>>>>> To summarize, we will be polishing the contents of 1150 & high
>>>> level
>>>>>>>> design
>>>>>>>>> in 1163 to prepare for a vote.
>>>>>>>>> We believe that the community should select the feature set of
>>>> 1150
>>>>>> to
>>>>>>>>> fully eliminate producer cross-zone costs, and make the
>>>> investment
>>>>>> in a
>>>>>>>>> high quality Diskless Topics implementation rather than in
>>>> stop-gap
>>>>>>>>> solutions.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Greg
>>>>>>>>>
>>>>>>>>> On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> This may be a tangent, but we needed to offload storage off
>>> of
>>>>>> Kafka
>>>>>>>> into
>>>>>>>>>> S3. We are keeping Kafka not as a source of truth, but as a
>>>> mostly
>>>>>>>>>> ephemeral broker that can come and go as it pleases. Be that
>>>>>> scaling
>>>>>>> or
>>>>>>>>>> outage. Disks can be destroyed and recreated at will, we
>>> still
>>>>>> retain
>>>>>>>>> data
>>>>>>>>>> and use broker for just that, brokering messages. Not only
>>>> that, we
>>>>>>>>> reduced
>>>>>>>>>> the requirement on the actual Kafka resources by reducing the
>>>> size
>>>>>>> of a
>>>>>>>>>> payload via a claim check pattern. Maybe this is an
>>>> anti–pattern,
>>>>>> but
>>>>>>>> it
>>>>>>>>> is
>>>>>>>>>> super fast and highly cost efficient. We reworked
>>>> ProducerRequest
>>>>>> to
>>>>>>>>> allow
>>>>>>>>>> plugins. We added a custom http plugin that submits every
>>>> request
>>>>>>> via a
>>>>>>>>>> persisted connection to a microservice. Microservice stores
>>> the
>>>>>>> payload
>>>>>>>>> and
>>>>>>>>>> returns a tiny json metadata object,a claim check, that can
>>> be
>>>> used
>>>>>>> to
>>>>>>>>> find
>>>>>>>>>> the actual data. Think of it as zipping the payload. This
>>> claim
>>>>>> check
>>>>>>>>>> metadata traverses the pipelines with consumers using the
>>> urls
>>>> in
>>>>>>>>> metadata
>>>>>>>>>> to pull what they need. Think unzipping. This allowed us to
>>>> also
>>>>>> pull
>>>>>>>>> ONLY
>>>>>>>>>> the data that we need in graphql like manner. So if you have
>>> a
>>>> 100K
>>>>>>>> json
>>>>>>>>>> payload and you need only a subsection, you can pull that by
>>>>>>> jmespath.
>>>>>>>>> When
>>>>>>>>>> you have multiple consumer groups yanking down huge payloads
>>>> it is
>>>>>>>>>> cumbersome on the broker. When you have the same consumer
>>>> groups
>>>>>>>> yanking
>>>>>>>>>> down a claim check, and then going out of band directly to
>>> the
>>>>>> source
>>>>>>>> of
>>>>>>>>>> truth, the broker has some breathing room. Obviously our
>>>>>> microservice
>>>>>>>>> does
>>>>>>>>>> not go directly to the cloud storage, as that would be too
>>>> slow. It
>>>>>>>>> stores
>>>>>>>>>> the payload in high speed memory cache and returns asap. That
>>>>>> memory
>>>>>>> is
>>>>>>>>>> eventually persisted into S3. The retrieval goest against the
>>>> cache
>>>>>>>>> first,
>>>>>>>>>> then against the S3. Overall a rather cheappy and zippy
>>>> solution. I
>>>>>>>> tried
>>>>>>>>>> proposing the KIP for this, but there was no excitement.
>>> Check
>>>> this
>>>>>>>> out:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Nov 7, 2025, at 5:49 PM, Jun Rao
>>> <jun@confluent.io.INVALID
>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi, Andrew,
>>>>>>>>>>>
>>>>>>>>>>> If we want to focus only on reducing cross-zone replication
>>>>>> costs,
>>>>>>>>> there
>>>>>>>>>> is
>>>>>>>>>>> an alternative design in the KIP-1176 discussion thread
>>> that
>>>>>> seems
>>>>>>>>>> simpler
>>>>>>>>>>> than the proposal here. I am copying the outline of that
>>>> approach
>>>>>>>>> below.
>>>>>>>>>>>
>>>>>>>>>>> 1. A new leader is elected.
>>>>>>>>>>> 2. Leader maintains a first tiered offset, which is
>>>> initialized
>>>>>> to
>>>>>>>> log
>>>>>>>>>> end
>>>>>>>>>>> offset.
>>>>>>>>>>> 3. Leader writes produced data from the client to local
>>> log.
>>>>>>>>>>> 4. Leader uploads produced data from all local logs as a
>>>> combined
>>>>>>>>> object
>>>>>>>>>>> 5. Leader stores the metadata for the combined object in
>>>> memory.
>>>>>>>>>>> 6. If a follower fetch request has an offset >= first
>>> tiered
>>>>>>> offset,
>>>>>>>>> the
>>>>>>>>>>> metadata for the corresponding combined object is returned.
>>>>>>>> Otherwise,
>>>>>>>>>> the
>>>>>>>>>>> local data is returned.
>>>>>>>>>>> 7. Leader periodically advances first tiered offset.
>>>>>>>>>>>
>>>>>>>>>>> It's still a bit unnatural, but it could work.
>>>>>>>>>>>
>>>>>>>>>>> Hi, Ivan,
>>>>>>>>>>>
>>>>>>>>>>> Are you still committed to proceeding with the original
>>>> design of
>>>>>>>>>> KIP-1150?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Jun
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
>>>>>>>>>> andrew_schofield@live.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> I’ve been following KIP-1150 and friends for a while. I’m
>>>> going
>>>>>> to
>>>>>>>>> jump
>>>>>>>>>>>> into the discussions too.
>>>>>>>>>>>>
>>>>>>>>>>>> Looking back at Jack Vanlightly’s message, I am not quite
>>> so
>>>>>>>> convinced
>>>>>>>>>>>> that it’s a kind of fork in the road. The primary aim of
>>> the
>>>>>>> effort
>>>>>>>> is
>>>>>>>>>> to
>>>>>>>>>>>> reduce cross-zone replication costs so Apache Kafka is not
>>>>>>>>> prohibitively
>>>>>>>>>>>> expensive to use on cloud storage. I think it would be
>>>> entirely
>>>>>>>> viable
>>>>>>>>>> to
>>>>>>>>>>>> prioritise code reuse for an initial implementation of
>>>> diskless
>>>>>>>>> topics,
>>>>>>>>>> and
>>>>>>>>>>>> we could still have a more cloud-native design in the
>>>> future.
>>>>>> It’s
>>>>>>>>> hard
>>>>>>>>>> to
>>>>>>>>>>>> predict what the community will prioritise in the future.
>>>>>>>>>>>>
>>>>>>>>>>>> Of the three major revisions, I’m in the rev3 camp. We can
>>>>>> support
>>>>>>>>>>>> leaderless produce requests, first writing WAL segments
>>> into
>>>>>>> object
>>>>>>>>>>>> storage, and then using the regular partition leaders to
>>>>>> sequence
>>>>>>>> the
>>>>>>>>>>>> records. The active log segment for a diskless topic will
>>>>>>> initially
>>>>>>>>>> contain
>>>>>>>>>>>> batch coordinates rather than record batches. The batch
>>>>>>> coordinates
>>>>>>>>> can
>>>>>>>>>> be
>>>>>>>>>>>> resolved from WAL segments for consumers, and also in
>>> order
>>>> to
>>>>>>>> prepare
>>>>>>>>>> log
>>>>>>>>>>>> segments for uploading to tiered storage. Jun is probably
>>>>>> correct
>>>>>>>> that
>>>>>>>>>> we
>>>>>>>>>>>> need a more frequent object merging process than tiered
>>>> storage
>>>>>>>>>> provides.
>>>>>>>>>>>> This is just the transition from write-optimised WAL
>>>> segments to
>>>>>>>>>>>> read-optimised tiered segments, and all of the object
>>>>>>> storage-based
>>>>>>>>>>>> implementations of Kafka that I’m aware of do this
>>>>>> rearrangement.
>>>>>>>> But
>>>>>>>>>>>> perhaps this more frequent object merging is a pre-GA
>>>>>> improvement,
>>>>>>>>>> rather
>>>>>>>>>>>> than a strict requirement for an initial implementation
>>> for
>>>>>> early
>>>>>>>>> access
>>>>>>>>>>>> use.
>>>>>>>>>>>>
>>>>>>>>>>>> For zone-aligned share consumers, the share group assignor
>>>> is
>>>>>>>> intended
>>>>>>>>>> to
>>>>>>>>>>>> be rack-aware. Consumers should be assigned to partitions
>>>> with
>>>>>>>> leaders
>>>>>>>>>> in
>>>>>>>>>>>> their zone. The simple assignor is not rack-aware, but it
>>>> easily
>>>>>>>> could
>>>>>>>>>> be
>>>>>>>>>>>> or we could have a rack-aware assignor.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Andrew
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 24 Oct 2025, at 23:14, Jun Rao
>>> <jun@confluent.io.INVALID
>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, Ivan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>
>>>>>>>>>>>>> "As I understand, you’re speaking about locally
>>>> materialized
>>>>>>>>> segments.
>>>>>>>>>>>> They
>>>>>>>>>>>>> will indeed consume some IOPS. See them as a cache that
>>>> could
>>>>>>>> always
>>>>>>>>> be
>>>>>>>>>>>>> restored from the remote storage. While it’s not ideal,
>>>> it's
>>>>>>> still
>>>>>>>> OK
>>>>>>>>>> to
>>>>>>>>>>>>> lose data in them due to a machine crash, for example.
>>>> Because
>>>>>> of
>>>>>>>>> this,
>>>>>>>>>>>> we
>>>>>>>>>>>>> can avoid explicit flushing on local materialized
>>> segments
>>>> at
>>>>>> all
>>>>>>>> and
>>>>>>>>>> let
>>>>>>>>>>>>> the file system and page cache figure out when to flush
>>>>>>> optimally.
>>>>>>>>> This
>>>>>>>>>>>>> would not eliminate the extra IOPS, but should reduce it
>>>>>>>>> dramatically,
>>>>>>>>>>>>> depending on throughput for each partition. We, of
>>> course,
>>>>>>> continue
>>>>>>>>>>>>> flushing the metadata segments as before."
>>>>>>>>>>>>>
>>>>>>>>>>>>> If we have a mix of classic and diskless topics on the
>>> same
>>>>>>> broker,
>>>>>>>>>> it's
>>>>>>>>>>>>> important that the classic topics' data is flushed to
>>> disk
>>>> as
>>>>>>>> quickly
>>>>>>>>>> as
>>>>>>>>>>>>> possible. To achieve this, users typically set
>>>>>>>> dirty_expire_centisecs
>>>>>>>>>> in
>>>>>>>>>>>>> the kernel based on the number of available disk IOPS.
>>>> Once you
>>>>>>> set
>>>>>>>>>> this
>>>>>>>>>>>>> number, it applies to all dirty files, including the
>>> cached
>>>>>> data
>>>>>>> in
>>>>>>>>>>>>> diskless topics. So, if there are more files actively
>>>>>>> accumulating
>>>>>>>>>> data,
>>>>>>>>>>>>> the flush frequency and therefore RPO is reduced for
>>>> classic
>>>>>>>> topics.
>>>>>>>>>>>>>
>>>>>>>>>>>>> "We should have mentioned this explicitly, but this step,
>>>> in
>>>>>>> fact,
>>>>>>>>>>>> remains
>>>>>>>>>>>>> in the form of segments offloading to tiered storage.
>>> When
>>>> we
>>>>>>>>> assemble
>>>>>>>>>> a
>>>>>>>>>>>>> segment and hand it over to RemoteLogManager, we’re
>>>> effectively
>>>>>>>> doing
>>>>>>>>>>>>> metadata compaction: replacing a big number of pieces of
>>>>>> metadata
>>>>>>>>> about
>>>>>>>>>>>>> individual batches with a single record in
>>>>>>> __remote_log_metadata."
>>>>>>>>>>>>>
>>>>>>>>>>>>> The object merging in tier storage typically only kicks
>>> in
>>>>>> after
>>>>>>> a
>>>>>>>>> few
>>>>>>>>>>>>> hours. The impact is (1) the amount of accumulated
>>>> metadata is
>>>>>>>> still
>>>>>>>>>>>> quite
>>>>>>>>>>>>> large; (2) there are many small objects, leading to poor
>>>> read
>>>>>>>>>>>> performance.
>>>>>>>>>>>>> I think we need a more frequent object merging process
>>> than
>>>>>> tier
>>>>>>>>>> storage
>>>>>>>>>>>>> provides.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
>>>>>> ivan@ivanyu.me>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello Jack, Jun, Luke, and all!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you for your messages.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me first address some of Jun’s comments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> First, it degrades the durability.
>>>>>>>>>>>>>>> For each partition, now there are two files being
>>>> actively
>>>>>>>> written
>>>>>>>>>> at a
>>>>>>>>>>>>>>> given point of time, one for the data and another for
>>> the
>>>>>>>> metadata.
>>>>>>>>>>>>>>> Flushing each file requires a separate IO. If the disk
>>>> has 1K
>>>>>>>> IOPS
>>>>>>>>>> and
>>>>>>>>>>>> we
>>>>>>>>>>>>>>> have 5K partitions in a broker, currently we can afford
>>>> to
>>>>>>> flush
>>>>>>>>> each
>>>>>>>>>>>>>>> partition every 5 seconds, achieving an RPO of 5
>>>> seconds. If
>>>>>> we
>>>>>>>>>> double
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> number of files per partition, we can only flush each
>>>>>> partition
>>>>>>>>> every
>>>>>>>>>>>> 10
>>>>>>>>>>>>>>> seconds, which makes RPO twice as bad.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As I understand, you’re speaking about locally
>>>> materialized
>>>>>>>>> segments.
>>>>>>>>>>>> They
>>>>>>>>>>>>>> will indeed consume some IOPS. See them as a cache that
>>>> could
>>>>>>>> always
>>>>>>>>>> be
>>>>>>>>>>>>>> restored from the remote storage. While it’s not ideal,
>>>> it's
>>>>>>> still
>>>>>>>>> OK
>>>>>>>>>> to
>>>>>>>>>>>>>> lose data in them due to a machine crash, for example.
>>>> Because
>>>>>>> of
>>>>>>>>>> this,
>>>>>>>>>>>> we
>>>>>>>>>>>>>> can avoid explicit flushing on local materialized
>>>> segments at
>>>>>>> all
>>>>>>>>> and
>>>>>>>>>>>> let
>>>>>>>>>>>>>> the file system and page cache figure out when to flush
>>>>>>> optimally.
>>>>>>>>>> This
>>>>>>>>>>>>>> would not eliminate the extra IOPS, but should reduce it
>>>>>>>>> dramatically,
>>>>>>>>>>>>>> depending on throughput for each partition. We, of
>>> course,
>>>>>>>> continue
>>>>>>>>>>>>>> flushing the metadata segments as before.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It’s worth making a note on caching. I think nobody will
>>>>>>> disagree
>>>>>>>>> that
>>>>>>>>>>>>>> doing direct reads from remote storage every time a
>>> batch
>>>> is
>>>>>>>>> requested
>>>>>>>>>>>> by a
>>>>>>>>>>>>>> consumer will not be practical neither from the
>>>> performance
>>>>>> nor
>>>>>>>> from
>>>>>>>>>> the
>>>>>>>>>>>>>> economy point of view. We need a way to keep the number
>>>> of GET
>>>>>>>>>> requests
>>>>>>>>>>>>>> down. There are multiple options, for example:
>>>>>>>>>>>>>> 1. Rack-aware distributed in-memory caching.
>>>>>>>>>>>>>> 2. Local in-memory caching. Comes with less network
>>>> chattiness
>>>>>>> and
>>>>>>>>>>>> works
>>>>>>>>>>>>>> well if we have more or less stable brokers to consume
>>>> from.
>>>>>>>>>>>>>> 3. Materialization of diskless logs on local disk. Way
>>>> lower
>>>>>>>> impact
>>>>>>>>> on
>>>>>>>>>>>>>> RAM and also requires stable brokers for consumption
>>>> (using
>>>>>> just
>>>>>>>>>>>> assigned
>>>>>>>>>>>>>> replicas will probably work well).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Materialization is one of possible options, but we can
>>>> choose
>>>>>>>>> another
>>>>>>>>>>>> one.
>>>>>>>>>>>>>> However, we will have this dilemma regardless of whether
>>>> we
>>>>>> have
>>>>>>>> an
>>>>>>>>>>>>>> explicit coordinator or we go “coordinator-less”.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Second, if we ever need this
>>>>>>>>>>>>>>> metadata somewhere else, say in the WAL file manager,
>>> the
>>>>>>>> consumer
>>>>>>>>>>>> needs
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> subscribe to every partition in the cluster, which is
>>>>>>>> inefficient.
>>>>>>>>>> The
>>>>>>>>>>>>>>> actual benefit of this approach is also questionable.
>>> On
>>>> the
>>>>>>>>> surface,
>>>>>>>>>>>> it
>>>>>>>>>>>>>>> might seem that we could reduce the number of lines
>>> that
>>>> need
>>>>>>> to
>>>>>>>> be
>>>>>>>>>>>>>> changed
>>>>>>>>>>>>>>> for this KIP. However, the changes are quite intrusive
>>>> to the
>>>>>>>>> classic
>>>>>>>>>>>>>>> partition's code path and will probably make the code
>>>> base
>>>>>>> harder
>>>>>>>>> to
>>>>>>>>>>>>>>> maintain in the long run. I like the original approach
>>>> based
>>>>>> on
>>>>>>>> the
>>>>>>>>>>>> batch
>>>>>>>>>>>>>>> coordinator much better than this one. We could
>>> probably
>>>>>>> refactor
>>>>>>>>> the
>>>>>>>>>>>>>>> producer state code so that it could be reused in the
>>>> batch
>>>>>>>>>>>> coordinator.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It’s hard to disagree with this. The explicit
>>> coordinator
>>>> is
>>>>>>> more
>>>>>>>> a
>>>>>>>>>> side
>>>>>>>>>>>>>> thing, while coordinator-less approach is more about
>>>> extending
>>>>>>>>>>>>>> ReplicaManager, UnifiedLog and others substantially.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for addressing the concerns on the number of
>>> RPCs
>>>> in
>>>>>> the
>>>>>>>>>> produce
>>>>>>>>>>>>>>> path. I agree that with the metadata crafting
>>> mechanism,
>>>> we
>>>>>>> could
>>>>>>>>>>>>>> mitigate
>>>>>>>>>>>>>>> the PRC problem. However, since we now require the
>>>> metadata
>>>>>> to
>>>>>>> be
>>>>>>>>>>>>>>> collocated with the data on the same set of brokers,
>>> it's
>>>>>> weird
>>>>>>>>> that
>>>>>>>>>>>> they
>>>>>>>>>>>>>>> are now managed by different mechanisms. The data
>>>> assignment
>>>>>>> now
>>>>>>>>> uses
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> metadata crafting mechanism, but the metadata is stored
>>>> in
>>>>>> the
>>>>>>>>>> classic
>>>>>>>>>>>>>>> partition using its own assignment strategy. It will be
>>>>>>>> complicated
>>>>>>>>>> to
>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>> them collocated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would like to note that the metadata crafting is
>>> needed
>>>> only
>>>>>>> to
>>>>>>>>> tell
>>>>>>>>>>>>>> producers which brokers they should send Produce
>>> requests
>>>> to,
>>>>>>> but
>>>>>>>>> data
>>>>>>>>>>>> (as
>>>>>>>>>>>>>> in “locally materialized log”) is located on partition
>>>>>> replicas,
>>>>>>>>> i.e.
>>>>>>>>>>>>>> automatically co-located with metadata.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As a side note, it would probably be better that instead
>>>> of
>>>>>>>>> implicitly
>>>>>>>>>>>>>> crafting partition metadata, we extend the metadata
>>>> protocol
>>>>>> so
>>>>>>>> that
>>>>>>>>>> for
>>>>>>>>>>>>>> diskless partitions we return not only the leader and
>>>>>> replicas,
>>>>>>>> but
>>>>>>>>>> also
>>>>>>>>>>>>>> some “recommended produce brokers”, selected for optimal
>>>>>>>> performance
>>>>>>>>>> and
>>>>>>>>>>>>>> costs. Producers will pick ones in their racks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am also concerned about the removal of the object
>>>>>>>>>> compaction/merging
>>>>>>>>>>>>>>> step.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We should have mentioned this explicitly, but this step,
>>>> in
>>>>>>> fact,
>>>>>>>>>>>> remains
>>>>>>>>>>>>>> in the form of segments offloading to tiered storage.
>>>> When we
>>>>>>>>>> assemble a
>>>>>>>>>>>>>> segment and hand it over to RemoteLogManager, we’re
>>>>>> effectively
>>>>>>>>> doing
>>>>>>>>>>>>>> metadata compaction: replacing a big number of pieces of
>>>>>>> metadata
>>>>>>>>>> about
>>>>>>>>>>>>>> individual batches with a single record in
>>>>>>> __remote_log_metadata.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We could create a Diskless-specific merging mechanism
>>>> instead
>>>>>> if
>>>>>>>>>> needed.
>>>>>>>>>>>>>> It’s rather easy with the explicit coordinator approach.
>>>> With
>>>>>>> the
>>>>>>>>>>>>>> coordinator-less approach, this would probably be a bit
>>>> more
>>>>>>>> tricky
>>>>>>>>>>>>>> (rewriting the tail of the log by the leader +
>>> replicating
>>>>>> this
>>>>>>>>> change
>>>>>>>>>>>>>> reliably).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I see a tendency toward primarily optimizing for the
>>>> fewest
>>>>>>> code
>>>>>>>>>>>> changes
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the KIP. Instead, our primary goal should be a clean
>>>> design
>>>>>>> that
>>>>>>>>> can
>>>>>>>>>>>> last
>>>>>>>>>>>>>>> for the long term.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, totally agree.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Luke,
>>>>>>>>>>>>>>> I'm wondering if the complexity of designing txn and
>>>> queue is
>>>>>>>>> because
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> leaderless cluster, do you think it will be simpler if
>>> we
>>>>>> only
>>>>>>>>> focus
>>>>>>>>>> on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> "diskless" design to handle object compaction/merging
>>>> to/from
>>>>>>> the
>>>>>>>>>>>> remote
>>>>>>>>>>>>>>> storage to save the cross-AZ cost?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After some evolution of the original proposal,
>>> leaderless
>>>> is
>>>>>> now
>>>>>>>>>>>> limited.
>>>>>>>>>>>>>> We only need to be able to accept Produce requests on
>>> more
>>>>>> than
>>>>>>>> one
>>>>>>>>>>>> broker
>>>>>>>>>>>>>> to eliminate the cross-AZ costs for producers. Do I get
>>> it
>>>>>> right
>>>>>>>>> that
>>>>>>>>>>>> you
>>>>>>>>>>>>>> propose to get rid of this? Or do I misunderstand?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let’s now look at this problem from a higher level, as
>>>> Jack
>>>>>>>>> proposed.
>>>>>>>>>> As
>>>>>>>>>>>>>> it was said, the big choice we need to make is whether
>>> we
>>>> 1)
>>>>>>>> create
>>>>>>>>> an
>>>>>>>>>>>>>> explicit batch coordinator; or 2) go for the
>>>> coordinator-less
>>>>>>>>>> approach,
>>>>>>>>>>>>>> where each diskless partition is managed by its leader
>>> as
>>>> in
>>>>>>>> classic
>>>>>>>>>>>> topics.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If we try to compare the two approaches:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pluggability:
>>>>>>>>>>>>>> - Explicit coordinator: Possible. For example, some
>>>> setups may
>>>>>>>>> benefit
>>>>>>>>>>>>>> from batch metadata being stored in a cloud database
>>>> (such as
>>>>>>> AWS
>>>>>>>>>>>> DynamoDB
>>>>>>>>>>>>>> or GCP Spanner).
>>>>>>>>>>>>>> - Coordinator-less: Impossible.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scalability and fault tolerance:
>>>>>>>>>>>>>> - Explicit coordinator: Depends on the implementation
>>> and
>>>> it’s
>>>>>>>> also
>>>>>>>>>>>>>> necessary to actively work for it.
>>>>>>>>>>>>>> - Coordinator-less: Closer to classic Kafka topics.
>>>> Scaling is
>>>>>>>> done
>>>>>>>>> by
>>>>>>>>>>>>>> partition placement, partitions could fail
>>> independently.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Separation of concerns:
>>>>>>>>>>>>>> - Explicit coordinator: Very good. Diskless remains more
>>>>>>>> independent
>>>>>>>>>>>> from
>>>>>>>>>>>>>> classic topics in terms of code and workflows. For
>>>> example,
>>>>>> the
>>>>>>>>>>>>>> above-mentioned non-tiered storage metadata compaction
>>>>>> mechanism
>>>>>>>>> could
>>>>>>>>>>>> be
>>>>>>>>>>>>>> relatively simply implemented with it. As a flip side of
>>>> this,
>>>>>>>> some
>>>>>>>>>>>>>> workflows (e.g. transactions) will have to be adapted.
>>>>>>>>>>>>>> - Coordinator-less: Less so. It leans to the opposite:
>>>>>> bringing
>>>>>>>>>> diskless
>>>>>>>>>>>>>> closer to classic topics. Some code paths and workflows
>>>> could
>>>>>> be
>>>>>>>>> more
>>>>>>>>>>>>>> straightforwardly reused, but they will inevitably have
>>>> to be
>>>>>>>>> adapted
>>>>>>>>>> to
>>>>>>>>>>>>>> accommodate both topic types as also discussed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cloud-nativeness. This is a vague concept, also related
>>>> to the
>>>>>>>>>> previous,
>>>>>>>>>>>>>> but let’s try:
>>>>>>>>>>>>>> - Explicit coordinator: Storing and processing metadata
>>>>>>> separately
>>>>>>>>>> makes
>>>>>>>>>>>>>> it easier for brokers to take different roles, be purely
>>>>>>> stateless
>>>>>>>>> if
>>>>>>>>>>>>>> needed, etc.
>>>>>>>>>>>>>> - Coordinator-less: Less so. Something could be achieved
>>>> with
>>>>>>>>> creative
>>>>>>>>>>>>>> partition placement, but not much.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Both seem to have their pros and cons. However,
>>> answering
>>>>>> Jack’s
>>>>>>>>>>>> question,
>>>>>>>>>>>>>> the explicit coordinator approach may indeed lead to a
>>>> more
>>>>>>>> flexible
>>>>>>>>>>>> design.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The purpose of this deviation in the discussion was to
>>>>>> receive a
>>>>>>>>>>>>>> preliminary community evaluation of the coordinator-less
>>>>>>> approach
>>>>>>>>>>>> without
>>>>>>>>>>>>>> taking on the task of writing a separate KIP and fitting
>>>> it in
>>>>>>> the
>>>>>>>>>>>> system
>>>>>>>>>>>>>> of KIP-1150 and its children. We’re open to stopping it
>>>> and
>>>>>>>> getting
>>>>>>>>>>>> back to
>>>>>>>>>>>>>> working out the coordinator design if the community
>>>> doesn’t
>>>>>>> favor
>>>>>>>>> the
>>>>>>>>>>>>>> proposed approach.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Ivan and Diskless team
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
>>>>>>>>>>>>>>> Hi Ivan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As Jun pointed out, the updated design seems to have
>>> some
>>>>>>>>>> shortcomings
>>>>>>>>>>>>>>> although it simplifies the implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm wondering if the complexity of designing txn and
>>>> queue is
>>>>>>>>> because
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> leaderless cluster, do you think it will be simpler if
>>> we
>>>>>> only
>>>>>>>>> focus
>>>>>>>>>> on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> "diskless" design to handle object compaction/merging
>>>> to/from
>>>>>>> the
>>>>>>>>>>>> remote
>>>>>>>>>>>>>>> storage to save the cross-AZ cost?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
>>>>>>> <jun@confluent.io.invalid
>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, Ivan,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the explanation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> "we write the reference to the WAL file with the batch
>>>> data"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I understand the approach now, but I think it is a
>>> hacky
>>>>>> one.
>>>>>>>>> There
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> multiple short comings with this design. First, it
>>>> degrades
>>>>>>> the
>>>>>>>>>>>>>> durability.
>>>>>>>>>>>>>>>> For each partition, now there are two files being
>>>> actively
>>>>>>>> written
>>>>>>>>>> at
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> given point of time, one for the data and another for
>>>> the
>>>>>>>>> metadata.
>>>>>>>>>>>>>>>> Flushing each file requires a separate IO. If the disk
>>>> has
>>>>>> 1K
>>>>>>>> IOPS
>>>>>>>>>> and
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> have 5K partitions in a broker, currently we can
>>> afford
>>>> to
>>>>>>> flush
>>>>>>>>>> each
>>>>>>>>>>>>>>>> partition every 5 seconds, achieving an RPO of 5
>>>> seconds. If
>>>>>>> we
>>>>>>>>>> double
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> number of files per partition, we can only flush each
>>>>>>> partition
>>>>>>>>>> every
>>>>>>>>>>>>>> 10
>>>>>>>>>>>>>>>> seconds, which makes RPO twice as bad. Second, if we
>>>> ever
>>>>>> need
>>>>>>>>> this
>>>>>>>>>>>>>>>> metadata somewhere else, say in the WAL file manager,
>>>> the
>>>>>>>> consumer
>>>>>>>>>>>>>> needs to
>>>>>>>>>>>>>>>> subscribe to every partition in the cluster, which is
>>>>>>>> inefficient.
>>>>>>>>>> The
>>>>>>>>>>>>>>>> actual benefit of this approach is also questionable.
>>>> On the
>>>>>>>>>> surface,
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> might seem that we could reduce the number of lines
>>> that
>>>>>> need
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>>>>> changed
>>>>>>>>>>>>>>>> for this KIP. However, the changes are quite intrusive
>>>> to
>>>>>> the
>>>>>>>>>> classic
>>>>>>>>>>>>>>>> partition's code path and will probably make the code
>>>> base
>>>>>>>> harder
>>>>>>>>> to
>>>>>>>>>>>>>>>> maintain in the long run. I like the original approach
>>>> based
>>>>>>> on
>>>>>>>>> the
>>>>>>>>>>>>>> batch
>>>>>>>>>>>>>>>> coordinator much better than this one. We could
>>> probably
>>>>>>>> refactor
>>>>>>>>>> the
>>>>>>>>>>>>>>>> producer state code so that it could be reused in the
>>>> batch
>>>>>>>>>>>>>> coordinator.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for addressing the concerns on the number of
>>>> RPCs in
>>>>>>> the
>>>>>>>>>>>> produce
>>>>>>>>>>>>>>>> path. I agree that with the metadata crafting
>>>> mechanism, we
>>>>>>>> could
>>>>>>>>>>>>>> mitigate
>>>>>>>>>>>>>>>> the PRC problem. However, since we now require the
>>>> metadata
>>>>>> to
>>>>>>>> be
>>>>>>>>>>>>>>>> collocated with the data on the same set of brokers,
>>>> it's
>>>>>>> weird
>>>>>>>>> that
>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>> are now managed by different mechanisms. The data
>>>> assignment
>>>>>>> now
>>>>>>>>>> uses
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> metadata crafting mechanism, but the metadata is
>>> stored
>>>> in
>>>>>> the
>>>>>>>>>> classic
>>>>>>>>>>>>>>>> partition using its own assignment strategy. It will
>>> be
>>>>>>>>> complicated
>>>>>>>>>> to
>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>>> them collocated.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am also concerned about the removal of the object
>>>>>>>>>> compaction/merging
>>>>>>>>>>>>>>>> step. My first concern is on the amount of metadata
>>> that
>>>>>> need
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>>>>> kept.
>>>>>>>>>>>>>>>> Without object compcation, the metadata generated in
>>> the
>>>>>>> produce
>>>>>>>>>> path
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> only be deleted after remote tiering kicks in. Let's
>>>> say for
>>>>>>>> every
>>>>>>>>>>>>>> 250ms we
>>>>>>>>>>>>>>>> produce 100 byte of metadata per partition. Let's say
>>>>>> remoting
>>>>>>>>>> tiering
>>>>>>>>>>>>>>>> kicks in after 5 hours. In a cluster with 100K
>>>> partitions,
>>>>>> we
>>>>>>>> need
>>>>>>>>>> to
>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
>>>> metadata,
>>>>>>>> quite
>>>>>>>>>>>>>>>> signficant. A second concern is on performance. Every
>>>> time
>>>>>> we
>>>>>>>> need
>>>>>>>>>> to
>>>>>>>>>>>>>>>> rebuild the caching data, we need to read a bunch of
>>>> small
>>>>>>>> objects
>>>>>>>>>>>>>> from S3,
>>>>>>>>>>>>>>>> slowing down the building process. If a consumer
>>>> happens to
>>>>>>> need
>>>>>>>>>> such
>>>>>>>>>>>>>> data,
>>>>>>>>>>>>>>>> it could slow down the application.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I see a tendency toward primarily optimizing for the
>>>> fewest
>>>>>>> code
>>>>>>>>>>>>>> changes in
>>>>>>>>>>>>>>>> the KIP. Instead, our primary goal should be a clean
>>>> design
>>>>>>> that
>>>>>>>>> can
>>>>>>>>>>>>>> last
>>>>>>>>>>>>>>>> for the long term.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
>>>>>>> ivan@ivanyu.me
>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you for your message. I’m sorry that I failed
>>> to
>>>>>>> clearly
>>>>>>>>>>>>>> explain
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> idea. Let me try to fix this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does each partition now have a metadata partition
>>> and
>>>> a
>>>>>>>> separate
>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>> partition? If so, I am concerned that it essentially
>>>>>> doubles
>>>>>>>> the
>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> partitions, which impacts the number of open file
>>>>>>> descriptors
>>>>>>>>> and
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> required IOPS, and so on. It also seems wasteful to
>>>> have a
>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>> partition just to store the metadata. It's as if we
>>>> are
>>>>>>>> creating
>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> internal topic with an unbounded number of
>>> partitions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> No. There will be only one physical partition per
>>>> diskless
>>>>>>>>>>>>>> partition. Let
>>>>>>>>>>>>>>>>> me explain this with an example. Let’s say we have a
>>>>>> diskless
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
>>>> leader. We
>>>>>>>>> produce
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>> batches to this partition. The content of the segment
>>>> file
>>>>>>> will
>>>>>>>>> be
>>>>>>>>>>>>>>>>> something like this (for each batch):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> BaseOffset: 00000000000000000000 (like in classic)
>>>>>>>>>>>>>>>>> Length: 123456 (like in classic)
>>>>>>>>>>>>>>>>> PartitionLeaderEpoch: like in classic
>>>>>>>>>>>>>>>>> Magic: like in classic
>>>>>>>>>>>>>>>>> CRC: like in classic
>>>>>>>>>>>>>>>>> Attributes: like in classic
>>>>>>>>>>>>>>>>> LastOffsetDelta: like in classic
>>>>>>>>>>>>>>>>> BaseTimestamp: like in classic
>>>>>>>>>>>>>>>>> MaxTimestamp: like in classic
>>>>>>>>>>>>>>>>> ProducerId: like in classic
>>>>>>>>>>>>>>>>> ProducerEpoch: like in classic
>>>>>>>>>>>>>>>>> BaseSequence: like in classic
>>>>>>>>>>>>>>>>> RecordsCount: like in classic
>>>>>>>>>>>>>>>>> Records:
>>>>>>>>>>>>>>>>>
>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
>>>>>> byte
>>>>>>>>> offset
>>>>>>>>>>>>>>>>> 123456
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It looks very much like classic log entries. The only
>>>>>>>> difference
>>>>>>>>> is
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> instead of writing real Records, we write the
>>>> reference to
>>>>>>> the
>>>>>>>>> WAL
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> with the batch data (I guess we need only the name
>>> and
>>>> the
>>>>>>> byte
>>>>>>>>>>>>>> offset,
>>>>>>>>>>>>>>>>> because the byte length is the standard field above).
>>>>>>>> Otherwise,
>>>>>>>>>>>>>> it’s a
>>>>>>>>>>>>>>>>> normal Kafka log with the leader and replicas.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So we have as many partitions for diskless as for
>>>> classic.
>>>>>> As
>>>>>>>> of
>>>>>>>>>> open
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> descriptors, let’s proceed to the following:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Are the metadata and
>>>>>>>>>>>>>>>>>> the data for the same partition always collocated on
>>>> the
>>>>>>> same
>>>>>>>>>>>>>> broker?
>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>> so, how do we enforce that when replicas are
>>>> reassigned?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The source of truth for the data is still in WAL
>>> files
>>>> on
>>>>>>>> object
>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>> The source of truth for the metadata is in segment
>>>> files on
>>>>>>> the
>>>>>>>>>>>>>> brokers
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the replica set. Two new mechanisms are planned, both
>>>>>>>> independent
>>>>>>>>>> of
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> new proposal, but I want to present them to give the
>>>> idea
>>>>>>> that
>>>>>>>>>> only a
>>>>>>>>>>>>>>>>> limited amount of data files will be operated
>>> locally:
>>>>>>>>>>>>>>>>> - We want to assemble batches into segment files and
>>>>>> offload
>>>>>>>> them
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> tiered storage in order to prevent the unbounded
>>>> growth of
>>>>>>>> batch
>>>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>>> For this, we need to open only  a few file
>>> descriptors
>>>> (for
>>>>>>> the
>>>>>>>>>>>>>> segment
>>>>>>>>>>>>>>>>> file itself + the necessary indexes) before the
>>>> segment is
>>>>>>>> fully
>>>>>>>>>>>>>> written
>>>>>>>>>>>>>>>>> and handed over to RemoteLogManager.
>>>>>>>>>>>>>>>>> - We want to assemble local segment files for caching
>>>>>>> purposes
>>>>>>>> as
>>>>>>>>>>>>>> well,
>>>>>>>>>>>>>>>>> i.e. to speed up fetching. This will not materialize
>>>> the
>>>>>> full
>>>>>>>>>>>>>> content of
>>>>>>>>>>>>>>>>> the log, but only the hot set according to some
>>> policy
>>>> (or
>>>>>>>>>>>>>> configurable
>>>>>>>>>>>>>>>>> policies), i.e. the number of segments and file
>>>> descriptors
>>>>>>>> will
>>>>>>>>>>>>>> also be
>>>>>>>>>>>>>>>>> limited.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The number of RPCs in the produce path is
>>>> significantly
>>>>>>>> higher.
>>>>>>>>>> For
>>>>>>>>>>>>>>>>>> example, if a produce request has 100 partitions,
>>> in a
>>>>>>> cluster
>>>>>>>>>>>>>> with 100
>>>>>>>>>>>>>>>>>> brokers, each produce request could generate 100
>>> more
>>>> RPC
>>>>>>>>>> requests.
>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>> will significantly increase the request rate.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is a valid concern that we considered, but this
>>>> issue
>>>>>>> can
>>>>>>>> be
>>>>>>>>>>>>>>>>> mitigated. I’ll try to explain the approach.
>>>>>>>>>>>>>>>>> The situation with a single broker is trivial: all
>>> the
>>>>>> commit
>>>>>>>>>>>>>> requests go
>>>>>>>>>>>>>>>>> from the broker to itself.
>>>>>>>>>>>>>>>>> Let’s scale this to a multi-broker cluster, but
>>>> located in
>>>>>>> the
>>>>>>>>>> single
>>>>>>>>>>>>>>>> rack
>>>>>>>>>>>>>>>>> (AZ). Any broker can accept Produce requests for
>>>> diskless
>>>>>>>>>>>>>> partitions, but
>>>>>>>>>>>>>>>>> we can tell producers (through metadata) to always
>>> send
>>>>>>> Produce
>>>>>>>>>>>>>> requests
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> leaders. For example, broker 0 hosts the leader
>>>> replicas
>>>>>> for
>>>>>>>>>> diskless
>>>>>>>>>>>>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
>>>>>> Produce
>>>>>>>>>>>>>> requests
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> these partitions in various combinations, but only
>>> for
>>>>>> them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                  Broker 0
>>>>>>>>>>>>>>>>>            +-----------------+
>>>>>>>>>>>>>>>>>            |    t1-0         |
>>>>>>>>>>>>>>>>>            |    t2-1 <--------------------+
>>>>>>>>>>>>>>>>>            |    t3-0         |            |
>>>>>>>>>>>>>>>>> produce      | +-------------+ |            |
>>>>>>>>>>>>>>>>> requests     | |  diskless   | |            |
>>>>>>>>>>>>>>>>> --------------->|   produce   +--------------+
>>>>>>>>>>>>>>>>> for these    | | WAL buffer  | |    commit requests
>>>>>>>>>>>>>>>>> partitions   | +-------------+ |    for these
>>>> partitions
>>>>>>>>>>>>>>>>>            |                 |
>>>>>>>>>>>>>>>>>            +-----------------+
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The same applies for other brokers in this cluster.
>>>>>>>> Effectively,
>>>>>>>>>> each
>>>>>>>>>>>>>>>>> broker will commit only to itself, which effectively
>>>> means
>>>>>> 1
>>>>>>>>> commit
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>> per WAL buffer (this may be 0 physical network calls,
>>>> if we
>>>>>>>> wish,
>>>>>>>>>>>>>> just a
>>>>>>>>>>>>>>>>> local function call).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Now let’s scale this to multiple racks (AZs).
>>>> Obviously, we
>>>>>>>>> cannot
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>> send Produce requests to the designated leaders of
>>>> diskless
>>>>>>>>>>>>>> partitions:
>>>>>>>>>>>>>>>>> this would mean inter-AZ network traffic, which we
>>>> would
>>>>>> like
>>>>>>>> to
>>>>>>>>>>>>>> avoid.
>>>>>>>>>>>>>>>> To
>>>>>>>>>>>>>>>>> avoid it, we say that every broker has a “diskless
>>>> produce
>>>>>>>>>>>>>>>> representative”
>>>>>>>>>>>>>>>>> in every AZ. If we continue our example: when a
>>> Produce
>>>>>>> request
>>>>>>>>> for
>>>>>>>>>>>>>> t1-0,
>>>>>>>>>>>>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands
>>>> on
>>>>>>>> broker 0
>>>>>>>>>>>>>> (in the
>>>>>>>>>>>>>>>>> broker’s AZ the representative is the broker itself).
>>>>>>> However,
>>>>>>>> if
>>>>>>>>>> it
>>>>>>>>>>>>>>>> comes
>>>>>>>>>>>>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker
>>>> 2.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> |produce requests         |produce requests
>>>> |produce
>>>>>>>>>>>>>> requests
>>>>>>>>>>>>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
>>> |for
>>>>>> t1-0,
>>>>>>>>> t2-1,
>>>>>>>>>>>>>>>> t3-0
>>>>>>>>>>>>>>>>> |from AZ 0                |from AZ 1
>>>> |from
>>>>>> AZ 2
>>>>>>>>>>>>>>>>> v                         v                        v
>>>>>>>>>>>>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker
>>> 2
>>>> (AZ
>>>>>> 2)
>>>>>>>>>>>>>>>>> +---------------+      +---------------+
>>>>>>> +---------------+
>>>>>>>>>>>>>>>>> |     t1-0      |      |               |      |
>>>>>>> |
>>>>>>>>>>>>>>>>> |     t2-1      |      |               |      |
>>>>>>> |
>>>>>>>>>>>>>>>>> |     t3-0      |      |               |      |
>>>>>>> |
>>>>>>>>>>>>>>>>> +---------------+      +--------+------+
>>>>>>> +--------+------+
>>>>>>>>>>>>>>>>>   ^     ^                    |
>>> |
>>>>>>>>>>>>>>>>>   |     +--------------------+
>>> |
>>>>>>>>>>>>>>>>>   |     commit requests for these partitions
>>> |
>>>>>>>>>>>>>>>>>   |
>>> |
>>>>>>>>>>>>>>>>>
>>> +-------------------------------------------------+
>>>>>>>>>>>>>>>>>         commit requests for these partitions
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> All the partitions that broker 0 is the leader of
>>> will
>>>> be
>>>>>>>>>>>>>> “represented”
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> brokers 1 and 2 in their AZs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Of course, this relationship goes both ways between
>>> AZs
>>>>>> (not
>>>>>>>>>>>>>> necessarily
>>>>>>>>>>>>>>>>> between the same brokers). It means that provided the
>>>>>> cluster
>>>>>>>> is
>>>>>>>>>>>>>> balanced
>>>>>>>>>>>>>>>>> by the number of brokers per AZ, each broker will
>>>> represent
>>>>>>>>>>>>>>>> (number_of_azs
>>>>>>>>>>>>>>>>> - 1) other brokers. This will result in the situation
>>>> that
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>>>>>> majority
>>>>>>>>>>>>>>>>> of commits, each broker will do up to (number_of_azs
>>> -
>>>> 1)
>>>>>>>> network
>>>>>>>>>>>>>> commit
>>>>>>>>>>>>>>>>> requests (plus one local). Cloud regions tend to
>>> have 3
>>>>>> AZs,
>>>>>>>> very
>>>>>>>>>>>>>> rarely
>>>>>>>>>>>>>>>>> more. That means, brokers will be doing up to 2
>>> network
>>>>>>> commit
>>>>>>>>>>>>>> requests
>>>>>>>>>>>>>>>> per
>>>>>>>>>>>>>>>>> WAL file.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There are the following exceptions:
>>>>>>>>>>>>>>>>> 1. Broker count imbalance between AZs. For example,
>>>> when we
>>>>>>>> have
>>>>>>>>> 2
>>>>>>>>>>>>>> AZs
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> one has three brokers and another AZ has one. This
>>> one
>>>>>> broker
>>>>>>>>> will
>>>>>>>>>> do
>>>>>>>>>>>>>>>>> between 1 and 3 commit requests per WAL file. This is
>>>> not
>>>>>> an
>>>>>>>>>> extreme
>>>>>>>>>>>>>>>>> amplification. Such an imbalance is not healthy in
>>> most
>>>>>>>> practical
>>>>>>>>>>>>>> setups
>>>>>>>>>>>>>>>>> and should be avoided anyway.
>>>>>>>>>>>>>>>>> 2. Leadership changes and metadata propagation
>>> period.
>>>> When
>>>>>>> the
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
>>>>>>> producers
>>>>>>>>>> will
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> know this immediately (unless we want to be strict
>>> and
>>>>>>> respond
>>>>>>>>> with
>>>>>>>>>>>>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
>>>> will
>>>>>> come
>>>>>>>>>>>>>> together
>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>> WAL buffer on broker 2, it will have to send two
>>> commit
>>>>>>>> requests:
>>>>>>>>>> to
>>>>>>>>>>>>>>>> broker
>>>>>>>>>>>>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
>>>> t3-0.
>>>>>>> This
>>>>>>>>>>>>>> situation
>>>>>>>>>>>>>>>>> is not permanent and as producers update the cluster
>>>>>>> metadata,
>>>>>>>> it
>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>> resolved.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This all could be built with the metadata crafting
>>>>>> mechanism
>>>>>>>> only
>>>>>>>>>>>>>> (which
>>>>>>>>>>>>>>>>> is anyway needed for Diskless in one way or another
>>> to
>>>>>> direct
>>>>>>>>>>>>>> producers
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> consumers where we need to avoid inter-AZ traffic),
>>>> just
>>>>>> with
>>>>>>>> the
>>>>>>>>>>>>>> right
>>>>>>>>>>>>>>>>> policy for it (for example, some deterministic
>>>> hash-based
>>>>>>>>> formula).
>>>>>>>>>>>>>> I.e.
>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>> explicit support for “produce representative” or
>>>> anything
>>>>>>> like
>>>>>>>>> this
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> needed on the cluster level, in KRaft, etc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The same WAL file metadata is now duplicated into
>>> two
>>>>>>> places,
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>> leader and WAL File Manager. Which one is the source
>>>> of
>>>>>>> truth,
>>>>>>>>> and
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> we maintain consistency between the two places?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We do only two operations on WAL files that span
>>>> multiple
>>>>>>>>> diskless
>>>>>>>>>>>>>>>>> partitions: committing and deleting. Commits can be
>>>> done
>>>>>>>>>>>>>> independently as
>>>>>>>>>>>>>>>>> described above. But deletes are different, because
>>>> when a
>>>>>>> file
>>>>>>>>> is
>>>>>>>>>>>>>>>> deleted,
>>>>>>>>>>>>>>>>> this affects all the partitions that still have alive
>>>>>> batches
>>>>>>>> in
>>>>>>>>>> this
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> (if any).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The WAL file manager is a necessary point of
>>>> coordination
>>>>>> to
>>>>>>>>> delete
>>>>>>>>>>>>>> WAL
>>>>>>>>>>>>>>>>> files safely. We can say it is the source of truth
>>>> about
>>>>>>> files
>>>>>>>>>>>>>>>> themselves,
>>>>>>>>>>>>>>>>> while the partition leaders and their logs hold the
>>>> truth
>>>>>>> about
>>>>>>>>>>>>>> whether a
>>>>>>>>>>>>>>>>> particular file contains live batches of this
>>>> particular
>>>>>>>>> partition.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The file manager will do this important task: be able
>>>> to
>>>>>> say
>>>>>>>> for
>>>>>>>>>> sure
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> a file does not contain any live batch of any
>>> existing
>>>>>>>> partition.
>>>>>>>>>> For
>>>>>>>>>>>>>>>> this,
>>>>>>>>>>>>>>>>> it will have to periodically check against the
>>>> partition
>>>>>>>> leaders.
>>>>>>>>>>>>>>>>> Considering that batch deletion is irreversible, when
>>>> we
>>>>>>>> declare
>>>>>>>>> a
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> “empty”, this is guaranteed to be and stay so.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The file manager has to know about files being
>>>> committed to
>>>>>>>> start
>>>>>>>>>>>>>> track
>>>>>>>>>>>>>>>>> them and periodically check if they are empty. We can
>>>>>>> consider
>>>>>>>>>>>>>> various
>>>>>>>>>>>>>>>> ways
>>>>>>>>>>>>>>>>> to achieve this:
>>>>>>>>>>>>>>>>> 1. As was proposed in my previous message: best
>>> effort
>>>>>> commit
>>>>>>>> by
>>>>>>>>>>>>>> brokers
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> periodic prefix scans of object storage to detect
>>> files
>>>>>> that
>>>>>>>> went
>>>>>>>>>>>>>> below
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> radar due to network issue or the file manager
>>>> temporary
>>>>>>>>>>>>>> unavailability.
>>>>>>>>>>>>>>>>> We’re speaking about listing the file names only and
>>>>>> opening
>>>>>>>> only
>>>>>>>>>>>>>>>>> previously unknown files in order to find the
>>>> partitions
>>>>>>>> involved
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>> 2. Only do scans without explicit commit, i.e. fill
>>> the
>>>>>> list
>>>>>>> of
>>>>>>>>>> files
>>>>>>>>>>>>>>>>> fully asynchronously and in the background. This may
>>>> be not
>>>>>>>> ideal
>>>>>>>>>>>>>> due to
>>>>>>>>>>>>>>>>> costs and performance of scanning tons of files.
>>>> However,
>>>>>> the
>>>>>>>>>> number
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> live WAL files should be limited due to tiered
>>> storage
>>>>>>>>> offloading +
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> optimize this if we give files some global soft order
>>>> in
>>>>>>> their
>>>>>>>>>> names.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not sure how this design simplifies the
>>>>>> implementation.
>>>>>>>> The
>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>> producer/replication code can't be simply reused.
>>>>>> Adjusting
>>>>>>>> both
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>>>> path in the leader and the replication path in the
>>>>>> follower
>>>>>>> to
>>>>>>>>>>>>>>>> understand
>>>>>>>>>>>>>>>>>> batch-header only data is quite intrusive to the
>>>> existing
>>>>>>>> logic.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It is true that we’ll have to change LocalLog and
>>>>>> UnifiedLog
>>>>>>> in
>>>>>>>>>>>>>> order to
>>>>>>>>>>>>>>>>> support these changes. However, it seems that
>>>> idempotence,
>>>>>>>>>>>>>> transactions,
>>>>>>>>>>>>>>>>> queues, tiered storage will have to be changed less
>>>> than
>>>>>> with
>>>>>>>> the
>>>>>>>>>>>>>>>> original
>>>>>>>>>>>>>>>>> design. This is because the partition leader state
>>>> would
>>>>>>> remain
>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> place (on brokers) and existing workflows that
>>> involve
>>>> it
>>>>>>> would
>>>>>>>>>> have
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> changed less compared to the situation where we
>>>> globalize
>>>>>> the
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> leader state in the batch coordinator. I admit this
>>> is
>>>> hard
>>>>>>> to
>>>>>>>>> make
>>>>>>>>>>>>>>>>> convincing without both real implementations to hand
>>> :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am also
>>>>>>>>>>>>>>>>>> not sure how this enables seamless switching the
>>> topic
>>>>>> modes
>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>> diskless and classic. Could you provide more details
>>>> on
>>>>>>> those?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Let’s consider the scenario of turning a classic
>>> topic
>>>> into
>>>>>>>>>>>>>> diskless. The
>>>>>>>>>>>>>>>>> user sets diskless.enabled=true, the leader receives
>>>> this
>>>>>>>>> metadata
>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>>>> and does the following:
>>>>>>>>>>>>>>>>> 1. Stop accepting normal append writes.
>>>>>>>>>>>>>>>>> 2. Close the current active segment.
>>>>>>>>>>>>>>>>> 3. Start a new segment that will be written in the
>>>> diskless
>>>>>>>>> format
>>>>>>>>>>>>>> (i.e.
>>>>>>>>>>>>>>>>> without data).
>>>>>>>>>>>>>>>>> 4. Start accepting diskless commits.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Since it’s the same log, the followers will know
>>> about
>>>> that
>>>>>>>>> switch
>>>>>>>>>>>>>>>>> consistently. They will finish replicating the
>>> classic
>>>>>>> segments
>>>>>>>>> and
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>> replicating the diskless ones. They will always know
>>>> where
>>>>>>> each
>>>>>>>>>>>>>> batch is
>>>>>>>>>>>>>>>>> located (either inside a classic segment or
>>> referenced
>>>> by a
>>>>>>>>>> diskless
>>>>>>>>>>>>>>>> one).
>>>>>>>>>>>>>>>>> Switching back should be similar.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Doing this with the coordinator is possible, but has
>>>> some
>>>>>>>>> caveats.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> leader must do the following:
>>>>>>>>>>>>>>>>> 1. Stop accepting normal append writes.
>>>>>>>>>>>>>>>>> 2. Close the current active segment.
>>>>>>>>>>>>>>>>> 3. Write a special control segment to persist and
>>>> replicate
>>>>>>> the
>>>>>>>>>> fact
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> from offset N the partition is now in the diskless
>>>> mode.
>>>>>>>>>>>>>>>>> 4. Inform the coordinator about the first offset N of
>>>> the
>>>>>>>>> “diskless
>>>>>>>>>>>>>> era”.
>>>>>>>>>>>>>>>>> 5. Inform the controller quorum that the transition
>>> has
>>>>>>>> finished
>>>>>>>>>> and
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> brokers now can process diskless writes for this
>>>> partition.
>>>>>>>>>>>>>>>>> This could fail at some points, so this will probably
>>>>>> require
>>>>>>>>> some
>>>>>>>>>>>>>>>>> explicit state machine with replication either in the
>>>>>>> partition
>>>>>>>>> log
>>>>>>>>>>>>>> or in
>>>>>>>>>>>>>>>>> KRaft.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It seems that the coordinator-less approach makes
>>> this
>>>>>>> simpler
>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> “coordinator” for the partition and the partition
>>>> leader
>>>>>> are
>>>>>>>> the
>>>>>>>>>>>>>> same and
>>>>>>>>>>>>>>>>> they store the partition metadata in the same log,
>>> too.
>>>>>> While
>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> coordinator approach we have to perform some kind of
>>> a
>>>>>>>>> distributed
>>>>>>>>>>>>>> commit
>>>>>>>>>>>>>>>>> to handover metadata management from the classic
>>>> partition
>>>>>>>> leader
>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> batch coordinator.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I hope these explanations help to clarify the idea.
>>>> Please
>>>>>>> let
>>>>>>>> me
>>>>>>>>>>>>>> know if
>>>>>>>>>>>>>>>>> I should go deeper anywhere.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Ivan and the Diskless team
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
>>>>>>>>>>>>>>>>>> Hi, Ivan,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the update.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not sure that I fully understand the new
>>> design,
>>>> but
>>>>>> it
>>>>>>>>> seems
>>>>>>>>>>>>>> less
>>>>>>>>>>>>>>>>>> clean than before.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does each partition now have a metadata partition
>>> and
>>>> a
>>>>>>>> separate
>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>> partition? If so, I am concerned that it essentially
>>>>>> doubles
>>>>>>>> the
>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> partitions, which impacts the number of open file
>>>>>>> descriptors
>>>>>>>>> and
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> required IOPS, and so on. It also seems wasteful to
>>>> have a
>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>> partition just to store the metadata. It's as if we
>>>> are
>>>>>>>> creating
>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> internal topic with an unbounded number of
>>>> partitions. Are
>>>>>>> the
>>>>>>>>>>>>>> metadata
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> the data for the same partition always collocated on
>>>> the
>>>>>>> same
>>>>>>>>>>>>>> broker?
>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>> so, how do we enforce that when replicas are
>>>> reassigned?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The number of RPCs in the produce path is
>>>> significantly
>>>>>>>> higher.
>>>>>>>>>> For
>>>>>>>>>>>>>>>>>> example, if a produce request has 100 partitions,
>>> in a
>>>>>>> cluster
>>>>>>>>>>>>>> with 100
>>>>>>>>>>>>>>>>>> brokers, each produce request could generate 100
>>> more
>>>> RPC
>>>>>>>>>> requests.
>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>> will significantly increase the request rate.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The same WAL file metadata is now duplicated into
>>> two
>>>>>>> places,
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>> leader and WAL File Manager. Which one is the source
>>>> of
>>>>>>> truth,
>>>>>>>>> and
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> we maintain consistency between the two places?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not sure how this design simplifies the
>>>>>> implementation.
>>>>>>>> The
>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>> producer/replication code can't be simply reused.
>>>>>> Adjusting
>>>>>>>> both
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>>>> path in the leader and the replication path in the
>>>>>> follower
>>>>>>> to
>>>>>>>>>>>>>>>> understand
>>>>>>>>>>>>>>>>>> batch-header only data is quite intrusive to the
>>>> existing
>>>>>>>>> logic. I
>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> not sure how this enables seamless switching the
>>> topic
>>>>>> modes
>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>> diskless and classic. Could you provide more details
>>>> on
>>>>>>> those?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
>>>>>>> ivan@ivanyu.me
>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi dear Kafka community,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In the initial Diskless proposal, we proposed to
>>>> have a
>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>> component, batch/diskless coordinator, whose role
>>>> would
>>>>>> be
>>>>>>> to
>>>>>>>>>>>>>>>> centrally
>>>>>>>>>>>>>>>>>>> manage the batch and WAL file metadata for diskless
>>>>>> topics.
>>>>>>>>> This
>>>>>>>>>>>>>>>>> component
>>>>>>>>>>>>>>>>>>> drew many reasonable comments from the community
>>>> about
>>>>>> how
>>>>>>> it
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>> various Kafka features (transactions, queues) and
>>> its
>>>>>>>>>>>>>> scalability.
>>>>>>>>>>>>>>>>> While we
>>>>>>>>>>>>>>>>>>> believe we have good answers to all the expressed
>>>>>> concerns,
>>>>>>>> we
>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>> step
>>>>>>>>>>>>>>>>>>> back and looked at the problem from a different
>>>>>>> perspective.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We would like to propose an alternative Diskless
>>>> design
>>>>>>>>> *without
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> centralized coordinator*. We believe this approach
>>>> has
>>>>>>>>> potential
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> propose to discuss it as it may be more appealing
>>> to
>>>> the
>>>>>>>>>>>>>> community.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Let us explain the idea. Most of the complications
>>>> with
>>>>>> the
>>>>>>>>>>>>>> original
>>>>>>>>>>>>>>>>>>> Diskless approach come from one necessary
>>>> architecture
>>>>>>>> change:
>>>>>>>>>>>>>>>>> globalizing
>>>>>>>>>>>>>>>>>>> the local state of partition leader in the batch
>>>>>>> coordinator.
>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>>> deviations to the established workflows in various
>>>>>> features
>>>>>>>>> like
>>>>>>>>>>>>>>>>> produce
>>>>>>>>>>>>>>>>>>> idempotence and transactions, queues, retention,
>>> etc.
>>>>>> These
>>>>>>>>>>>>>>>> deviations
>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>> to be carefully considered, designed, and later
>>>>>> implemented
>>>>>>>> and
>>>>>>>>>>>>>>>>> tested. In
>>>>>>>>>>>>>>>>>>> the new approach we want to avoid this by making
>>>>>> partition
>>>>>>>>>>>>>> leaders
>>>>>>>>>>>>>>>>> again
>>>>>>>>>>>>>>>>>>> responsible for managing their partitions, even in
>>>>>> diskless
>>>>>>>>>>>>>> topics.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In classic Kafka topics, batch data and metadata
>>> are
>>>>>>> blended
>>>>>>>>>>>>>> together
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> the one partition log. The crux of the Diskless
>>> idea
>>>> is
>>>>>> to
>>>>>>>>>>>>>> decouple
>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>> and move data to the remote storage, while keeping
>>>>>> metadata
>>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>>>>>> else.
>>>>>>>>>>>>>>>>>>> Using the central batch coordinator for managing
>>>> batch
>>>>>>>> metadata
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>>>>>>> but not the only.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Let’s now think about managing metadata for each
>>> user
>>>>>>>> partition
>>>>>>>>>>>>>>>>>>> independently. Generally partitions are independent
>>>> and
>>>>>>> don’t
>>>>>>>>>>>>>> share
>>>>>>>>>>>>>>>>>>> anything apart from that their data are mixed in
>>> WAL
>>>>>> files.
>>>>>>>> If
>>>>>>>>> we
>>>>>>>>>>>>>>>>> figure
>>>>>>>>>>>>>>>>>>> out how to commit and later delete WAL files
>>> safely,
>>>> we
>>>>>>> will
>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> necessary autonomy that allows us to get rid of the
>>>>>> central
>>>>>>>>> batch
>>>>>>>>>>>>>>>>>>> coordinator. Instead, *each diskless user partition
>>>> will
>>>>>> be
>>>>>>>>>>>>>> managed
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> its
>>>>>>>>>>>>>>>>>>> leader*, as in classic Kafka topics. Also like in
>>>> classic
>>>>>>>>>>>>>> topics, the
>>>>>>>>>>>>>>>>>>> leader uses the partition log as the way to persist
>>>> batch
>>>>>>>>>>>>>> metadata,
>>>>>>>>>>>>>>>>> i.e.
>>>>>>>>>>>>>>>>>>> the regular batch header + the information about
>>> how
>>>> to
>>>>>>> find
>>>>>>>>> this
>>>>>>>>>>>>>>>>> batch on
>>>>>>>>>>>>>>>>>>> remote storage. In contrast to classic topics,
>>> batch
>>>> data
>>>>>>> is
>>>>>>>> in
>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For clarity, let’s compare the three designs:
>>>>>>>>>>>>>>>>>>> • Classic topics:
>>>>>>>>>>>>>>>>>>> • Data and metadata are co-located in the
>>> partition
>>>> log.
>>>>>>>>>>>>>>>>>>> • The partition log content: [Batch header
>>>>>>> (metadata)|Batch
>>>>>>>>>>>>>> data].
>>>>>>>>>>>>>>>>>>> • The partition log is replicated to the
>>> followers.
>>>>>>>>>>>>>>>>>>> • The replicas and leader have local state built
>>>> from
>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>>>>> • Original Diskless:
>>>>>>>>>>>>>>>>>>> • Metadata is in the batch coordinator, data is on
>>>>>> remote
>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>>>> • The partition state is global in the batch
>>>>>> coordinator.
>>>>>>>>>>>>>>>>>>> • New Diskless:
>>>>>>>>>>>>>>>>>>> • Metadata is in the partition log, data is on
>>>> remote
>>>>>>>> storage.
>>>>>>>>>>>>>>>>>>> • Partition log content: [Batch header
>>>> (metadata)|Batch
>>>>>>>>>>>>>>>> coordinates
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> remote storage].
>>>>>>>>>>>>>>>>>>> • The partition log is replicated to the
>>> followers.
>>>>>>>>>>>>>>>>>>> • The replicas and leader have local state built
>>>> from
>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Let’s consider the produce path. Here’s the
>>> reminder
>>>> of
>>>>>> the
>>>>>>>>>>>>>> original
>>>>>>>>>>>>>>>>>>> Diskless design:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The new approach could be depicted as the
>>> following:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As you can see, the main difference is that now
>>>> instead
>>>>>> of
>>>>>>> a
>>>>>>>>>>>>>> single
>>>>>>>>>>>>>>>>> commit
>>>>>>>>>>>>>>>>>>> request to the batch coordinator, we send multiple
>>>>>> parallel
>>>>>>>>>>>>>> commit
>>>>>>>>>>>>>>>>> requests
>>>>>>>>>>>>>>>>>>> to all the leaders of each partition involved in
>>> the
>>>> WAL
>>>>>>>> file.
>>>>>>>>>>>>>> Each
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>> will commit its batches independently, without
>>>>>> coordinating
>>>>>>>>> with
>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> leaders and any other components. Batch data is
>>>> addressed
>>>>>>> by
>>>>>>>>> the
>>>>>>>>>>>>>> WAL
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>> name, the byte offset and size, which allows
>>>> partitions
>>>>>> to
>>>>>>>> know
>>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>>>>>> about other partitions to access their data in
>>>> shared WAL
>>>>>>>>> files.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The number of partitions involved in a single WAL
>>>> file
>>>>>> may
>>>>>>> be
>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>> large,
>>>>>>>>>>>>>>>>>>> e.g. a hundred. A hundred network requests to
>>> commit
>>>> one
>>>>>>> WAL
>>>>>>>>>>>>>> file is
>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>> impractical. However, there are ways to reduce this
>>>>>> number:
>>>>>>>>>>>>>>>>>>> 1. Partition leaders are located on brokers.
>>>> Requests to
>>>>>>>>>>>>>> leaders on
>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>> broker could be grouped together into a single
>>>> physical
>>>>>>>> network
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>>> (resembling the normal Produce request that may
>>> carry
>>>>>>> batches
>>>>>>>>> for
>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>> partitions inside). This will cap the number of
>>>> network
>>>>>>>>> requests
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> number of brokers in the cluster.
>>>>>>>>>>>>>>>>>>> 2. If we craft the cluster metadata to make
>>> producers
>>>>>> send
>>>>>>>>> their
>>>>>>>>>>>>>>>>> requests
>>>>>>>>>>>>>>>>>>> to the right brokers (with respect to AZs), we may
>>>>>> achieve
>>>>>>>> the
>>>>>>>>>>>>>> higher
>>>>>>>>>>>>>>>>>>> concentration of logical commit requests in
>>> physical
>>>>>>> network
>>>>>>>>>>>>>> requests
>>>>>>>>>>>>>>>>>>> reducing the number of the latter ones even
>>> further,
>>>>>>> ideally
>>>>>>>> to
>>>>>>>>>>>>>> one.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Obviously, out of multiple commit requests some may
>>>> fail
>>>>>> or
>>>>>>>>> time
>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>> variety of reasons. This is fine. Some producers
>>> will
>>>>>>> receive
>>>>>>>>>>>>>> totally
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> partially failed responses to their Produce
>>> requests,
>>>>>>> similar
>>>>>>>>> to
>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>> would have received when appending to a classic
>>> topic
>>>>>> fails
>>>>>>>> or
>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>> If a partition experiences problems, other
>>> partitions
>>>>>> will
>>>>>>>> not
>>>>>>>>> be
>>>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>> (again, like in classic topics). Of course, the
>>>>>> uncommitted
>>>>>>>>> data
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> garbage in WAL files. But WAL files are short-lived
>>>>>>> (batches
>>>>>>>>> are
>>>>>>>>>>>>>>>>> constantly
>>>>>>>>>>>>>>>>>>> assembled into segments and offloaded to tiered
>>>> storage),
>>>>>>> so
>>>>>>>>> this
>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>> will be eventually deleted.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For safely deleting WAL files we now need to
>>>> centrally
>>>>>>> manage
>>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> this is the only state and logic that spans
>>> multiple
>>>>>>>>> partitions.
>>>>>>>>>>>>>> On
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> diagram, you can see another commit request called
>>>>>> “Commit
>>>>>>>> file
>>>>>>>>>>>>>> (best
>>>>>>>>>>>>>>>>>>> effort)” going to the WAL File Manager. This
>>> manager
>>>> will
>>>>>>> be
>>>>>>>>>>>>>>>>> responsible
>>>>>>>>>>>>>>>>>>> for the following:
>>>>>>>>>>>>>>>>>>> 1. Collecting (by requests from brokers) and
>>>> persisting
>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> committed WAL files.
>>>>>>>>>>>>>>>>>>> 2. To handle potential failures in file information
>>>>>>> delivery,
>>>>>>>>> it
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> doing prefix scan on the remote storage
>>> periodically
>>>> to
>>>>>>> find
>>>>>>>>> and
>>>>>>>>>>>>>>>>> register
>>>>>>>>>>>>>>>>>>> unknown files. The period of this scan will be
>>>>>> configurable
>>>>>>>> and
>>>>>>>>>>>>>>>> ideally
>>>>>>>>>>>>>>>>>>> should be quite long.
>>>>>>>>>>>>>>>>>>> 3. Checking with the relevant partition leaders
>>>> (after a
>>>>>>>> grace
>>>>>>>>>>>>>>>>> period) if
>>>>>>>>>>>>>>>>>>> they still have batches in a particular file.
>>>>>>>>>>>>>>>>>>> 4. Physically deleting files when they aren’t
>>> anymore
>>>>>>>> referred
>>>>>>>>>>>>>> to by
>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> partition.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This new design offers the following advantages:
>>>>>>>>>>>>>>>>>>> 1. It simplifies the implementation of many Kafka
>>>>>> features
>>>>>>>> such
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> idempotence, transactions, queues, tiered storage,
>>>>>>> retention.
>>>>>>>>>>>>>> Now we
>>>>>>>>>>>>>>>>> don’t
>>>>>>>>>>>>>>>>>>> need to abstract away and reuse the code from
>>>> partition
>>>>>>>> leaders
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> batch coordinator. Instead, we will literally use
>>> the
>>>>>> same
>>>>>>>> code
>>>>>>>>>>>>>> paths
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> leaders, with little adaptation. Workflows from
>>>> classic
>>>>>>>> topics
>>>>>>>>>>>>>> mostly
>>>>>>>>>>>>>>>>>>> remain unchanged.
>>>>>>>>>>>>>>>>>>> For example, it seems that
>>>>>>>>>>>>>>>>>>>
>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator
>>>>>>>> and
>>>>>>>>>>>>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
>>>>>> transaction
>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> partition leader side could be used for diskless
>>>> topics
>>>>>>> with
>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>>>>>> adaptation. ProducerStateManager, needed for both
>>>>>>> idempotent
>>>>>>>>>>>>>> produce
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> transactions, would be reused.
>>>>>>>>>>>>>>>>>>> Another example is share groups support, where the
>>>> share
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> leader,
>>>>>>>>>>>>>>>>>>> being co-located with the partition leader, would
>>>> execute
>>>>>>> the
>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>> for both diskless and classic topics.
>>>>>>>>>>>>>>>>>>> 2. It returns to the familiar partition-based
>>> scaling
>>>>>>> model,
>>>>>>>>>>>>>> where
>>>>>>>>>>>>>>>>>>> partitions are independent.
>>>>>>>>>>>>>>>>>>> 3. It makes the operation and failure patterns
>>>> closer to
>>>>>>> the
>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>> ones from classic topics.
>>>>>>>>>>>>>>>>>>> 4. It opens a straightforward path to seamless
>>>> switching
>>>>>>> the
>>>>>>>>>>>>>> topics
>>>>>>>>>>>>>>>>> modes
>>>>>>>>>>>>>>>>>>> between diskless and classic.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The rest of the things remain unchanged compared to
>>>> the
>>>>>>>>> previous
>>>>>>>>>>>>>>>>> Diskless
>>>>>>>>>>>>>>>>>>> design (after all previous discussions). Such
>>> things
>>>> as
>>>>>>> local
>>>>>>>>>>>>>> segment
>>>>>>>>>>>>>>>>>>> materialization by replicas, the consume path,
>>> tiered
>>>>>>> storage
>>>>>>>>>>>>>>>>> integration,
>>>>>>>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If the community finds this design more suitable,
>>> we
>>>> will
>>>>>>>>> update
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> KIP(s) accordingly and continue working on it.
>>>> Please let
>>>>>>> us
>>>>>>>>> know
>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> think.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>> Ivan and Diskless team
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
>>> wrote:
>>>>>>>>>>>>>>>>>>>> Hi Justine,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes, you're right. We need to track the aborted
>>>>>>> transactions
>>>>>>>>>>>>>> for in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> diskless coordinator for as long as the
>>> corresponding
>>>>>>> offsets
>>>>>>>>> are
>>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>>>> With the tiered storage unification Greg mentioned
>>>>>> earlier,
>>>>>>>>> this
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> finite time even for infinite data retention.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Ivan
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> Hey Ivan,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the response. I think most of what you
>>>> said
>>>>>>> made
>>>>>>>>>>>>>>>> sense,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> did have some questions about this part:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> As we understand this, the partition leader in
>>>> classic
>>>>>>>>>>>>>> topics
>>>>>>>>>>>>>>>>> forgets
>>>>>>>>>>>>>>>>>>>>> about a transaction once it’s replicated (HWM
>>>>>> overpasses
>>>>>>>>>>>>>> it). The
>>>>>>>>>>>>>>>>>>>>> transaction coordinator acts like the main
>>>> guardian,
>>>>>>>> allowing
>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>> leaders to do this safely. Please correct me if
>>>> this is
>>>>>>>>>>>>>> wrong. We
>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>> about relying on this with the batch coordinator
>>>> and
>>>>>>> delete
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>>>>>> about a transaction once it’s finished (as
>>> there’s
>>>> no
>>>>>>>>>>>>>> replication
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> HWM
>>>>>>>>>>>>>>>>>>>>> advances immediately).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I didn't quite understand this. In classic
>>> topics,
>>>> we
>>>>>>> have
>>>>>>>>>>>>>> maps
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> ongoing
>>>>>>>>>>>>>>>>>>>>> transactions which remove state when the
>>>> transaction is
>>>>>>>>>>>>>> completed
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> aborted transactions index which is retained for
>>>> much
>>>>>>>> longer.
>>>>>>>>>>>>>>>> Once
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> transaction is completed, the coordinator is no
>>>> longer
>>>>>>>>>>>>>> involved
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> maintaining this partition side state, and it is
>>>>>> subject
>>>>>>> to
>>>>>>>>>>>>>>>>> compaction
>>>>>>>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>>> Looking back at the outline provided above, I
>>>> didn't
>>>>>> see
>>>>>>>> much
>>>>>>>>>>>>>>>>> about the
>>>>>>>>>>>>>>>>>>>>> fetch path, so maybe that could be expanded a bit
>>>>>>> further.
>>>>>>>> I
>>>>>>>>>>>>>> saw
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> following in a response:
>>>>>>>>>>>>>>>>>>>>>> When the broker constructs a fully valid local
>>>>>> segment,
>>>>>>>>>>>>>> all the
>>>>>>>>>>>>>>>>>>> necessary
>>>>>>>>>>>>>>>>>>>>> control batches will be inserted and indices,
>>>> including
>>>>>>> the
>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>> index will be built to serve FetchRequests
>>> exactly
>>>> as
>>>>>>> they
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Based on this, it seems like we need to retain
>>> the
>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>> aborted txns for longer.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Justine
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
>>>>>>>>>>>>>> ivan@ivanyu.me>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Justine and all,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you for your questions!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
>>>>>> identified
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> producer ID
>>>>>>>>>>>>>>>>>>>>>>> and epoch, the positive result of this check
>>>> could be
>>>>>>>>>>>>>> cached
>>>>>>>>>>>>>>>>>>> locally
>>>>>>>>>>>>>>>>>>>>>>> Are we saying that only new transaction
>>> version 2
>>>>>>>>>>>>>>>> transactions
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>> here? If not, we can't uniquely identify
>>>> transactions
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> producer id +
>>>>>>>>>>>>>>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> You’re right that we (probably unintentionally)
>>>>>> focused
>>>>>>>>>>>>>> only on
>>>>>>>>>>>>>>>>>>> version 2.
>>>>>>>>>>>>>>>>>>>>>> We can either limit the support to version 2 or
>>>>>> consider
>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>> surrogates to support version 1.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 2. >The batch coordinator does the final
>>>>>>> transactional
>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>> batches. This procedure would output the same
>>>> errors
>>>>>>>>>>>>>> like the
>>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>>> leader in classic topics would do.
>>>>>>>>>>>>>>>>>>>>>>> Can you expand on what these checks are? Would
>>>> you be
>>>>>>>>>>>>>>>> checking
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> transaction was still ongoing for example?* *
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes, the producer epoch, that the transaction is
>>>>>>> ongoing,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> course
>>>>>>>>>>>>>>>>>>>>>> the normal idempotence checks. What the
>>> partition
>>>>>> leader
>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>>>>> classic
>>>>>>>>>>>>>>>>>>>>>> topics does before appending a batch to the
>>> local
>>>> log
>>>>>>>>>>>>>> (e.g. in
>>>>>>>>>>>>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
>>>>>>>>>>>>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
>>>>>>> Diskless,
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> unfortunately
>>>>>>>>>>>>>>>>>>>>>> cannot do these checks before appending the data
>>>> to
>>>>>> the
>>>>>>>> WAL
>>>>>>>>>>>>>>>>> segment
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> uploading it, but we can “tombstone” these
>>>> batches in
>>>>>>> the
>>>>>>>>>>>>>> batch
>>>>>>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>>>>>>> during the final commit.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Is there state about ongoing
>>>>>>>>>>>>>>>>>>>>>>> transactions in the batch coordinator? I see
>>> some
>>>>>> other
>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> the End transaction section, but it's not super
>>>> clear
>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>> state is
>>>>>>>>>>>>>>>>>>>>>> stored
>>>>>>>>>>>>>>>>>>>>>>> and when it is stored.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Right, this should have been more explicit. As
>>> the
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> leader
>>>>>>>>>>>>>>>>>>> tracks
>>>>>>>>>>>>>>>>>>>>>> ongoing transactions for classic topics, the
>>> batch
>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>> to as
>>>>>>>>>>>>>>>>>>>>>> well. So when a transaction starts and ends, the
>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>>>>>>> must inform the batch coordinator about this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
>>>> LSO --
>>>>>>>>>>>>>> perhaps
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> stored in the batch coordinator?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes. This could be deduced from the committed
>>>> batches
>>>>>>> and
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>> information, but for the sake of performance
>>> we’d
>>>>>> better
>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> explicitly.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 4. Are there any thoughts about how long
>>>>>>> transactional
>>>>>>>>>>>>>>>>> state is
>>>>>>>>>>>>>>>>>>>>>>> maintained in the batch coordinator and how it
>>>> will
>>>>>> be
>>>>>>>>>>>>>>>> cleaned
>>>>>>>>>>>>>>>>> up?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> As we understand this, the partition leader in
>>>> classic
>>>>>>>>>>>>>> topics
>>>>>>>>>>>>>>>>> forgets
>>>>>>>>>>>>>>>>>>>>>> about a transaction once it’s replicated (HWM
>>>>>> overpasses
>>>>>>>>>>>>>> it).
>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>> transaction coordinator acts like the main
>>>> guardian,
>>>>>>>>>>>>>> allowing
>>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>> leaders to do this safely. Please correct me if
>>>> this
>>>>>> is
>>>>>>>>>>>>>> wrong.
>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>> about relying on this with the batch coordinator
>>>> and
>>>>>>>>>>>>>> delete the
>>>>>>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>>>>>>> about a transaction once it’s finished (as
>>>> there’s no
>>>>>>>>>>>>>>>> replication
>>>>>>>>>>>>>>>>>>> and HWM
>>>>>>>>>>>>>>>>>>>>>> advances immediately).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Ivan
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> Hey folks,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Excited to see some updates related to
>>>> transactions!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I had a few questions.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
>>>>>> identified
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> producer ID
>>>>>>>>>>>>>>>>>>>>>>> and epoch, the positive result of this check
>>>> could be
>>>>>>>>>>>>>> cached
>>>>>>>>>>>>>>>>>>> locally
>>>>>>>>>>>>>>>>>>>>>>> Are we saying that only new transaction
>>> version 2
>>>>>>>>>>>>>>>> transactions
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>> here? If not, we can't uniquely identify
>>>> transactions
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> producer id +
>>>>>>>>>>>>>>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 2. >The batch coordinator does the final
>>>>>>> transactional
>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>> batches. This procedure would output the same
>>>> errors
>>>>>>>>>>>>>> like the
>>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>>> leader in classic topics would do.
>>>>>>>>>>>>>>>>>>>>>>> Can you expand on what these checks are? Would
>>>> you be
>>>>>>>>>>>>>>>> checking
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> transaction was still ongoing for example? Is
>>>> there
>>>>>>> state
>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> ongoing
>>>>>>>>>>>>>>>>>>>>>>> transactions in the batch coordinator? I see
>>> some
>>>>>> other
>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> the End transaction section, but it's not super
>>>> clear
>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>> state is
>>>>>>>>>>>>>>>>>>>>>> stored
>>>>>>>>>>>>>>>>>>>>>>> and when it is stored.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
>>>> LSO --
>>>>>>>>>>>>>> perhaps
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> stored in the batch coordinator?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> JO 4. Are there any thoughts about how long
>>>>>>> transactional
>>>>>>>>>>>>>>>>> state is
>>>>>>>>>>>>>>>>>>>>>>> maintained in the batch coordinator and how it
>>>> will
>>>>>> be
>>>>>>>>>>>>>>>> cleaned
>>>>>>>>>>>>>>>>> up?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
>>>>>>>>>>>>>>>>> <ju...@confluent.io.invalid>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Greg and Ivan,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the update. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
>>>> local
>>>>>>>>>>>>>>>> segments,
>>>>>>>>>>>>>>>>>>> making
>>>>>>>>>>>>>>>>>>>>>> use of
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
>>>>>> zero-copy
>>>>>>>>>>>>>>>>>>> functionality
>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>> built into classic topics."
>>>>>>>>>>>>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
>>>> state for
>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>> partition in
>>>>>>>>>>>>>>>>>>>>>>>> diskless topics?
>>>>>>>>>>>>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
>>>> fetches
>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>> aborted
>>>>>>>>>>>>>>>>>>>>>>>> records. How is that achieved?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
>>>>>>>>>>>>>> transaction is
>>>>>>>>>>>>>>>>>>> finished
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> also inserts the control batches in the
>>>>>> corresponding
>>>>>>>>>>>>>> logs
>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>> involved
>>>>>>>>>>>>>>>>>>>>>>>> Diskless topics. This happens only on the
>>>> metadata
>>>>>>>>>>>>>> level,
>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>> actual
>>>>>>>>>>>>>>>>>>>>>> control
>>>>>>>>>>>>>>>>>>>>>>>> batches are written to any file. "
>>>>>>>>>>>>>>>>>>>>>>>> A fetch response could include multiple
>>>>>> transactional
>>>>>>>>>>>>>>>>> batches.
>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>> does the
>>>>>>>>>>>>>>>>>>>>>>>> broker obtain the information about the ending
>>>>>> control
>>>>>>>>>>>>>>>> batch
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>> batch? Does that mean that a fetch response
>>>> needs to
>>>>>>> be
>>>>>>>>>>>>>>>>> built by
>>>>>>>>>>>>>>>>>>>>>>>> stitching record batches and generated control
>>>>>> batches
>>>>>>>>>>>>>>>>> together?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> JR 12. Queues: Is there still a share
>>> partition
>>>>>> leader
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>> consumers
>>>>>>>>>>>>>>>>>>>>>>>> are routed to?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
>>>> this
>>>>>> or
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>>>>>>>> implementation-focused?" It would be useful to
>>>>>> include
>>>>>>>>>>>>>>>> enough
>>>>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> understand correctness and performance impact.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
>>>> given
>>>>>>>>>>>>>>>> producer
>>>>>>>>>>>>>>>>>>> contain a
>>>>>>>>>>>>>>>>>>>>>>>> sequence number, which is ordered. If a
>>> producer
>>>>>> sends
>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>> Produce
>>>>>>>>>>>>>>>>>>>>>>>> request to an arbitrary broker, those requests
>>>> could
>>>>>>>>>>>>>> reach
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> batch
>>>>>>>>>>>>>>>>>>>>>>>> coordinator in different order and lead to
>>>> rejection
>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>> produce
>>>>>>>>>>>>>>>>>>>>>>>> requests.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
>>> Yurchenko <
>>>>>>>>>>>>>>>>> ivan@ivanyu.me>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> We have also thought in a bit more details
>>>> about
>>>>>>>>>>>>>>>>> transactions
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> queues,
>>>>>>>>>>>>>>>>>>>>>>>>> here's the plan.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> *Transactions*
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The support for transactions in *classic
>>>> topics* is
>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> precise
>>>>>>>>>>>>>>>>>>>>>>>>> interactions between three actors: clients
>>>> (mostly
>>>>>>>>>>>>>>>>> producers,
>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
>>>>>>>>>>>>>> classes),
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>>>>> coordinators. Brokers also run partition
>>>> leaders
>>>>>> with
>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>> local
>>>>>>>>>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>>>>>>>> (ProducerStateManager and others).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The high level (some details skipped)
>>> workflow
>>>> is
>>>>>> the
>>>>>>>>>>>>>>>>>>> following.
>>>>>>>>>>>>>>>>>>>>>> When a
>>>>>>>>>>>>>>>>>>>>>>>>> transactional Produce request is received by
>>>> the
>>>>>>>>>>>>>> broker:
>>>>>>>>>>>>>>>>>>>>>>>>> 1. For each partition, the partition leader
>>>> checks
>>>>>>>>>>>>>> if a
>>>>>>>>>>>>>>>>>>> non-empty
>>>>>>>>>>>>>>>>>>>>>>>>> transaction is running for this partition.
>>>> This is
>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>> its
>>>>>>>>>>>>>>>>>>>>>> local
>>>>>>>>>>>>>>>>>>>>>>>>> state derived from the log metadata
>>>>>>>>>>>>>>>> (ProducerStateManager,
>>>>>>>>>>>>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
>>>>>>>>>>>>>>>>>>>>>>>>> 2. The transaction coordinator is informed
>>>> about
>>>>>> all
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> aren’t part of the transaction to include
>>> them.
>>>>>>>>>>>>>>>>>>>>>>>>> 3. The partition leaders do additional
>>>>>> transactional
>>>>>>>>>>>>>>>>> checks.
>>>>>>>>>>>>>>>>>>>>>>>>> 4. The partition leaders append the
>>>> transactional
>>>>>>>>>>>>>> data to
>>>>>>>>>>>>>>>>>>> their logs
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> update some of their state (for example, log
>>>> the
>>>>>> fact
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>>>>> is running for the partition and its first
>>>> offset).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> When the transaction is committed or aborted:
>>>>>>>>>>>>>>>>>>>>>>>>> 1. The producer contacts the transaction
>>>>>> coordinator
>>>>>>>>>>>>>>>>> directly
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> EndTxnRequest.
>>>>>>>>>>>>>>>>>>>>>>>>> 2. The transaction coordinator writes
>>>>>> PREPARE_COMMIT
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> PREPARE_ABORT to
>>>>>>>>>>>>>>>>>>>>>>>>> its log and responds to the producer.
>>>>>>>>>>>>>>>>>>>>>>>>> 3. The transaction coordinator sends
>>>>>>>>>>>>>>>>> WriteTxnMarkersRequest to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> leaders
>>>>>>>>>>>>>>>>>>>>>>>>> of the involved partitions.
>>>>>>>>>>>>>>>>>>>>>>>>> 4. The partition leaders write the
>>> transaction
>>>>>>>>>>>>>> markers to
>>>>>>>>>>>>>>>>>>> their logs
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> respond to the coordinator.
>>>>>>>>>>>>>>>>>>>>>>>>> 5. The coordinator writes the final
>>> transaction
>>>>>> state
>>>>>>>>>>>>>>>>>>>>>> COMPLETE_COMMIT or
>>>>>>>>>>>>>>>>>>>>>>>>> COMPLETE_ABORT.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> In classic topics, partitions have leaders
>>> and
>>>> lots
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>>>>>>>> necessary for supporting this workflow is
>>>> local.
>>>>>> The
>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>> challenge
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
>>>> there
>>>>>>>>>>>>>> are no
>>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
>>>> need
>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>> globalized
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> batch coordinator. We are already doing this
>>> to
>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>> idempotent
>>>>>>>>>>>>>>>>>>>>>>>> produce.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The high level workflow for *diskless topics*
>>>> would
>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>> similar:
>>>>>>>>>>>>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
>>>>>>>>>>>>>> non-empty
>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>> running for this partition. In contrast to
>>>> classic
>>>>>>>>>>>>>>>> topics,
>>>>>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>>>>>>>>>>> checked
>>>>>>>>>>>>>>>>>>>>>>>>> against the batch coordinator with a single
>>>> RPC.
>>>>>>>>>>>>>> Since a
>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>> be uniquely identified with producer ID and
>>>> epoch,
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> positive
>>>>>>>>>>>>>>>>>>>>>> result of
>>>>>>>>>>>>>>>>>>>>>>>>> this check could be cached locally (for the
>>>> double
>>>>>>>>>>>>>>>>> configured
>>>>>>>>>>>>>>>>>>>>>> duration
>>>>>>>>>>>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>>>>>>>> transaction, for example).
>>>>>>>>>>>>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
>>>>>> informed
>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> partitions that aren’t part of the
>>> transaction
>>>> to
>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>>>>>>>>>> 3. No transactional checks are done on the
>>>> broker
>>>>>>>>>>>>>> side.
>>>>>>>>>>>>>>>>>>>>>>>>> 4. The broker appends the transactional data
>>>> to the
>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>> shared
>>>>>>>>>>>>>>>>>>>>>> WAL
>>>>>>>>>>>>>>>>>>>>>>>>> segment. It doesn’t update any
>>>> transaction-related
>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> Diskless
>>>>>>>>>>>>>>>>>>>>>>>>> topics, because it doesn’t have any.
>>>>>>>>>>>>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>>>>>>>>>>> normal produce flow.
>>>>>>>>>>>>>>>>>>>>>>>>> 6. The batch coordinator does the final
>>>>>> transactional
>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>> batches. This procedure would output the same
>>>>>> errors
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
>>>>>> batches
>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> rejected.
>>>>>>>>>>>>>>>>>>>>>>>>> This means, there will potentially be garbage
>>>> in
>>>>>> the
>>>>>>>>>>>>>> WAL
>>>>>>>>>>>>>>>>>>> segment
>>>>>>>>>>>>>>>>>>>>>> file in
>>>>>>>>>>>>>>>>>>>>>>>>> case of transactional errors. This is
>>>> preferable to
>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>> network
>>>>>>>>>>>>>>>>>>>>>>>>> round trips, especially considering the WAL
>>>>>> segments
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> relatively
>>>>>>>>>>>>>>>>>>>>>>>>> short-living (see the Greg's update above).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> When the transaction is committed or aborted:
>>>>>>>>>>>>>>>>>>>>>>>>> 1. The producer contacts the transaction
>>>>>> coordinator
>>>>>>>>>>>>>>>>> directly
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> EndTxnRequest.
>>>>>>>>>>>>>>>>>>>>>>>>> 2. The transaction coordinator writes
>>>>>> PREPARE_COMMIT
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> PREPARE_ABORT to
>>>>>>>>>>>>>>>>>>>>>>>>> its log and responds to the producer.
>>>>>>>>>>>>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
>>> informs
>>>> the
>>>>>>>>>>>>>> batch
>>>>>>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> the transaction is finished.
>>>>>>>>>>>>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
>>> the
>>>>>>>>>>>>>>>>> transaction is
>>>>>>>>>>>>>>>>>>>>>> finished
>>>>>>>>>>>>>>>>>>>>>>>>> and also inserts the control batches in the
>>>>>>>>>>>>>> corresponding
>>>>>>>>>>>>>>>>> logs
>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>> involved Diskless topics. This happens only
>>> on
>>>> the
>>>>>>>>>>>>>>>> metadata
>>>>>>>>>>>>>>>>>>> level, no
>>>>>>>>>>>>>>>>>>>>>>>>> actual control batches are written to any
>>> file.
>>>>>> They
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> dynamically
>>>>>>>>>>>>>>>>>>>>>>>>> created on Fetch and other read operations.
>>> We
>>>>>> could
>>>>>>>>>>>>>>>>>>> technically
>>>>>>>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>>>>>>>>>>> these control batches for real, but this
>>> would
>>>> mean
>>>>>>>>>>>>>> extra
>>>>>>>>>>>>>>>>>>> produce
>>>>>>>>>>>>>>>>>>>>>>>> latency,
>>>>>>>>>>>>>>>>>>>>>>>>> so it's better just to mark them in the batch
>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>> milliseconds.
>>>>>>>>>>>>>>>>>>>>>>>>> 5. The transaction coordinator sends
>>>>>>>>>>>>>>>>> WriteTxnMarkersRequest to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> leaders
>>>>>>>>>>>>>>>>>>>>>>>>> of the involved partitions. – Now only to
>>>> classic
>>>>>>>>>>>>>> topics
>>>>>>>>>>>>>>>>> now.
>>>>>>>>>>>>>>>>>>>>>>>>> 6. The partition leaders of classic topics
>>>> write
>>>>>> the
>>>>>>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>>>>>>>>> markers
>>>>>>>>>>>>>>>>>>>>>>>>> to their logs and respond to the coordinator.
>>>>>>>>>>>>>>>>>>>>>>>>> 7. The coordinator writes the final
>>> transaction
>>>>>> state
>>>>>>>>>>>>>>>>>>>>>> COMPLETE_COMMIT or
>>>>>>>>>>>>>>>>>>>>>>>>> COMPLETE_ABORT.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Compared to the non-transactional produce
>>>> flow, we
>>>>>>>>>>>>>> get:
>>>>>>>>>>>>>>>>>>>>>>>>> 1. An extra network round trip between
>>> brokers
>>>> and
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> batch
>>>>>>>>>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>>>>>>>>>> when a new partition appear in the
>>>> transaction. To
>>>>>>>>>>>>>>>>> mitigate the
>>>>>>>>>>>>>>>>>>>>>> impact of
>>>>>>>>>>>>>>>>>>>>>>>>> them:
>>>>>>>>>>>>>>>>>>>>>>>>> - The results will be cached.
>>>>>>>>>>>>>>>>>>>>>>>>> - The calls for multiple partitions in one
>>>> Produce
>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>> grouped.
>>>>>>>>>>>>>>>>>>>>>>>>> - The batch coordinator should be optimized
>>> for
>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>>>>>>> response to
>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>> RPCs.
>>>>>>>>>>>>>>>>>>>>>>>>> - The fact that a single producer normally
>>> will
>>>>>>>>>>>>>>>>> communicate
>>>>>>>>>>>>>>>>>>> with a
>>>>>>>>>>>>>>>>>>>>>>>>> single broker for the duration of the
>>>> transaction
>>>>>>>>>>>>>> further
>>>>>>>>>>>>>>>>>>> reduces the
>>>>>>>>>>>>>>>>>>>>>>>>> expected number of round trips.
>>>>>>>>>>>>>>>>>>>>>>>>> 2. An extra round trip between the
>>> transaction
>>>>>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> batch
>>>>>>>>>>>>>>>>>>>>>>>>> coordinator when a transaction is finished.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> With this proposal, transactions will also be
>>>> able
>>>>>> to
>>>>>>>>>>>>>>>> span
>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>> classic
>>>>>>>>>>>>>>>>>>>>>>>>> and Diskless topics.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> *Queues*
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The share group coordination and management
>>> is
>>>> a
>>>>>>>>>>>>>> side job
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>>> interfere with the topic itself (leadership,
>>>>>>>>>>>>>> replicas,
>>>>>>>>>>>>>>>>> physical
>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> records, etc.) and non-queue producers and
>>>>>> consumers
>>>>>>>>>>>>>>>>> (Fetch and
>>>>>>>>>>>>>>>>>>>>>> Produce
>>>>>>>>>>>>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
>>>>>> affected.)
>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>> see any
>>>>>>>>>>>>>>>>>>>>>>>>> reason why we can't make Diskless topics
>>>> compatible
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> share
>>>>>>>>>>>>>>>>>>>>>> groups the
>>>>>>>>>>>>>>>>>>>>>>>>> same way as classic topics are. Even on the
>>>> code
>>>>>>>>>>>>>> level,
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>> expect
>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>> serious refactoring: the same reading
>>> routines
>>>> are
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>> used for
>>>>>>>>>>>>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Should the KIPs be modified to include this
>>> or
>>>> it's
>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>>>>>>>>> implementation-focused?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Ivan
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you all for your questions and design
>>>> input
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> KIP-1150.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
>>>> with a
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> design. To
>>>>>>>>>>>>>>>>>>>>>>>>> summarize
>>>>>>>>>>>>>>>>>>>>>>>>>> the changes:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
>>> the
>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>> KIP-405
>>>>>>>>>>>>>>>>>>>>>> Tiered
>>>>>>>>>>>>>>>>>>>>>>>>>> Storage interfaces, permitting data produced
>>>> to a
>>>>>>>>>>>>>>>>> Diskless
>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>>>>>>>>> moved to tiered storage.
>>>>>>>>>>>>>>>>>>>>>>>>>> This lowers the scalability requirements for
>>>> the
>>>>>>>>>>>>>> Batch
>>>>>>>>>>>>>>>>>>> Coordinator
>>>>>>>>>>>>>>>>>>>>>>>>>> component, and allows Diskless to compose
>>> with
>>>>>>>>>>>>>> Tiered
>>>>>>>>>>>>>>>>> Storage
>>>>>>>>>>>>>>>>>>>>>> plugin
>>>>>>>>>>>>>>>>>>>>>>>>>> features such as encryption and alternative
>>>> data
>>>>>>>>>>>>>>>> formats.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
>>> local
>>>>>>>>>>>>>> segments,
>>>>>>>>>>>>>>>>>>> making use
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
>>>>>>>>>>>>>> zero-copy
>>>>>>>>>>>>>>>>>>> functionality
>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>> built into classic topics.
>>>>>>>>>>>>>>>>>>>>>>>>>> However, local segments are now considered
>>>> cache
>>>>>>>>>>>>>>>>> elements,
>>>>>>>>>>>>>>>>>>> do not
>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> be durably stored, and can be built without
>>>>>>>>>>>>>> contacting
>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>> replicas.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 3. The design has been simplified
>>>> substantially,
>>>>>> by
>>>>>>>>>>>>>>>>> removing
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
>>>>>>>>>>>>>> component, and
>>>>>>>>>>>>>>>>>>> "object
>>>>>>>>>>>>>>>>>>>>>>>>>> compaction/merging" step.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> The design maintains leaderless produces as
>>>>>>>>>>>>>> enabled by
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> Batch
>>>>>>>>>>>>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
>>> as
>>>> the
>>>>>>>>>>>>>>>> earlier
>>>>>>>>>>>>>>>>>>> design,
>>>>>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>>>>>>>>>>> being simpler and integrating better into
>>> the
>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>> ecosystem.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
>>>> feedback on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>>>>>> Greg Harris
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
>>>>>>>>>>>>>>>>>>> <ju...@confluent.io.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
>>>> support
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>> client
>>>>>>>>>>>>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
>>>>>>>>>>>>>> supporting
>>>>>>>>>>>>>>>> APIs
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>> transactions
>>>>>>>>>>>>>>>>>>>>>>>>>>> and queues.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
>>>> Siekierski
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
>>>> current
>>>>>>>>>>>>>>>>> status of
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> KIP?
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> haven't seen much activity here for the
>>>> past 2
>>>>>>>>>>>>>>>>> months,
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> vote got
>>>>>>>>>>>>>>>>>>>>>>>>>>> vetoed
>>>>>>>>>>>>>>>>>>>>>>>>>>>> but I think the pending questions have
>>> been
>>>>>>>>>>>>>>>> answered
>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>> then.
>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1183
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
>>>>>>>>>>>>>> activity
>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>> May.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
>>>> real
>>>>>>>>>>>>>>>> choices
>>>>>>>>>>>>>>>>>>> that can
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
>>>> being
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> dominant
>>>>>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it comes to market adoption - but all
>>> these
>>>> are
>>>>>>>>>>>>>>>>>>> standalone
>>>>>>>>>>>>>>>>>>>>>>>> products.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
>>> would
>>>>>>>>>>>>>> hate to
>>>>>>>>>>>>>>>>> see a
>>>>>>>>>>>>>>>>>>>>>> stall. So
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> question is: can we get an update?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
>>> Colin
>>>>>>>>>>>>>>>> McCabe -
>>>>>>>>>>>>>>>>>>> have your
>>>>>>>>>>>>>>>>>>>>>>>>> questions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> been answered? If not, is there anything I
>>>> can
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> help? I'm
>>>>>>>>>>>>>>>>>>>>>>>> deeply
>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar with both architectures and have
>>>>>>>>>>>>>> written
>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> both?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
>>>>>>>>>>>>>>>> Kozlovski
>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>> (just a
>>>>>>>>>>>>>>>>>>>>>> bunch
>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> links
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to everything)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) create the email threads
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
>>>>>>>>>>>>>> example, I
>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>> searching
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while;
>>> As
>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>> tell, it
>>>>>>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exist yet.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
>>> of
>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> root
>>>>>>>>>>>>>>>>>>>>>> KIP be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
>>>> links
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> sub-KIPs
>>>>>>>>>>>>>>>>>>>>>> where
>>>>>>>>>>>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>>>>>>>>>>>> aimed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to move the discussion towards), I think
>>> it
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> good to
>>>>>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DISCUSS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> threads for them all.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
>>>> about
>>>>>>>>>>>>>>>>>>> introducing a
>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topics that would make use of Object
>>>>>>>>>>>>>> Storage as
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> primary
>>>>>>>>>>>>>>>>>>>>>>>>> source of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
>>>>>>>>>>>>>> decided
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> split it
>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> related KIPs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
>>>>>>>>>>>>>> should aim
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> how to
>>>>>>>>>>>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This follows the same approach used when
>>>> we
>>>>>>>>>>>>>>>>> discussed
>>>>>>>>>>>>>>>>>>>>>> KRaft.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
>>> really
>>>>>>>>>>>>>> hard
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> discuss
>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> meta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
>>>>>>>>>>>>>> (linked
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> KIP-1150)
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> offer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation of this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
>>>> DISCUSS
>>>>>>>>>>>>>>>>> threads
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>> Apache
>>>>>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this feature or not.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
>>>>>>>>>>>>>> authors
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Josep Prat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
>>> +491715557497 |
>>>>>>>>>>>>>>>>> aiven.io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
>>>>>>>>>>>>>>>>> Valtonen,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anatolii Popov
>>>>>> Senior Software Developer, *Aiven OY*
>>>>>> m: +358505126242
>>>>>> w: aiven.io  e: anatolii.popov@aiven.io
>>>>>> <https://www.facebook.com/aivencloud>
>>>>>> <https://www.linkedin.com/company/aiven/>   <
>>>> https://twitter.com/aiven_io>
>>>>>>
>>>>>
>>>>
>>>
>>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Robin,

Your proposal actually resonates with what we proposed [1] a little while ago (nicknamed revision 3), so we exercised similar ideas. We abandoned them in favor of the original coordinator-based approach for the reasons described in the mailing list and summarized in the rejected alternatives of KIP-1163 [2].

If you consider your proposal the alternative to KIP-1163 under the umbrella of KIP-1150, we propose to file a formal KIP so we can have a more visible community discussion of the merits of both approach. What do you think?

Best,
Ivan

[1] https://lists.apache.org/thread/w03qffwllynkx5ygvcd48vb3vjvsf8k7
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=350783976#KIP1163:DisklessCore-Coordinator-lessapproaches

On Wed, Jan 21, 2026, at 09:24, Robin Han wrote:
> Hi Greg,
> 
> Thanks for the reply
> 
> > Based on your description, the GET operations on the leader and followers
> > must complete before offsets are assigned, adding Produce request latency
> > that is not present in the KIP-1163 design.
> 
> Yes, the ProduceRequestLatency is 1 * PUT(In Broker N) + 1 GET(In Broker M)
> + 1 GET(Follower Synchronization).
> - Compared with KIP-1163: Produce Latency has two more GET time costs;
> Consume E2E Latency has one more GET time cost.
> - Compared with KIP-1163: There is no need for coordination by a
> centralized Coordinator, thus eliminating the delay caused by Coordinator
> confirmation and avoiding the impact of Coordinator single-point failure on
> the availability of the entire cluster.
> 
> In the design of Diskless Topic, I suspect that adding the time cost of two
> more GET operations will not bring about an essential change (with P50
> increasing from 500ms to 700ms).
> 
> > KIP-1163 also assumes that object storage is more reliable than disk
> > storage, which is the case in the hyperscaler clouds. The parts of your
> > design that make it compatible with low-reliability object storage become
> > counterproductive when Diskless is used with a high-reliability object
> > storage, such as re-introducing direct replication and cross-az costs.
> 
> It might be that my description is not accurate enough. The design is also
> strongly dependent on object storage. Regarding the part about object
> management and recycling, what I want to emphasize more is that the current
> design does not require complex object metadata management, and objects are
> just a temporary state.
> 
> > We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> > in favor of a complete solution.
> 
> I understand that the main reasons why "KIP-1176 was rejected" are that it
> did not solve the cross-AZ traffic costs for Produce, and that implementing
> Active Segment Tiered Storage added a lot of complexity to metadata
> management.
> The design I mentioned can not only solve all cross-AZ traffic costs
> including sending and replication (the core goal of KIP-1150), but also
> does not add complex management of Object metadata, nor does it require
> large-scale modifications to Apache Kafka.
> It could be a good design that can be adopted by KIP-1150.
> 
> Greg Harris via dev <de...@kafka.apache.org> 于2026年1月21日周三 03:20写道：
> 
> > Hi Robin,
> >
> > Thanks for your design sketch.
> >
> > Based on your description, the GET operations on the leader and followers
> > must complete before offsets are assigned, adding Produce request latency
> > that is not present in the KIP-1163 design.
> >
> > KIP-1163 also assumes that object storage is more reliable than disk
> > storage, which is the case in the hyperscaler clouds. The parts of your
> > design that make it compatible with low-reliability object storage become
> > counterproductive when Diskless is used with a high-reliability object
> > storage, such as re-introducing direct replication and cross-az costs.
> >
> > We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> > in favor of a complete solution. The folks who championed that proposal are
> > now working with us on KIP-1163, and are currently designing Compaction in
> > Tiered Storage to make Diskless a complete solution for compacted topics as
> > well. Similarly, we would appreciate your contributions to the design of
> > KIP-1163, and your vote on the motivational KIP-1150.
> >
> > Thanks!
> >
> > On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi Jun,
> > >
> > > Here are our answers to your questions and comments:
> > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > >
> > > Thank you, good point, added this.
> > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > >
> > > Yes, there is a per-request cost for object storage which is not directly
> > > comparable to inter-zone transfer costs. We have found that the design in
> > > KIP-1163 amortizes those request costs such that they are insignificant
> > > compared to the $0.02/gb cost which is called out in KIP-1150.
> > > We currently have very precise cost models that were put together for an
> > > earlier iteration of the Diskless topics. These are somewhat stale and
> > > inaccurate for the latest design, so we will work in the background to
> > post
> > > concrete numbers in KIP-1163 before voting on that proposal. Would this
> > be
> > > OK?
> > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > >
> > > Thanks for calling out. We have updated KIP-1164 with the latest design.
> > > It's missing public interfaces, but we think it may be a bit too early to
> > > define them as some rounds of discussion is ahead, with potential to
> > change
> > > the design. But otherwise the KIP is good for the first review.
> > > We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
> > >
> > > Best,
> > > Ivan
> > >
> > > On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> > > > Hi, Anatolii,
> > > >
> > > > Thanks for the updated KIP. Looks good to me overall. A few more
> > > comments.
> > > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > > >
> > > > Jun
> > > >
> > > > On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> > > dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thank you for the extensive feedback.
> > > > >
> > > > > We have substantially updated the KIP to address the points raised.
> > > Given
> > > > > the scope of these changes, we have started a new VOTE thread to
> > > restart
> > > > > the voting process cleanly.
> > > > >
> > > > > You can find the new thread here:
> > > > > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> > > > >
> > > > > Best,
> > > > > Anatolii
> > > > >
> > > > > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > > > > dev@kafka.apache.org> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We (the team at Slack) have been following the recent discussion
> > > > > regarding
> > > > > > the scope and timeline of KIP-1150. We agree with the community
> > > sentiment
> > > > > > that Diskless Topics represents the right long-term architecture
> > for
> > > > > Kafka
> > > > > > in the cloud, but we also recognize the valid concerns raised
> > > regarding
> > > > > the
> > > > > > engineering resources required to deliver such an ambitious change.
> > > > > >
> > > > > > To help address these concerns and accelerate the timeline, we are
> > > happy
> > > > > to
> > > > > > announce that we are partnering with the KIP-1150 authors to
> > > co-develop
> > > > > > this feature.
> > > > > >
> > > > > > We previously proposed KIP-1176: Tiered Storage for Active Log
> > > Segments
> > > > > to
> > > > > > solve similar problems. However, rather than fragmenting the
> > > community's
> > > > > > efforts across competing designs, we have decided to withdraw
> > > KIP-1176
> > > > > and
> > > > > > consolidate our engineering resources behind KIP-1150.
> > > > > >
> > > > > > To start, we plan to take ownership of Compaction for Tiered
> > Storage.
> > > > > This
> > > > > > has long been a missing feature in KIP-405, and it becomes critical
> > > in a
> > > > > > Diskless architecture where long-lived data is obligated to be
> > > tiered. By
> > > > > > driving this related prerequisite feature, we hope to allow for
> > > faster
> > > > > > delivery of KIP-1150.
> > > > > >
> > > > > > We are excited to collaborate on this to ensure a robust and timely
> > > > > > delivery for the community.
> > > > > >
> > > > > > Best,
> > > > > > Tom & Henry
> > > > > >
> > > > > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Thanks for sharing the meeting notes.
> > > > > > > I agree we should keep polishing the contents of 1150 & high
> > level
> > > > > design
> > > > > > > in 1163 to prepare for a vote.
> > > > > > >
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > There was a video call between myself, Ivan Yurchenko, Jun Rao,
> > > and
> > > > > > > Andrew
> > > > > > > > Schofield pertaining to KIP-1150. Here are the notes from that
> > > > > meeting:
> > > > > > > >
> > > > > > > > Ivan: What is the future state of Kafka in this area, in 5
> > years?
> > > > > > > > Jun: Do we want something more cloud native? Yes, started with
> > > Tiered
> > > > > > > > Storage. If there’s a better way, we should explore it. In the
> > > long
> > > > > > term
> > > > > > > > this will be useful
> > > > > > > > Because Kafka is used so widely, we need to make sure
> > everything
> > > we
> > > > > add
> > > > > > > is
> > > > > > > > for the long term and for everyone, not just for a single
> > > company.
> > > > > > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > > > > > something
> > > > > > > > that’s high quality/lasts/maintainable, and can work with all
> > > > > existing
> > > > > > > > capabilities.
> > > > > > > > If both 1150 and 1176 proceed at the same time, it’s confusing.
> > > They
> > > > > > > > overlap, but Diskless is more ambitious.
> > > > > > > > If both KIPs are being seriously worked on, then we don’t
> > really
> > > need
> > > > > > > both,
> > > > > > > > because Diskless clearly is better. Having multiple will
> > confuse
> > > > > > people.
> > > > > > > It
> > > > > > > > will duplicate some of the effort.
> > > > > > > > If we want diskless ultimately, what is the short term
> > strategy,
> > > to
> > > > > get
> > > > > > > > some early wins first?
> > > > > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > > > > Andrew: Eventually the architecture will change substantially,
> > > it may
> > > > > > not
> > > > > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > > > > Greg: We all agree on having a high quality feature merged
> > > upstream,
> > > > > > and
> > > > > > > > supporting all APIs
> > > > > > > > Jun: We should try and keep things simple, but there is some
> > > minimum
> > > > > > > > complexity needed.
> > > > > > > > When doing the short term changes (1176), it doesn’t really
> > > progress
> > > > > in
> > > > > > > > changing to a more modern architecture.
> > > > > > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > > > > > Jun: The danger of only applying changes to some part of the
> > > API, you
> > > > > > set
> > > > > > > > the precedence that you only have to implement part of the API.
> > > > > > > Supporting
> > > > > > > > the full API set should be a minimum requirement.
> > > > > > > > Andrew: When we started Kraft, how much did we know the design?
> > > > > > > > Jun: For Kraft we didn’t really know much about the migration,
> > > but
> > > > > the
> > > > > > > > high-level was clear.
> > > > > > > > Greg: Is 1150 votable in its current state?
> > > > > > > > Jun: 1150 should promise to support all APIs. It doesn’t have
> > to
> > > have
> > > > > > all
> > > > > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > > > > We do need some high-level design enough to give confidence
> > that
> > > the
> > > > > > > > promise is able to be fulfilled.
> > > > > > > > Greg: Is the draft version in 1163 enough detail or is more
> > > needed?
> > > > > > > > Jun: We need to agree on the core design, such as leaderless
> > > etc. And
> > > > > > how
> > > > > > > > the APIs will be supported.
> > > > > > > > Greg: Okay we can include these things, and provide a sketch of
> > > how
> > > > > the
> > > > > > > > other leader-based features operate.
> > > > > > > > Jun: Yeah if at a high level the sketch appears to work, we can
> > > > > approve
> > > > > > > > that functionality.
> > > > > > > > Are you committed to doing the more involved and big project?
> > > > > > > > Greg: Yes, we’re committed to the 1163 design and can’t really
> > > accept
> > > > > > > 1176.
> > > > > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > > > > Greg: We’ll push internally for resources, and use the
> > community
> > > > > > > sentiment
> > > > > > > > to motivate Aiven.
> > > > > > > > How far into the future should we look? What sort of scale?
> > > > > > > > Jun: As long as there’s a path forward, and we’re not closing
> > off
> > > > > > future
> > > > > > > > improvements, we can figure out how to handle a larger scale
> > > when it
> > > > > > > > arises.
> > > > > > > > Greg: Random replica placement is very harmful, can we
> > recommend
> > > > > users
> > > > > > to
> > > > > > > > use an external tool like CruiseControl?
> > > > > > > > Jun: Not everyone uses CruiseControl, we would probably need
> > some
> > > > > > > solution
> > > > > > > > for this out of the box
> > > > > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > > > > Jun: Out-of-box experience should be good, good to allow other
> > > > > > > > implementations
> > > > > > > > Greg: But it could hurt Kafka feature/upgrade velocity when we
> > > wait
> > > > > for
> > > > > > > > plugin providers to implement it
> > > > > > > > Ivan: We imagined that maybe cloud hyperscalers could implement
> > > it
> > > > > with
> > > > > > > > e.g. dynamodb
> > > > > > > > Greg: Could we bake more details of the different providers
> > into
> > > > > Kafka,
> > > > > > > or
> > > > > > > > does it still make sense for it to be pluggable?
> > > > > > > > Jun: Make it whatever is easiest to roll out and add new
> > clients
> > > > > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > > > > Ivan: The vote is already open, we’re not too pressed for time.
> > > We’ll
> > > > > > go
> > > > > > > > improve the 1163 design and communication.
> > > > > > > > Is 1176 a competing design? Someone will ask.
> > > > > > > > Jun: If we are seriously working on something more ambitious,
> > > yeah we
> > > > > > > > shouldn’t do the stop-gap solution.
> > > > > > > > It’s diverting review resources. If we can get the short term
> > > thing
> > > > > in
> > > > > > > 1yr
> > > > > > > > but Diskless solution is 2y it makes sense to go for Diskless.
> > If
> > > > > it’s
> > > > > > > 5yr,
> > > > > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case.
> > Should
> > > we
> > > > > > > > explicitly exclude 1176?
> > > > > > > > Andrew: Put your arms around the feature set you actually want,
> > > and
> > > > > use
> > > > > > > > that to rule out 1176.
> > > > > > > > Probably don’t need -1 votes, most likely KIPs just don’t
> > receive
> > > > > > votes.
> > > > > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > > > > Jun: Satish posted meeting notes regularly, we should do the
> > > same.
> > > > > > > >
> > > > > > > > To summarize, we will be polishing the contents of 1150 & high
> > > level
> > > > > > > design
> > > > > > > > in 1163 to prepare for a vote.
> > > > > > > > We believe that the community should select the feature set of
> > > 1150
> > > > > to
> > > > > > > > fully eliminate producer cross-zone costs, and make the
> > > investment
> > > > > in a
> > > > > > > > high quality Diskless Topics implementation rather than in
> > > stop-gap
> > > > > > > > solutions.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
> > wrote:
> > > > > > > >
> > > > > > > > > This may be a tangent, but we needed to offload storage off
> > of
> > > > > Kafka
> > > > > > > into
> > > > > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> > > mostly
> > > > > > > > > ephemeral broker that can come and go as it pleases. Be that
> > > > > scaling
> > > > > > or
> > > > > > > > > outage. Disks can be destroyed and recreated at will, we
> > still
> > > > > retain
> > > > > > > > data
> > > > > > > > > and use broker for just that, brokering messages. Not only
> > > that, we
> > > > > > > > reduced
> > > > > > > > > the requirement on the actual Kafka resources by reducing the
> > > size
> > > > > > of a
> > > > > > > > > payload via a claim check pattern. Maybe this is an
> > > anti–pattern,
> > > > > but
> > > > > > > it
> > > > > > > > is
> > > > > > > > > super fast and highly cost efficient. We reworked
> > > ProducerRequest
> > > > > to
> > > > > > > > allow
> > > > > > > > > plugins. We added a custom http plugin that submits every
> > > request
> > > > > > via a
> > > > > > > > > persisted connection to a microservice. Microservice stores
> > the
> > > > > > payload
> > > > > > > > and
> > > > > > > > > returns a tiny json metadata object,a claim check, that can
> > be
> > > used
> > > > > > to
> > > > > > > > find
> > > > > > > > > the actual data. Think of it as zipping the payload. This
> > claim
> > > > > check
> > > > > > > > > metadata traverses the pipelines with consumers using the
> > urls
> > > in
> > > > > > > > metadata
> > > > > > > > > to pull what they need. Think unzipping. This allowed us to
> > > also
> > > > > pull
> > > > > > > > ONLY
> > > > > > > > > the data that we need in graphql like manner. So if you have
> > a
> > > 100K
> > > > > > > json
> > > > > > > > > payload and you need only a subsection, you can pull that by
> > > > > > jmespath.
> > > > > > > > When
> > > > > > > > > you have multiple consumer groups yanking down huge payloads
> > > it is
> > > > > > > > > cumbersome on the broker. When you have the same consumer
> > > groups
> > > > > > > yanking
> > > > > > > > > down a claim check, and then going out of band directly to
> > the
> > > > > source
> > > > > > > of
> > > > > > > > > truth, the broker has some breathing room. Obviously our
> > > > > microservice
> > > > > > > > does
> > > > > > > > > not go directly to the cloud storage, as that would be too
> > > slow. It
> > > > > > > > stores
> > > > > > > > > the payload in high speed memory cache and returns asap. That
> > > > > memory
> > > > > > is
> > > > > > > > > eventually persisted into S3. The retrieval goest against the
> > > cache
> > > > > > > > first,
> > > > > > > > > then against the S3. Overall a rather cheappy and zippy
> > > solution. I
> > > > > > > tried
> > > > > > > > > proposing the KIP for this, but there was no excitement.
> > Check
> > > this
> > > > > > > out:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi, Andrew,
> > > > > > > > > >
> > > > > > > > > > If we want to focus only on reducing cross-zone replication
> > > > > costs,
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > an alternative design in the KIP-1176 discussion thread
> > that
> > > > > seems
> > > > > > > > > simpler
> > > > > > > > > > than the proposal here. I am copying the outline of that
> > > approach
> > > > > > > > below.
> > > > > > > > > >
> > > > > > > > > > 1. A new leader is elected.
> > > > > > > > > > 2. Leader maintains a first tiered offset, which is
> > > initialized
> > > > > to
> > > > > > > log
> > > > > > > > > end
> > > > > > > > > > offset.
> > > > > > > > > > 3. Leader writes produced data from the client to local
> > log.
> > > > > > > > > > 4. Leader uploads produced data from all local logs as a
> > > combined
> > > > > > > > object
> > > > > > > > > > 5. Leader stores the metadata for the combined object in
> > > memory.
> > > > > > > > > > 6. If a follower fetch request has an offset >= first
> > tiered
> > > > > > offset,
> > > > > > > > the
> > > > > > > > > > metadata for the corresponding combined object is returned.
> > > > > > > Otherwise,
> > > > > > > > > the
> > > > > > > > > > local data is returned.
> > > > > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > > > > >
> > > > > > > > > > It's still a bit unnatural, but it could work.
> > > > > > > > > >
> > > > > > > > > > Hi, Ivan,
> > > > > > > > > >
> > > > > > > > > > Are you still committed to proceeding with the original
> > > design of
> > > > > > > > > KIP-1150?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >> I’ve been following KIP-1150 and friends for a while. I’m
> > > going
> > > > > to
> > > > > > > > jump
> > > > > > > > > >> into the discussions too.
> > > > > > > > > >>
> > > > > > > > > >> Looking back at Jack Vanlightly’s message, I am not quite
> > so
> > > > > > > convinced
> > > > > > > > > >> that it’s a kind of fork in the road. The primary aim of
> > the
> > > > > > effort
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > > > > > prohibitively
> > > > > > > > > >> expensive to use on cloud storage. I think it would be
> > > entirely
> > > > > > > viable
> > > > > > > > > to
> > > > > > > > > >> prioritise code reuse for an initial implementation of
> > > diskless
> > > > > > > > topics,
> > > > > > > > > and
> > > > > > > > > >> we could still have a more cloud-native design in the
> > > future.
> > > > > It’s
> > > > > > > > hard
> > > > > > > > > to
> > > > > > > > > >> predict what the community will prioritise in the future.
> > > > > > > > > >>
> > > > > > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> > > > > support
> > > > > > > > > >> leaderless produce requests, first writing WAL segments
> > into
> > > > > > object
> > > > > > > > > >> storage, and then using the regular partition leaders to
> > > > > sequence
> > > > > > > the
> > > > > > > > > >> records. The active log segment for a diskless topic will
> > > > > > initially
> > > > > > > > > contain
> > > > > > > > > >> batch coordinates rather than record batches. The batch
> > > > > > coordinates
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > > >> resolved from WAL segments for consumers, and also in
> > order
> > > to
> > > > > > > prepare
> > > > > > > > > log
> > > > > > > > > >> segments for uploading to tiered storage. Jun is probably
> > > > > correct
> > > > > > > that
> > > > > > > > > we
> > > > > > > > > >> need a more frequent object merging process than tiered
> > > storage
> > > > > > > > > provides.
> > > > > > > > > >> This is just the transition from write-optimised WAL
> > > segments to
> > > > > > > > > >> read-optimised tiered segments, and all of the object
> > > > > > storage-based
> > > > > > > > > >> implementations of Kafka that I’m aware of do this
> > > > > rearrangement.
> > > > > > > But
> > > > > > > > > >> perhaps this more frequent object merging is a pre-GA
> > > > > improvement,
> > > > > > > > > rather
> > > > > > > > > >> than a strict requirement for an initial implementation
> > for
> > > > > early
> > > > > > > > access
> > > > > > > > > >> use.
> > > > > > > > > >>
> > > > > > > > > >> For zone-aligned share consumers, the share group assignor
> > > is
> > > > > > > intended
> > > > > > > > > to
> > > > > > > > > >> be rack-aware. Consumers should be assigned to partitions
> > > with
> > > > > > > leaders
> > > > > > > > > in
> > > > > > > > > >> their zone. The simple assignor is not rack-aware, but it
> > > easily
> > > > > > > could
> > > > > > > > > be
> > > > > > > > > >> or we could have a rack-aware assignor.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Andrew
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> Hi, Ivan,
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > >>>
> > > > > > > > > >>> "As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>> will indeed consume some IOPS. See them as a cache that
> > > could
> > > > > > > always
> > > > > > > > be
> > > > > > > > > >>> restored from the remote storage. While it’s not ideal,
> > > it's
> > > > > > still
> > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > of
> > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>> can avoid explicit flushing on local materialized
> > segments
> > > at
> > > > > all
> > > > > > > and
> > > > > > > > > let
> > > > > > > > > >>> the file system and page cache figure out when to flush
> > > > > > optimally.
> > > > > > > > This
> > > > > > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > > dramatically,
> > > > > > > > > >>> depending on throughput for each partition. We, of
> > course,
> > > > > > continue
> > > > > > > > > >>> flushing the metadata segments as before."
> > > > > > > > > >>>
> > > > > > > > > >>> If we have a mix of classic and diskless topics on the
> > same
> > > > > > broker,
> > > > > > > > > it's
> > > > > > > > > >>> important that the classic topics' data is flushed to
> > disk
> > > as
> > > > > > > quickly
> > > > > > > > > as
> > > > > > > > > >>> possible. To achieve this, users typically set
> > > > > > > dirty_expire_centisecs
> > > > > > > > > in
> > > > > > > > > >>> the kernel based on the number of available disk IOPS.
> > > Once you
> > > > > > set
> > > > > > > > > this
> > > > > > > > > >>> number, it applies to all dirty files, including the
> > cached
> > > > > data
> > > > > > in
> > > > > > > > > >>> diskless topics. So, if there are more files actively
> > > > > > accumulating
> > > > > > > > > data,
> > > > > > > > > >>> the flush frequency and therefore RPO is reduced for
> > > classic
> > > > > > > topics.
> > > > > > > > > >>>
> > > > > > > > > >>> "We should have mentioned this explicitly, but this step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>> in the form of segments offloading to tiered storage.
> > When
> > > we
> > > > > > > > assemble
> > > > > > > > > a
> > > > > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> > > effectively
> > > > > > > doing
> > > > > > > > > >>> metadata compaction: replacing a big number of pieces of
> > > > > metadata
> > > > > > > > about
> > > > > > > > > >>> individual batches with a single record in
> > > > > > __remote_log_metadata."
> > > > > > > > > >>>
> > > > > > > > > >>> The object merging in tier storage typically only kicks
> > in
> > > > > after
> > > > > > a
> > > > > > > > few
> > > > > > > > > >>> hours. The impact is (1) the amount of accumulated
> > > metadata is
> > > > > > > still
> > > > > > > > > >> quite
> > > > > > > > > >>> large; (2) there are many small objects, leading to poor
> > > read
> > > > > > > > > >> performance.
> > > > > > > > > >>> I think we need a more frequent object merging process
> > than
> > > > > tier
> > > > > > > > > storage
> > > > > > > > > >>> provides.
> > > > > > > > > >>>
> > > > > > > > > >>> Jun
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me>
> > > > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thank you for your messages.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> First, it degrades the durability.
> > > > > > > > > >>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at a
> > > > > > > > > >>>>> given point of time, one for the data and another for
> > the
> > > > > > > metadata.
> > > > > > > > > >>>>> Flushing each file requires a separate IO. If the disk
> > > has 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >> we
> > > > > > > > > >>>>> have 5K partitions in a broker, currently we can afford
> > > to
> > > > > > flush
> > > > > > > > each
> > > > > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> number of files per partition, we can only flush each
> > > > > partition
> > > > > > > > every
> > > > > > > > > >> 10
> > > > > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>>> will indeed consume some IOPS. See them as a cache that
> > > could
> > > > > > > always
> > > > > > > > > be
> > > > > > > > > >>>> restored from the remote storage. While it’s not ideal,
> > > it's
> > > > > > still
> > > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > > of
> > > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>>> can avoid explicit flushing on local materialized
> > > segments at
> > > > > > all
> > > > > > > > and
> > > > > > > > > >> let
> > > > > > > > > >>>> the file system and page cache figure out when to flush
> > > > > > optimally.
> > > > > > > > > This
> > > > > > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > > dramatically,
> > > > > > > > > >>>> depending on throughput for each partition. We, of
> > course,
> > > > > > > continue
> > > > > > > > > >>>> flushing the metadata segments as before.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s worth making a note on caching. I think nobody will
> > > > > > disagree
> > > > > > > > that
> > > > > > > > > >>>> doing direct reads from remote storage every time a
> > batch
> > > is
> > > > > > > > requested
> > > > > > > > > >> by a
> > > > > > > > > >>>> consumer will not be practical neither from the
> > > performance
> > > > > nor
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > >>>> economy point of view. We need a way to keep the number
> > > of GET
> > > > > > > > > requests
> > > > > > > > > >>>> down. There are multiple options, for example:
> > > > > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > > > > >>>> 2. Local in-memory caching. Comes with less network
> > > chattiness
> > > > > > and
> > > > > > > > > >> works
> > > > > > > > > >>>> well if we have more or less stable brokers to consume
> > > from.
> > > > > > > > > >>>> 3. Materialization of diskless logs on local disk. Way
> > > lower
> > > > > > > impact
> > > > > > > > on
> > > > > > > > > >>>> RAM and also requires stable brokers for consumption
> > > (using
> > > > > just
> > > > > > > > > >> assigned
> > > > > > > > > >>>> replicas will probably work well).
> > > > > > > > > >>>>
> > > > > > > > > >>>> Materialization is one of possible options, but we can
> > > choose
> > > > > > > > another
> > > > > > > > > >> one.
> > > > > > > > > >>>> However, we will have this dilemma regardless of whether
> > > we
> > > > > have
> > > > > > > an
> > > > > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Second, if we ever need this
> > > > > > > > > >>>>> metadata somewhere else, say in the WAL file manager,
> > the
> > > > > > > consumer
> > > > > > > > > >> needs
> > > > > > > > > >>>> to
> > > > > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>> actual benefit of this approach is also questionable.
> > On
> > > the
> > > > > > > > surface,
> > > > > > > > > >> it
> > > > > > > > > >>>>> might seem that we could reduce the number of lines
> > that
> > > need
> > > > > > to
> > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>> for this KIP. However, the changes are quite intrusive
> > > to the
> > > > > > > > classic
> > > > > > > > > >>>>> partition's code path and will probably make the code
> > > base
> > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>> maintain in the long run. I like the original approach
> > > based
> > > > > on
> > > > > > > the
> > > > > > > > > >> batch
> > > > > > > > > >>>>> coordinator much better than this one. We could
> > probably
> > > > > > refactor
> > > > > > > > the
> > > > > > > > > >>>>> producer state code so that it could be reused in the
> > > batch
> > > > > > > > > >> coordinator.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s hard to disagree with this. The explicit
> > coordinator
> > > is
> > > > > > more
> > > > > > > a
> > > > > > > > > side
> > > > > > > > > >>>> thing, while coordinator-less approach is more about
> > > extending
> > > > > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Thanks for addressing the concerns on the number of
> > RPCs
> > > in
> > > > > the
> > > > > > > > > produce
> > > > > > > > > >>>>> path. I agree that with the metadata crafting
> > mechanism,
> > > we
> > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > be
> > > > > > > > > >>>>> collocated with the data on the same set of brokers,
> > it's
> > > > > weird
> > > > > > > > that
> > > > > > > > > >> they
> > > > > > > > > >>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > uses
> > > > > > > > > >> the
> > > > > > > > > >>>>> metadata crafting mechanism, but the metadata is stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>> partition using its own assignment strategy. It will be
> > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>> them collocated.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I would like to note that the metadata crafting is
> > needed
> > > only
> > > > > > to
> > > > > > > > tell
> > > > > > > > > >>>> producers which brokers they should send Produce
> > requests
> > > to,
> > > > > > but
> > > > > > > > data
> > > > > > > > > >> (as
> > > > > > > > > >>>> in “locally materialized log”) is located on partition
> > > > > replicas,
> > > > > > > > i.e.
> > > > > > > > > >>>> automatically co-located with metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As a side note, it would probably be better that instead
> > > of
> > > > > > > > implicitly
> > > > > > > > > >>>> crafting partition metadata, we extend the metadata
> > > protocol
> > > > > so
> > > > > > > that
> > > > > > > > > for
> > > > > > > > > >>>> diskless partitions we return not only the leader and
> > > > > replicas,
> > > > > > > but
> > > > > > > > > also
> > > > > > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > > > > > performance
> > > > > > > > > and
> > > > > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>> step.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We should have mentioned this explicitly, but this step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>>> in the form of segments offloading to tiered storage.
> > > When we
> > > > > > > > > assemble a
> > > > > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > > > > effectively
> > > > > > > > doing
> > > > > > > > > >>>> metadata compaction: replacing a big number of pieces of
> > > > > > metadata
> > > > > > > > > about
> > > > > > > > > >>>> individual batches with a single record in
> > > > > > __remote_log_metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We could create a Diskless-specific merging mechanism
> > > instead
> > > > > if
> > > > > > > > > needed.
> > > > > > > > > >>>> It’s rather easy with the explicit coordinator approach.
> > > With
> > > > > > the
> > > > > > > > > >>>> coordinator-less approach, this would probably be a bit
> > > more
> > > > > > > tricky
> > > > > > > > > >>>> (rewriting the tail of the log by the leader +
> > replicating
> > > > > this
> > > > > > > > change
> > > > > > > > > >>>> reliably).
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >> changes
> > > > > > > > > >>>> in
> > > > > > > > > >>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >> last
> > > > > > > > > >>>>> for the long term.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Yes, totally agree.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Luke,
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>
> > > > > > > > > >>>> After some evolution of the original proposal,
> > leaderless
> > > is
> > > > > now
> > > > > > > > > >> limited.
> > > > > > > > > >>>> We only need to be able to accept Produce requests on
> > more
> > > > > than
> > > > > > > one
> > > > > > > > > >> broker
> > > > > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get
> > it
> > > > > right
> > > > > > > > that
> > > > > > > > > >> you
> > > > > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let’s now look at this problem from a higher level, as
> > > Jack
> > > > > > > > proposed.
> > > > > > > > > As
> > > > > > > > > >>>> it was said, the big choice we need to make is whether
> > we
> > > 1)
> > > > > > > create
> > > > > > > > an
> > > > > > > > > >>>> explicit batch coordinator; or 2) go for the
> > > coordinator-less
> > > > > > > > > approach,
> > > > > > > > > >>>> where each diskless partition is managed by its leader
> > as
> > > in
> > > > > > > classic
> > > > > > > > > >> topics.
> > > > > > > > > >>>>
> > > > > > > > > >>>> If we try to compare the two approaches:
> > > > > > > > > >>>>
> > > > > > > > > >>>> Pluggability:
> > > > > > > > > >>>> - Explicit coordinator: Possible. For example, some
> > > setups may
> > > > > > > > benefit
> > > > > > > > > >>>> from batch metadata being stored in a cloud database
> > > (such as
> > > > > > AWS
> > > > > > > > > >> DynamoDB
> > > > > > > > > >>>> or GCP Spanner).
> > > > > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Scalability and fault tolerance:
> > > > > > > > > >>>> - Explicit coordinator: Depends on the implementation
> > and
> > > it’s
> > > > > > > also
> > > > > > > > > >>>> necessary to actively work for it.
> > > > > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics.
> > > Scaling is
> > > > > > > done
> > > > > > > > by
> > > > > > > > > >>>> partition placement, partitions could fail
> > independently.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Separation of concerns:
> > > > > > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > > > > > independent
> > > > > > > > > >> from
> > > > > > > > > >>>> classic topics in terms of code and workflows. For
> > > example,
> > > > > the
> > > > > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > > > > mechanism
> > > > > > > > could
> > > > > > > > > >> be
> > > > > > > > > >>>> relatively simply implemented with it. As a flip side of
> > > this,
> > > > > > > some
> > > > > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > > > > bringing
> > > > > > > > > diskless
> > > > > > > > > >>>> closer to classic topics. Some code paths and workflows
> > > could
> > > > > be
> > > > > > > > more
> > > > > > > > > >>>> straightforwardly reused, but they will inevitably have
> > > to be
> > > > > > > > adapted
> > > > > > > > > to
> > > > > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Cloud-nativeness. This is a vague concept, also related
> > > to the
> > > > > > > > > previous,
> > > > > > > > > >>>> but let’s try:
> > > > > > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > > > > > separately
> > > > > > > > > makes
> > > > > > > > > >>>> it easier for brokers to take different roles, be purely
> > > > > > stateless
> > > > > > > > if
> > > > > > > > > >>>> needed, etc.
> > > > > > > > > >>>> - Coordinator-less: Less so. Something could be achieved
> > > with
> > > > > > > > creative
> > > > > > > > > >>>> partition placement, but not much.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Both seem to have their pros and cons. However,
> > answering
> > > > > Jack’s
> > > > > > > > > >> question,
> > > > > > > > > >>>> the explicit coordinator approach may indeed lead to a
> > > more
> > > > > > > flexible
> > > > > > > > > >> design.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> The purpose of this deviation in the discussion was to
> > > > > receive a
> > > > > > > > > >>>> preliminary community evaluation of the coordinator-less
> > > > > > approach
> > > > > > > > > >> without
> > > > > > > > > >>>> taking on the task of writing a separate KIP and fitting
> > > it in
> > > > > > the
> > > > > > > > > >> system
> > > > > > > > > >>>> of KIP-1150 and its children. We’re open to stopping it
> > > and
> > > > > > > getting
> > > > > > > > > >> back to
> > > > > > > > > >>>> working out the coordinator design if the community
> > > doesn’t
> > > > > > favor
> > > > > > > > the
> > > > > > > > > >>>> proposed approach.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best,
> > > > > > > > > >>>> Ivan and Diskless team
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > > > > >>>>> Hi Ivan,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> As Jun pointed out, the updated design seems to have
> > some
> > > > > > > > > shortcomings
> > > > > > > > > >>>>> although it simplifies the implementation.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thank you,
> > > > > > > > > >>>>> Luke
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for the explanation.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> "we write the reference to the WAL file with the batch
> > > data"
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I understand the approach now, but I think it is a
> > hacky
> > > > > one.
> > > > > > > > There
> > > > > > > > > >> are
> > > > > > > > > >>>>>> multiple short comings with this design. First, it
> > > degrades
> > > > > > the
> > > > > > > > > >>>> durability.
> > > > > > > > > >>>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at
> > > > > > > > > >> a
> > > > > > > > > >>>>>> given point of time, one for the data and another for
> > > the
> > > > > > > > metadata.
> > > > > > > > > >>>>>> Flushing each file requires a separate IO. If the disk
> > > has
> > > > > 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> have 5K partitions in a broker, currently we can
> > afford
> > > to
> > > > > > flush
> > > > > > > > > each
> > > > > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> number of files per partition, we can only flush each
> > > > > > partition
> > > > > > > > > every
> > > > > > > > > >>>> 10
> > > > > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we
> > > ever
> > > > > need
> > > > > > > > this
> > > > > > > > > >>>>>> metadata somewhere else, say in the WAL file manager,
> > > the
> > > > > > > consumer
> > > > > > > > > >>>> needs to
> > > > > > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>>> actual benefit of this approach is also questionable.
> > > On the
> > > > > > > > > surface,
> > > > > > > > > >>>> it
> > > > > > > > > >>>>>> might seem that we could reduce the number of lines
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>>> for this KIP. However, the changes are quite intrusive
> > > to
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition's code path and will probably make the code
> > > base
> > > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>>> maintain in the long run. I like the original approach
> > > based
> > > > > > on
> > > > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>> coordinator much better than this one. We could
> > probably
> > > > > > > refactor
> > > > > > > > > the
> > > > > > > > > >>>>>> producer state code so that it could be reused in the
> > > batch
> > > > > > > > > >>>> coordinator.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for addressing the concerns on the number of
> > > RPCs in
> > > > > > the
> > > > > > > > > >> produce
> > > > > > > > > >>>>>> path. I agree that with the metadata crafting
> > > mechanism, we
> > > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > > be
> > > > > > > > > >>>>>> collocated with the data on the same set of brokers,
> > > it's
> > > > > > weird
> > > > > > > > that
> > > > > > > > > >>>> they
> > > > > > > > > >>>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > > uses
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> metadata crafting mechanism, but the metadata is
> > stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition using its own assignment strategy. It will
> > be
> > > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> them collocated.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>>> step. My first concern is on the amount of metadata
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> kept.
> > > > > > > > > >>>>>> Without object compcation, the metadata generated in
> > the
> > > > > > produce
> > > > > > > > > path
> > > > > > > > > >>>> can
> > > > > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's
> > > say for
> > > > > > > every
> > > > > > > > > >>>> 250ms we
> > > > > > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> > > > > remoting
> > > > > > > > > tiering
> > > > > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K
> > > partitions,
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> > > metadata,
> > > > > > > quite
> > > > > > > > > >>>>>> signficant. A second concern is on performance. Every
> > > time
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>>>> rebuild the caching data, we need to read a bunch of
> > > small
> > > > > > > objects
> > > > > > > > > >>>> from S3,
> > > > > > > > > >>>>>> slowing down the building process. If a consumer
> > > happens to
> > > > > > need
> > > > > > > > > such
> > > > > > > > > >>>> data,
> > > > > > > > > >>>>>> it could slow down the application.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >>>> changes in
> > > > > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >>>> last
> > > > > > > > > >>>>>> for the long term.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Jun
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hi Jun,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed
> > to
> > > > > > clearly
> > > > > > > > > >>>> explain
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > partitions.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> No. There will be only one physical partition per
> > > diskless
> > > > > > > > > >>>> partition. Let
> > > > > > > > > >>>>>>> me explain this with an example. Let’s say we have a
> > > > > diskless
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> > > leader. We
> > > > > > > > produce
> > > > > > > > > >>>> some
> > > > > > > > > >>>>>>> batches to this partition. The content of the segment
> > > file
> > > > > > will
> > > > > > > > be
> > > > > > > > > >>>>>>> something like this (for each batch):
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > > > > >>>>>>> Magic: like in classic
> > > > > > > > > >>>>>>> CRC: like in classic
> > > > > > > > > >>>>>>> Attributes: like in classic
> > > > > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > > > > >>>>>>> ProducerId: like in classic
> > > > > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > > > > >>>>>>> Records:
> > > > > > > > > >>>>>>>
> > path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > > > > byte
> > > > > > > > offset
> > > > > > > > > >>>>>>> 123456
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It looks very much like classic log entries. The only
> > > > > > > difference
> > > > > > > > is
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> instead of writing real Records, we write the
> > > reference to
> > > > > > the
> > > > > > > > WAL
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> with the batch data (I guess we need only the name
> > and
> > > the
> > > > > > byte
> > > > > > > > > >>>> offset,
> > > > > > > > > >>>>>>> because the byte length is the standard field above).
> > > > > > > Otherwise,
> > > > > > > > > >>>> it’s a
> > > > > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> So we have as many partitions for diskless as for
> > > classic.
> > > > > As
> > > > > > > of
> > > > > > > > > open
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Are the metadata and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The source of truth for the data is still in WAL
> > files
> > > on
> > > > > > > object
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>> The source of truth for the metadata is in segment
> > > files on
> > > > > > the
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > > > > > independent
> > > > > > > > > of
> > > > > > > > > >>>> this
> > > > > > > > > >>>>>>> new proposal, but I want to present them to give the
> > > idea
> > > > > > that
> > > > > > > > > only a
> > > > > > > > > >>>>>>> limited amount of data files will be operated
> > locally:
> > > > > > > > > >>>>>>> - We want to assemble batches into segment files and
> > > > > offload
> > > > > > > them
> > > > > > > > > to
> > > > > > > > > >>>>>>> tiered storage in order to prevent the unbounded
> > > growth of
> > > > > > > batch
> > > > > > > > > >>>>>> metadata.
> > > > > > > > > >>>>>>> For this, we need to open only  a few file
> > descriptors
> > > (for
> > > > > > the
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>> file itself + the necessary indexes) before the
> > > segment is
> > > > > > > fully
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > > > > >>>>>>> - We want to assemble local segment files for caching
> > > > > > purposes
> > > > > > > as
> > > > > > > > > >>>> well,
> > > > > > > > > >>>>>>> i.e. to speed up fetching. This will not materialize
> > > the
> > > > > full
> > > > > > > > > >>>> content of
> > > > > > > > > >>>>>>> the log, but only the hot set according to some
> > policy
> > > (or
> > > > > > > > > >>>> configurable
> > > > > > > > > >>>>>>> policies), i.e. the number of segments and file
> > > descriptors
> > > > > > > will
> > > > > > > > > >>>> also be
> > > > > > > > > >>>>>>> limited.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This is a valid concern that we considered, but this
> > > issue
> > > > > > can
> > > > > > > be
> > > > > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > > > > >>>>>>> The situation with a single broker is trivial: all
> > the
> > > > > commit
> > > > > > > > > >>>> requests go
> > > > > > > > > >>>>>>> from the broker to itself.
> > > > > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but
> > > located in
> > > > > > the
> > > > > > > > > single
> > > > > > > > > >>>>>> rack
> > > > > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for
> > > diskless
> > > > > > > > > >>>> partitions, but
> > > > > > > > > >>>>>>> we can tell producers (through metadata) to always
> > send
> > > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader
> > > replicas
> > > > > for
> > > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>> these partitions in various combinations, but only
> > for
> > > > > them.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>                   Broker 0
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>             |    t1-0         |
> > > > > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > > > > >>>>>>> partitions   | +-------------+ |    for these
> > > partitions
> > > > > > > > > >>>>>>>             |                 |
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > > > > Effectively,
> > > > > > > > > each
> > > > > > > > > >>>>>>> broker will commit only to itself, which effectively
> > > means
> > > > > 1
> > > > > > > > commit
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls,
> > > if we
> > > > > > > wish,
> > > > > > > > > >>>> just a
> > > > > > > > > >>>>>>> local function call).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs).
> > > Obviously, we
> > > > > > > > cannot
> > > > > > > > > >>>> always
> > > > > > > > > >>>>>>> send Produce requests to the designated leaders of
> > > diskless
> > > > > > > > > >>>> partitions:
> > > > > > > > > >>>>>>> this would mean inter-AZ network traffic, which we
> > > would
> > > > > like
> > > > > > > to
> > > > > > > > > >>>> avoid.
> > > > > > > > > >>>>>> To
> > > > > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> > > produce
> > > > > > > > > >>>>>> representative”
> > > > > > > > > >>>>>>> in every AZ. If we continue our example: when a
> > Produce
> > > > > > request
> > > > > > > > for
> > > > > > > > > >>>> t1-0,
> > > > > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands
> > > on
> > > > > > > broker 0
> > > > > > > > > >>>> (in the
> > > > > > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > > > > > However,
> > > > > > > if
> > > > > > > > > it
> > > > > > > > > >>>>>> comes
> > > > > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker
> > > 2.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> |produce requests         |produce requests
> > > |produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
> > |for
> > > > > t1-0,
> > > > > > > > t2-1,
> > > > > > > > > >>>>>> t3-0
> > > > > > > > > >>>>>>> |from AZ 0                |from AZ 1
> > >  |from
> > > > > AZ 2
> > > > > > > > > >>>>>>> v                         v                        v
> > > > > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker
> > 2
> > > (AZ
> > > > > 2)
> > > > > > > > > >>>>>>> +---------------+      +---------------+
> > > > > > +---------------+
> > > > > > > > > >>>>>>> |     t1-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t2-1      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t3-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> +---------------+      +--------+------+
> > > > > > +--------+------+
> > > > > > > > > >>>>>>>    ^     ^                    |
> > |
> > > > > > > > > >>>>>>>    |     +--------------------+
> > |
> > > > > > > > > >>>>>>>    |     commit requests for these partitions
> > |
> > > > > > > > > >>>>>>>    |
> >  |
> > > > > > > > > >>>>>>>
> > +-------------------------------------------------+
> > > > > > > > > >>>>>>>          commit requests for these partitions
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> All the partitions that broker 0 is the leader of
> > will
> > > be
> > > > > > > > > >>>> “represented”
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Of course, this relationship goes both ways between
> > AZs
> > > > > (not
> > > > > > > > > >>>> necessarily
> > > > > > > > > >>>>>>> between the same brokers). It means that provided the
> > > > > cluster
> > > > > > > is
> > > > > > > > > >>>> balanced
> > > > > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> > > represent
> > > > > > > > > >>>>>> (number_of_azs
> > > > > > > > > >>>>>>> - 1) other brokers. This will result in the situation
> > > that
> > > > > > for
> > > > > > > > the
> > > > > > > > > >>>>>> majority
> > > > > > > > > >>>>>>> of commits, each broker will do up to (number_of_azs
> > -
> > > 1)
> > > > > > > network
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests (plus one local). Cloud regions tend to
> > have 3
> > > > > AZs,
> > > > > > > very
> > > > > > > > > >>>> rarely
> > > > > > > > > >>>>>>> more. That means, brokers will be doing up to 2
> > network
> > > > > > commit
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> per
> > > > > > > > > >>>>>>> WAL file.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> There are the following exceptions:
> > > > > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example,
> > > when we
> > > > > > > have
> > > > > > > > 2
> > > > > > > > > >>>> AZs
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> one has three brokers and another AZ has one. This
> > one
> > > > > broker
> > > > > > > > will
> > > > > > > > > do
> > > > > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is
> > > not
> > > > > an
> > > > > > > > > extreme
> > > > > > > > > >>>>>>> amplification. Such an imbalance is not healthy in
> > most
> > > > > > > practical
> > > > > > > > > >>>> setups
> > > > > > > > > >>>>>>> and should be avoided anyway.
> > > > > > > > > >>>>>>> 2. Leadership changes and metadata propagation
> > period.
> > > When
> > > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > > > > > producers
> > > > > > > > > will
> > > > > > > > > >>>> not
> > > > > > > > > >>>>>>> know this immediately (unless we want to be strict
> > and
> > > > > > respond
> > > > > > > > with
> > > > > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> > > will
> > > > > come
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>> in a
> > > > > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two
> > commit
> > > > > > > requests:
> > > > > > > > > to
> > > > > > > > > >>>>>> broker
> > > > > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
> > > t3-0.
> > > > > > This
> > > > > > > > > >>>> situation
> > > > > > > > > >>>>>>> is not permanent and as producers update the cluster
> > > > > > metadata,
> > > > > > > it
> > > > > > > > > >>>> will be
> > > > > > > > > >>>>>>> resolved.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This all could be built with the metadata crafting
> > > > > mechanism
> > > > > > > only
> > > > > > > > > >>>> (which
> > > > > > > > > >>>>>>> is anyway needed for Diskless in one way or another
> > to
> > > > > direct
> > > > > > > > > >>>> producers
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic),
> > > just
> > > > > with
> > > > > > > the
> > > > > > > > > >>>> right
> > > > > > > > > >>>>>>> policy for it (for example, some deterministic
> > > hash-based
> > > > > > > > formula).
> > > > > > > > > >>>> I.e.
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>> explicit support for “produce representative” or
> > > anything
> > > > > > like
> > > > > > > > this
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> We do only two operations on WAL files that span
> > > multiple
> > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions: committing and deleting. Commits can be
> > > done
> > > > > > > > > >>>> independently as
> > > > > > > > > >>>>>>> described above. But deletes are different, because
> > > when a
> > > > > > file
> > > > > > > > is
> > > > > > > > > >>>>>> deleted,
> > > > > > > > > >>>>>>> this affects all the partitions that still have alive
> > > > > batches
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> (if any).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The WAL file manager is a necessary point of
> > > coordination
> > > > > to
> > > > > > > > delete
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> files safely. We can say it is the source of truth
> > > about
> > > > > > files
> > > > > > > > > >>>>>> themselves,
> > > > > > > > > >>>>>>> while the partition leaders and their logs hold the
> > > truth
> > > > > > about
> > > > > > > > > >>>> whether a
> > > > > > > > > >>>>>>> particular file contains live batches of this
> > > particular
> > > > > > > > partition.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager will do this important task: be able
> > > to
> > > > > say
> > > > > > > for
> > > > > > > > > sure
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> a file does not contain any live batch of any
> > existing
> > > > > > > partition.
> > > > > > > > > For
> > > > > > > > > >>>>>> this,
> > > > > > > > > >>>>>>> it will have to periodically check against the
> > > partition
> > > > > > > leaders.
> > > > > > > > > >>>>>>> Considering that batch deletion is irreversible, when
> > > we
> > > > > > > declare
> > > > > > > > a
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager has to know about files being
> > > committed to
> > > > > > > start
> > > > > > > > > >>>> track
> > > > > > > > > >>>>>>> them and periodically check if they are empty. We can
> > > > > > consider
> > > > > > > > > >>>> various
> > > > > > > > > >>>>>> ways
> > > > > > > > > >>>>>>> to achieve this:
> > > > > > > > > >>>>>>> 1. As was proposed in my previous message: best
> > effort
> > > > > commit
> > > > > > > by
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> +
> > > > > > > > > >>>>>>> periodic prefix scans of object storage to detect
> > files
> > > > > that
> > > > > > > went
> > > > > > > > > >>>> below
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> radar due to network issue or the file manager
> > > temporary
> > > > > > > > > >>>> unavailability.
> > > > > > > > > >>>>>>> We’re speaking about listing the file names only and
> > > > > opening
> > > > > > > only
> > > > > > > > > >>>>>>> previously unknown files in order to find the
> > > partitions
> > > > > > > involved
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>> them.
> > > > > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill
> > the
> > > > > list
> > > > > > of
> > > > > > > > > files
> > > > > > > > > >>>>>>> fully asynchronously and in the background. This may
> > > be not
> > > > > > > ideal
> > > > > > > > > >>>> due to
> > > > > > > > > >>>>>>> costs and performance of scanning tons of files.
> > > However,
> > > > > the
> > > > > > > > > number
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>> live WAL files should be limited due to tiered
> > storage
> > > > > > > > offloading +
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> can
> > > > > > > > > >>>>>>> optimize this if we give files some global soft order
> > > in
> > > > > > their
> > > > > > > > > names.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > logic.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > > > > UnifiedLog
> > > > > > in
> > > > > > > > > >>>> order to
> > > > > > > > > >>>>>>> support these changes. However, it seems that
> > > idempotence,
> > > > > > > > > >>>> transactions,
> > > > > > > > > >>>>>>> queues, tiered storage will have to be changed less
> > > than
> > > > > with
> > > > > > > the
> > > > > > > > > >>>>>> original
> > > > > > > > > >>>>>>> design. This is because the partition leader state
> > > would
> > > > > > remain
> > > > > > > > in
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> same
> > > > > > > > > >>>>>>> place (on brokers) and existing workflows that
> > involve
> > > it
> > > > > > would
> > > > > > > > > have
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> be
> > > > > > > > > >>>>>>> changed less compared to the situation where we
> > > globalize
> > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader state in the batch coordinator. I admit this
> > is
> > > hard
> > > > > > to
> > > > > > > > make
> > > > > > > > > >>>>>>> convincing without both real implementations to hand
> > :)
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Let’s consider the scenario of turning a classic
> > topic
> > > into
> > > > > > > > > >>>> diskless. The
> > > > > > > > > >>>>>>> user sets diskless.enabled=true, the leader receives
> > > this
> > > > > > > > metadata
> > > > > > > > > >>>> update
> > > > > > > > > >>>>>>> and does the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Start a new segment that will be written in the
> > > diskless
> > > > > > > > format
> > > > > > > > > >>>> (i.e.
> > > > > > > > > >>>>>>> without data).
> > > > > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Since it’s the same log, the followers will know
> > about
> > > that
> > > > > > > > switch
> > > > > > > > > >>>>>>> consistently. They will finish replicating the
> > classic
> > > > > > segments
> > > > > > > > and
> > > > > > > > > >>>> start
> > > > > > > > > >>>>>>> replicating the diskless ones. They will always know
> > > where
> > > > > > each
> > > > > > > > > >>>> batch is
> > > > > > > > > >>>>>>> located (either inside a classic segment or
> > referenced
> > > by a
> > > > > > > > > diskless
> > > > > > > > > >>>>>> one).
> > > > > > > > > >>>>>>> Switching back should be similar.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Doing this with the coordinator is possible, but has
> > > some
> > > > > > > > caveats.
> > > > > > > > > >>>> The
> > > > > > > > > >>>>>>> leader must do the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Write a special control segment to persist and
> > > replicate
> > > > > > the
> > > > > > > > > fact
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> from offset N the partition is now in the diskless
> > > mode.
> > > > > > > > > >>>>>>> 4. Inform the coordinator about the first offset N of
> > > the
> > > > > > > > “diskless
> > > > > > > > > >>>> era”.
> > > > > > > > > >>>>>>> 5. Inform the controller quorum that the transition
> > has
> > > > > > > finished
> > > > > > > > > and
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> brokers now can process diskless writes for this
> > > partition.
> > > > > > > > > >>>>>>> This could fail at some points, so this will probably
> > > > > require
> > > > > > > > some
> > > > > > > > > >>>>>>> explicit state machine with replication either in the
> > > > > > partition
> > > > > > > > log
> > > > > > > > > >>>> or in
> > > > > > > > > >>>>>>> KRaft.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It seems that the coordinator-less approach makes
> > this
> > > > > > simpler
> > > > > > > > > >>>> because
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> “coordinator” for the partition and the partition
> > > leader
> > > > > are
> > > > > > > the
> > > > > > > > > >>>> same and
> > > > > > > > > >>>>>>> they store the partition metadata in the same log,
> > too.
> > > > > While
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>>>>>> coordinator approach we have to perform some kind of
> > a
> > > > > > > > distributed
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> to handover metadata management from the classic
> > > partition
> > > > > > > leader
> > > > > > > > > to
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch coordinator.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> > > Please
> > > > > > let
> > > > > > > me
> > > > > > > > > >>>> know if
> > > > > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Best,
> > > > > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > > > > >>>>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Thanks for the update.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure that I fully understand the new
> > design,
> > > but
> > > > > it
> > > > > > > > seems
> > > > > > > > > >>>> less
> > > > > > > > > >>>>>>>> clean than before.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > > partitions. Are
> > > > > > the
> > > > > > > > > >>>> metadata
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > > logic. I
> > > > > > > > > >>>> am
> > > > > > > > > >>>>>>> also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Jun
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to
> > > have a
> > > > > > > > separate
> > > > > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role
> > > would
> > > > > be
> > > > > > to
> > > > > > > > > >>>>>> centrally
> > > > > > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> > > > > topics.
> > > > > > > > This
> > > > > > > > > >>>>>>> component
> > > > > > > > > >>>>>>>>> drew many reasonable comments from the community
> > > about
> > > > > how
> > > > > > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> support
> > > > > > > > > >>>>>>>>> various Kafka features (transactions, queues) and
> > its
> > > > > > > > > >>>> scalability.
> > > > > > > > > >>>>>>> While we
> > > > > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > > > > concerns,
> > > > > > > we
> > > > > > > > > >>>> took a
> > > > > > > > > >>>>>>> step
> > > > > > > > > >>>>>>>>> back and looked at the problem from a different
> > > > > > perspective.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> We would like to propose an alternative Diskless
> > > design
> > > > > > > > *without
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>>>>> centralized coordinator*. We believe this approach
> > > has
> > > > > > > > potential
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>>>>> propose to discuss it as it may be more appealing
> > to
> > > the
> > > > > > > > > >>>> community.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let us explain the idea. Most of the complications
> > > with
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless approach come from one necessary
> > > architecture
> > > > > > > change:
> > > > > > > > > >>>>>>> globalizing
> > > > > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > > > > coordinator.
> > > > > > > > > >>>> This
> > > > > > > > > >>>>>>> causes
> > > > > > > > > >>>>>>>>> deviations to the established workflows in various
> > > > > features
> > > > > > > > like
> > > > > > > > > >>>>>>> produce
> > > > > > > > > >>>>>>>>> idempotence and transactions, queues, retention,
> > etc.
> > > > > These
> > > > > > > > > >>>>>> deviations
> > > > > > > > > >>>>>>> need
> > > > > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > > > > implemented
> > > > > > > and
> > > > > > > > > >>>>>>> tested. In
> > > > > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > > > > partition
> > > > > > > > > >>>> leaders
> > > > > > > > > >>>>>>> again
> > > > > > > > > >>>>>>>>> responsible for managing their partitions, even in
> > > > > diskless
> > > > > > > > > >>>> topics.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata
> > are
> > > > > > blended
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> the one partition log. The crux of the Diskless
> > idea
> > > is
> > > > > to
> > > > > > > > > >>>> decouple
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> and move data to the remote storage, while keeping
> > > > > metadata
> > > > > > > > > >>>> somewhere
> > > > > > > > > >>>>>>> else.
> > > > > > > > > >>>>>>>>> Using the central batch coordinator for managing
> > > batch
> > > > > > > metadata
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>> way,
> > > > > > > > > >>>>>>>>> but not the only.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s now think about managing metadata for each
> > user
> > > > > > > partition
> > > > > > > > > >>>>>>>>> independently. Generally partitions are independent
> > > and
> > > > > > don’t
> > > > > > > > > >>>> share
> > > > > > > > > >>>>>>>>> anything apart from that their data are mixed in
> > WAL
> > > > > files.
> > > > > > > If
> > > > > > > > we
> > > > > > > > > >>>>>>> figure
> > > > > > > > > >>>>>>>>> out how to commit and later delete WAL files
> > safely,
> > > we
> > > > > > will
> > > > > > > > > >>>> achieve
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> > > > > central
> > > > > > > > batch
> > > > > > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition
> > > will
> > > > > be
> > > > > > > > > >>>> managed
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> its
> > > > > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> > > classic
> > > > > > > > > >>>> topics, the
> > > > > > > > > >>>>>>>>> leader uses the partition log as the way to persist
> > > batch
> > > > > > > > > >>>> metadata,
> > > > > > > > > >>>>>>> i.e.
> > > > > > > > > >>>>>>>>> the regular batch header + the information about
> > how
> > > to
> > > > > > find
> > > > > > > > this
> > > > > > > > > >>>>>>> batch on
> > > > > > > > > >>>>>>>>> remote storage. In contrast to classic topics,
> > batch
> > > data
> > > > > > is
> > > > > > > in
> > > > > > > > > >>>>>> remote
> > > > > > > > > >>>>>>>>> storage.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > > > > >>>>>>>>> • Classic topics:
> > > > > > > > > >>>>>>>>>  • Data and metadata are co-located in the
> > partition
> > > log.
> > > > > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > > > > (metadata)|Batch
> > > > > > > > > >>>> data].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>> • Original Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> > > > > remote
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>>>>  • The partition state is global in the batch
> > > > > coordinator.
> > > > > > > > > >>>>>>>>> • New Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on
> > > remote
> > > > > > > storage.
> > > > > > > > > >>>>>>>>>  • Partition log content: [Batch header
> > > (metadata)|Batch
> > > > > > > > > >>>>>> coordinates
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> remote storage].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the
> > reminder
> > > of
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless design:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The new approach could be depicted as the
> > following:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> As you can see, the main difference is that now
> > > instead
> > > > > of
> > > > > > a
> > > > > > > > > >>>> single
> > > > > > > > > >>>>>>> commit
> > > > > > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> > > > > parallel
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to all the leaders of each partition involved in
> > the
> > > WAL
> > > > > > > file.
> > > > > > > > > >>>> Each
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> will commit its batches independently, without
> > > > > coordinating
> > > > > > > > with
> > > > > > > > > >>>>>> other
> > > > > > > > > >>>>>>>>> leaders and any other components. Batch data is
> > > addressed
> > > > > > by
> > > > > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> file
> > > > > > > > > >>>>>>>>> name, the byte offset and size, which allows
> > > partitions
> > > > > to
> > > > > > > know
> > > > > > > > > >>>>>> nothing
> > > > > > > > > >>>>>>>>> about other partitions to access their data in
> > > shared WAL
> > > > > > > > files.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The number of partitions involved in a single WAL
> > > file
> > > > > may
> > > > > > be
> > > > > > > > > >>>> quite
> > > > > > > > > >>>>>>> large,
> > > > > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to
> > commit
> > > one
> > > > > > WAL
> > > > > > > > > >>>> file is
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> > > > > number:
> > > > > > > > > >>>>>>>>> 1. Partition leaders are located on brokers.
> > > Requests to
> > > > > > > > > >>>> leaders on
> > > > > > > > > >>>>>>> one
> > > > > > > > > >>>>>>>>> broker could be grouped together into a single
> > > physical
> > > > > > > network
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>>>> (resembling the normal Produce request that may
> > carry
> > > > > > batches
> > > > > > > > for
> > > > > > > > > >>>>>> many
> > > > > > > > > >>>>>>>>> partitions inside). This will cap the number of
> > > network
> > > > > > > > requests
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make
> > producers
> > > > > send
> > > > > > > > their
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> > > > > achieve
> > > > > > > the
> > > > > > > > > >>>> higher
> > > > > > > > > >>>>>>>>> concentration of logical commit requests in
> > physical
> > > > > > network
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>>>> reducing the number of the latter ones even
> > further,
> > > > > > ideally
> > > > > > > to
> > > > > > > > > >>>> one.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Obviously, out of multiple commit requests some may
> > > fail
> > > > > or
> > > > > > > > time
> > > > > > > > > >>>> out
> > > > > > > > > >>>>>>> for a
> > > > > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers
> > will
> > > > > > receive
> > > > > > > > > >>>> totally
> > > > > > > > > >>>>>>> or
> > > > > > > > > >>>>>>>>> partially failed responses to their Produce
> > requests,
> > > > > > similar
> > > > > > > > to
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> they
> > > > > > > > > >>>>>>>>> would have received when appending to a classic
> > topic
> > > > > fails
> > > > > > > or
> > > > > > > > > >>>> times
> > > > > > > > > >>>>>>> out.
> > > > > > > > > >>>>>>>>> If a partition experiences problems, other
> > partitions
> > > > > will
> > > > > > > not
> > > > > > > > be
> > > > > > > > > >>>>>>> affected
> > > > > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > > > > uncommitted
> > > > > > > > data
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > > > > > (batches
> > > > > > > > are
> > > > > > > > > >>>>>>> constantly
> > > > > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> > > storage),
> > > > > > so
> > > > > > > > this
> > > > > > > > > >>>>>>> garbage
> > > > > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For safely deleting WAL files we now need to
> > > centrally
> > > > > > manage
> > > > > > > > > >>>> them,
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>>>> this is the only state and logic that spans
> > multiple
> > > > > > > > partitions.
> > > > > > > > > >>>> On
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> diagram, you can see another commit request called
> > > > > “Commit
> > > > > > > file
> > > > > > > > > >>>> (best
> > > > > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This
> > manager
> > > will
> > > > > > be
> > > > > > > > > >>>>>>> responsible
> > > > > > > > > >>>>>>>>> for the following:
> > > > > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and
> > > persisting
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> committed WAL files.
> > > > > > > > > >>>>>>>>> 2. To handle potential failures in file information
> > > > > > delivery,
> > > > > > > > it
> > > > > > > > > >>>>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> doing prefix scan on the remote storage
> > periodically
> > > to
> > > > > > find
> > > > > > > > and
> > > > > > > > > >>>>>>> register
> > > > > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > > > > configurable
> > > > > > > and
> > > > > > > > > >>>>>> ideally
> > > > > > > > > >>>>>>>>> should be quite long.
> > > > > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders
> > > (after a
> > > > > > > grace
> > > > > > > > > >>>>>>> period) if
> > > > > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t
> > anymore
> > > > > > > referred
> > > > > > > > > >>>> to by
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> partition.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > > > > features
> > > > > > > such
> > > > > > > > > >>>> as
> > > > > > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > > > > > retention.
> > > > > > > > > >>>> Now we
> > > > > > > > > >>>>>>> don’t
> > > > > > > > > >>>>>>>>> need to abstract away and reuse the code from
> > > partition
> > > > > > > leaders
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use
> > the
> > > > > same
> > > > > > > code
> > > > > > > > > >>>> paths
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from
> > > classic
> > > > > > > topics
> > > > > > > > > >>>> mostly
> > > > > > > > > >>>>>>>>> remain unchanged.
> > > > > > > > > >>>>>>>>> For example, it seems that
> > > > > > > > > >>>>>>>>>
> > > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > > > > and
> > > > > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > > > > transaction
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>> on
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> partition leader side could be used for diskless
> > > topics
> > > > > > with
> > > > > > > > > >>>> little
> > > > > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > > > > idempotent
> > > > > > > > > >>>> produce
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > > > > >>>>>>>>> Another example is share groups support, where the
> > > share
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader,
> > > > > > > > > >>>>>>>>> being co-located with the partition leader, would
> > > execute
> > > > > > the
> > > > > > > > > >>>> same
> > > > > > > > > >>>>>>> logic
> > > > > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > > > > >>>>>>>>> 2. It returns to the familiar partition-based
> > scaling
> > > > > > model,
> > > > > > > > > >>>> where
> > > > > > > > > >>>>>>>>> partitions are independent.
> > > > > > > > > >>>>>>>>> 3. It makes the operation and failure patterns
> > > closer to
> > > > > > the
> > > > > > > > > >>>>>> familiar
> > > > > > > > > >>>>>>>>> ones from classic topics.
> > > > > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> > > switching
> > > > > > the
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> modes
> > > > > > > > > >>>>>>>>> between diskless and classic.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The rest of the things remain unchanged compared to
> > > the
> > > > > > > > previous
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> design (after all previous discussions). Such
> > things
> > > as
> > > > > > local
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>>>> materialization by replicas, the consume path,
> > tiered
> > > > > > storage
> > > > > > > > > >>>>>>> integration,
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> If the community finds this design more suitable,
> > we
> > > will
> > > > > > > > update
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it.
> > > Please let
> > > > > > us
> > > > > > > > know
> > > > > > > > > >>>>>> what
> > > > > > > > > >>>>>>> you
> > > > > > > > > >>>>>>>>> think.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
> > wrote:
> > > > > > > > > >>>>>>>>>> Hi Justine,
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > > > > transactions
> > > > > > > > > >>>> for in
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> diskless coordinator for as long as the
> > corresponding
> > > > > > offsets
> > > > > > > > are
> > > > > > > > > >>>>>>> there.
> > > > > > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> > > > > earlier,
> > > > > > > > this
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks for the response. I think most of what you
> > > said
> > > > > > made
> > > > > > > > > >>>>>> sense,
> > > > > > > > > >>>>>>> but
> > > > > > > > > >>>>>>>>> I
> > > > > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it). The
> > > > > > > > > >>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > allowing
> > > > > > > > > >>>>>>> partition
> > > > > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if
> > > this is
> > > > > > > > > >>>> wrong. We
> > > > > > > > > >>>>>>> think
> > > > > > > > > >>>>>>>>>>> about relying on this with the batch coordinator
> > > and
> > > > > > delete
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s finished (as
> > there’s
> > > no
> > > > > > > > > >>>> replication
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> HWM
> > > > > > > > > >>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I didn't quite understand this. In classic
> > topics,
> > > we
> > > > > > have
> > > > > > > > > >>>> maps
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>> transactions which remove state when the
> > > transaction is
> > > > > > > > > >>>> completed
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> an
> > > > > > > > > >>>>>>>>>>> aborted transactions index which is retained for
> > > much
> > > > > > > longer.
> > > > > > > > > >>>>>> Once
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> > > longer
> > > > > > > > > >>>> involved
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> > > > > subject
> > > > > > to
> > > > > > > > > >>>>>>> compaction
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>>> Looking back at the outline provided above, I
> > > didn't
> > > > > see
> > > > > > > much
> > > > > > > > > >>>>>>> about the
> > > > > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > > > > > further.
> > > > > > > I
> > > > > > > > > >>>> saw
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> following in a response:
> > > > > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > > > > segment,
> > > > > > > > > >>>> all the
> > > > > > > > > >>>>>>>>> necessary
> > > > > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> > > including
> > > > > > the
> > > > > > > > > >>>>>>> transaction
> > > > > > > > > >>>>>>>>>>> index will be built to serve FetchRequests
> > exactly
> > > as
> > > > > > they
> > > > > > > > > >>>> are
> > > > > > > > > >>>>>>> today.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain
> > the
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>> Justine
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > > > > > >>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> > > > > focused
> > > > > > > > > >>>> only on
> > > > > > > > > >>>>>>>>> version 2.
> > > > > > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> > > > > consider
> > > > > > > > > >>>> using
> > > > > > > > > >>>>>>> some
> > > > > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > > > > > ongoing,
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> course
> > > > > > > > > >>>>>>>>>>>> the normal idempotence checks. What the
> > partition
> > > > > leader
> > > > > > > > > >>>> in the
> > > > > > > > > >>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>> topics does before appending a batch to the
> > local
> > > log
> > > > > > > > > >>>> (e.g. in
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > > > > > Diskless,
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>>>>> unfortunately
> > > > > > > > > >>>>>>>>>>>> cannot do these checks before appending the data
> > > to
> > > > > the
> > > > > > > WAL
> > > > > > > > > >>>>>>> segment
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these
> > > batches in
> > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As
> > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader
> > > > > > > > > >>>>>>>>> tracks
> > > > > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the
> > batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> has
> > > > > > > > > >>>>>>>>> to as
> > > > > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > > > > > >>>> transaction
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> > > batches
> > > > > > and
> > > > > > > > > >>>> other
> > > > > > > > > >>>>>>>>>>>> information, but for the sake of performance
> > we’d
> > > > > better
> > > > > > > > > >>>> store
> > > > > > > > > >>>>>> it
> > > > > > > > > >>>>>>>>>>>> explicitly.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it).
> > > > > > > > > >>>>>> The
> > > > > > > > > >>>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > > > >>>> allowing
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if
> > > this
> > > > > is
> > > > > > > > > >>>> wrong.
> > > > > > > > > >>>>>> We
> > > > > > > > > >>>>>>>>> think
> > > > > > > > > >>>>>>>>>>>> about relying on this with the batch coordinator
> > > and
> > > > > > > > > >>>> delete the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as
> > > there’s no
> > > > > > > > > >>>>>> replication
> > > > > > > > > >>>>>>>>> and HWM
> > > > > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> > > transactions!
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is
> > > there
> > > > > > state
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> > > local
> > > > > > > > > >>>>>> segments,
> > > > > > > > > >>>>>>>>> making
> > > > > > > > > >>>>>>>>>>>> use of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> > > state for
> > > > > > > > > >>>> each
> > > > > > > > > >>>>>>>>> partition in
> > > > > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> > > fetches
> > > > > > > > > >>>> need
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> know
> > > > > > > > > >>>>>>>>>>>> aborted
> > > > > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > > > > >>>> transaction is
> > > > > > > > > >>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > > > > corresponding
> > > > > > > > > >>>> logs
> > > > > > > > > >>>>>>> of the
> > > > > > > > > >>>>>>>>>>>> involved
> > > > > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the
> > > metadata
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>>>> actual
> > > > > > > > > >>>>>>>>>>>> control
> > > > > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > > > > transactional
> > > > > > > > > >>>>>>> batches.
> > > > > > > > > >>>>>>>>> How
> > > > > > > > > >>>>>>>>>>>> does the
> > > > > > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> > > > > control
> > > > > > > > > >>>>>> batch
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response
> > > needs to
> > > > > > be
> > > > > > > > > >>>>>>> built by
> > > > > > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> > > > > batches
> > > > > > > > > >>>>>>> together?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share
> > partition
> > > > > leader
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>>>>> consumers
> > > > > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
> > > this
> > > > > or
> > > > > > > > > >>>> it's
> > > > > > > > > >>>>>>> too
> > > > > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> > > > > include
> > > > > > > > > >>>>>> enough
> > > > > > > > > >>>>>>>>> details
> > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
> > > given
> > > > > > > > > >>>>>> producer
> > > > > > > > > >>>>>>>>> contain a
> > > > > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a
> > producer
> > > > > sends
> > > > > > > > > >>>>>> every
> > > > > > > > > >>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests
> > > could
> > > > > > > > > >>>> reach
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> > > rejection
> > > > > > > > > >>>> of the
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> requests.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
> > Yurchenko <
> > > > > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details
> > > about
> > > > > > > > > >>>>>>> transactions
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> queues,
> > > > > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic
> > > topics* is
> > > > > > > > > >>>> based
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> precise
> > > > > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> > > (mostly
> > > > > > > > > >>>>>>> producers,
> > > > > > > > > >>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > > > > > >>>> classes),
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition
> > > leaders
> > > > > with
> > > > > > > > > >>>>>> their
> > > > > > > > > >>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level (some details skipped)
> > workflow
> > > is
> > > > > the
> > > > > > > > > >>>>>>>>> following.
> > > > > > > > > >>>>>>>>>>>> When a
> > > > > > > > > >>>>>>>>>>>>>>> transactional Produce request is received by
> > > the
> > > > > > > > > >>>> broker:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> > > checks
> > > > > > > > > >>>> if a
> > > > > > > > > >>>>>>>>> non-empty
> > > > > > > > > >>>>>>>>>>>>>>> transaction is running for this partition.
> > > This is
> > > > > > > > > >>>> done
> > > > > > > > > >>>>>>> using
> > > > > > > > > >>>>>>>>> its
> > > > > > > > > >>>>>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > > > > >>>>>> (ProducerStateManager,
> > > > > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> > > about
> > > > > all
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> partitions
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include
> > them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > > > > transactional
> > > > > > > > > >>>>>>> checks.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the
> > > transactional
> > > > > > > > > >>>> data to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> update some of their state (for example, log
> > > the
> > > > > fact
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> > > offset).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the
> > transaction
> > > > > > > > > >>>> markers to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders
> > and
> > > lots
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>>>> important
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is
> > > local.
> > > > > The
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>>>>> challenge
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
> > > there
> > > > > > > > > >>>> are no
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
> > > need
> > > > > > > > > >>>> to be
> > > > > > > > > >>>>>>>>> globalized
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this
> > to
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>>>>> idempotent
> > > > > > > > > >>>>>>>>>>>>>> produce.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics*
> > > would
> > > > > > > > > >>>> look
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>>>>> similar:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > > > > > >>>> non-empty
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> > > classic
> > > > > > > > > >>>>>> topics,
> > > > > > > > > >>>>>>>>> this is
> > > > > > > > > >>>>>>>>>>>>>> checked
> > > > > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single
> > > RPC.
> > > > > > > > > >>>> Since a
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>> could
> > > > > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> > > epoch,
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> positive
> > > > > > > > > >>>>>>>>>>>> result of
> > > > > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> > > double
> > > > > > > > > >>>>>>> configured
> > > > > > > > > >>>>>>>>>>>> duration
> > > > > > > > > >>>>>>>>>>>>>> of a
> > > > > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > > > > informed
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the
> > transaction
> > > to
> > > > > > > > > >>>> include
> > > > > > > > > >>>>>>> them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the
> > > broker
> > > > > > > > > >>>> side.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data
> > > to the
> > > > > > > > > >>>>>> current
> > > > > > > > > >>>>>>>>> shared
> > > > > > > > > >>>>>>>>>>>> WAL
> > > > > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> > > transaction-related
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> Diskless
> > > > > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> like
> > > > > > > > > >>>>>>>>> in the
> > > > > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> > > > > errors
> > > > > > > > > >>>> like
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> > > > > batches
> > > > > > > > > >>>>>> could
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> rejected.
> > > > > > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage
> > > in
> > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>>>> segment
> > > > > > > > > >>>>>>>>>>>> file in
> > > > > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is
> > > preferable to
> > > > > > > > > >>>> doing
> > > > > > > > > >>>>>>> more
> > > > > > > > > >>>>>>>>>>>> network
> > > > > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > > > > segments
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> relatively
> > > > > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
> > informs
> > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
> > the
> > > > > > > > > >>>>>>> transaction is
> > > > > > > > > >>>>>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > > > > >>>> corresponding
> > > > > > > > > >>>>>>> logs
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only
> > on
> > > the
> > > > > > > > > >>>>>> metadata
> > > > > > > > > >>>>>>>>> level, no
> > > > > > > > > >>>>>>>>>>>>>>> actual control batches are written to any
> > file.
> > > > > They
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> dynamically
> > > > > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations.
> > We
> > > > > could
> > > > > > > > > >>>>>>>>> technically
> > > > > > > > > >>>>>>>>>>>> write
> > > > > > > > > >>>>>>>>>>>>>>> these control batches for real, but this
> > would
> > > mean
> > > > > > > > > >>>> extra
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> latency,
> > > > > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> save
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> now.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> > > write
> > > > > the
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> markers
> > > > > > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce
> > > flow, we
> > > > > > > > > >>>> get:
> > > > > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between
> > brokers
> > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch
> > > > > > > > > >>>>>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>>>>> when a new partition appear in the
> > > transaction. To
> > > > > > > > > >>>>>>> mitigate the
> > > > > > > > > >>>>>>>>>>>> impact of
> > > > > > > > > >>>>>>>>>>>>>>> them:
> > > > > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> > > Produce
> > > > > > > > > >>>>>>> request
> > > > > > > > > >>>>>>>>> will be
> > > > > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized
> > for
> > > > > > > > > >>>> fast
> > > > > > > > > >>>>>>>>> response to
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally
> > will
> > > > > > > > > >>>>>>> communicate
> > > > > > > > > >>>>>>>>> with a
> > > > > > > > > >>>>>>>>>>>>>>> single broker for the duration of the
> > > transaction
> > > > > > > > > >>>> further
> > > > > > > > > >>>>>>>>> reduces the
> > > > > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the
> > transaction
> > > > > > > > > >>>>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be
> > > able
> > > > > to
> > > > > > > > > >>>>>> span
> > > > > > > > > >>>>>>> both
> > > > > > > > > >>>>>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The share group coordination and management
> > is
> > > a
> > > > > > > > > >>>> side job
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > > > > > >>>> replicas,
> > > > > > > > > >>>>>>> physical
> > > > > > > > > >>>>>>>>>>>> storage
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > > > > consumers
> > > > > > > > > >>>>>>> (Fetch and
> > > > > > > > > >>>>>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > > > > affected.)
> > > > > > > > > >>>> We
> > > > > > > > > >>>>>>> don't
> > > > > > > > > >>>>>>>>> see any
> > > > > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> > > compatible
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>> share
> > > > > > > > > >>>>>>>>>>>> groups the
> > > > > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the
> > > code
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> we
> > > > > > > > > >>>>>>>>> don't
> > > > > > > > > >>>>>>>>>>>> expect
> > > > > > > > > >>>>>>>>>>>>>> any
> > > > > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading
> > routines
> > > are
> > > > > > > > > >>>> used
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>> used for
> > > > > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this
> > or
> > > it's
> > > > > > > > > >>>> too
> > > > > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design
> > > input
> > > > > > > > > >>>> on
> > > > > > > > > >>>>>>>>> KIP-1150.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> > > with a
> > > > > > > > > >>>> new
> > > > > > > > > >>>>>>>>> design. To
> > > > > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> KIP-405
> > > > > > > > > >>>>>>>>>>>> Tiered
> > > > > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced
> > > to a
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> topic
> > > > > > > > > >>>>>>>>>>>> to be
> > > > > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for
> > > the
> > > > > > > > > >>>> Batch
> > > > > > > > > >>>>>>>>> Coordinator
> > > > > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose
> > with
> > > > > > > > > >>>> Tiered
> > > > > > > > > >>>>>>> Storage
> > > > > > > > > >>>>>>>>>>>> plugin
> > > > > > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative
> > > data
> > > > > > > > > >>>>>> formats.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
> > local
> > > > > > > > > >>>> segments,
> > > > > > > > > >>>>>>>>> making use
> > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > > > > > >>>> zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered
> > > cache
> > > > > > > > > >>>>>>> elements,
> > > > > > > > > >>>>>>>>> do not
> > > > > > > > > >>>>>>>>>>>> need
> > > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > > > > > >>>> contacting
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified
> > > substantially,
> > > > > by
> > > > > > > > > >>>>>>> removing
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> previous
> > > > > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > > > > >>>> component, and
> > > > > > > > > >>>>>>>>> "object
> > > > > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > > > > > >>>> enabled by
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> Batch
> > > > > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
> > as
> > > the
> > > > > > > > > >>>>>> earlier
> > > > > > > > > >>>>>>>>> design,
> > > > > > > > > >>>>>>>>>>>> while
> > > > > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> ecosystem.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> > > feedback on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> new
> > > > > > > > > >>>>>>>>> design.
> > > > > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> > > support
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>> all
> > > > > > > > > >>>>>>>>> existing
> > > > > > > > > >>>>>>>>>>>>>> client
> > > > > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > > > > >>>> supporting
> > > > > > > > > >>>>>> APIs
> > > > > > > > > >>>>>>>>> like
> > > > > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> > > Siekierski
> > > > > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> > > current
> > > > > > > > > >>>>>>> status of
> > > > > > > > > >>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>> KIP?
> > > > > > > > > >>>>>>>>>>>>>> I
> > > > > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> > > past 2
> > > > > > > > > >>>>>>> months,
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>> vote got
> > > > > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have
> > been
> > > > > > > > > >>>>>> answered
> > > > > > > > > >>>>>>>>> since
> > > > > > > > > >>>>>>>>>>>> then.
> > > > > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > > > > >>>> activity
> > > > > > > > > >>>>>>> since
> > > > > > > > > >>>>>>>>> May.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> > > real
> > > > > > > > > >>>>>> choices
> > > > > > > > > >>>>>>>>> that can
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> > > being
> > > > > > > > > >>>> by
> > > > > > > > > >>>>>> far
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> dominant
> > > > > > > > > >>>>>>>>>>>>>> one
> > > > > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all
> > these
> > > are
> > > > > > > > > >>>>>>>>> standalone
> > > > > > > > > >>>>>>>>>>>>>> products.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
> > would
> > > > > > > > > >>>> hate to
> > > > > > > > > >>>>>>> see a
> > > > > > > > > >>>>>>>>>>>> stall. So
> > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
> > Colin
> > > > > > > > > >>>>>> McCabe -
> > > > > > > > > >>>>>>>>> have your
> > > > > > > > > >>>>>>>>>>>>>>> questions
> > > > > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I
> > > can
> > > > > > > > > >>>> do
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>>>> help? I'm
> > > > > > > > > >>>>>>>>>>>>>> deeply
> > > > > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> both?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > > > > > >>>>>> Kozlovski
> > > > > > > > > >>>>>>> <
> > > > > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>>>> (just a
> > > > > > > > > >>>>>>>>>>>> bunch
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > > > > >>>> example, I
> > > > > > > > > >>>>>>> was
> > > > > > > > > >>>>>>>>>>>> searching
> > > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while;
> > As
> > > > > > > > > >>>> far
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>> I
> > > > > > > > > >>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>> tell, it
> > > > > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
> > of
> > > > > > > > > >>>> having
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> root
> > > > > > > > > >>>>>>>>>>>> KIP be
> > > > > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> > > links
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>> sub-KIPs
> > > > > > > > > >>>>>>>>>>>> where
> > > > > > > > > >>>>>>>>>>>>>> were
> > > > > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think
> > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> good to
> > > > > > > > > >>>>>>>>>>>>>> create
> > > > > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> > > about
> > > > > > > > > >>>>>>>>> introducing a
> > > > > > > > > >>>>>>>>>>>> new
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > > > > >>>> Storage as
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> primary
> > > > > > > > > >>>>>>>>>>>>>>> source of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > > > > > >>>> decided
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> split it
> > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > > > > >>>> should aim
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > > > > > >>>>>> details
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> how to
> > > > > > > > > >>>>>>>>>>>>>>> implement
> > > > > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when
> > > we
> > > > > > > > > >>>>>>> discussed
> > > > > > > > > >>>>>>>>>>>> KRaft.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
> > really
> > > > > > > > > >>>> hard
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > >>>>>>>>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > > > > > >>>> (linked
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > > > > >>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> > > DISCUSS
> > > > > > > > > >>>>>>> threads
> > > > > > > > > >>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > > > > >>>> whether
> > > > > > > > > >>>>>>> Apache
> > > > > > > > > >>>>>>>>> Kafka
> > > > > > > > > >>>>>>>>>>>> wants
> > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > > > > >>>> authors
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> this KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
> >  +491715557497 |
> > > > > > > > > >>>>>>> aiven.io
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > > > > > >>>>>>> Valtonen,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Anatolii Popov
> > > > > Senior Software Developer, *Aiven OY*
> > > > > m: +358505126242
> > > > > w: aiven.io  e: anatolii.popov@aiven.io
> > > > > <https://www.facebook.com/aivencloud>
> > > > > <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Robin Han <ha...@gmail.com>]

Hi Greg,

Thanks for the reply

> Based on your description, the GET operations on the leader and followers
> must complete before offsets are assigned, adding Produce request latency
> that is not present in the KIP-1163 design.

Yes, the ProduceRequestLatency is 1 * PUT(In Broker N) + 1 GET(In Broker M)
+ 1 GET(Follower Synchronization).
- Compared with KIP-1163: Produce Latency has two more GET time costs;
Consume E2E Latency has one more GET time cost.
- Compared with KIP-1163: There is no need for coordination by a
centralized Coordinator, thus eliminating the delay caused by Coordinator
confirmation and avoiding the impact of Coordinator single-point failure on
the availability of the entire cluster.

In the design of Diskless Topic, I suspect that adding the time cost of two
more GET operations will not bring about an essential change (with P50
increasing from 500ms to 700ms).

> KIP-1163 also assumes that object storage is more reliable than disk
> storage, which is the case in the hyperscaler clouds. The parts of your
> design that make it compatible with low-reliability object storage become
> counterproductive when Diskless is used with a high-reliability object
> storage, such as re-introducing direct replication and cross-az costs.

It might be that my description is not accurate enough. The design is also
strongly dependent on object storage. Regarding the part about object
management and recycling, what I want to emphasize more is that the current
design does not require complex object metadata management, and objects are
just a temporary state.

> We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> in favor of a complete solution.

I understand that the main reasons why "KIP-1176 was rejected" are that it
did not solve the cross-AZ traffic costs for Produce, and that implementing
Active Segment Tiered Storage added a lot of complexity to metadata
management.
The design I mentioned can not only solve all cross-AZ traffic costs
including sending and replication (the core goal of KIP-1150), but also
does not add complex management of Object metadata, nor does it require
large-scale modifications to Apache Kafka.
It could be a good design that can be adopted by KIP-1150.

Greg Harris via dev <de...@kafka.apache.org> 于2026年1月21日周三 03:20写道：

> Hi Robin,
>
> Thanks for your design sketch.
>
> Based on your description, the GET operations on the leader and followers
> must complete before offsets are assigned, adding Produce request latency
> that is not present in the KIP-1163 design.
>
> KIP-1163 also assumes that object storage is more reliable than disk
> storage, which is the case in the hyperscaler clouds. The parts of your
> design that make it compatible with low-reliability object storage become
> counterproductive when Diskless is used with a high-reliability object
> storage, such as re-introducing direct replication and cross-az costs.
>
> We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> in favor of a complete solution. The folks who championed that proposal are
> now working with us on KIP-1163, and are currently designing Compaction in
> Tiered Storage to make Diskless a complete solution for compacted topics as
> well. Similarly, we would appreciate your contributions to the design of
> KIP-1163, and your vote on the motivational KIP-1150.
>
> Thanks!
>
> On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>
> > Hi Jun,
> >
> > Here are our answers to your questions and comments:
> >
> > > JR10. "Even in the case where network traffic is not accounted; we
> > believe
> > > that the operational benefits of Diskless topics are still appealing to
> > > Kafka users."
> > > Could you list other benefits such as better scalability and
> durability?
> >
> > Thank you, good point, added this.
> >
> > > JR11. "AWS charges 0.02$ per GiB"
> > > The object store is not free and has a different cost model based on
> > > requests. It would be useful summarize the tradeoff on cost and
> latency,
> > > and when diskless topics are cost effective.
> >
> > Yes, there is a per-request cost for object storage which is not directly
> > comparable to inter-zone transfer costs. We have found that the design in
> > KIP-1163 amortizes those request costs such that they are insignificant
> > compared to the $0.02/gb cost which is called out in KIP-1150.
> > We currently have very precise cost models that were put together for an
> > earlier iteration of the Diskless topics. These are somewhat stale and
> > inaccurate for the latest design, so we will work in the background to
> post
> > concrete numbers in KIP-1163 before voting on that proposal. Would this
> be
> > OK?
> >
> > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > been
> > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > included in KIP-1163.
> >
> > Thanks for calling out. We have updated KIP-1164 with the latest design.
> > It's missing public interfaces, but we think it may be a bit too early to
> > define them as some rounds of discussion is ahead, with potential to
> change
> > the design. But otherwise the KIP is good for the first review.
> > We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
> >
> > Best,
> > Ivan
> >
> > On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> > > Hi, Anatolii,
> > >
> > > Thanks for the updated KIP. Looks good to me overall. A few more
> > comments.
> > >
> > > JR10. "Even in the case where network traffic is not accounted; we
> > believe
> > > that the operational benefits of Diskless topics are still appealing to
> > > Kafka users."
> > > Could you list other benefits such as better scalability and
> durability?
> > >
> > > JR11. "AWS charges 0.02$ per GiB"
> > > The object store is not free and has a different cost model based on
> > > requests. It would be useful summarize the tradeoff on cost and
> latency,
> > > and when diskless topics are cost effective.
> > >
> > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > been
> > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > included in KIP-1163.
> > >
> > > Jun
> > >
> > > On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> > dev@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thank you for the extensive feedback.
> > > >
> > > > We have substantially updated the KIP to address the points raised.
> > Given
> > > > the scope of these changes, we have started a new VOTE thread to
> > restart
> > > > the voting process cleanly.
> > > >
> > > > You can find the new thread here:
> > > > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> > > >
> > > > Best,
> > > > Anatolii
> > > >
> > > > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > > > dev@kafka.apache.org> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We (the team at Slack) have been following the recent discussion
> > > > regarding
> > > > > the scope and timeline of KIP-1150. We agree with the community
> > sentiment
> > > > > that Diskless Topics represents the right long-term architecture
> for
> > > > Kafka
> > > > > in the cloud, but we also recognize the valid concerns raised
> > regarding
> > > > the
> > > > > engineering resources required to deliver such an ambitious change.
> > > > >
> > > > > To help address these concerns and accelerate the timeline, we are
> > happy
> > > > to
> > > > > announce that we are partnering with the KIP-1150 authors to
> > co-develop
> > > > > this feature.
> > > > >
> > > > > We previously proposed KIP-1176: Tiered Storage for Active Log
> > Segments
> > > > to
> > > > > solve similar problems. However, rather than fragmenting the
> > community's
> > > > > efforts across competing designs, we have decided to withdraw
> > KIP-1176
> > > > and
> > > > > consolidate our engineering resources behind KIP-1150.
> > > > >
> > > > > To start, we plan to take ownership of Compaction for Tiered
> Storage.
> > > > This
> > > > > has long been a missing feature in KIP-405, and it becomes critical
> > in a
> > > > > Diskless architecture where long-lived data is obligated to be
> > tiered. By
> > > > > driving this related prerequisite feature, we hope to allow for
> > faster
> > > > > delivery of KIP-1150.
> > > > >
> > > > > We are excited to collaborate on this to ensure a robust and timely
> > > > > delivery for the community.
> > > > >
> > > > > Best,
> > > > > Tom & Henry
> > > > >
> > > > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > Thanks for sharing the meeting notes.
> > > > > > I agree we should keep polishing the contents of 1150 & high
> level
> > > > design
> > > > > > in 1163 to prepare for a vote.
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > > > <greg.harris@aiven.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > There was a video call between myself, Ivan Yurchenko, Jun Rao,
> > and
> > > > > > Andrew
> > > > > > > Schofield pertaining to KIP-1150. Here are the notes from that
> > > > meeting:
> > > > > > >
> > > > > > > Ivan: What is the future state of Kafka in this area, in 5
> years?
> > > > > > > Jun: Do we want something more cloud native? Yes, started with
> > Tiered
> > > > > > > Storage. If there’s a better way, we should explore it. In the
> > long
> > > > > term
> > > > > > > this will be useful
> > > > > > > Because Kafka is used so widely, we need to make sure
> everything
> > we
> > > > add
> > > > > > is
> > > > > > > for the long term and for everyone, not just for a single
> > company.
> > > > > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > > > > something
> > > > > > > that’s high quality/lasts/maintainable, and can work with all
> > > > existing
> > > > > > > capabilities.
> > > > > > > If both 1150 and 1176 proceed at the same time, it’s confusing.
> > They
> > > > > > > overlap, but Diskless is more ambitious.
> > > > > > > If both KIPs are being seriously worked on, then we don’t
> really
> > need
> > > > > > both,
> > > > > > > because Diskless clearly is better. Having multiple will
> confuse
> > > > > people.
> > > > > > It
> > > > > > > will duplicate some of the effort.
> > > > > > > If we want diskless ultimately, what is the short term
> strategy,
> > to
> > > > get
> > > > > > > some early wins first?
> > > > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > > > Andrew: Eventually the architecture will change substantially,
> > it may
> > > > > not
> > > > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > > > Greg: We all agree on having a high quality feature merged
> > upstream,
> > > > > and
> > > > > > > supporting all APIs
> > > > > > > Jun: We should try and keep things simple, but there is some
> > minimum
> > > > > > > complexity needed.
> > > > > > > When doing the short term changes (1176), it doesn’t really
> > progress
> > > > in
> > > > > > > changing to a more modern architecture.
> > > > > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > > > > Jun: The danger of only applying changes to some part of the
> > API, you
> > > > > set
> > > > > > > the precedence that you only have to implement part of the API.
> > > > > > Supporting
> > > > > > > the full API set should be a minimum requirement.
> > > > > > > Andrew: When we started Kraft, how much did we know the design?
> > > > > > > Jun: For Kraft we didn’t really know much about the migration,
> > but
> > > > the
> > > > > > > high-level was clear.
> > > > > > > Greg: Is 1150 votable in its current state?
> > > > > > > Jun: 1150 should promise to support all APIs. It doesn’t have
> to
> > have
> > > > > all
> > > > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > > > We do need some high-level design enough to give confidence
> that
> > the
> > > > > > > promise is able to be fulfilled.
> > > > > > > Greg: Is the draft version in 1163 enough detail or is more
> > needed?
> > > > > > > Jun: We need to agree on the core design, such as leaderless
> > etc. And
> > > > > how
> > > > > > > the APIs will be supported.
> > > > > > > Greg: Okay we can include these things, and provide a sketch of
> > how
> > > > the
> > > > > > > other leader-based features operate.
> > > > > > > Jun: Yeah if at a high level the sketch appears to work, we can
> > > > approve
> > > > > > > that functionality.
> > > > > > > Are you committed to doing the more involved and big project?
> > > > > > > Greg: Yes, we’re committed to the 1163 design and can’t really
> > accept
> > > > > > 1176.
> > > > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > > > Greg: We’ll push internally for resources, and use the
> community
> > > > > > sentiment
> > > > > > > to motivate Aiven.
> > > > > > > How far into the future should we look? What sort of scale?
> > > > > > > Jun: As long as there’s a path forward, and we’re not closing
> off
> > > > > future
> > > > > > > improvements, we can figure out how to handle a larger scale
> > when it
> > > > > > > arises.
> > > > > > > Greg: Random replica placement is very harmful, can we
> recommend
> > > > users
> > > > > to
> > > > > > > use an external tool like CruiseControl?
> > > > > > > Jun: Not everyone uses CruiseControl, we would probably need
> some
> > > > > > solution
> > > > > > > for this out of the box
> > > > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > > > Jun: Out-of-box experience should be good, good to allow other
> > > > > > > implementations
> > > > > > > Greg: But it could hurt Kafka feature/upgrade velocity when we
> > wait
> > > > for
> > > > > > > plugin providers to implement it
> > > > > > > Ivan: We imagined that maybe cloud hyperscalers could implement
> > it
> > > > with
> > > > > > > e.g. dynamodb
> > > > > > > Greg: Could we bake more details of the different providers
> into
> > > > Kafka,
> > > > > > or
> > > > > > > does it still make sense for it to be pluggable?
> > > > > > > Jun: Make it whatever is easiest to roll out and add new
> clients
> > > > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > > > Ivan: The vote is already open, we’re not too pressed for time.
> > We’ll
> > > > > go
> > > > > > > improve the 1163 design and communication.
> > > > > > > Is 1176 a competing design? Someone will ask.
> > > > > > > Jun: If we are seriously working on something more ambitious,
> > yeah we
> > > > > > > shouldn’t do the stop-gap solution.
> > > > > > > It’s diverting review resources. If we can get the short term
> > thing
> > > > in
> > > > > > 1yr
> > > > > > > but Diskless solution is 2y it makes sense to go for Diskless.
> If
> > > > it’s
> > > > > > 5yr,
> > > > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case.
> Should
> > we
> > > > > > > explicitly exclude 1176?
> > > > > > > Andrew: Put your arms around the feature set you actually want,
> > and
> > > > use
> > > > > > > that to rule out 1176.
> > > > > > > Probably don’t need -1 votes, most likely KIPs just don’t
> receive
> > > > > votes.
> > > > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > > > Jun: Satish posted meeting notes regularly, we should do the
> > same.
> > > > > > >
> > > > > > > To summarize, we will be polishing the contents of 1150 & high
> > level
> > > > > > design
> > > > > > > in 1163 to prepare for a vote.
> > > > > > > We believe that the community should select the feature set of
> > 1150
> > > > to
> > > > > > > fully eliminate producer cross-zone costs, and make the
> > investment
> > > > in a
> > > > > > > high quality Diskless Topics implementation rather than in
> > stop-gap
> > > > > > > solutions.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
> wrote:
> > > > > > >
> > > > > > > > This may be a tangent, but we needed to offload storage off
> of
> > > > Kafka
> > > > > > into
> > > > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> > mostly
> > > > > > > > ephemeral broker that can come and go as it pleases. Be that
> > > > scaling
> > > > > or
> > > > > > > > outage. Disks can be destroyed and recreated at will, we
> still
> > > > retain
> > > > > > > data
> > > > > > > > and use broker for just that, brokering messages. Not only
> > that, we
> > > > > > > reduced
> > > > > > > > the requirement on the actual Kafka resources by reducing the
> > size
> > > > > of a
> > > > > > > > payload via a claim check pattern. Maybe this is an
> > anti–pattern,
> > > > but
> > > > > > it
> > > > > > > is
> > > > > > > > super fast and highly cost efficient. We reworked
> > ProducerRequest
> > > > to
> > > > > > > allow
> > > > > > > > plugins. We added a custom http plugin that submits every
> > request
> > > > > via a
> > > > > > > > persisted connection to a microservice. Microservice stores
> the
> > > > > payload
> > > > > > > and
> > > > > > > > returns a tiny json metadata object,a claim check, that can
> be
> > used
> > > > > to
> > > > > > > find
> > > > > > > > the actual data. Think of it as zipping the payload. This
> claim
> > > > check
> > > > > > > > metadata traverses the pipelines with consumers using the
> urls
> > in
> > > > > > > metadata
> > > > > > > > to pull what they need. Think unzipping. This allowed us to
> > also
> > > > pull
> > > > > > > ONLY
> > > > > > > > the data that we need in graphql like manner. So if you have
> a
> > 100K
> > > > > > json
> > > > > > > > payload and you need only a subsection, you can pull that by
> > > > > jmespath.
> > > > > > > When
> > > > > > > > you have multiple consumer groups yanking down huge payloads
> > it is
> > > > > > > > cumbersome on the broker. When you have the same consumer
> > groups
> > > > > > yanking
> > > > > > > > down a claim check, and then going out of band directly to
> the
> > > > source
> > > > > > of
> > > > > > > > truth, the broker has some breathing room. Obviously our
> > > > microservice
> > > > > > > does
> > > > > > > > not go directly to the cloud storage, as that would be too
> > slow. It
> > > > > > > stores
> > > > > > > > the payload in high speed memory cache and returns asap. That
> > > > memory
> > > > > is
> > > > > > > > eventually persisted into S3. The retrieval goest against the
> > cache
> > > > > > > first,
> > > > > > > > then against the S3. Overall a rather cheappy and zippy
> > solution. I
> > > > > > tried
> > > > > > > > proposing the KIP for this, but there was no excitement.
> Check
> > this
> > > > > > out:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > > > >
> > > > > > > >
> > > > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao
> <jun@confluent.io.INVALID
> > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi, Andrew,
> > > > > > > > >
> > > > > > > > > If we want to focus only on reducing cross-zone replication
> > > > costs,
> > > > > > > there
> > > > > > > > is
> > > > > > > > > an alternative design in the KIP-1176 discussion thread
> that
> > > > seems
> > > > > > > > simpler
> > > > > > > > > than the proposal here. I am copying the outline of that
> > approach
> > > > > > > below.
> > > > > > > > >
> > > > > > > > > 1. A new leader is elected.
> > > > > > > > > 2. Leader maintains a first tiered offset, which is
> > initialized
> > > > to
> > > > > > log
> > > > > > > > end
> > > > > > > > > offset.
> > > > > > > > > 3. Leader writes produced data from the client to local
> log.
> > > > > > > > > 4. Leader uploads produced data from all local logs as a
> > combined
> > > > > > > object
> > > > > > > > > 5. Leader stores the metadata for the combined object in
> > memory.
> > > > > > > > > 6. If a follower fetch request has an offset >= first
> tiered
> > > > > offset,
> > > > > > > the
> > > > > > > > > metadata for the corresponding combined object is returned.
> > > > > > Otherwise,
> > > > > > > > the
> > > > > > > > > local data is returned.
> > > > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > > > >
> > > > > > > > > It's still a bit unnatural, but it could work.
> > > > > > > > >
> > > > > > > > > Hi, Ivan,
> > > > > > > > >
> > > > > > > > > Are you still committed to proceeding with the original
> > design of
> > > > > > > > KIP-1150?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > > > andrew_schofield@live.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >> I’ve been following KIP-1150 and friends for a while. I’m
> > going
> > > > to
> > > > > > > jump
> > > > > > > > >> into the discussions too.
> > > > > > > > >>
> > > > > > > > >> Looking back at Jack Vanlightly’s message, I am not quite
> so
> > > > > > convinced
> > > > > > > > >> that it’s a kind of fork in the road. The primary aim of
> the
> > > > > effort
> > > > > > is
> > > > > > > > to
> > > > > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > > > > prohibitively
> > > > > > > > >> expensive to use on cloud storage. I think it would be
> > entirely
> > > > > > viable
> > > > > > > > to
> > > > > > > > >> prioritise code reuse for an initial implementation of
> > diskless
> > > > > > > topics,
> > > > > > > > and
> > > > > > > > >> we could still have a more cloud-native design in the
> > future.
> > > > It’s
> > > > > > > hard
> > > > > > > > to
> > > > > > > > >> predict what the community will prioritise in the future.
> > > > > > > > >>
> > > > > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> > > > support
> > > > > > > > >> leaderless produce requests, first writing WAL segments
> into
> > > > > object
> > > > > > > > >> storage, and then using the regular partition leaders to
> > > > sequence
> > > > > > the
> > > > > > > > >> records. The active log segment for a diskless topic will
> > > > > initially
> > > > > > > > contain
> > > > > > > > >> batch coordinates rather than record batches. The batch
> > > > > coordinates
> > > > > > > can
> > > > > > > > be
> > > > > > > > >> resolved from WAL segments for consumers, and also in
> order
> > to
> > > > > > prepare
> > > > > > > > log
> > > > > > > > >> segments for uploading to tiered storage. Jun is probably
> > > > correct
> > > > > > that
> > > > > > > > we
> > > > > > > > >> need a more frequent object merging process than tiered
> > storage
> > > > > > > > provides.
> > > > > > > > >> This is just the transition from write-optimised WAL
> > segments to
> > > > > > > > >> read-optimised tiered segments, and all of the object
> > > > > storage-based
> > > > > > > > >> implementations of Kafka that I’m aware of do this
> > > > rearrangement.
> > > > > > But
> > > > > > > > >> perhaps this more frequent object merging is a pre-GA
> > > > improvement,
> > > > > > > > rather
> > > > > > > > >> than a strict requirement for an initial implementation
> for
> > > > early
> > > > > > > access
> > > > > > > > >> use.
> > > > > > > > >>
> > > > > > > > >> For zone-aligned share consumers, the share group assignor
> > is
> > > > > > intended
> > > > > > > > to
> > > > > > > > >> be rack-aware. Consumers should be assigned to partitions
> > with
> > > > > > leaders
> > > > > > > > in
> > > > > > > > >> their zone. The simple assignor is not rack-aware, but it
> > easily
> > > > > > could
> > > > > > > > be
> > > > > > > > >> or we could have a rack-aware assignor.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Andrew
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao
> <jun@confluent.io.INVALID
> > >
> > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> Hi, Ivan,
> > > > > > > > >>>
> > > > > > > > >>> Thanks for the reply.
> > > > > > > > >>>
> > > > > > > > >>> "As I understand, you’re speaking about locally
> > materialized
> > > > > > > segments.
> > > > > > > > >> They
> > > > > > > > >>> will indeed consume some IOPS. See them as a cache that
> > could
> > > > > > always
> > > > > > > be
> > > > > > > > >>> restored from the remote storage. While it’s not ideal,
> > it's
> > > > > still
> > > > > > OK
> > > > > > > > to
> > > > > > > > >>> lose data in them due to a machine crash, for example.
> > Because
> > > > of
> > > > > > > this,
> > > > > > > > >> we
> > > > > > > > >>> can avoid explicit flushing on local materialized
> segments
> > at
> > > > all
> > > > > > and
> > > > > > > > let
> > > > > > > > >>> the file system and page cache figure out when to flush
> > > > > optimally.
> > > > > > > This
> > > > > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > dramatically,
> > > > > > > > >>> depending on throughput for each partition. We, of
> course,
> > > > > continue
> > > > > > > > >>> flushing the metadata segments as before."
> > > > > > > > >>>
> > > > > > > > >>> If we have a mix of classic and diskless topics on the
> same
> > > > > broker,
> > > > > > > > it's
> > > > > > > > >>> important that the classic topics' data is flushed to
> disk
> > as
> > > > > > quickly
> > > > > > > > as
> > > > > > > > >>> possible. To achieve this, users typically set
> > > > > > dirty_expire_centisecs
> > > > > > > > in
> > > > > > > > >>> the kernel based on the number of available disk IOPS.
> > Once you
> > > > > set
> > > > > > > > this
> > > > > > > > >>> number, it applies to all dirty files, including the
> cached
> > > > data
> > > > > in
> > > > > > > > >>> diskless topics. So, if there are more files actively
> > > > > accumulating
> > > > > > > > data,
> > > > > > > > >>> the flush frequency and therefore RPO is reduced for
> > classic
> > > > > > topics.
> > > > > > > > >>>
> > > > > > > > >>> "We should have mentioned this explicitly, but this step,
> > in
> > > > > fact,
> > > > > > > > >> remains
> > > > > > > > >>> in the form of segments offloading to tiered storage.
> When
> > we
> > > > > > > assemble
> > > > > > > > a
> > > > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> > effectively
> > > > > > doing
> > > > > > > > >>> metadata compaction: replacing a big number of pieces of
> > > > metadata
> > > > > > > about
> > > > > > > > >>> individual batches with a single record in
> > > > > __remote_log_metadata."
> > > > > > > > >>>
> > > > > > > > >>> The object merging in tier storage typically only kicks
> in
> > > > after
> > > > > a
> > > > > > > few
> > > > > > > > >>> hours. The impact is (1) the amount of accumulated
> > metadata is
> > > > > > still
> > > > > > > > >> quite
> > > > > > > > >>> large; (2) there are many small objects, leading to poor
> > read
> > > > > > > > >> performance.
> > > > > > > > >>> I think we need a more frequent object merging process
> than
> > > > tier
> > > > > > > > storage
> > > > > > > > >>> provides.
> > > > > > > > >>>
> > > > > > > > >>> Jun
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > > > ivan@ivanyu.me>
> > > > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > > > >>>>
> > > > > > > > >>>> Thank you for your messages.
> > > > > > > > >>>>
> > > > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > > > >>>>
> > > > > > > > >>>>> First, it degrades the durability.
> > > > > > > > >>>>> For each partition, now there are two files being
> > actively
> > > > > > written
> > > > > > > > at a
> > > > > > > > >>>>> given point of time, one for the data and another for
> the
> > > > > > metadata.
> > > > > > > > >>>>> Flushing each file requires a separate IO. If the disk
> > has 1K
> > > > > > IOPS
> > > > > > > > and
> > > > > > > > >> we
> > > > > > > > >>>>> have 5K partitions in a broker, currently we can afford
> > to
> > > > > flush
> > > > > > > each
> > > > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5
> > seconds. If
> > > > we
> > > > > > > > double
> > > > > > > > >>>> the
> > > > > > > > >>>>> number of files per partition, we can only flush each
> > > > partition
> > > > > > > every
> > > > > > > > >> 10
> > > > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > > > >>>>
> > > > > > > > >>>> As I understand, you’re speaking about locally
> > materialized
> > > > > > > segments.
> > > > > > > > >> They
> > > > > > > > >>>> will indeed consume some IOPS. See them as a cache that
> > could
> > > > > > always
> > > > > > > > be
> > > > > > > > >>>> restored from the remote storage. While it’s not ideal,
> > it's
> > > > > still
> > > > > > > OK
> > > > > > > > to
> > > > > > > > >>>> lose data in them due to a machine crash, for example.
> > Because
> > > > > of
> > > > > > > > this,
> > > > > > > > >> we
> > > > > > > > >>>> can avoid explicit flushing on local materialized
> > segments at
> > > > > all
> > > > > > > and
> > > > > > > > >> let
> > > > > > > > >>>> the file system and page cache figure out when to flush
> > > > > optimally.
> > > > > > > > This
> > > > > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > dramatically,
> > > > > > > > >>>> depending on throughput for each partition. We, of
> course,
> > > > > > continue
> > > > > > > > >>>> flushing the metadata segments as before.
> > > > > > > > >>>>
> > > > > > > > >>>> It’s worth making a note on caching. I think nobody will
> > > > > disagree
> > > > > > > that
> > > > > > > > >>>> doing direct reads from remote storage every time a
> batch
> > is
> > > > > > > requested
> > > > > > > > >> by a
> > > > > > > > >>>> consumer will not be practical neither from the
> > performance
> > > > nor
> > > > > > from
> > > > > > > > the
> > > > > > > > >>>> economy point of view. We need a way to keep the number
> > of GET
> > > > > > > > requests
> > > > > > > > >>>> down. There are multiple options, for example:
> > > > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > > > >>>> 2. Local in-memory caching. Comes with less network
> > chattiness
> > > > > and
> > > > > > > > >> works
> > > > > > > > >>>> well if we have more or less stable brokers to consume
> > from.
> > > > > > > > >>>> 3. Materialization of diskless logs on local disk. Way
> > lower
> > > > > > impact
> > > > > > > on
> > > > > > > > >>>> RAM and also requires stable brokers for consumption
> > (using
> > > > just
> > > > > > > > >> assigned
> > > > > > > > >>>> replicas will probably work well).
> > > > > > > > >>>>
> > > > > > > > >>>> Materialization is one of possible options, but we can
> > choose
> > > > > > > another
> > > > > > > > >> one.
> > > > > > > > >>>> However, we will have this dilemma regardless of whether
> > we
> > > > have
> > > > > > an
> > > > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > > > >>>>
> > > > > > > > >>>>> Second, if we ever need this
> > > > > > > > >>>>> metadata somewhere else, say in the WAL file manager,
> the
> > > > > > consumer
> > > > > > > > >> needs
> > > > > > > > >>>> to
> > > > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > > > inefficient.
> > > > > > > > The
> > > > > > > > >>>>> actual benefit of this approach is also questionable.
> On
> > the
> > > > > > > surface,
> > > > > > > > >> it
> > > > > > > > >>>>> might seem that we could reduce the number of lines
> that
> > need
> > > > > to
> > > > > > be
> > > > > > > > >>>> changed
> > > > > > > > >>>>> for this KIP. However, the changes are quite intrusive
> > to the
> > > > > > > classic
> > > > > > > > >>>>> partition's code path and will probably make the code
> > base
> > > > > harder
> > > > > > > to
> > > > > > > > >>>>> maintain in the long run. I like the original approach
> > based
> > > > on
> > > > > > the
> > > > > > > > >> batch
> > > > > > > > >>>>> coordinator much better than this one. We could
> probably
> > > > > refactor
> > > > > > > the
> > > > > > > > >>>>> producer state code so that it could be reused in the
> > batch
> > > > > > > > >> coordinator.
> > > > > > > > >>>>
> > > > > > > > >>>> It’s hard to disagree with this. The explicit
> coordinator
> > is
> > > > > more
> > > > > > a
> > > > > > > > side
> > > > > > > > >>>> thing, while coordinator-less approach is more about
> > extending
> > > > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > > > >>>>
> > > > > > > > >>>>> Thanks for addressing the concerns on the number of
> RPCs
> > in
> > > > the
> > > > > > > > produce
> > > > > > > > >>>>> path. I agree that with the metadata crafting
> mechanism,
> > we
> > > > > could
> > > > > > > > >>>> mitigate
> > > > > > > > >>>>> the PRC problem. However, since we now require the
> > metadata
> > > > to
> > > > > be
> > > > > > > > >>>>> collocated with the data on the same set of brokers,
> it's
> > > > weird
> > > > > > > that
> > > > > > > > >> they
> > > > > > > > >>>>> are now managed by different mechanisms. The data
> > assignment
> > > > > now
> > > > > > > uses
> > > > > > > > >> the
> > > > > > > > >>>>> metadata crafting mechanism, but the metadata is stored
> > in
> > > > the
> > > > > > > > classic
> > > > > > > > >>>>> partition using its own assignment strategy. It will be
> > > > > > complicated
> > > > > > > > to
> > > > > > > > >>>> keep
> > > > > > > > >>>>> them collocated.
> > > > > > > > >>>>
> > > > > > > > >>>> I would like to note that the metadata crafting is
> needed
> > only
> > > > > to
> > > > > > > tell
> > > > > > > > >>>> producers which brokers they should send Produce
> requests
> > to,
> > > > > but
> > > > > > > data
> > > > > > > > >> (as
> > > > > > > > >>>> in “locally materialized log”) is located on partition
> > > > replicas,
> > > > > > > i.e.
> > > > > > > > >>>> automatically co-located with metadata.
> > > > > > > > >>>>
> > > > > > > > >>>> As a side note, it would probably be better that instead
> > of
> > > > > > > implicitly
> > > > > > > > >>>> crafting partition metadata, we extend the metadata
> > protocol
> > > > so
> > > > > > that
> > > > > > > > for
> > > > > > > > >>>> diskless partitions we return not only the leader and
> > > > replicas,
> > > > > > but
> > > > > > > > also
> > > > > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > > > > performance
> > > > > > > > and
> > > > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > > > >>>>
> > > > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > > > compaction/merging
> > > > > > > > >>>>> step.
> > > > > > > > >>>>
> > > > > > > > >>>> We should have mentioned this explicitly, but this step,
> > in
> > > > > fact,
> > > > > > > > >> remains
> > > > > > > > >>>> in the form of segments offloading to tiered storage.
> > When we
> > > > > > > > assemble a
> > > > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > > > effectively
> > > > > > > doing
> > > > > > > > >>>> metadata compaction: replacing a big number of pieces of
> > > > > metadata
> > > > > > > > about
> > > > > > > > >>>> individual batches with a single record in
> > > > > __remote_log_metadata.
> > > > > > > > >>>>
> > > > > > > > >>>> We could create a Diskless-specific merging mechanism
> > instead
> > > > if
> > > > > > > > needed.
> > > > > > > > >>>> It’s rather easy with the explicit coordinator approach.
> > With
> > > > > the
> > > > > > > > >>>> coordinator-less approach, this would probably be a bit
> > more
> > > > > > tricky
> > > > > > > > >>>> (rewriting the tail of the log by the leader +
> replicating
> > > > this
> > > > > > > change
> > > > > > > > >>>> reliably).
> > > > > > > > >>>>
> > > > > > > > >>>>> I see a tendency toward primarily optimizing for the
> > fewest
> > > > > code
> > > > > > > > >> changes
> > > > > > > > >>>> in
> > > > > > > > >>>>> the KIP. Instead, our primary goal should be a clean
> > design
> > > > > that
> > > > > > > can
> > > > > > > > >> last
> > > > > > > > >>>>> for the long term.
> > > > > > > > >>>>
> > > > > > > > >>>> Yes, totally agree.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Luke,
> > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > queue is
> > > > > > > because
> > > > > > > > >> of
> > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> we
> > > > only
> > > > > > > focus
> > > > > > > > on
> > > > > > > > >>>> the
> > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > to/from
> > > > > the
> > > > > > > > >> remote
> > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > >>>>
> > > > > > > > >>>> After some evolution of the original proposal,
> leaderless
> > is
> > > > now
> > > > > > > > >> limited.
> > > > > > > > >>>> We only need to be able to accept Produce requests on
> more
> > > > than
> > > > > > one
> > > > > > > > >> broker
> > > > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get
> it
> > > > right
> > > > > > > that
> > > > > > > > >> you
> > > > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Let’s now look at this problem from a higher level, as
> > Jack
> > > > > > > proposed.
> > > > > > > > As
> > > > > > > > >>>> it was said, the big choice we need to make is whether
> we
> > 1)
> > > > > > create
> > > > > > > an
> > > > > > > > >>>> explicit batch coordinator; or 2) go for the
> > coordinator-less
> > > > > > > > approach,
> > > > > > > > >>>> where each diskless partition is managed by its leader
> as
> > in
> > > > > > classic
> > > > > > > > >> topics.
> > > > > > > > >>>>
> > > > > > > > >>>> If we try to compare the two approaches:
> > > > > > > > >>>>
> > > > > > > > >>>> Pluggability:
> > > > > > > > >>>> - Explicit coordinator: Possible. For example, some
> > setups may
> > > > > > > benefit
> > > > > > > > >>>> from batch metadata being stored in a cloud database
> > (such as
> > > > > AWS
> > > > > > > > >> DynamoDB
> > > > > > > > >>>> or GCP Spanner).
> > > > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > > > >>>>
> > > > > > > > >>>> Scalability and fault tolerance:
> > > > > > > > >>>> - Explicit coordinator: Depends on the implementation
> and
> > it’s
> > > > > > also
> > > > > > > > >>>> necessary to actively work for it.
> > > > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics.
> > Scaling is
> > > > > > done
> > > > > > > by
> > > > > > > > >>>> partition placement, partitions could fail
> independently.
> > > > > > > > >>>>
> > > > > > > > >>>> Separation of concerns:
> > > > > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > > > > independent
> > > > > > > > >> from
> > > > > > > > >>>> classic topics in terms of code and workflows. For
> > example,
> > > > the
> > > > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > > > mechanism
> > > > > > > could
> > > > > > > > >> be
> > > > > > > > >>>> relatively simply implemented with it. As a flip side of
> > this,
> > > > > > some
> > > > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > > > bringing
> > > > > > > > diskless
> > > > > > > > >>>> closer to classic topics. Some code paths and workflows
> > could
> > > > be
> > > > > > > more
> > > > > > > > >>>> straightforwardly reused, but they will inevitably have
> > to be
> > > > > > > adapted
> > > > > > > > to
> > > > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > > > >>>>
> > > > > > > > >>>> Cloud-nativeness. This is a vague concept, also related
> > to the
> > > > > > > > previous,
> > > > > > > > >>>> but let’s try:
> > > > > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > > > > separately
> > > > > > > > makes
> > > > > > > > >>>> it easier for brokers to take different roles, be purely
> > > > > stateless
> > > > > > > if
> > > > > > > > >>>> needed, etc.
> > > > > > > > >>>> - Coordinator-less: Less so. Something could be achieved
> > with
> > > > > > > creative
> > > > > > > > >>>> partition placement, but not much.
> > > > > > > > >>>>
> > > > > > > > >>>> Both seem to have their pros and cons. However,
> answering
> > > > Jack’s
> > > > > > > > >> question,
> > > > > > > > >>>> the explicit coordinator approach may indeed lead to a
> > more
> > > > > > flexible
> > > > > > > > >> design.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> The purpose of this deviation in the discussion was to
> > > > receive a
> > > > > > > > >>>> preliminary community evaluation of the coordinator-less
> > > > > approach
> > > > > > > > >> without
> > > > > > > > >>>> taking on the task of writing a separate KIP and fitting
> > it in
> > > > > the
> > > > > > > > >> system
> > > > > > > > >>>> of KIP-1150 and its children. We’re open to stopping it
> > and
> > > > > > getting
> > > > > > > > >> back to
> > > > > > > > >>>> working out the coordinator design if the community
> > doesn’t
> > > > > favor
> > > > > > > the
> > > > > > > > >>>> proposed approach.
> > > > > > > > >>>>
> > > > > > > > >>>> Best,
> > > > > > > > >>>> Ivan and Diskless team
> > > > > > > > >>>>
> > > > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > > > >>>>> Hi Ivan,
> > > > > > > > >>>>>
> > > > > > > > >>>>> As Jun pointed out, the updated design seems to have
> some
> > > > > > > > shortcomings
> > > > > > > > >>>>> although it simplifies the implementation.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > queue is
> > > > > > > because
> > > > > > > > >> of
> > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> we
> > > > only
> > > > > > > focus
> > > > > > > > on
> > > > > > > > >>>> the
> > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > to/from
> > > > > the
> > > > > > > > >> remote
> > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thank you,
> > > > > > > > >>>>> Luke
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Hi, Ivan,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks for the explanation.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> "we write the reference to the WAL file with the batch
> > data"
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I understand the approach now, but I think it is a
> hacky
> > > > one.
> > > > > > > There
> > > > > > > > >> are
> > > > > > > > >>>>>> multiple short comings with this design. First, it
> > degrades
> > > > > the
> > > > > > > > >>>> durability.
> > > > > > > > >>>>>> For each partition, now there are two files being
> > actively
> > > > > > written
> > > > > > > > at
> > > > > > > > >> a
> > > > > > > > >>>>>> given point of time, one for the data and another for
> > the
> > > > > > > metadata.
> > > > > > > > >>>>>> Flushing each file requires a separate IO. If the disk
> > has
> > > > 1K
> > > > > > IOPS
> > > > > > > > and
> > > > > > > > >>>> we
> > > > > > > > >>>>>> have 5K partitions in a broker, currently we can
> afford
> > to
> > > > > flush
> > > > > > > > each
> > > > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5
> > seconds. If
> > > > > we
> > > > > > > > double
> > > > > > > > >>>> the
> > > > > > > > >>>>>> number of files per partition, we can only flush each
> > > > > partition
> > > > > > > > every
> > > > > > > > >>>> 10
> > > > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we
> > ever
> > > > need
> > > > > > > this
> > > > > > > > >>>>>> metadata somewhere else, say in the WAL file manager,
> > the
> > > > > > consumer
> > > > > > > > >>>> needs to
> > > > > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > > > > inefficient.
> > > > > > > > The
> > > > > > > > >>>>>> actual benefit of this approach is also questionable.
> > On the
> > > > > > > > surface,
> > > > > > > > >>>> it
> > > > > > > > >>>>>> might seem that we could reduce the number of lines
> that
> > > > need
> > > > > to
> > > > > > > be
> > > > > > > > >>>> changed
> > > > > > > > >>>>>> for this KIP. However, the changes are quite intrusive
> > to
> > > > the
> > > > > > > > classic
> > > > > > > > >>>>>> partition's code path and will probably make the code
> > base
> > > > > > harder
> > > > > > > to
> > > > > > > > >>>>>> maintain in the long run. I like the original approach
> > based
> > > > > on
> > > > > > > the
> > > > > > > > >>>> batch
> > > > > > > > >>>>>> coordinator much better than this one. We could
> probably
> > > > > > refactor
> > > > > > > > the
> > > > > > > > >>>>>> producer state code so that it could be reused in the
> > batch
> > > > > > > > >>>> coordinator.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks for addressing the concerns on the number of
> > RPCs in
> > > > > the
> > > > > > > > >> produce
> > > > > > > > >>>>>> path. I agree that with the metadata crafting
> > mechanism, we
> > > > > > could
> > > > > > > > >>>> mitigate
> > > > > > > > >>>>>> the PRC problem. However, since we now require the
> > metadata
> > > > to
> > > > > > be
> > > > > > > > >>>>>> collocated with the data on the same set of brokers,
> > it's
> > > > > weird
> > > > > > > that
> > > > > > > > >>>> they
> > > > > > > > >>>>>> are now managed by different mechanisms. The data
> > assignment
> > > > > now
> > > > > > > > uses
> > > > > > > > >>>> the
> > > > > > > > >>>>>> metadata crafting mechanism, but the metadata is
> stored
> > in
> > > > the
> > > > > > > > classic
> > > > > > > > >>>>>> partition using its own assignment strategy. It will
> be
> > > > > > > complicated
> > > > > > > > to
> > > > > > > > >>>> keep
> > > > > > > > >>>>>> them collocated.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > > > compaction/merging
> > > > > > > > >>>>>> step. My first concern is on the amount of metadata
> that
> > > > need
> > > > > to
> > > > > > > be
> > > > > > > > >>>> kept.
> > > > > > > > >>>>>> Without object compcation, the metadata generated in
> the
> > > > > produce
> > > > > > > > path
> > > > > > > > >>>> can
> > > > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's
> > say for
> > > > > > every
> > > > > > > > >>>> 250ms we
> > > > > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> > > > remoting
> > > > > > > > tiering
> > > > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K
> > partitions,
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > >>>> keep
> > > > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> > metadata,
> > > > > > quite
> > > > > > > > >>>>>> signficant. A second concern is on performance. Every
> > time
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > >>>>>> rebuild the caching data, we need to read a bunch of
> > small
> > > > > > objects
> > > > > > > > >>>> from S3,
> > > > > > > > >>>>>> slowing down the building process. If a consumer
> > happens to
> > > > > need
> > > > > > > > such
> > > > > > > > >>>> data,
> > > > > > > > >>>>>> it could slow down the application.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> > fewest
> > > > > code
> > > > > > > > >>>> changes in
> > > > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> > design
> > > > > that
> > > > > > > can
> > > > > > > > >>>> last
> > > > > > > > >>>>>> for the long term.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Jun
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me
> > > > > > >
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Hi Jun,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed
> to
> > > > > clearly
> > > > > > > > >>>> explain
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> and
> > a
> > > > > > separate
> > > > > > > > >>>> data
> > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > doubles
> > > > > > the
> > > > > > > > >>>> number
> > > > > > > > >>>>>>> of
> > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > descriptors
> > > > > > > and
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > have a
> > > > > > > > separate
> > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > are
> > > > > > creating
> > > > > > > > an
> > > > > > > > >>>>>>>> internal topic with an unbounded number of
> partitions.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> No. There will be only one physical partition per
> > diskless
> > > > > > > > >>>> partition. Let
> > > > > > > > >>>>>>> me explain this with an example. Let’s say we have a
> > > > diskless
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> > leader. We
> > > > > > > produce
> > > > > > > > >>>> some
> > > > > > > > >>>>>>> batches to this partition. The content of the segment
> > file
> > > > > will
> > > > > > > be
> > > > > > > > >>>>>>> something like this (for each batch):
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > > > >>>>>>> Magic: like in classic
> > > > > > > > >>>>>>> CRC: like in classic
> > > > > > > > >>>>>>> Attributes: like in classic
> > > > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > > > >>>>>>> ProducerId: like in classic
> > > > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > > > >>>>>>> Records:
> > > > > > > > >>>>>>>
> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > > > byte
> > > > > > > offset
> > > > > > > > >>>>>>> 123456
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> It looks very much like classic log entries. The only
> > > > > > difference
> > > > > > > is
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> instead of writing real Records, we write the
> > reference to
> > > > > the
> > > > > > > WAL
> > > > > > > > >>>> file
> > > > > > > > >>>>>>> with the batch data (I guess we need only the name
> and
> > the
> > > > > byte
> > > > > > > > >>>> offset,
> > > > > > > > >>>>>>> because the byte length is the standard field above).
> > > > > > Otherwise,
> > > > > > > > >>>> it’s a
> > > > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> So we have as many partitions for diskless as for
> > classic.
> > > > As
> > > > > > of
> > > > > > > > open
> > > > > > > > >>>>>> file
> > > > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Are the metadata and
> > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > the
> > > > > same
> > > > > > > > >>>> broker?
> > > > > > > > >>>>>> If
> > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > reassigned?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The source of truth for the data is still in WAL
> files
> > on
> > > > > > object
> > > > > > > > >>>> storage.
> > > > > > > > >>>>>>> The source of truth for the metadata is in segment
> > files on
> > > > > the
> > > > > > > > >>>> brokers
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > > > > independent
> > > > > > > > of
> > > > > > > > >>>> this
> > > > > > > > >>>>>>> new proposal, but I want to present them to give the
> > idea
> > > > > that
> > > > > > > > only a
> > > > > > > > >>>>>>> limited amount of data files will be operated
> locally:
> > > > > > > > >>>>>>> - We want to assemble batches into segment files and
> > > > offload
> > > > > > them
> > > > > > > > to
> > > > > > > > >>>>>>> tiered storage in order to prevent the unbounded
> > growth of
> > > > > > batch
> > > > > > > > >>>>>> metadata.
> > > > > > > > >>>>>>> For this, we need to open only  a few file
> descriptors
> > (for
> > > > > the
> > > > > > > > >>>> segment
> > > > > > > > >>>>>>> file itself + the necessary indexes) before the
> > segment is
> > > > > > fully
> > > > > > > > >>>> written
> > > > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > > > >>>>>>> - We want to assemble local segment files for caching
> > > > > purposes
> > > > > > as
> > > > > > > > >>>> well,
> > > > > > > > >>>>>>> i.e. to speed up fetching. This will not materialize
> > the
> > > > full
> > > > > > > > >>>> content of
> > > > > > > > >>>>>>> the log, but only the hot set according to some
> policy
> > (or
> > > > > > > > >>>> configurable
> > > > > > > > >>>>>>> policies), i.e. the number of segments and file
> > descriptors
> > > > > > will
> > > > > > > > >>>> also be
> > > > > > > > >>>>>>> limited.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > significantly
> > > > > > higher.
> > > > > > > > For
> > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> in a
> > > > > cluster
> > > > > > > > >>>> with 100
> > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> more
> > RPC
> > > > > > > > requests.
> > > > > > > > >>>>>> This
> > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> This is a valid concern that we considered, but this
> > issue
> > > > > can
> > > > > > be
> > > > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > > > >>>>>>> The situation with a single broker is trivial: all
> the
> > > > commit
> > > > > > > > >>>> requests go
> > > > > > > > >>>>>>> from the broker to itself.
> > > > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but
> > located in
> > > > > the
> > > > > > > > single
> > > > > > > > >>>>>> rack
> > > > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for
> > diskless
> > > > > > > > >>>> partitions, but
> > > > > > > > >>>>>>> we can tell producers (through metadata) to always
> send
> > > > > Produce
> > > > > > > > >>>> requests
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader
> > replicas
> > > > for
> > > > > > > > diskless
> > > > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> > > > Produce
> > > > > > > > >>>> requests
> > > > > > > > >>>>>> for
> > > > > > > > >>>>>>> these partitions in various combinations, but only
> for
> > > > them.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>                   Broker 0
> > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > >>>>>>>             |    t1-0         |
> > > > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > > > >>>>>>> partitions   | +-------------+ |    for these
> > partitions
> > > > > > > > >>>>>>>             |                 |
> > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > > > Effectively,
> > > > > > > > each
> > > > > > > > >>>>>>> broker will commit only to itself, which effectively
> > means
> > > > 1
> > > > > > > commit
> > > > > > > > >>>>>> request
> > > > > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls,
> > if we
> > > > > > wish,
> > > > > > > > >>>> just a
> > > > > > > > >>>>>>> local function call).
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs).
> > Obviously, we
> > > > > > > cannot
> > > > > > > > >>>> always
> > > > > > > > >>>>>>> send Produce requests to the designated leaders of
> > diskless
> > > > > > > > >>>> partitions:
> > > > > > > > >>>>>>> this would mean inter-AZ network traffic, which we
> > would
> > > > like
> > > > > > to
> > > > > > > > >>>> avoid.
> > > > > > > > >>>>>> To
> > > > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> > produce
> > > > > > > > >>>>>> representative”
> > > > > > > > >>>>>>> in every AZ. If we continue our example: when a
> Produce
> > > > > request
> > > > > > > for
> > > > > > > > >>>> t1-0,
> > > > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands
> > on
> > > > > > broker 0
> > > > > > > > >>>> (in the
> > > > > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > > > > However,
> > > > > > if
> > > > > > > > it
> > > > > > > > >>>>>> comes
> > > > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker
> > 2.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> |produce requests         |produce requests
> > |produce
> > > > > > > > >>>> requests
> > > > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
> |for
> > > > t1-0,
> > > > > > > t2-1,
> > > > > > > > >>>>>> t3-0
> > > > > > > > >>>>>>> |from AZ 0                |from AZ 1
> >  |from
> > > > AZ 2
> > > > > > > > >>>>>>> v                         v                        v
> > > > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker
> 2
> > (AZ
> > > > 2)
> > > > > > > > >>>>>>> +---------------+      +---------------+
> > > > > +---------------+
> > > > > > > > >>>>>>> |     t1-0      |      |               |      |
> > > > >  |
> > > > > > > > >>>>>>> |     t2-1      |      |               |      |
> > > > >  |
> > > > > > > > >>>>>>> |     t3-0      |      |               |      |
> > > > >  |
> > > > > > > > >>>>>>> +---------------+      +--------+------+
> > > > > +--------+------+
> > > > > > > > >>>>>>>    ^     ^                    |
> |
> > > > > > > > >>>>>>>    |     +--------------------+
> |
> > > > > > > > >>>>>>>    |     commit requests for these partitions
> |
> > > > > > > > >>>>>>>    |
>  |
> > > > > > > > >>>>>>>
> +-------------------------------------------------+
> > > > > > > > >>>>>>>          commit requests for these partitions
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> All the partitions that broker 0 is the leader of
> will
> > be
> > > > > > > > >>>> “represented”
> > > > > > > > >>>>>> by
> > > > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Of course, this relationship goes both ways between
> AZs
> > > > (not
> > > > > > > > >>>> necessarily
> > > > > > > > >>>>>>> between the same brokers). It means that provided the
> > > > cluster
> > > > > > is
> > > > > > > > >>>> balanced
> > > > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> > represent
> > > > > > > > >>>>>> (number_of_azs
> > > > > > > > >>>>>>> - 1) other brokers. This will result in the situation
> > that
> > > > > for
> > > > > > > the
> > > > > > > > >>>>>> majority
> > > > > > > > >>>>>>> of commits, each broker will do up to (number_of_azs
> -
> > 1)
> > > > > > network
> > > > > > > > >>>> commit
> > > > > > > > >>>>>>> requests (plus one local). Cloud regions tend to
> have 3
> > > > AZs,
> > > > > > very
> > > > > > > > >>>> rarely
> > > > > > > > >>>>>>> more. That means, brokers will be doing up to 2
> network
> > > > > commit
> > > > > > > > >>>> requests
> > > > > > > > >>>>>> per
> > > > > > > > >>>>>>> WAL file.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> There are the following exceptions:
> > > > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example,
> > when we
> > > > > > have
> > > > > > > 2
> > > > > > > > >>>> AZs
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>> one has three brokers and another AZ has one. This
> one
> > > > broker
> > > > > > > will
> > > > > > > > do
> > > > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is
> > not
> > > > an
> > > > > > > > extreme
> > > > > > > > >>>>>>> amplification. Such an imbalance is not healthy in
> most
> > > > > > practical
> > > > > > > > >>>> setups
> > > > > > > > >>>>>>> and should be avoided anyway.
> > > > > > > > >>>>>>> 2. Leadership changes and metadata propagation
> period.
> > When
> > > > > the
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > > > > producers
> > > > > > > > will
> > > > > > > > >>>> not
> > > > > > > > >>>>>>> know this immediately (unless we want to be strict
> and
> > > > > respond
> > > > > > > with
> > > > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> > will
> > > > come
> > > > > > > > >>>> together
> > > > > > > > >>>>>> in a
> > > > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two
> commit
> > > > > > requests:
> > > > > > > > to
> > > > > > > > >>>>>> broker
> > > > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
> > t3-0.
> > > > > This
> > > > > > > > >>>> situation
> > > > > > > > >>>>>>> is not permanent and as producers update the cluster
> > > > > metadata,
> > > > > > it
> > > > > > > > >>>> will be
> > > > > > > > >>>>>>> resolved.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> This all could be built with the metadata crafting
> > > > mechanism
> > > > > > only
> > > > > > > > >>>> (which
> > > > > > > > >>>>>>> is anyway needed for Diskless in one way or another
> to
> > > > direct
> > > > > > > > >>>> producers
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic),
> > just
> > > > with
> > > > > > the
> > > > > > > > >>>> right
> > > > > > > > >>>>>>> policy for it (for example, some deterministic
> > hash-based
> > > > > > > formula).
> > > > > > > > >>>> I.e.
> > > > > > > > >>>>>> no
> > > > > > > > >>>>>>> explicit support for “produce representative” or
> > anything
> > > > > like
> > > > > > > this
> > > > > > > > >>>> is
> > > > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> two
> > > > > places,
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > of
> > > > > truth,
> > > > > > > and
> > > > > > > > >>>> how
> > > > > > > > >>>>>> do
> > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> We do only two operations on WAL files that span
> > multiple
> > > > > > > diskless
> > > > > > > > >>>>>>> partitions: committing and deleting. Commits can be
> > done
> > > > > > > > >>>> independently as
> > > > > > > > >>>>>>> described above. But deletes are different, because
> > when a
> > > > > file
> > > > > > > is
> > > > > > > > >>>>>> deleted,
> > > > > > > > >>>>>>> this affects all the partitions that still have alive
> > > > batches
> > > > > > in
> > > > > > > > this
> > > > > > > > >>>>>> file
> > > > > > > > >>>>>>> (if any).
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The WAL file manager is a necessary point of
> > coordination
> > > > to
> > > > > > > delete
> > > > > > > > >>>> WAL
> > > > > > > > >>>>>>> files safely. We can say it is the source of truth
> > about
> > > > > files
> > > > > > > > >>>>>> themselves,
> > > > > > > > >>>>>>> while the partition leaders and their logs hold the
> > truth
> > > > > about
> > > > > > > > >>>> whether a
> > > > > > > > >>>>>>> particular file contains live batches of this
> > particular
> > > > > > > partition.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The file manager will do this important task: be able
> > to
> > > > say
> > > > > > for
> > > > > > > > sure
> > > > > > > > >>>>>> that
> > > > > > > > >>>>>>> a file does not contain any live batch of any
> existing
> > > > > > partition.
> > > > > > > > For
> > > > > > > > >>>>>> this,
> > > > > > > > >>>>>>> it will have to periodically check against the
> > partition
> > > > > > leaders.
> > > > > > > > >>>>>>> Considering that batch deletion is irreversible, when
> > we
> > > > > > declare
> > > > > > > a
> > > > > > > > >>>> file
> > > > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The file manager has to know about files being
> > committed to
> > > > > > start
> > > > > > > > >>>> track
> > > > > > > > >>>>>>> them and periodically check if they are empty. We can
> > > > > consider
> > > > > > > > >>>> various
> > > > > > > > >>>>>> ways
> > > > > > > > >>>>>>> to achieve this:
> > > > > > > > >>>>>>> 1. As was proposed in my previous message: best
> effort
> > > > commit
> > > > > > by
> > > > > > > > >>>> brokers
> > > > > > > > >>>>>> +
> > > > > > > > >>>>>>> periodic prefix scans of object storage to detect
> files
> > > > that
> > > > > > went
> > > > > > > > >>>> below
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>> radar due to network issue or the file manager
> > temporary
> > > > > > > > >>>> unavailability.
> > > > > > > > >>>>>>> We’re speaking about listing the file names only and
> > > > opening
> > > > > > only
> > > > > > > > >>>>>>> previously unknown files in order to find the
> > partitions
> > > > > > involved
> > > > > > > > >>>> with
> > > > > > > > >>>>>> them.
> > > > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill
> the
> > > > list
> > > > > of
> > > > > > > > files
> > > > > > > > >>>>>>> fully asynchronously and in the background. This may
> > be not
> > > > > > ideal
> > > > > > > > >>>> due to
> > > > > > > > >>>>>>> costs and performance of scanning tons of files.
> > However,
> > > > the
> > > > > > > > number
> > > > > > > > >>>> of
> > > > > > > > >>>>>>> live WAL files should be limited due to tiered
> storage
> > > > > > > offloading +
> > > > > > > > >>>> we
> > > > > > > > >>>>>> can
> > > > > > > > >>>>>>> optimize this if we give files some global soft order
> > in
> > > > > their
> > > > > > > > names.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > implementation.
> > > > > > The
> > > > > > > > >>>>>> existing
> > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > Adjusting
> > > > > > both
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> write
> > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > follower
> > > > > to
> > > > > > > > >>>>>> understand
> > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > existing
> > > > > > logic.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > > > UnifiedLog
> > > > > in
> > > > > > > > >>>> order to
> > > > > > > > >>>>>>> support these changes. However, it seems that
> > idempotence,
> > > > > > > > >>>> transactions,
> > > > > > > > >>>>>>> queues, tiered storage will have to be changed less
> > than
> > > > with
> > > > > > the
> > > > > > > > >>>>>> original
> > > > > > > > >>>>>>> design. This is because the partition leader state
> > would
> > > > > remain
> > > > > > > in
> > > > > > > > >>>> the
> > > > > > > > >>>>>> same
> > > > > > > > >>>>>>> place (on brokers) and existing workflows that
> involve
> > it
> > > > > would
> > > > > > > > have
> > > > > > > > >>>> to
> > > > > > > > >>>>>> be
> > > > > > > > >>>>>>> changed less compared to the situation where we
> > globalize
> > > > the
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> leader state in the batch coordinator. I admit this
> is
> > hard
> > > > > to
> > > > > > > make
> > > > > > > > >>>>>>> convincing without both real implementations to hand
> :)
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> I am also
> > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> topic
> > > > modes
> > > > > > > > >>>> between
> > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > on
> > > > > those?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Let’s consider the scenario of turning a classic
> topic
> > into
> > > > > > > > >>>> diskless. The
> > > > > > > > >>>>>>> user sets diskless.enabled=true, the leader receives
> > this
> > > > > > > metadata
> > > > > > > > >>>> update
> > > > > > > > >>>>>>> and does the following:
> > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > >>>>>>> 3. Start a new segment that will be written in the
> > diskless
> > > > > > > format
> > > > > > > > >>>> (i.e.
> > > > > > > > >>>>>>> without data).
> > > > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Since it’s the same log, the followers will know
> about
> > that
> > > > > > > switch
> > > > > > > > >>>>>>> consistently. They will finish replicating the
> classic
> > > > > segments
> > > > > > > and
> > > > > > > > >>>> start
> > > > > > > > >>>>>>> replicating the diskless ones. They will always know
> > where
> > > > > each
> > > > > > > > >>>> batch is
> > > > > > > > >>>>>>> located (either inside a classic segment or
> referenced
> > by a
> > > > > > > > diskless
> > > > > > > > >>>>>> one).
> > > > > > > > >>>>>>> Switching back should be similar.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Doing this with the coordinator is possible, but has
> > some
> > > > > > > caveats.
> > > > > > > > >>>> The
> > > > > > > > >>>>>>> leader must do the following:
> > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > >>>>>>> 3. Write a special control segment to persist and
> > replicate
> > > > > the
> > > > > > > > fact
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> from offset N the partition is now in the diskless
> > mode.
> > > > > > > > >>>>>>> 4. Inform the coordinator about the first offset N of
> > the
> > > > > > > “diskless
> > > > > > > > >>>> era”.
> > > > > > > > >>>>>>> 5. Inform the controller quorum that the transition
> has
> > > > > > finished
> > > > > > > > and
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> brokers now can process diskless writes for this
> > partition.
> > > > > > > > >>>>>>> This could fail at some points, so this will probably
> > > > require
> > > > > > > some
> > > > > > > > >>>>>>> explicit state machine with replication either in the
> > > > > partition
> > > > > > > log
> > > > > > > > >>>> or in
> > > > > > > > >>>>>>> KRaft.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> It seems that the coordinator-less approach makes
> this
> > > > > simpler
> > > > > > > > >>>> because
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>> “coordinator” for the partition and the partition
> > leader
> > > > are
> > > > > > the
> > > > > > > > >>>> same and
> > > > > > > > >>>>>>> they store the partition metadata in the same log,
> too.
> > > > While
> > > > > > in
> > > > > > > > the
> > > > > > > > >>>>>>> coordinator approach we have to perform some kind of
> a
> > > > > > > distributed
> > > > > > > > >>>> commit
> > > > > > > > >>>>>>> to handover metadata management from the classic
> > partition
> > > > > > leader
> > > > > > > > to
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> batch coordinator.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> > Please
> > > > > let
> > > > > > me
> > > > > > > > >>>> know if
> > > > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Best,
> > > > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > > > >>>>>>>> Hi, Ivan,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thanks for the update.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> I am not sure that I fully understand the new
> design,
> > but
> > > > it
> > > > > > > seems
> > > > > > > > >>>> less
> > > > > > > > >>>>>>>> clean than before.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> and
> > a
> > > > > > separate
> > > > > > > > >>>> data
> > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > doubles
> > > > > > the
> > > > > > > > >>>> number
> > > > > > > > >>>>>>> of
> > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > descriptors
> > > > > > > and
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > have a
> > > > > > > > separate
> > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > are
> > > > > > creating
> > > > > > > > an
> > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > partitions. Are
> > > > > the
> > > > > > > > >>>> metadata
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > the
> > > > > same
> > > > > > > > >>>> broker?
> > > > > > > > >>>>>> If
> > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > reassigned?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > significantly
> > > > > > higher.
> > > > > > > > For
> > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> in a
> > > > > cluster
> > > > > > > > >>>> with 100
> > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> more
> > RPC
> > > > > > > > requests.
> > > > > > > > >>>>>> This
> > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> two
> > > > > places,
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > of
> > > > > truth,
> > > > > > > and
> > > > > > > > >>>> how
> > > > > > > > >>>>>> do
> > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > implementation.
> > > > > > The
> > > > > > > > >>>>>> existing
> > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > Adjusting
> > > > > > both
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> write
> > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > follower
> > > > > to
> > > > > > > > >>>>>> understand
> > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > existing
> > > > > > > logic. I
> > > > > > > > >>>> am
> > > > > > > > >>>>>>> also
> > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> topic
> > > > modes
> > > > > > > > >>>> between
> > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > on
> > > > > those?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Jun
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me
> > > > > > >
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to
> > have a
> > > > > > > separate
> > > > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role
> > would
> > > > be
> > > > > to
> > > > > > > > >>>>>> centrally
> > > > > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> > > > topics.
> > > > > > > This
> > > > > > > > >>>>>>> component
> > > > > > > > >>>>>>>>> drew many reasonable comments from the community
> > about
> > > > how
> > > > > it
> > > > > > > > >>>> would
> > > > > > > > >>>>>>> support
> > > > > > > > >>>>>>>>> various Kafka features (transactions, queues) and
> its
> > > > > > > > >>>> scalability.
> > > > > > > > >>>>>>> While we
> > > > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > > > concerns,
> > > > > > we
> > > > > > > > >>>> took a
> > > > > > > > >>>>>>> step
> > > > > > > > >>>>>>>>> back and looked at the problem from a different
> > > > > perspective.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> We would like to propose an alternative Diskless
> > design
> > > > > > > *without
> > > > > > > > >>>> a
> > > > > > > > >>>>>>>>> centralized coordinator*. We believe this approach
> > has
> > > > > > > potential
> > > > > > > > >>>> and
> > > > > > > > >>>>>>>>> propose to discuss it as it may be more appealing
> to
> > the
> > > > > > > > >>>> community.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Let us explain the idea. Most of the complications
> > with
> > > > the
> > > > > > > > >>>> original
> > > > > > > > >>>>>>>>> Diskless approach come from one necessary
> > architecture
> > > > > > change:
> > > > > > > > >>>>>>> globalizing
> > > > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > > > coordinator.
> > > > > > > > >>>> This
> > > > > > > > >>>>>>> causes
> > > > > > > > >>>>>>>>> deviations to the established workflows in various
> > > > features
> > > > > > > like
> > > > > > > > >>>>>>> produce
> > > > > > > > >>>>>>>>> idempotence and transactions, queues, retention,
> etc.
> > > > These
> > > > > > > > >>>>>> deviations
> > > > > > > > >>>>>>> need
> > > > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > > > implemented
> > > > > > and
> > > > > > > > >>>>>>> tested. In
> > > > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > > > partition
> > > > > > > > >>>> leaders
> > > > > > > > >>>>>>> again
> > > > > > > > >>>>>>>>> responsible for managing their partitions, even in
> > > > diskless
> > > > > > > > >>>> topics.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata
> are
> > > > > blended
> > > > > > > > >>>> together
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>> the one partition log. The crux of the Diskless
> idea
> > is
> > > > to
> > > > > > > > >>>> decouple
> > > > > > > > >>>>>>> them
> > > > > > > > >>>>>>>>> and move data to the remote storage, while keeping
> > > > metadata
> > > > > > > > >>>> somewhere
> > > > > > > > >>>>>>> else.
> > > > > > > > >>>>>>>>> Using the central batch coordinator for managing
> > batch
> > > > > > metadata
> > > > > > > > >>>> is
> > > > > > > > >>>>>> one
> > > > > > > > >>>>>>> way,
> > > > > > > > >>>>>>>>> but not the only.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Let’s now think about managing metadata for each
> user
> > > > > > partition
> > > > > > > > >>>>>>>>> independently. Generally partitions are independent
> > and
> > > > > don’t
> > > > > > > > >>>> share
> > > > > > > > >>>>>>>>> anything apart from that their data are mixed in
> WAL
> > > > files.
> > > > > > If
> > > > > > > we
> > > > > > > > >>>>>>> figure
> > > > > > > > >>>>>>>>> out how to commit and later delete WAL files
> safely,
> > we
> > > > > will
> > > > > > > > >>>> achieve
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> > > > central
> > > > > > > batch
> > > > > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition
> > will
> > > > be
> > > > > > > > >>>> managed
> > > > > > > > >>>>>> by
> > > > > > > > >>>>>>> its
> > > > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> > classic
> > > > > > > > >>>> topics, the
> > > > > > > > >>>>>>>>> leader uses the partition log as the way to persist
> > batch
> > > > > > > > >>>> metadata,
> > > > > > > > >>>>>>> i.e.
> > > > > > > > >>>>>>>>> the regular batch header + the information about
> how
> > to
> > > > > find
> > > > > > > this
> > > > > > > > >>>>>>> batch on
> > > > > > > > >>>>>>>>> remote storage. In contrast to classic topics,
> batch
> > data
> > > > > is
> > > > > > in
> > > > > > > > >>>>>> remote
> > > > > > > > >>>>>>>>> storage.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > > > >>>>>>>>> • Classic topics:
> > > > > > > > >>>>>>>>>  • Data and metadata are co-located in the
> partition
> > log.
> > > > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > > > (metadata)|Batch
> > > > > > > > >>>> data].
> > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> followers.
> > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > from
> > > > > > > > >>>> metadata.
> > > > > > > > >>>>>>>>> • Original Diskless:
> > > > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> > > > remote
> > > > > > > > >>>> storage.
> > > > > > > > >>>>>>>>>  • The partition state is global in the batch
> > > > coordinator.
> > > > > > > > >>>>>>>>> • New Diskless:
> > > > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on
> > remote
> > > > > > storage.
> > > > > > > > >>>>>>>>>  • Partition log content: [Batch header
> > (metadata)|Batch
> > > > > > > > >>>>>> coordinates
> > > > > > > > >>>>>>> on
> > > > > > > > >>>>>>>>> remote storage].
> > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> followers.
> > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > from
> > > > > > > > >>>> metadata.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the
> reminder
> > of
> > > > the
> > > > > > > > >>>> original
> > > > > > > > >>>>>>>>> Diskless design:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The new approach could be depicted as the
> following:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> As you can see, the main difference is that now
> > instead
> > > > of
> > > > > a
> > > > > > > > >>>> single
> > > > > > > > >>>>>>> commit
> > > > > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> > > > parallel
> > > > > > > > >>>> commit
> > > > > > > > >>>>>>> requests
> > > > > > > > >>>>>>>>> to all the leaders of each partition involved in
> the
> > WAL
> > > > > > file.
> > > > > > > > >>>> Each
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>> them
> > > > > > > > >>>>>>>>> will commit its batches independently, without
> > > > coordinating
> > > > > > > with
> > > > > > > > >>>>>> other
> > > > > > > > >>>>>>>>> leaders and any other components. Batch data is
> > addressed
> > > > > by
> > > > > > > the
> > > > > > > > >>>> WAL
> > > > > > > > >>>>>>> file
> > > > > > > > >>>>>>>>> name, the byte offset and size, which allows
> > partitions
> > > > to
> > > > > > know
> > > > > > > > >>>>>> nothing
> > > > > > > > >>>>>>>>> about other partitions to access their data in
> > shared WAL
> > > > > > > files.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The number of partitions involved in a single WAL
> > file
> > > > may
> > > > > be
> > > > > > > > >>>> quite
> > > > > > > > >>>>>>> large,
> > > > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to
> commit
> > one
> > > > > WAL
> > > > > > > > >>>> file is
> > > > > > > > >>>>>>> very
> > > > > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> > > > number:
> > > > > > > > >>>>>>>>> 1. Partition leaders are located on brokers.
> > Requests to
> > > > > > > > >>>> leaders on
> > > > > > > > >>>>>>> one
> > > > > > > > >>>>>>>>> broker could be grouped together into a single
> > physical
> > > > > > network
> > > > > > > > >>>>>> request
> > > > > > > > >>>>>>>>> (resembling the normal Produce request that may
> carry
> > > > > batches
> > > > > > > for
> > > > > > > > >>>>>> many
> > > > > > > > >>>>>>>>> partitions inside). This will cap the number of
> > network
> > > > > > > requests
> > > > > > > > >>>> to
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make
> producers
> > > > send
> > > > > > > their
> > > > > > > > >>>>>>> requests
> > > > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> > > > achieve
> > > > > > the
> > > > > > > > >>>> higher
> > > > > > > > >>>>>>>>> concentration of logical commit requests in
> physical
> > > > > network
> > > > > > > > >>>> requests
> > > > > > > > >>>>>>>>> reducing the number of the latter ones even
> further,
> > > > > ideally
> > > > > > to
> > > > > > > > >>>> one.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Obviously, out of multiple commit requests some may
> > fail
> > > > or
> > > > > > > time
> > > > > > > > >>>> out
> > > > > > > > >>>>>>> for a
> > > > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers
> will
> > > > > receive
> > > > > > > > >>>> totally
> > > > > > > > >>>>>>> or
> > > > > > > > >>>>>>>>> partially failed responses to their Produce
> requests,
> > > > > similar
> > > > > > > to
> > > > > > > > >>>> what
> > > > > > > > >>>>>>> they
> > > > > > > > >>>>>>>>> would have received when appending to a classic
> topic
> > > > fails
> > > > > > or
> > > > > > > > >>>> times
> > > > > > > > >>>>>>> out.
> > > > > > > > >>>>>>>>> If a partition experiences problems, other
> partitions
> > > > will
> > > > > > not
> > > > > > > be
> > > > > > > > >>>>>>> affected
> > > > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > > > uncommitted
> > > > > > > data
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > > > > (batches
> > > > > > > are
> > > > > > > > >>>>>>> constantly
> > > > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> > storage),
> > > > > so
> > > > > > > this
> > > > > > > > >>>>>>> garbage
> > > > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> For safely deleting WAL files we now need to
> > centrally
> > > > > manage
> > > > > > > > >>>> them,
> > > > > > > > >>>>>> as
> > > > > > > > >>>>>>>>> this is the only state and logic that spans
> multiple
> > > > > > > partitions.
> > > > > > > > >>>> On
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>> diagram, you can see another commit request called
> > > > “Commit
> > > > > > file
> > > > > > > > >>>> (best
> > > > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This
> manager
> > will
> > > > > be
> > > > > > > > >>>>>>> responsible
> > > > > > > > >>>>>>>>> for the following:
> > > > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and
> > persisting
> > > > > > > > >>>> information
> > > > > > > > >>>>>>> about
> > > > > > > > >>>>>>>>> committed WAL files.
> > > > > > > > >>>>>>>>> 2. To handle potential failures in file information
> > > > > delivery,
> > > > > > > it
> > > > > > > > >>>>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> doing prefix scan on the remote storage
> periodically
> > to
> > > > > find
> > > > > > > and
> > > > > > > > >>>>>>> register
> > > > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > > > configurable
> > > > > > and
> > > > > > > > >>>>>> ideally
> > > > > > > > >>>>>>>>> should be quite long.
> > > > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders
> > (after a
> > > > > > grace
> > > > > > > > >>>>>>> period) if
> > > > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t
> anymore
> > > > > > referred
> > > > > > > > >>>> to by
> > > > > > > > >>>>>>> any
> > > > > > > > >>>>>>>>> partition.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > > > features
> > > > > > such
> > > > > > > > >>>> as
> > > > > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > > > > retention.
> > > > > > > > >>>> Now we
> > > > > > > > >>>>>>> don’t
> > > > > > > > >>>>>>>>> need to abstract away and reuse the code from
> > partition
> > > > > > leaders
> > > > > > > > >>>> in
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use
> the
> > > > same
> > > > > > code
> > > > > > > > >>>> paths
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from
> > classic
> > > > > > topics
> > > > > > > > >>>> mostly
> > > > > > > > >>>>>>>>> remain unchanged.
> > > > > > > > >>>>>>>>> For example, it seems that
> > > > > > > > >>>>>>>>>
> > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > > > and
> > > > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > > > transaction
> > > > > > > > >>>> support
> > > > > > > > >>>>>> on
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> partition leader side could be used for diskless
> > topics
> > > > > with
> > > > > > > > >>>> little
> > > > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > > > idempotent
> > > > > > > > >>>> produce
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > > > >>>>>>>>> Another example is share groups support, where the
> > share
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> leader,
> > > > > > > > >>>>>>>>> being co-located with the partition leader, would
> > execute
> > > > > the
> > > > > > > > >>>> same
> > > > > > > > >>>>>>> logic
> > > > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > > > >>>>>>>>> 2. It returns to the familiar partition-based
> scaling
> > > > > model,
> > > > > > > > >>>> where
> > > > > > > > >>>>>>>>> partitions are independent.
> > > > > > > > >>>>>>>>> 3. It makes the operation and failure patterns
> > closer to
> > > > > the
> > > > > > > > >>>>>> familiar
> > > > > > > > >>>>>>>>> ones from classic topics.
> > > > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> > switching
> > > > > the
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> modes
> > > > > > > > >>>>>>>>> between diskless and classic.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The rest of the things remain unchanged compared to
> > the
> > > > > > > previous
> > > > > > > > >>>>>>> Diskless
> > > > > > > > >>>>>>>>> design (after all previous discussions). Such
> things
> > as
> > > > > local
> > > > > > > > >>>> segment
> > > > > > > > >>>>>>>>> materialization by replicas, the consume path,
> tiered
> > > > > storage
> > > > > > > > >>>>>>> integration,
> > > > > > > > >>>>>>>>> etc.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> If the community finds this design more suitable,
> we
> > will
> > > > > > > update
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it.
> > Please let
> > > > > us
> > > > > > > know
> > > > > > > > >>>>>> what
> > > > > > > > >>>>>>> you
> > > > > > > > >>>>>>>>> think.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Best regards,
> > > > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
> wrote:
> > > > > > > > >>>>>>>>>> Hi Justine,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > > > transactions
> > > > > > > > >>>> for in
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> diskless coordinator for as long as the
> corresponding
> > > > > offsets
> > > > > > > are
> > > > > > > > >>>>>>> there.
> > > > > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> > > > earlier,
> > > > > > > this
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>> Ivan
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> > wrote:
> > > > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks for the response. I think most of what you
> > said
> > > > > made
> > > > > > > > >>>>>> sense,
> > > > > > > > >>>>>>> but
> > > > > > > > >>>>>>>>> I
> > > > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > classic
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> forgets
> > > > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > overpasses
> > > > > > > > >>>> it). The
> > > > > > > > >>>>>>>>>>> transaction coordinator acts like the main
> > guardian,
> > > > > > allowing
> > > > > > > > >>>>>>> partition
> > > > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if
> > this is
> > > > > > > > >>>> wrong. We
> > > > > > > > >>>>>>> think
> > > > > > > > >>>>>>>>>>> about relying on this with the batch coordinator
> > and
> > > > > delete
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>> information
> > > > > > > > >>>>>>>>>>> about a transaction once it’s finished (as
> there’s
> > no
> > > > > > > > >>>> replication
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> HWM
> > > > > > > > >>>>>>>>>>> advances immediately).
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I didn't quite understand this. In classic
> topics,
> > we
> > > > > have
> > > > > > > > >>>> maps
> > > > > > > > >>>>>> for
> > > > > > > > >>>>>>>>> ongoing
> > > > > > > > >>>>>>>>>>> transactions which remove state when the
> > transaction is
> > > > > > > > >>>> completed
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> an
> > > > > > > > >>>>>>>>>>> aborted transactions index which is retained for
> > much
> > > > > > longer.
> > > > > > > > >>>>>> Once
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> > longer
> > > > > > > > >>>> involved
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> > > > subject
> > > > > to
> > > > > > > > >>>>>>> compaction
> > > > > > > > >>>>>>>>> etc.
> > > > > > > > >>>>>>>>>>> Looking back at the outline provided above, I
> > didn't
> > > > see
> > > > > > much
> > > > > > > > >>>>>>> about the
> > > > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > > > > further.
> > > > > > I
> > > > > > > > >>>> saw
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> following in a response:
> > > > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > > > segment,
> > > > > > > > >>>> all the
> > > > > > > > >>>>>>>>> necessary
> > > > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> > including
> > > > > the
> > > > > > > > >>>>>>> transaction
> > > > > > > > >>>>>>>>>>> index will be built to serve FetchRequests
> exactly
> > as
> > > > > they
> > > > > > > > >>>> are
> > > > > > > > >>>>>>> today.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain
> the
> > > > > > > > >>>> information
> > > > > > > > >>>>>>> about
> > > > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>> Justine
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > > > > >>>> ivan@ivanyu.me>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > identified
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer ID
> > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > could be
> > > > > > > > >>>> cached
> > > > > > > > >>>>>>>>> locally
> > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> version 2
> > > > > > > > >>>>>> transactions
> > > > > > > > >>>>>>> can
> > > > > > > > >>>>>>>>> be
> > > > > > > > >>>>>>>>>>>> used
> > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > transactions
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer id +
> > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> > > > focused
> > > > > > > > >>>> only on
> > > > > > > > >>>>>>>>> version 2.
> > > > > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> > > > consider
> > > > > > > > >>>> using
> > > > > > > > >>>>>>> some
> > > > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > transactional
> > > > > > > > >>>>>>> checks
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > errors
> > > > > > > > >>>> like the
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > you be
> > > > > > > > >>>>>> checking
> > > > > > > > >>>>>>> if
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > > > > ongoing,
> > > > > > > > >>>> and
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>> course
> > > > > > > > >>>>>>>>>>>> the normal idempotence checks. What the
> partition
> > > > leader
> > > > > > > > >>>> in the
> > > > > > > > >>>>>>>>> classic
> > > > > > > > >>>>>>>>>>>> topics does before appending a batch to the
> local
> > log
> > > > > > > > >>>> (e.g. in
> > > > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > > > > Diskless,
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>> unfortunately
> > > > > > > > >>>>>>>>>>>> cannot do these checks before appending the data
> > to
> > > > the
> > > > > > WAL
> > > > > > > > >>>>>>> segment
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these
> > batches in
> > > > > the
> > > > > > > > >>>> batch
> > > > > > > > >>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> some
> > > > other
> > > > > > > > >>>> state
> > > > > > > > >>>>>>>>> mentioned
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > clear
> > > > > > > > >>>> what
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As
> the
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> leader
> > > > > > > > >>>>>>>>> tracks
> > > > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the
> batch
> > > > > > > > >>>> coordinator
> > > > > > > > >>>>>>> has
> > > > > > > > >>>>>>>>> to as
> > > > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > > > > >>>> transaction
> > > > > > > > >>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > LSO --
> > > > > > > > >>>> perhaps
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>> would
> > > > > > > > >>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> > batches
> > > > > and
> > > > > > > > >>>> other
> > > > > > > > >>>>>>>>>>>> information, but for the sake of performance
> we’d
> > > > better
> > > > > > > > >>>> store
> > > > > > > > >>>>>> it
> > > > > > > > >>>>>>>>>>>> explicitly.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > transactional
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > will
> > > > be
> > > > > > > > >>>>>> cleaned
> > > > > > > > >>>>>>> up?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > classic
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> forgets
> > > > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > overpasses
> > > > > > > > >>>> it).
> > > > > > > > >>>>>> The
> > > > > > > > >>>>>>>>>>>> transaction coordinator acts like the main
> > guardian,
> > > > > > > > >>>> allowing
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if
> > this
> > > > is
> > > > > > > > >>>> wrong.
> > > > > > > > >>>>>> We
> > > > > > > > >>>>>>>>> think
> > > > > > > > >>>>>>>>>>>> about relying on this with the batch coordinator
> > and
> > > > > > > > >>>> delete the
> > > > > > > > >>>>>>>>> information
> > > > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as
> > there’s no
> > > > > > > > >>>>>> replication
> > > > > > > > >>>>>>>>> and HWM
> > > > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>> Ivan
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> > wrote:
> > > > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> > transactions!
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > identified
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer ID
> > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > could be
> > > > > > > > >>>> cached
> > > > > > > > >>>>>>>>> locally
> > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> version 2
> > > > > > > > >>>>>> transactions
> > > > > > > > >>>>>>> can
> > > > > > > > >>>>>>>>> be
> > > > > > > > >>>>>>>>>>>> used
> > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > transactions
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer id +
> > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > transactional
> > > > > > > > >>>>>>> checks
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > errors
> > > > > > > > >>>> like the
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > you be
> > > > > > > > >>>>>> checking
> > > > > > > > >>>>>>> if
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is
> > there
> > > > > state
> > > > > > > > >>>>>> about
> > > > > > > > >>>>>>>>> ongoing
> > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> some
> > > > other
> > > > > > > > >>>> state
> > > > > > > > >>>>>>>>> mentioned
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > clear
> > > > > > > > >>>> what
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > LSO --
> > > > > > > > >>>> perhaps
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>> would
> > > > > > > > >>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > transactional
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > will
> > > > be
> > > > > > > > >>>>>> cleaned
> > > > > > > > >>>>>>> up?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> > local
> > > > > > > > >>>>>> segments,
> > > > > > > > >>>>>>>>> making
> > > > > > > > >>>>>>>>>>>> use of
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > zero-copy
> > > > > > > > >>>>>>>>> functionality
> > > > > > > > >>>>>>>>>>>> already
> > > > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> > state for
> > > > > > > > >>>> each
> > > > > > > > >>>>>>>>> partition in
> > > > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> > fetches
> > > > > > > > >>>> need
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> know
> > > > > > > > >>>>>>>>>>>> aborted
> > > > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > > > >>>> transaction is
> > > > > > > > >>>>>>>>> finished
> > > > > > > > >>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > > > corresponding
> > > > > > > > >>>> logs
> > > > > > > > >>>>>>> of the
> > > > > > > > >>>>>>>>>>>> involved
> > > > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the
> > metadata
> > > > > > > > >>>> level,
> > > > > > > > >>>>>> no
> > > > > > > > >>>>>>>>> actual
> > > > > > > > >>>>>>>>>>>> control
> > > > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > > > transactional
> > > > > > > > >>>>>>> batches.
> > > > > > > > >>>>>>>>> How
> > > > > > > > >>>>>>>>>>>> does the
> > > > > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> > > > control
> > > > > > > > >>>>>> batch
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> each
> > > > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response
> > needs to
> > > > > be
> > > > > > > > >>>>>>> built by
> > > > > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> > > > batches
> > > > > > > > >>>>>>> together?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share
> partition
> > > > leader
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> all
> > > > > > > > >>>>>>>>>>>> consumers
> > > > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
> > this
> > > > or
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>> too
> > > > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> > > > include
> > > > > > > > >>>>>> enough
> > > > > > > > >>>>>>>>> details
> > > > > > > > >>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
> > given
> > > > > > > > >>>>>> producer
> > > > > > > > >>>>>>>>> contain a
> > > > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a
> producer
> > > > sends
> > > > > > > > >>>>>> every
> > > > > > > > >>>>>>>>> Produce
> > > > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests
> > could
> > > > > > > > >>>> reach
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> batch
> > > > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> > rejection
> > > > > > > > >>>> of the
> > > > > > > > >>>>>>>>> produce
> > > > > > > > >>>>>>>>>>>>>> requests.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
> Yurchenko <
> > > > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details
> > about
> > > > > > > > >>>>>>> transactions
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>>>> queues,
> > > > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic
> > topics* is
> > > > > > > > >>>> based
> > > > > > > > >>>>>>> on
> > > > > > > > >>>>>>>>> precise
> > > > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> > (mostly
> > > > > > > > >>>>>>> producers,
> > > > > > > > >>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > > > > >>>> classes),
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition
> > leaders
> > > > with
> > > > > > > > >>>>>> their
> > > > > > > > >>>>>>>>> local
> > > > > > > > >>>>>>>>>>>> state
> > > > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The high level (some details skipped)
> workflow
> > is
> > > > the
> > > > > > > > >>>>>>>>> following.
> > > > > > > > >>>>>>>>>>>> When a
> > > > > > > > >>>>>>>>>>>>>>> transactional Produce request is received by
> > the
> > > > > > > > >>>> broker:
> > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> > checks
> > > > > > > > >>>> if a
> > > > > > > > >>>>>>>>> non-empty
> > > > > > > > >>>>>>>>>>>>>>> transaction is running for this partition.
> > This is
> > > > > > > > >>>> done
> > > > > > > > >>>>>>> using
> > > > > > > > >>>>>>>>> its
> > > > > > > > >>>>>>>>>>>> local
> > > > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > > > >>>>>> (ProducerStateManager,
> > > > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> > about
> > > > all
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>> partitions
> > > > > > > > >>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include
> them.
> > > > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > > > transactional
> > > > > > > > >>>>>>> checks.
> > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the
> > transactional
> > > > > > > > >>>> data to
> > > > > > > > >>>>>>>>> their logs
> > > > > > > > >>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>> update some of their state (for example, log
> > the
> > > > fact
> > > > > > > > >>>>>> that
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> > offset).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > coordinator
> > > > > > > > >>>>>>> directly
> > > > > > > > >>>>>>>>> with
> > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > PREPARE_COMMIT
> > > > > > > > >>>> or
> > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the
> transaction
> > > > > > > > >>>> markers to
> > > > > > > > >>>>>>>>> their logs
> > > > > > > > >>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final
> transaction
> > > > state
> > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders
> and
> > lots
> > > > > > > > >>>> of
> > > > > > > > >>>>>>>>> important
> > > > > > > > >>>>>>>>>>>> state
> > > > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is
> > local.
> > > > The
> > > > > > > > >>>> main
> > > > > > > > >>>>>>>>> challenge
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
> > there
> > > > > > > > >>>> are no
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
> > need
> > > > > > > > >>>> to be
> > > > > > > > >>>>>>>>> globalized
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this
> to
> > > > > > > > >>>> support
> > > > > > > > >>>>>>>>> idempotent
> > > > > > > > >>>>>>>>>>>>>> produce.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics*
> > would
> > > > > > > > >>>> look
> > > > > > > > >>>>>>> very
> > > > > > > > >>>>>>>>>>>> similar:
> > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > > > > >>>> non-empty
> > > > > > > > >>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> > classic
> > > > > > > > >>>>>> topics,
> > > > > > > > >>>>>>>>> this is
> > > > > > > > >>>>>>>>>>>>>> checked
> > > > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single
> > RPC.
> > > > > > > > >>>> Since a
> > > > > > > > >>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>>>> could
> > > > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> > epoch,
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> positive
> > > > > > > > >>>>>>>>>>>> result of
> > > > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> > double
> > > > > > > > >>>>>>> configured
> > > > > > > > >>>>>>>>>>>> duration
> > > > > > > > >>>>>>>>>>>>>> of a
> > > > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > > > informed
> > > > > > > > >>>>>> about
> > > > > > > > >>>>>>> all
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the
> transaction
> > to
> > > > > > > > >>>> include
> > > > > > > > >>>>>>> them.
> > > > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the
> > broker
> > > > > > > > >>>> side.
> > > > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data
> > to the
> > > > > > > > >>>>>> current
> > > > > > > > >>>>>>>>> shared
> > > > > > > > >>>>>>>>>>>> WAL
> > > > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> > transaction-related
> > > > > > > > >>>> state
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> Diskless
> > > > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > > > > >>>> coordinator
> > > > > > > > >>>>>>> like
> > > > > > > > >>>>>>>>> in the
> > > > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > > > transactional
> > > > > > > > >>>>>>> checks
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> > > > errors
> > > > > > > > >>>> like
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> > > > batches
> > > > > > > > >>>>>> could
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>>>>> rejected.
> > > > > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage
> > in
> > > > the
> > > > > > > > >>>> WAL
> > > > > > > > >>>>>>>>> segment
> > > > > > > > >>>>>>>>>>>> file in
> > > > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is
> > preferable to
> > > > > > > > >>>> doing
> > > > > > > > >>>>>>> more
> > > > > > > > >>>>>>>>>>>> network
> > > > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > > > segments
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>>>>> relatively
> > > > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > coordinator
> > > > > > > > >>>>>>> directly
> > > > > > > > >>>>>>>>> with
> > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > PREPARE_COMMIT
> > > > > > > > >>>> or
> > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
> informs
> > the
> > > > > > > > >>>> batch
> > > > > > > > >>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
> the
> > > > > > > > >>>>>>> transaction is
> > > > > > > > >>>>>>>>>>>> finished
> > > > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > > > >>>> corresponding
> > > > > > > > >>>>>>> logs
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only
> on
> > the
> > > > > > > > >>>>>> metadata
> > > > > > > > >>>>>>>>> level, no
> > > > > > > > >>>>>>>>>>>>>>> actual control batches are written to any
> file.
> > > > They
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>>>>> dynamically
> > > > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations.
> We
> > > > could
> > > > > > > > >>>>>>>>> technically
> > > > > > > > >>>>>>>>>>>> write
> > > > > > > > >>>>>>>>>>>>>>> these control batches for real, but this
> would
> > mean
> > > > > > > > >>>> extra
> > > > > > > > >>>>>>>>> produce
> > > > > > > > >>>>>>>>>>>>>> latency,
> > > > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > > > > >>>> coordinator
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> save
> > > > > > > > >>>>>>>>>>>> these
> > > > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to
> > classic
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> now.
> > > > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> > write
> > > > the
> > > > > > > > >>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>> markers
> > > > > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final
> transaction
> > > > state
> > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce
> > flow, we
> > > > > > > > >>>> get:
> > > > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between
> brokers
> > and
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> batch
> > > > > > > > >>>>>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>>>>> when a new partition appear in the
> > transaction. To
> > > > > > > > >>>>>>> mitigate the
> > > > > > > > >>>>>>>>>>>> impact of
> > > > > > > > >>>>>>>>>>>>>>> them:
> > > > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> > Produce
> > > > > > > > >>>>>>> request
> > > > > > > > >>>>>>>>> will be
> > > > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized
> for
> > > > > > > > >>>> fast
> > > > > > > > >>>>>>>>> response to
> > > > > > > > >>>>>>>>>>>> these
> > > > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally
> will
> > > > > > > > >>>>>>> communicate
> > > > > > > > >>>>>>>>> with a
> > > > > > > > >>>>>>>>>>>>>>> single broker for the duration of the
> > transaction
> > > > > > > > >>>> further
> > > > > > > > >>>>>>>>> reduces the
> > > > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the
> transaction
> > > > > > > > >>>>>> coordinator
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> batch
> > > > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be
> > able
> > > > to
> > > > > > > > >>>>>> span
> > > > > > > > >>>>>>> both
> > > > > > > > >>>>>>>>>>>> classic
> > > > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The share group coordination and management
> is
> > a
> > > > > > > > >>>> side job
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > > > > >>>> replicas,
> > > > > > > > >>>>>>> physical
> > > > > > > > >>>>>>>>>>>> storage
> > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > > > consumers
> > > > > > > > >>>>>>> (Fetch and
> > > > > > > > >>>>>>>>>>>> Produce
> > > > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > > > affected.)
> > > > > > > > >>>> We
> > > > > > > > >>>>>>> don't
> > > > > > > > >>>>>>>>> see any
> > > > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> > compatible
> > > > > > > > >>>> with
> > > > > > > > >>>>>>> share
> > > > > > > > >>>>>>>>>>>> groups the
> > > > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the
> > code
> > > > > > > > >>>> level,
> > > > > > > > >>>>>> we
> > > > > > > > >>>>>>>>> don't
> > > > > > > > >>>>>>>>>>>> expect
> > > > > > > > >>>>>>>>>>>>>> any
> > > > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading
> routines
> > are
> > > > > > > > >>>> used
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>> are
> > > > > > > > >>>>>>>>>>>> used for
> > > > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this
> or
> > it's
> > > > > > > > >>>> too
> > > > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> > wrote:
> > > > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design
> > input
> > > > > > > > >>>> on
> > > > > > > > >>>>>>>>> KIP-1150.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> > with a
> > > > > > > > >>>> new
> > > > > > > > >>>>>>>>> design. To
> > > > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
> the
> > > > > > > > >>>> existing
> > > > > > > > >>>>>>>>> KIP-405
> > > > > > > > >>>>>>>>>>>> Tiered
> > > > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced
> > to a
> > > > > > > > >>>>>>> Diskless
> > > > > > > > >>>>>>>>> topic
> > > > > > > > >>>>>>>>>>>> to be
> > > > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for
> > the
> > > > > > > > >>>> Batch
> > > > > > > > >>>>>>>>> Coordinator
> > > > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose
> with
> > > > > > > > >>>> Tiered
> > > > > > > > >>>>>>> Storage
> > > > > > > > >>>>>>>>>>>> plugin
> > > > > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative
> > data
> > > > > > > > >>>>>> formats.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
> local
> > > > > > > > >>>> segments,
> > > > > > > > >>>>>>>>> making use
> > > > > > > > >>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > > > > >>>> zero-copy
> > > > > > > > >>>>>>>>> functionality
> > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered
> > cache
> > > > > > > > >>>>>>> elements,
> > > > > > > > >>>>>>>>> do not
> > > > > > > > >>>>>>>>>>>> need
> > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > > > > >>>> contacting
> > > > > > > > >>>>>>> any
> > > > > > > > >>>>>>>>> other
> > > > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified
> > substantially,
> > > > by
> > > > > > > > >>>>>>> removing
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> previous
> > > > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > > > >>>> component, and
> > > > > > > > >>>>>>>>> "object
> > > > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > > > > >>>> enabled by
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> Batch
> > > > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
> as
> > the
> > > > > > > > >>>>>> earlier
> > > > > > > > >>>>>>>>> design,
> > > > > > > > >>>>>>>>>>>> while
> > > > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into
> the
> > > > > > > > >>>> existing
> > > > > > > > >>>>>>>>> ecosystem.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> > feedback on
> > > > > > > > >>>> the
> > > > > > > > >>>>>> new
> > > > > > > > >>>>>>>>> design.
> > > > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> > support
> > > > > > > > >>>> of
> > > > > > > > >>>>>> all
> > > > > > > > >>>>>>>>> existing
> > > > > > > > >>>>>>>>>>>>>> client
> > > > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > > > >>>> supporting
> > > > > > > > >>>>>> APIs
> > > > > > > > >>>>>>>>> like
> > > > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> > Siekierski
> > > > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> > current
> > > > > > > > >>>>>>> status of
> > > > > > > > >>>>>>>>> this
> > > > > > > > >>>>>>>>>>>> KIP?
> > > > > > > > >>>>>>>>>>>>>> I
> > > > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> > past 2
> > > > > > > > >>>>>>> months,
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>> vote got
> > > > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have
> been
> > > > > > > > >>>>>> answered
> > > > > > > > >>>>>>>>> since
> > > > > > > > >>>>>>>>>>>> then.
> > > > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > > > >>>> activity
> > > > > > > > >>>>>>> since
> > > > > > > > >>>>>>>>> May.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> > real
> > > > > > > > >>>>>> choices
> > > > > > > > >>>>>>>>> that can
> > > > > > > > >>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> > being
> > > > > > > > >>>> by
> > > > > > > > >>>>>> far
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>> dominant
> > > > > > > > >>>>>>>>>>>>>> one
> > > > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all
> these
> > are
> > > > > > > > >>>>>>>>> standalone
> > > > > > > > >>>>>>>>>>>>>> products.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
> would
> > > > > > > > >>>> hate to
> > > > > > > > >>>>>>> see a
> > > > > > > > >>>>>>>>>>>> stall. So
> > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
> Colin
> > > > > > > > >>>>>> McCabe -
> > > > > > > > >>>>>>>>> have your
> > > > > > > > >>>>>>>>>>>>>>> questions
> > > > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I
> > can
> > > > > > > > >>>> do
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>>>> help? I'm
> > > > > > > > >>>>>>>>>>>>>> deeply
> > > > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > > > > >>>> written
> > > > > > > > >>>>>>> about
> > > > > > > > >>>>>>>>> both?
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > > > > >>>>>> Kozlovski
> > > > > > > > >>>>>>> <
> > > > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > > > > >>>> main
> > > > > > > > >>>>>> one
> > > > > > > > >>>>>>>>> (just a
> > > > > > > > >>>>>>>>>>>> bunch
> > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > > > >>>> example, I
> > > > > > > > >>>>>>> was
> > > > > > > > >>>>>>>>>>>> searching
> > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while;
> As
> > > > > > > > >>>> far
> > > > > > > > >>>>>> as
> > > > > > > > >>>>>>> I
> > > > > > > > >>>>>>>>> can
> > > > > > > > >>>>>>>>>>>> tell, it
> > > > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
> of
> > > > > > > > >>>> having
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> root
> > > > > > > > >>>>>>>>>>>> KIP be
> > > > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> > links
> > > > > > > > >>>> to
> > > > > > > > >>>>>>>>> sub-KIPs
> > > > > > > > >>>>>>>>>>>> where
> > > > > > > > >>>>>>>>>>>>>> were
> > > > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think
> it
> > > > > > > > >>>> would
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> good to
> > > > > > > > >>>>>>>>>>>>>> create
> > > > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> > about
> > > > > > > > >>>>>>>>> introducing a
> > > > > > > > >>>>>>>>>>>> new
> > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > > > >>>> Storage as
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> primary
> > > > > > > > >>>>>>>>>>>>>>> source of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > > > > >>>> decided
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>> split it
> > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > > > >>>> should aim
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>> have
> > > > > > > > >>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > > > > >>>>>> details
> > > > > > > > >>>>>>> on
> > > > > > > > >>>>>>>>> how to
> > > > > > > > >>>>>>>>>>>>>>> implement
> > > > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when
> > we
> > > > > > > > >>>>>>> discussed
> > > > > > > > >>>>>>>>>>>> KRaft.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
> really
> > > > > > > > >>>> hard
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>> discuss
> > > > > > > > >>>>>>>>>>>> on
> > > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > > > > >>>> (linked
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > > > >>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> > DISCUSS
> > > > > > > > >>>>>>> threads
> > > > > > > > >>>>>>>>> for
> > > > > > > > >>>>>>>>>>>> each
> > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > > > >>>> whether
> > > > > > > > >>>>>>> Apache
> > > > > > > > >>>>>>>>> Kafka
> > > > > > > > >>>>>>>>>>>> wants
> > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > > > >>>> authors
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>> this KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
>  +491715557497 |
> > > > > > > > >>>>>>> aiven.io
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > > > > >>>>>>> Valtonen,
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Anatolii Popov
> > > > Senior Software Developer, *Aiven OY*
> > > > m: +358505126242
> > > > w: aiven.io  e: anatolii.popov@aiven.io
> > > > <https://www.facebook.com/aivencloud>
> > > > <https://www.linkedin.com/company/aiven/>   <
> > https://twitter.com/aiven_io>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Robin Han <ha...@gmail.com>]

Hi Jun,

The 'simpler approach' I mentioned earlier can solve the cross-AZ traffic
costs for both sending and replication simultaneously. And it can easily
integrate with existing Apache Kafka features without having to rewrite
code for Transaction, Queue, Compaction, and so on. And there is no need to
implement it twice in Classic Topic and Diskless Topic when Apache Kafka
adds new features in the future.

Robin

Jun Rao via dev <de...@kafka.apache.org> 于2026年1月21日周三 07:30写道：

> Hi, Ivan,
>
> Thanks for the reply.
>
> JR10. "Cluster scalability increases when less data is required to be
> stored on disks and rebalanced between brokers."
> This seems to contradict a bit with what KIP-1163 says: "Start building it
> from some configurable amount, e.g. 100 MiB prior to the current log end or
> earliest available offset, whichever is later.".  I just want to confirm
> that we plan to make data rebalancing much faster in this KIP.
>
> JR11. Sounds good.
>
> JR13. Since people keep asking about the "simpler" approach, it would be
> useful to include in the rejected section why we chose not to pursue the
> "simpler" approach like the one in KIP-1176. To me, there are two main
> reasons: (1) this KIP provides more benefits (saving all cross AZ network
> costs instead of just the replication part, better scalability and
> durability); (2) it provides an opportunity for a cleaner long-term design
> instead of retrofitting to the existing framework unnaturally.
>
> Jun
>
>
>
> On Tue, Jan 20, 2026 at 11:22 AM Greg Harris via dev <dev@kafka.apache.org
> >
> wrote:
>
> > Hi Robin,
> >
> > Thanks for your design sketch.
> >
> > Based on your description, the GET operations on the leader and followers
> > must complete before offsets are assigned, adding Produce request latency
> > that is not present in the KIP-1163 design.
> >
> > KIP-1163 also assumes that object storage is more reliable than disk
> > storage, which is the case in the hyperscaler clouds. The parts of your
> > design that make it compatible with low-reliability object storage become
> > counterproductive when Diskless is used with a high-reliability object
> > storage, such as re-introducing direct replication and cross-az costs.
> >
> > We’ve already discussed and rejected KIP-1176 (another “simpler”
> proposal)
> > in favor of a complete solution. The folks who championed that proposal
> are
> > now working with us on KIP-1163, and are currently designing Compaction
> in
> > Tiered Storage to make Diskless a complete solution for compacted topics
> as
> > well. Similarly, we would appreciate your contributions to the design of
> > KIP-1163, and your vote on the motivational KIP-1150.
> >
> > Thanks!
> >
> > On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi Jun,
> > >
> > > Here are our answers to your questions and comments:
> > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing
> to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > >
> > > Thank you, good point, added this.
> > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > >
> > > Yes, there is a per-request cost for object storage which is not
> directly
> > > comparable to inter-zone transfer costs. We have found that the design
> in
> > > KIP-1163 amortizes those request costs such that they are insignificant
> > > compared to the $0.02/gb cost which is called out in KIP-1150.
> > > We currently have very precise cost models that were put together for
> an
> > > earlier iteration of the Diskless topics. These are somewhat stale and
> > > inaccurate for the latest design, so we will work in the background to
> > post
> > > concrete numbers in KIP-1163 before voting on that proposal. Would this
> > be
> > > OK?
> > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > >
> > > Thanks for calling out. We have updated KIP-1164 with the latest
> design.
> > > It's missing public interfaces, but we think it may be a bit too early
> to
> > > define them as some rounds of discussion is ahead, with potential to
> > change
> > > the design. But otherwise the KIP is good for the first review.
> > > We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
> > >
> > > Best,
> > > Ivan
> > >
> > > On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> > > > Hi, Anatolii,
> > > >
> > > > Thanks for the updated KIP. Looks good to me overall. A few more
> > > comments.
> > > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing
> to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > > >
> > > > Jun
> > > >
> > > > On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> > > dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thank you for the extensive feedback.
> > > > >
> > > > > We have substantially updated the KIP to address the points raised.
> > > Given
> > > > > the scope of these changes, we have started a new VOTE thread to
> > > restart
> > > > > the voting process cleanly.
> > > > >
> > > > > You can find the new thread here:
> > > > > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> > > > >
> > > > > Best,
> > > > > Anatolii
> > > > >
> > > > > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > > > > dev@kafka.apache.org> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We (the team at Slack) have been following the recent discussion
> > > > > regarding
> > > > > > the scope and timeline of KIP-1150. We agree with the community
> > > sentiment
> > > > > > that Diskless Topics represents the right long-term architecture
> > for
> > > > > Kafka
> > > > > > in the cloud, but we also recognize the valid concerns raised
> > > regarding
> > > > > the
> > > > > > engineering resources required to deliver such an ambitious
> change.
> > > > > >
> > > > > > To help address these concerns and accelerate the timeline, we
> are
> > > happy
> > > > > to
> > > > > > announce that we are partnering with the KIP-1150 authors to
> > > co-develop
> > > > > > this feature.
> > > > > >
> > > > > > We previously proposed KIP-1176: Tiered Storage for Active Log
> > > Segments
> > > > > to
> > > > > > solve similar problems. However, rather than fragmenting the
> > > community's
> > > > > > efforts across competing designs, we have decided to withdraw
> > > KIP-1176
> > > > > and
> > > > > > consolidate our engineering resources behind KIP-1150.
> > > > > >
> > > > > > To start, we plan to take ownership of Compaction for Tiered
> > Storage.
> > > > > This
> > > > > > has long been a missing feature in KIP-405, and it becomes
> critical
> > > in a
> > > > > > Diskless architecture where long-lived data is obligated to be
> > > tiered. By
> > > > > > driving this related prerequisite feature, we hope to allow for
> > > faster
> > > > > > delivery of KIP-1150.
> > > > > >
> > > > > > We are excited to collaborate on this to ensure a robust and
> timely
> > > > > > delivery for the community.
> > > > > >
> > > > > > Best,
> > > > > > Tom & Henry
> > > > > >
> > > > > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Thanks for sharing the meeting notes.
> > > > > > > I agree we should keep polishing the contents of 1150 & high
> > level
> > > > > design
> > > > > > > in 1163 to prepare for a vote.
> > > > > > >
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > There was a video call between myself, Ivan Yurchenko, Jun
> Rao,
> > > and
> > > > > > > Andrew
> > > > > > > > Schofield pertaining to KIP-1150. Here are the notes from
> that
> > > > > meeting:
> > > > > > > >
> > > > > > > > Ivan: What is the future state of Kafka in this area, in 5
> > years?
> > > > > > > > Jun: Do we want something more cloud native? Yes, started
> with
> > > Tiered
> > > > > > > > Storage. If there’s a better way, we should explore it. In
> the
> > > long
> > > > > > term
> > > > > > > > this will be useful
> > > > > > > > Because Kafka is used so widely, we need to make sure
> > everything
> > > we
> > > > > add
> > > > > > > is
> > > > > > > > for the long term and for everyone, not just for a single
> > > company.
> > > > > > > > When we add TS, it doesn’t just solve Uber’s use-case. We
> want
> > > > > > something
> > > > > > > > that’s high quality/lasts/maintainable, and can work with all
> > > > > existing
> > > > > > > > capabilities.
> > > > > > > > If both 1150 and 1176 proceed at the same time, it’s
> confusing.
> > > They
> > > > > > > > overlap, but Diskless is more ambitious.
> > > > > > > > If both KIPs are being seriously worked on, then we don’t
> > really
> > > need
> > > > > > > both,
> > > > > > > > because Diskless clearly is better. Having multiple will
> > confuse
> > > > > > people.
> > > > > > > It
> > > > > > > > will duplicate some of the effort.
> > > > > > > > If we want diskless ultimately, what is the short term
> > strategy,
> > > to
> > > > > get
> > > > > > > > some early wins first?
> > > > > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > > > > Andrew: Eventually the architecture will change
> substantially,
> > > it may
> > > > > > not
> > > > > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > > > > Greg: We all agree on having a high quality feature merged
> > > upstream,
> > > > > > and
> > > > > > > > supporting all APIs
> > > > > > > > Jun: We should try and keep things simple, but there is some
> > > minimum
> > > > > > > > complexity needed.
> > > > > > > > When doing the short term changes (1176), it doesn’t really
> > > progress
> > > > > in
> > > > > > > > changing to a more modern architecture.
> > > > > > > > Greg: Was TS+Compaction the only feature miss we’ve had so
> far?
> > > > > > > > Jun: The danger of only applying changes to some part of the
> > > API, you
> > > > > > set
> > > > > > > > the precedence that you only have to implement part of the
> API.
> > > > > > > Supporting
> > > > > > > > the full API set should be a minimum requirement.
> > > > > > > > Andrew: When we started Kraft, how much did we know the
> design?
> > > > > > > > Jun: For Kraft we didn’t really know much about the
> migration,
> > > but
> > > > > the
> > > > > > > > high-level was clear.
> > > > > > > > Greg: Is 1150 votable in its current state?
> > > > > > > > Jun: 1150 should promise to support all APIs. It doesn’t have
> > to
> > > have
> > > > > > all
> > > > > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > > > > We do need some high-level design enough to give confidence
> > that
> > > the
> > > > > > > > promise is able to be fulfilled.
> > > > > > > > Greg: Is the draft version in 1163 enough detail or is more
> > > needed?
> > > > > > > > Jun: We need to agree on the core design, such as leaderless
> > > etc. And
> > > > > > how
> > > > > > > > the APIs will be supported.
> > > > > > > > Greg: Okay we can include these things, and provide a sketch
> of
> > > how
> > > > > the
> > > > > > > > other leader-based features operate.
> > > > > > > > Jun: Yeah if at a high level the sketch appears to work, we
> can
> > > > > approve
> > > > > > > > that functionality.
> > > > > > > > Are you committed to doing the more involved and big project?
> > > > > > > > Greg: Yes, we’re committed to the 1163 design and can’t
> really
> > > accept
> > > > > > > 1176.
> > > > > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > > > > Greg: We’ll push internally for resources, and use the
> > community
> > > > > > > sentiment
> > > > > > > > to motivate Aiven.
> > > > > > > > How far into the future should we look? What sort of scale?
> > > > > > > > Jun: As long as there’s a path forward, and we’re not closing
> > off
> > > > > > future
> > > > > > > > improvements, we can figure out how to handle a larger scale
> > > when it
> > > > > > > > arises.
> > > > > > > > Greg: Random replica placement is very harmful, can we
> > recommend
> > > > > users
> > > > > > to
> > > > > > > > use an external tool like CruiseControl?
> > > > > > > > Jun: Not everyone uses CruiseControl, we would probably need
> > some
> > > > > > > solution
> > > > > > > > for this out of the box
> > > > > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > > > > Jun: Out-of-box experience should be good, good to allow
> other
> > > > > > > > implementations
> > > > > > > > Greg: But it could hurt Kafka feature/upgrade velocity when
> we
> > > wait
> > > > > for
> > > > > > > > plugin providers to implement it
> > > > > > > > Ivan: We imagined that maybe cloud hyperscalers could
> implement
> > > it
> > > > > with
> > > > > > > > e.g. dynamodb
> > > > > > > > Greg: Could we bake more details of the different providers
> > into
> > > > > Kafka,
> > > > > > > or
> > > > > > > > does it still make sense for it to be pluggable?
> > > > > > > > Jun: Make it whatever is easiest to roll out and add new
> > clients
> > > > > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > > > > Ivan: The vote is already open, we’re not too pressed for
> time.
> > > We’ll
> > > > > > go
> > > > > > > > improve the 1163 design and communication.
> > > > > > > > Is 1176 a competing design? Someone will ask.
> > > > > > > > Jun: If we are seriously working on something more ambitious,
> > > yeah we
> > > > > > > > shouldn’t do the stop-gap solution.
> > > > > > > > It’s diverting review resources. If we can get the short term
> > > thing
> > > > > in
> > > > > > > 1yr
> > > > > > > > but Diskless solution is 2y it makes sense to go for
> Diskless.
> > If
> > > > > it’s
> > > > > > > 5yr,
> > > > > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case.
> > Should
> > > we
> > > > > > > > explicitly exclude 1176?
> > > > > > > > Andrew: Put your arms around the feature set you actually
> want,
> > > and
> > > > > use
> > > > > > > > that to rule out 1176.
> > > > > > > > Probably don’t need -1 votes, most likely KIPs just don’t
> > receive
> > > > > > votes.
> > > > > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > > > > Jun: Satish posted meeting notes regularly, we should do the
> > > same.
> > > > > > > >
> > > > > > > > To summarize, we will be polishing the contents of 1150 &
> high
> > > level
> > > > > > > design
> > > > > > > > in 1163 to prepare for a vote.
> > > > > > > > We believe that the community should select the feature set
> of
> > > 1150
> > > > > to
> > > > > > > > fully eliminate producer cross-zone costs, and make the
> > > investment
> > > > > in a
> > > > > > > > high quality Diskless Topics implementation rather than in
> > > stop-gap
> > > > > > > > solutions.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
> > wrote:
> > > > > > > >
> > > > > > > > > This may be a tangent, but we needed to offload storage off
> > of
> > > > > Kafka
> > > > > > > into
> > > > > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> > > mostly
> > > > > > > > > ephemeral broker that can come and go as it pleases. Be
> that
> > > > > scaling
> > > > > > or
> > > > > > > > > outage. Disks can be destroyed and recreated at will, we
> > still
> > > > > retain
> > > > > > > > data
> > > > > > > > > and use broker for just that, brokering messages. Not only
> > > that, we
> > > > > > > > reduced
> > > > > > > > > the requirement on the actual Kafka resources by reducing
> the
> > > size
> > > > > > of a
> > > > > > > > > payload via a claim check pattern. Maybe this is an
> > > anti–pattern,
> > > > > but
> > > > > > > it
> > > > > > > > is
> > > > > > > > > super fast and highly cost efficient. We reworked
> > > ProducerRequest
> > > > > to
> > > > > > > > allow
> > > > > > > > > plugins. We added a custom http plugin that submits every
> > > request
> > > > > > via a
> > > > > > > > > persisted connection to a microservice. Microservice stores
> > the
> > > > > > payload
> > > > > > > > and
> > > > > > > > > returns a tiny json metadata object,a claim check, that can
> > be
> > > used
> > > > > > to
> > > > > > > > find
> > > > > > > > > the actual data. Think of it as zipping the payload. This
> > claim
> > > > > check
> > > > > > > > > metadata traverses the pipelines with consumers using the
> > urls
> > > in
> > > > > > > > metadata
> > > > > > > > > to pull what they need. Think unzipping. This allowed us to
> > > also
> > > > > pull
> > > > > > > > ONLY
> > > > > > > > > the data that we need in graphql like manner. So if you
> have
> > a
> > > 100K
> > > > > > > json
> > > > > > > > > payload and you need only a subsection, you can pull that
> by
> > > > > > jmespath.
> > > > > > > > When
> > > > > > > > > you have multiple consumer groups yanking down huge
> payloads
> > > it is
> > > > > > > > > cumbersome on the broker. When you have the same consumer
> > > groups
> > > > > > > yanking
> > > > > > > > > down a claim check, and then going out of band directly to
> > the
> > > > > source
> > > > > > > of
> > > > > > > > > truth, the broker has some breathing room. Obviously our
> > > > > microservice
> > > > > > > > does
> > > > > > > > > not go directly to the cloud storage, as that would be too
> > > slow. It
> > > > > > > > stores
> > > > > > > > > the payload in high speed memory cache and returns asap.
> That
> > > > > memory
> > > > > > is
> > > > > > > > > eventually persisted into S3. The retrieval goest against
> the
> > > cache
> > > > > > > > first,
> > > > > > > > > then against the S3. Overall a rather cheappy and zippy
> > > solution. I
> > > > > > > tried
> > > > > > > > > proposing the KIP for this, but there was no excitement.
> > Check
> > > this
> > > > > > > out:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi, Andrew,
> > > > > > > > > >
> > > > > > > > > > If we want to focus only on reducing cross-zone
> replication
> > > > > costs,
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > an alternative design in the KIP-1176 discussion thread
> > that
> > > > > seems
> > > > > > > > > simpler
> > > > > > > > > > than the proposal here. I am copying the outline of that
> > > approach
> > > > > > > > below.
> > > > > > > > > >
> > > > > > > > > > 1. A new leader is elected.
> > > > > > > > > > 2. Leader maintains a first tiered offset, which is
> > > initialized
> > > > > to
> > > > > > > log
> > > > > > > > > end
> > > > > > > > > > offset.
> > > > > > > > > > 3. Leader writes produced data from the client to local
> > log.
> > > > > > > > > > 4. Leader uploads produced data from all local logs as a
> > > combined
> > > > > > > > object
> > > > > > > > > > 5. Leader stores the metadata for the combined object in
> > > memory.
> > > > > > > > > > 6. If a follower fetch request has an offset >= first
> > tiered
> > > > > > offset,
> > > > > > > > the
> > > > > > > > > > metadata for the corresponding combined object is
> returned.
> > > > > > > Otherwise,
> > > > > > > > > the
> > > > > > > > > > local data is returned.
> > > > > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > > > > >
> > > > > > > > > > It's still a bit unnatural, but it could work.
> > > > > > > > > >
> > > > > > > > > > Hi, Ivan,
> > > > > > > > > >
> > > > > > > > > > Are you still committed to proceeding with the original
> > > design of
> > > > > > > > > KIP-1150?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >> I’ve been following KIP-1150 and friends for a while.
> I’m
> > > going
> > > > > to
> > > > > > > > jump
> > > > > > > > > >> into the discussions too.
> > > > > > > > > >>
> > > > > > > > > >> Looking back at Jack Vanlightly’s message, I am not
> quite
> > so
> > > > > > > convinced
> > > > > > > > > >> that it’s a kind of fork in the road. The primary aim of
> > the
> > > > > > effort
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > >> reduce cross-zone replication costs so Apache Kafka is
> not
> > > > > > > > prohibitively
> > > > > > > > > >> expensive to use on cloud storage. I think it would be
> > > entirely
> > > > > > > viable
> > > > > > > > > to
> > > > > > > > > >> prioritise code reuse for an initial implementation of
> > > diskless
> > > > > > > > topics,
> > > > > > > > > and
> > > > > > > > > >> we could still have a more cloud-native design in the
> > > future.
> > > > > It’s
> > > > > > > > hard
> > > > > > > > > to
> > > > > > > > > >> predict what the community will prioritise in the
> future.
> > > > > > > > > >>
> > > > > > > > > >> Of the three major revisions, I’m in the rev3 camp. We
> can
> > > > > support
> > > > > > > > > >> leaderless produce requests, first writing WAL segments
> > into
> > > > > > object
> > > > > > > > > >> storage, and then using the regular partition leaders to
> > > > > sequence
> > > > > > > the
> > > > > > > > > >> records. The active log segment for a diskless topic
> will
> > > > > > initially
> > > > > > > > > contain
> > > > > > > > > >> batch coordinates rather than record batches. The batch
> > > > > > coordinates
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > > >> resolved from WAL segments for consumers, and also in
> > order
> > > to
> > > > > > > prepare
> > > > > > > > > log
> > > > > > > > > >> segments for uploading to tiered storage. Jun is
> probably
> > > > > correct
> > > > > > > that
> > > > > > > > > we
> > > > > > > > > >> need a more frequent object merging process than tiered
> > > storage
> > > > > > > > > provides.
> > > > > > > > > >> This is just the transition from write-optimised WAL
> > > segments to
> > > > > > > > > >> read-optimised tiered segments, and all of the object
> > > > > > storage-based
> > > > > > > > > >> implementations of Kafka that I’m aware of do this
> > > > > rearrangement.
> > > > > > > But
> > > > > > > > > >> perhaps this more frequent object merging is a pre-GA
> > > > > improvement,
> > > > > > > > > rather
> > > > > > > > > >> than a strict requirement for an initial implementation
> > for
> > > > > early
> > > > > > > > access
> > > > > > > > > >> use.
> > > > > > > > > >>
> > > > > > > > > >> For zone-aligned share consumers, the share group
> assignor
> > > is
> > > > > > > intended
> > > > > > > > > to
> > > > > > > > > >> be rack-aware. Consumers should be assigned to
> partitions
> > > with
> > > > > > > leaders
> > > > > > > > > in
> > > > > > > > > >> their zone. The simple assignor is not rack-aware, but
> it
> > > easily
> > > > > > > could
> > > > > > > > > be
> > > > > > > > > >> or we could have a rack-aware assignor.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Andrew
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> Hi, Ivan,
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > >>>
> > > > > > > > > >>> "As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>> will indeed consume some IOPS. See them as a cache that
> > > could
> > > > > > > always
> > > > > > > > be
> > > > > > > > > >>> restored from the remote storage. While it’s not ideal,
> > > it's
> > > > > > still
> > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > of
> > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>> can avoid explicit flushing on local materialized
> > segments
> > > at
> > > > > all
> > > > > > > and
> > > > > > > > > let
> > > > > > > > > >>> the file system and page cache figure out when to flush
> > > > > > optimally.
> > > > > > > > This
> > > > > > > > > >>> would not eliminate the extra IOPS, but should reduce
> it
> > > > > > > > dramatically,
> > > > > > > > > >>> depending on throughput for each partition. We, of
> > course,
> > > > > > continue
> > > > > > > > > >>> flushing the metadata segments as before."
> > > > > > > > > >>>
> > > > > > > > > >>> If we have a mix of classic and diskless topics on the
> > same
> > > > > > broker,
> > > > > > > > > it's
> > > > > > > > > >>> important that the classic topics' data is flushed to
> > disk
> > > as
> > > > > > > quickly
> > > > > > > > > as
> > > > > > > > > >>> possible. To achieve this, users typically set
> > > > > > > dirty_expire_centisecs
> > > > > > > > > in
> > > > > > > > > >>> the kernel based on the number of available disk IOPS.
> > > Once you
> > > > > > set
> > > > > > > > > this
> > > > > > > > > >>> number, it applies to all dirty files, including the
> > cached
> > > > > data
> > > > > > in
> > > > > > > > > >>> diskless topics. So, if there are more files actively
> > > > > > accumulating
> > > > > > > > > data,
> > > > > > > > > >>> the flush frequency and therefore RPO is reduced for
> > > classic
> > > > > > > topics.
> > > > > > > > > >>>
> > > > > > > > > >>> "We should have mentioned this explicitly, but this
> step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>> in the form of segments offloading to tiered storage.
> > When
> > > we
> > > > > > > > assemble
> > > > > > > > > a
> > > > > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> > > effectively
> > > > > > > doing
> > > > > > > > > >>> metadata compaction: replacing a big number of pieces
> of
> > > > > metadata
> > > > > > > > about
> > > > > > > > > >>> individual batches with a single record in
> > > > > > __remote_log_metadata."
> > > > > > > > > >>>
> > > > > > > > > >>> The object merging in tier storage typically only kicks
> > in
> > > > > after
> > > > > > a
> > > > > > > > few
> > > > > > > > > >>> hours. The impact is (1) the amount of accumulated
> > > metadata is
> > > > > > > still
> > > > > > > > > >> quite
> > > > > > > > > >>> large; (2) there are many small objects, leading to
> poor
> > > read
> > > > > > > > > >> performance.
> > > > > > > > > >>> I think we need a more frequent object merging process
> > than
> > > > > tier
> > > > > > > > > storage
> > > > > > > > > >>> provides.
> > > > > > > > > >>>
> > > > > > > > > >>> Jun
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me>
> > > > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thank you for your messages.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> First, it degrades the durability.
> > > > > > > > > >>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at a
> > > > > > > > > >>>>> given point of time, one for the data and another for
> > the
> > > > > > > metadata.
> > > > > > > > > >>>>> Flushing each file requires a separate IO. If the
> disk
> > > has 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >> we
> > > > > > > > > >>>>> have 5K partitions in a broker, currently we can
> afford
> > > to
> > > > > > flush
> > > > > > > > each
> > > > > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> number of files per partition, we can only flush each
> > > > > partition
> > > > > > > > every
> > > > > > > > > >> 10
> > > > > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>>> will indeed consume some IOPS. See them as a cache
> that
> > > could
> > > > > > > always
> > > > > > > > > be
> > > > > > > > > >>>> restored from the remote storage. While it’s not
> ideal,
> > > it's
> > > > > > still
> > > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > > of
> > > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>>> can avoid explicit flushing on local materialized
> > > segments at
> > > > > > all
> > > > > > > > and
> > > > > > > > > >> let
> > > > > > > > > >>>> the file system and page cache figure out when to
> flush
> > > > > > optimally.
> > > > > > > > > This
> > > > > > > > > >>>> would not eliminate the extra IOPS, but should reduce
> it
> > > > > > > > dramatically,
> > > > > > > > > >>>> depending on throughput for each partition. We, of
> > course,
> > > > > > > continue
> > > > > > > > > >>>> flushing the metadata segments as before.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s worth making a note on caching. I think nobody
> will
> > > > > > disagree
> > > > > > > > that
> > > > > > > > > >>>> doing direct reads from remote storage every time a
> > batch
> > > is
> > > > > > > > requested
> > > > > > > > > >> by a
> > > > > > > > > >>>> consumer will not be practical neither from the
> > > performance
> > > > > nor
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > >>>> economy point of view. We need a way to keep the
> number
> > > of GET
> > > > > > > > > requests
> > > > > > > > > >>>> down. There are multiple options, for example:
> > > > > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > > > > >>>> 2. Local in-memory caching. Comes with less network
> > > chattiness
> > > > > > and
> > > > > > > > > >> works
> > > > > > > > > >>>> well if we have more or less stable brokers to consume
> > > from.
> > > > > > > > > >>>> 3. Materialization of diskless logs on local disk. Way
> > > lower
> > > > > > > impact
> > > > > > > > on
> > > > > > > > > >>>> RAM and also requires stable brokers for consumption
> > > (using
> > > > > just
> > > > > > > > > >> assigned
> > > > > > > > > >>>> replicas will probably work well).
> > > > > > > > > >>>>
> > > > > > > > > >>>> Materialization is one of possible options, but we can
> > > choose
> > > > > > > > another
> > > > > > > > > >> one.
> > > > > > > > > >>>> However, we will have this dilemma regardless of
> whether
> > > we
> > > > > have
> > > > > > > an
> > > > > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Second, if we ever need this
> > > > > > > > > >>>>> metadata somewhere else, say in the WAL file manager,
> > the
> > > > > > > consumer
> > > > > > > > > >> needs
> > > > > > > > > >>>> to
> > > > > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>> actual benefit of this approach is also questionable.
> > On
> > > the
> > > > > > > > surface,
> > > > > > > > > >> it
> > > > > > > > > >>>>> might seem that we could reduce the number of lines
> > that
> > > need
> > > > > > to
> > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>> for this KIP. However, the changes are quite
> intrusive
> > > to the
> > > > > > > > classic
> > > > > > > > > >>>>> partition's code path and will probably make the code
> > > base
> > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>> maintain in the long run. I like the original
> approach
> > > based
> > > > > on
> > > > > > > the
> > > > > > > > > >> batch
> > > > > > > > > >>>>> coordinator much better than this one. We could
> > probably
> > > > > > refactor
> > > > > > > > the
> > > > > > > > > >>>>> producer state code so that it could be reused in the
> > > batch
> > > > > > > > > >> coordinator.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s hard to disagree with this. The explicit
> > coordinator
> > > is
> > > > > > more
> > > > > > > a
> > > > > > > > > side
> > > > > > > > > >>>> thing, while coordinator-less approach is more about
> > > extending
> > > > > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Thanks for addressing the concerns on the number of
> > RPCs
> > > in
> > > > > the
> > > > > > > > > produce
> > > > > > > > > >>>>> path. I agree that with the metadata crafting
> > mechanism,
> > > we
> > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > be
> > > > > > > > > >>>>> collocated with the data on the same set of brokers,
> > it's
> > > > > weird
> > > > > > > > that
> > > > > > > > > >> they
> > > > > > > > > >>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > uses
> > > > > > > > > >> the
> > > > > > > > > >>>>> metadata crafting mechanism, but the metadata is
> stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>> partition using its own assignment strategy. It will
> be
> > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>> them collocated.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I would like to note that the metadata crafting is
> > needed
> > > only
> > > > > > to
> > > > > > > > tell
> > > > > > > > > >>>> producers which brokers they should send Produce
> > requests
> > > to,
> > > > > > but
> > > > > > > > data
> > > > > > > > > >> (as
> > > > > > > > > >>>> in “locally materialized log”) is located on partition
> > > > > replicas,
> > > > > > > > i.e.
> > > > > > > > > >>>> automatically co-located with metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As a side note, it would probably be better that
> instead
> > > of
> > > > > > > > implicitly
> > > > > > > > > >>>> crafting partition metadata, we extend the metadata
> > > protocol
> > > > > so
> > > > > > > that
> > > > > > > > > for
> > > > > > > > > >>>> diskless partitions we return not only the leader and
> > > > > replicas,
> > > > > > > but
> > > > > > > > > also
> > > > > > > > > >>>> some “recommended produce brokers”, selected for
> optimal
> > > > > > > performance
> > > > > > > > > and
> > > > > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>> step.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We should have mentioned this explicitly, but this
> step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>>> in the form of segments offloading to tiered storage.
> > > When we
> > > > > > > > > assemble a
> > > > > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > > > > effectively
> > > > > > > > doing
> > > > > > > > > >>>> metadata compaction: replacing a big number of pieces
> of
> > > > > > metadata
> > > > > > > > > about
> > > > > > > > > >>>> individual batches with a single record in
> > > > > > __remote_log_metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We could create a Diskless-specific merging mechanism
> > > instead
> > > > > if
> > > > > > > > > needed.
> > > > > > > > > >>>> It’s rather easy with the explicit coordinator
> approach.
> > > With
> > > > > > the
> > > > > > > > > >>>> coordinator-less approach, this would probably be a
> bit
> > > more
> > > > > > > tricky
> > > > > > > > > >>>> (rewriting the tail of the log by the leader +
> > replicating
> > > > > this
> > > > > > > > change
> > > > > > > > > >>>> reliably).
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >> changes
> > > > > > > > > >>>> in
> > > > > > > > > >>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >> last
> > > > > > > > > >>>>> for the long term.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Yes, totally agree.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Luke,
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler
> if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>
> > > > > > > > > >>>> After some evolution of the original proposal,
> > leaderless
> > > is
> > > > > now
> > > > > > > > > >> limited.
> > > > > > > > > >>>> We only need to be able to accept Produce requests on
> > more
> > > > > than
> > > > > > > one
> > > > > > > > > >> broker
> > > > > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I
> get
> > it
> > > > > right
> > > > > > > > that
> > > > > > > > > >> you
> > > > > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let’s now look at this problem from a higher level, as
> > > Jack
> > > > > > > > proposed.
> > > > > > > > > As
> > > > > > > > > >>>> it was said, the big choice we need to make is whether
> > we
> > > 1)
> > > > > > > create
> > > > > > > > an
> > > > > > > > > >>>> explicit batch coordinator; or 2) go for the
> > > coordinator-less
> > > > > > > > > approach,
> > > > > > > > > >>>> where each diskless partition is managed by its leader
> > as
> > > in
> > > > > > > classic
> > > > > > > > > >> topics.
> > > > > > > > > >>>>
> > > > > > > > > >>>> If we try to compare the two approaches:
> > > > > > > > > >>>>
> > > > > > > > > >>>> Pluggability:
> > > > > > > > > >>>> - Explicit coordinator: Possible. For example, some
> > > setups may
> > > > > > > > benefit
> > > > > > > > > >>>> from batch metadata being stored in a cloud database
> > > (such as
> > > > > > AWS
> > > > > > > > > >> DynamoDB
> > > > > > > > > >>>> or GCP Spanner).
> > > > > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Scalability and fault tolerance:
> > > > > > > > > >>>> - Explicit coordinator: Depends on the implementation
> > and
> > > it’s
> > > > > > > also
> > > > > > > > > >>>> necessary to actively work for it.
> > > > > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics.
> > > Scaling is
> > > > > > > done
> > > > > > > > by
> > > > > > > > > >>>> partition placement, partitions could fail
> > independently.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Separation of concerns:
> > > > > > > > > >>>> - Explicit coordinator: Very good. Diskless remains
> more
> > > > > > > independent
> > > > > > > > > >> from
> > > > > > > > > >>>> classic topics in terms of code and workflows. For
> > > example,
> > > > > the
> > > > > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > > > > mechanism
> > > > > > > > could
> > > > > > > > > >> be
> > > > > > > > > >>>> relatively simply implemented with it. As a flip side
> of
> > > this,
> > > > > > > some
> > > > > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > > > > bringing
> > > > > > > > > diskless
> > > > > > > > > >>>> closer to classic topics. Some code paths and
> workflows
> > > could
> > > > > be
> > > > > > > > more
> > > > > > > > > >>>> straightforwardly reused, but they will inevitably
> have
> > > to be
> > > > > > > > adapted
> > > > > > > > > to
> > > > > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Cloud-nativeness. This is a vague concept, also
> related
> > > to the
> > > > > > > > > previous,
> > > > > > > > > >>>> but let’s try:
> > > > > > > > > >>>> - Explicit coordinator: Storing and processing
> metadata
> > > > > > separately
> > > > > > > > > makes
> > > > > > > > > >>>> it easier for brokers to take different roles, be
> purely
> > > > > > stateless
> > > > > > > > if
> > > > > > > > > >>>> needed, etc.
> > > > > > > > > >>>> - Coordinator-less: Less so. Something could be
> achieved
> > > with
> > > > > > > > creative
> > > > > > > > > >>>> partition placement, but not much.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Both seem to have their pros and cons. However,
> > answering
> > > > > Jack’s
> > > > > > > > > >> question,
> > > > > > > > > >>>> the explicit coordinator approach may indeed lead to a
> > > more
> > > > > > > flexible
> > > > > > > > > >> design.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> The purpose of this deviation in the discussion was to
> > > > > receive a
> > > > > > > > > >>>> preliminary community evaluation of the
> coordinator-less
> > > > > > approach
> > > > > > > > > >> without
> > > > > > > > > >>>> taking on the task of writing a separate KIP and
> fitting
> > > it in
> > > > > > the
> > > > > > > > > >> system
> > > > > > > > > >>>> of KIP-1150 and its children. We’re open to stopping
> it
> > > and
> > > > > > > getting
> > > > > > > > > >> back to
> > > > > > > > > >>>> working out the coordinator design if the community
> > > doesn’t
> > > > > > favor
> > > > > > > > the
> > > > > > > > > >>>> proposed approach.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best,
> > > > > > > > > >>>> Ivan and Diskless team
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > > > > >>>>> Hi Ivan,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> As Jun pointed out, the updated design seems to have
> > some
> > > > > > > > > shortcomings
> > > > > > > > > >>>>> although it simplifies the implementation.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler
> if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thank you,
> > > > > > > > > >>>>> Luke
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for the explanation.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> "we write the reference to the WAL file with the
> batch
> > > data"
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I understand the approach now, but I think it is a
> > hacky
> > > > > one.
> > > > > > > > There
> > > > > > > > > >> are
> > > > > > > > > >>>>>> multiple short comings with this design. First, it
> > > degrades
> > > > > > the
> > > > > > > > > >>>> durability.
> > > > > > > > > >>>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at
> > > > > > > > > >> a
> > > > > > > > > >>>>>> given point of time, one for the data and another
> for
> > > the
> > > > > > > > metadata.
> > > > > > > > > >>>>>> Flushing each file requires a separate IO. If the
> disk
> > > has
> > > > > 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> have 5K partitions in a broker, currently we can
> > afford
> > > to
> > > > > > flush
> > > > > > > > > each
> > > > > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> number of files per partition, we can only flush
> each
> > > > > > partition
> > > > > > > > > every
> > > > > > > > > >>>> 10
> > > > > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we
> > > ever
> > > > > need
> > > > > > > > this
> > > > > > > > > >>>>>> metadata somewhere else, say in the WAL file
> manager,
> > > the
> > > > > > > consumer
> > > > > > > > > >>>> needs to
> > > > > > > > > >>>>>> subscribe to every partition in the cluster, which
> is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>>> actual benefit of this approach is also
> questionable.
> > > On the
> > > > > > > > > surface,
> > > > > > > > > >>>> it
> > > > > > > > > >>>>>> might seem that we could reduce the number of lines
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>>> for this KIP. However, the changes are quite
> intrusive
> > > to
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition's code path and will probably make the
> code
> > > base
> > > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>>> maintain in the long run. I like the original
> approach
> > > based
> > > > > > on
> > > > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>> coordinator much better than this one. We could
> > probably
> > > > > > > refactor
> > > > > > > > > the
> > > > > > > > > >>>>>> producer state code so that it could be reused in
> the
> > > batch
> > > > > > > > > >>>> coordinator.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for addressing the concerns on the number of
> > > RPCs in
> > > > > > the
> > > > > > > > > >> produce
> > > > > > > > > >>>>>> path. I agree that with the metadata crafting
> > > mechanism, we
> > > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > > be
> > > > > > > > > >>>>>> collocated with the data on the same set of brokers,
> > > it's
> > > > > > weird
> > > > > > > > that
> > > > > > > > > >>>> they
> > > > > > > > > >>>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > > uses
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> metadata crafting mechanism, but the metadata is
> > stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition using its own assignment strategy. It will
> > be
> > > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> them collocated.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>>> step. My first concern is on the amount of metadata
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> kept.
> > > > > > > > > >>>>>> Without object compcation, the metadata generated in
> > the
> > > > > > produce
> > > > > > > > > path
> > > > > > > > > >>>> can
> > > > > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's
> > > say for
> > > > > > > every
> > > > > > > > > >>>> 250ms we
> > > > > > > > > >>>>>> produce 100 byte of metadata per partition. Let's
> say
> > > > > remoting
> > > > > > > > > tiering
> > > > > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K
> > > partitions,
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> > > metadata,
> > > > > > > quite
> > > > > > > > > >>>>>> signficant. A second concern is on performance.
> Every
> > > time
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>>>> rebuild the caching data, we need to read a bunch of
> > > small
> > > > > > > objects
> > > > > > > > > >>>> from S3,
> > > > > > > > > >>>>>> slowing down the building process. If a consumer
> > > happens to
> > > > > > need
> > > > > > > > > such
> > > > > > > > > >>>> data,
> > > > > > > > > >>>>>> it could slow down the application.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >>>> changes in
> > > > > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >>>> last
> > > > > > > > > >>>>>> for the long term.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Jun
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hi Jun,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed
> > to
> > > > > > clearly
> > > > > > > > > >>>> explain
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it
> essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful
> to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if
> we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > partitions.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> No. There will be only one physical partition per
> > > diskless
> > > > > > > > > >>>> partition. Let
> > > > > > > > > >>>>>>> me explain this with an example. Let’s say we have
> a
> > > > > diskless
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> > > leader. We
> > > > > > > > produce
> > > > > > > > > >>>> some
> > > > > > > > > >>>>>>> batches to this partition. The content of the
> segment
> > > file
> > > > > > will
> > > > > > > > be
> > > > > > > > > >>>>>>> something like this (for each batch):
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > > > > >>>>>>> Magic: like in classic
> > > > > > > > > >>>>>>> CRC: like in classic
> > > > > > > > > >>>>>>> Attributes: like in classic
> > > > > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > > > > >>>>>>> ProducerId: like in classic
> > > > > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > > > > >>>>>>> Records:
> > > > > > > > > >>>>>>>
> > path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > > > > byte
> > > > > > > > offset
> > > > > > > > > >>>>>>> 123456
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It looks very much like classic log entries. The
> only
> > > > > > > difference
> > > > > > > > is
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> instead of writing real Records, we write the
> > > reference to
> > > > > > the
> > > > > > > > WAL
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> with the batch data (I guess we need only the name
> > and
> > > the
> > > > > > byte
> > > > > > > > > >>>> offset,
> > > > > > > > > >>>>>>> because the byte length is the standard field
> above).
> > > > > > > Otherwise,
> > > > > > > > > >>>> it’s a
> > > > > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> So we have as many partitions for diskless as for
> > > classic.
> > > > > As
> > > > > > > of
> > > > > > > > > open
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Are the metadata and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated
> on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The source of truth for the data is still in WAL
> > files
> > > on
> > > > > > > object
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>> The source of truth for the metadata is in segment
> > > files on
> > > > > > the
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>> the replica set. Two new mechanisms are planned,
> both
> > > > > > > independent
> > > > > > > > > of
> > > > > > > > > >>>> this
> > > > > > > > > >>>>>>> new proposal, but I want to present them to give
> the
> > > idea
> > > > > > that
> > > > > > > > > only a
> > > > > > > > > >>>>>>> limited amount of data files will be operated
> > locally:
> > > > > > > > > >>>>>>> - We want to assemble batches into segment files
> and
> > > > > offload
> > > > > > > them
> > > > > > > > > to
> > > > > > > > > >>>>>>> tiered storage in order to prevent the unbounded
> > > growth of
> > > > > > > batch
> > > > > > > > > >>>>>> metadata.
> > > > > > > > > >>>>>>> For this, we need to open only  a few file
> > descriptors
> > > (for
> > > > > > the
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>> file itself + the necessary indexes) before the
> > > segment is
> > > > > > > fully
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > > > > >>>>>>> - We want to assemble local segment files for
> caching
> > > > > > purposes
> > > > > > > as
> > > > > > > > > >>>> well,
> > > > > > > > > >>>>>>> i.e. to speed up fetching. This will not
> materialize
> > > the
> > > > > full
> > > > > > > > > >>>> content of
> > > > > > > > > >>>>>>> the log, but only the hot set according to some
> > policy
> > > (or
> > > > > > > > > >>>> configurable
> > > > > > > > > >>>>>>> policies), i.e. the number of segments and file
> > > descriptors
> > > > > > > will
> > > > > > > > > >>>> also be
> > > > > > > > > >>>>>>> limited.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This is a valid concern that we considered, but
> this
> > > issue
> > > > > > can
> > > > > > > be
> > > > > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > > > > >>>>>>> The situation with a single broker is trivial: all
> > the
> > > > > commit
> > > > > > > > > >>>> requests go
> > > > > > > > > >>>>>>> from the broker to itself.
> > > > > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but
> > > located in
> > > > > > the
> > > > > > > > > single
> > > > > > > > > >>>>>> rack
> > > > > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for
> > > diskless
> > > > > > > > > >>>> partitions, but
> > > > > > > > > >>>>>>> we can tell producers (through metadata) to always
> > send
> > > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader
> > > replicas
> > > > > for
> > > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive
> diskless
> > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>> these partitions in various combinations, but only
> > for
> > > > > them.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>                   Broker 0
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>             |    t1-0         |
> > > > > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > > > > >>>>>>> partitions   | +-------------+ |    for these
> > > partitions
> > > > > > > > > >>>>>>>             |                 |
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > > > > Effectively,
> > > > > > > > > each
> > > > > > > > > >>>>>>> broker will commit only to itself, which
> effectively
> > > means
> > > > > 1
> > > > > > > > commit
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>> per WAL buffer (this may be 0 physical network
> calls,
> > > if we
> > > > > > > wish,
> > > > > > > > > >>>> just a
> > > > > > > > > >>>>>>> local function call).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs).
> > > Obviously, we
> > > > > > > > cannot
> > > > > > > > > >>>> always
> > > > > > > > > >>>>>>> send Produce requests to the designated leaders of
> > > diskless
> > > > > > > > > >>>> partitions:
> > > > > > > > > >>>>>>> this would mean inter-AZ network traffic, which we
> > > would
> > > > > like
> > > > > > > to
> > > > > > > > > >>>> avoid.
> > > > > > > > > >>>>>> To
> > > > > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> > > produce
> > > > > > > > > >>>>>> representative”
> > > > > > > > > >>>>>>> in every AZ. If we continue our example: when a
> > Produce
> > > > > > request
> > > > > > > > for
> > > > > > > > > >>>> t1-0,
> > > > > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it
> lands
> > > on
> > > > > > > broker 0
> > > > > > > > > >>>> (in the
> > > > > > > > > >>>>>>> broker’s AZ the representative is the broker
> itself).
> > > > > > However,
> > > > > > > if
> > > > > > > > > it
> > > > > > > > > >>>>>> comes
> > > > > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s
> broker
> > > 2.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> |produce requests         |produce requests
> > > |produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
> > |for
> > > > > t1-0,
> > > > > > > > t2-1,
> > > > > > > > > >>>>>> t3-0
> > > > > > > > > >>>>>>> |from AZ 0                |from AZ 1
> > >  |from
> > > > > AZ 2
> > > > > > > > > >>>>>>> v                         v
> v
> > > > > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)
> Broker
> > 2
> > > (AZ
> > > > > 2)
> > > > > > > > > >>>>>>> +---------------+      +---------------+
> > > > > > +---------------+
> > > > > > > > > >>>>>>> |     t1-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t2-1      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t3-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> +---------------+      +--------+------+
> > > > > > +--------+------+
> > > > > > > > > >>>>>>>    ^     ^                    |
> > |
> > > > > > > > > >>>>>>>    |     +--------------------+
> > |
> > > > > > > > > >>>>>>>    |     commit requests for these partitions
> > |
> > > > > > > > > >>>>>>>    |
> >  |
> > > > > > > > > >>>>>>>
> > +-------------------------------------------------+
> > > > > > > > > >>>>>>>          commit requests for these partitions
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> All the partitions that broker 0 is the leader of
> > will
> > > be
> > > > > > > > > >>>> “represented”
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Of course, this relationship goes both ways between
> > AZs
> > > > > (not
> > > > > > > > > >>>> necessarily
> > > > > > > > > >>>>>>> between the same brokers). It means that provided
> the
> > > > > cluster
> > > > > > > is
> > > > > > > > > >>>> balanced
> > > > > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> > > represent
> > > > > > > > > >>>>>> (number_of_azs
> > > > > > > > > >>>>>>> - 1) other brokers. This will result in the
> situation
> > > that
> > > > > > for
> > > > > > > > the
> > > > > > > > > >>>>>> majority
> > > > > > > > > >>>>>>> of commits, each broker will do up to
> (number_of_azs
> > -
> > > 1)
> > > > > > > network
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests (plus one local). Cloud regions tend to
> > have 3
> > > > > AZs,
> > > > > > > very
> > > > > > > > > >>>> rarely
> > > > > > > > > >>>>>>> more. That means, brokers will be doing up to 2
> > network
> > > > > > commit
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> per
> > > > > > > > > >>>>>>> WAL file.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> There are the following exceptions:
> > > > > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example,
> > > when we
> > > > > > > have
> > > > > > > > 2
> > > > > > > > > >>>> AZs
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> one has three brokers and another AZ has one. This
> > one
> > > > > broker
> > > > > > > > will
> > > > > > > > > do
> > > > > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This
> is
> > > not
> > > > > an
> > > > > > > > > extreme
> > > > > > > > > >>>>>>> amplification. Such an imbalance is not healthy in
> > most
> > > > > > > practical
> > > > > > > > > >>>> setups
> > > > > > > > > >>>>>>> and should be avoided anyway.
> > > > > > > > > >>>>>>> 2. Leadership changes and metadata propagation
> > period.
> > > When
> > > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3,
> the
> > > > > > producers
> > > > > > > > > will
> > > > > > > > > >>>> not
> > > > > > > > > >>>>>>> know this immediately (unless we want to be strict
> > and
> > > > > > respond
> > > > > > > > with
> > > > > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> > > will
> > > > > come
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>> in a
> > > > > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two
> > commit
> > > > > > > requests:
> > > > > > > > > to
> > > > > > > > > >>>>>> broker
> > > > > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to
> commit
> > > t3-0.
> > > > > > This
> > > > > > > > > >>>> situation
> > > > > > > > > >>>>>>> is not permanent and as producers update the
> cluster
> > > > > > metadata,
> > > > > > > it
> > > > > > > > > >>>> will be
> > > > > > > > > >>>>>>> resolved.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This all could be built with the metadata crafting
> > > > > mechanism
> > > > > > > only
> > > > > > > > > >>>> (which
> > > > > > > > > >>>>>>> is anyway needed for Diskless in one way or another
> > to
> > > > > direct
> > > > > > > > > >>>> producers
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic),
> > > just
> > > > > with
> > > > > > > the
> > > > > > > > > >>>> right
> > > > > > > > > >>>>>>> policy for it (for example, some deterministic
> > > hash-based
> > > > > > > > formula).
> > > > > > > > > >>>> I.e.
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>> explicit support for “produce representative” or
> > > anything
> > > > > > like
> > > > > > > > this
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the
> source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> We do only two operations on WAL files that span
> > > multiple
> > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions: committing and deleting. Commits can be
> > > done
> > > > > > > > > >>>> independently as
> > > > > > > > > >>>>>>> described above. But deletes are different, because
> > > when a
> > > > > > file
> > > > > > > > is
> > > > > > > > > >>>>>> deleted,
> > > > > > > > > >>>>>>> this affects all the partitions that still have
> alive
> > > > > batches
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> (if any).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The WAL file manager is a necessary point of
> > > coordination
> > > > > to
> > > > > > > > delete
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> files safely. We can say it is the source of truth
> > > about
> > > > > > files
> > > > > > > > > >>>>>> themselves,
> > > > > > > > > >>>>>>> while the partition leaders and their logs hold the
> > > truth
> > > > > > about
> > > > > > > > > >>>> whether a
> > > > > > > > > >>>>>>> particular file contains live batches of this
> > > particular
> > > > > > > > partition.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager will do this important task: be
> able
> > > to
> > > > > say
> > > > > > > for
> > > > > > > > > sure
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> a file does not contain any live batch of any
> > existing
> > > > > > > partition.
> > > > > > > > > For
> > > > > > > > > >>>>>> this,
> > > > > > > > > >>>>>>> it will have to periodically check against the
> > > partition
> > > > > > > leaders.
> > > > > > > > > >>>>>>> Considering that batch deletion is irreversible,
> when
> > > we
> > > > > > > declare
> > > > > > > > a
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager has to know about files being
> > > committed to
> > > > > > > start
> > > > > > > > > >>>> track
> > > > > > > > > >>>>>>> them and periodically check if they are empty. We
> can
> > > > > > consider
> > > > > > > > > >>>> various
> > > > > > > > > >>>>>> ways
> > > > > > > > > >>>>>>> to achieve this:
> > > > > > > > > >>>>>>> 1. As was proposed in my previous message: best
> > effort
> > > > > commit
> > > > > > > by
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> +
> > > > > > > > > >>>>>>> periodic prefix scans of object storage to detect
> > files
> > > > > that
> > > > > > > went
> > > > > > > > > >>>> below
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> radar due to network issue or the file manager
> > > temporary
> > > > > > > > > >>>> unavailability.
> > > > > > > > > >>>>>>> We’re speaking about listing the file names only
> and
> > > > > opening
> > > > > > > only
> > > > > > > > > >>>>>>> previously unknown files in order to find the
> > > partitions
> > > > > > > involved
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>> them.
> > > > > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill
> > the
> > > > > list
> > > > > > of
> > > > > > > > > files
> > > > > > > > > >>>>>>> fully asynchronously and in the background. This
> may
> > > be not
> > > > > > > ideal
> > > > > > > > > >>>> due to
> > > > > > > > > >>>>>>> costs and performance of scanning tons of files.
> > > However,
> > > > > the
> > > > > > > > > number
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>> live WAL files should be limited due to tiered
> > storage
> > > > > > > > offloading +
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> can
> > > > > > > > > >>>>>>> optimize this if we give files some global soft
> order
> > > in
> > > > > > their
> > > > > > > > > names.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > logic.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > > > > UnifiedLog
> > > > > > in
> > > > > > > > > >>>> order to
> > > > > > > > > >>>>>>> support these changes. However, it seems that
> > > idempotence,
> > > > > > > > > >>>> transactions,
> > > > > > > > > >>>>>>> queues, tiered storage will have to be changed less
> > > than
> > > > > with
> > > > > > > the
> > > > > > > > > >>>>>> original
> > > > > > > > > >>>>>>> design. This is because the partition leader state
> > > would
> > > > > > remain
> > > > > > > > in
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> same
> > > > > > > > > >>>>>>> place (on brokers) and existing workflows that
> > involve
> > > it
> > > > > > would
> > > > > > > > > have
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> be
> > > > > > > > > >>>>>>> changed less compared to the situation where we
> > > globalize
> > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader state in the batch coordinator. I admit this
> > is
> > > hard
> > > > > > to
> > > > > > > > make
> > > > > > > > > >>>>>>> convincing without both real implementations to
> hand
> > :)
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more
> details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Let’s consider the scenario of turning a classic
> > topic
> > > into
> > > > > > > > > >>>> diskless. The
> > > > > > > > > >>>>>>> user sets diskless.enabled=true, the leader
> receives
> > > this
> > > > > > > > metadata
> > > > > > > > > >>>> update
> > > > > > > > > >>>>>>> and does the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Start a new segment that will be written in the
> > > diskless
> > > > > > > > format
> > > > > > > > > >>>> (i.e.
> > > > > > > > > >>>>>>> without data).
> > > > > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Since it’s the same log, the followers will know
> > about
> > > that
> > > > > > > > switch
> > > > > > > > > >>>>>>> consistently. They will finish replicating the
> > classic
> > > > > > segments
> > > > > > > > and
> > > > > > > > > >>>> start
> > > > > > > > > >>>>>>> replicating the diskless ones. They will always
> know
> > > where
> > > > > > each
> > > > > > > > > >>>> batch is
> > > > > > > > > >>>>>>> located (either inside a classic segment or
> > referenced
> > > by a
> > > > > > > > > diskless
> > > > > > > > > >>>>>> one).
> > > > > > > > > >>>>>>> Switching back should be similar.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Doing this with the coordinator is possible, but
> has
> > > some
> > > > > > > > caveats.
> > > > > > > > > >>>> The
> > > > > > > > > >>>>>>> leader must do the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Write a special control segment to persist and
> > > replicate
> > > > > > the
> > > > > > > > > fact
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> from offset N the partition is now in the diskless
> > > mode.
> > > > > > > > > >>>>>>> 4. Inform the coordinator about the first offset N
> of
> > > the
> > > > > > > > “diskless
> > > > > > > > > >>>> era”.
> > > > > > > > > >>>>>>> 5. Inform the controller quorum that the transition
> > has
> > > > > > > finished
> > > > > > > > > and
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> brokers now can process diskless writes for this
> > > partition.
> > > > > > > > > >>>>>>> This could fail at some points, so this will
> probably
> > > > > require
> > > > > > > > some
> > > > > > > > > >>>>>>> explicit state machine with replication either in
> the
> > > > > > partition
> > > > > > > > log
> > > > > > > > > >>>> or in
> > > > > > > > > >>>>>>> KRaft.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It seems that the coordinator-less approach makes
> > this
> > > > > > simpler
> > > > > > > > > >>>> because
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> “coordinator” for the partition and the partition
> > > leader
> > > > > are
> > > > > > > the
> > > > > > > > > >>>> same and
> > > > > > > > > >>>>>>> they store the partition metadata in the same log,
> > too.
> > > > > While
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>>>>>> coordinator approach we have to perform some kind
> of
> > a
> > > > > > > > distributed
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> to handover metadata management from the classic
> > > partition
> > > > > > > leader
> > > > > > > > > to
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch coordinator.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> > > Please
> > > > > > let
> > > > > > > me
> > > > > > > > > >>>> know if
> > > > > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Best,
> > > > > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > > > > >>>>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Thanks for the update.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure that I fully understand the new
> > design,
> > > but
> > > > > it
> > > > > > > > seems
> > > > > > > > > >>>> less
> > > > > > > > > >>>>>>>> clean than before.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it
> essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful
> to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if
> we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > > partitions. Are
> > > > > > the
> > > > > > > > > >>>> metadata
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated
> on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the
> source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > > logic. I
> > > > > > > > > >>>> am
> > > > > > > > > >>>>>>> also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more
> details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Jun
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to
> > > have a
> > > > > > > > separate
> > > > > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role
> > > would
> > > > > be
> > > > > > to
> > > > > > > > > >>>>>> centrally
> > > > > > > > > >>>>>>>>> manage the batch and WAL file metadata for
> diskless
> > > > > topics.
> > > > > > > > This
> > > > > > > > > >>>>>>> component
> > > > > > > > > >>>>>>>>> drew many reasonable comments from the community
> > > about
> > > > > how
> > > > > > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> support
> > > > > > > > > >>>>>>>>> various Kafka features (transactions, queues) and
> > its
> > > > > > > > > >>>> scalability.
> > > > > > > > > >>>>>>> While we
> > > > > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > > > > concerns,
> > > > > > > we
> > > > > > > > > >>>> took a
> > > > > > > > > >>>>>>> step
> > > > > > > > > >>>>>>>>> back and looked at the problem from a different
> > > > > > perspective.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> We would like to propose an alternative Diskless
> > > design
> > > > > > > > *without
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>>>>> centralized coordinator*. We believe this
> approach
> > > has
> > > > > > > > potential
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>>>>> propose to discuss it as it may be more appealing
> > to
> > > the
> > > > > > > > > >>>> community.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let us explain the idea. Most of the
> complications
> > > with
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless approach come from one necessary
> > > architecture
> > > > > > > change:
> > > > > > > > > >>>>>>> globalizing
> > > > > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > > > > coordinator.
> > > > > > > > > >>>> This
> > > > > > > > > >>>>>>> causes
> > > > > > > > > >>>>>>>>> deviations to the established workflows in
> various
> > > > > features
> > > > > > > > like
> > > > > > > > > >>>>>>> produce
> > > > > > > > > >>>>>>>>> idempotence and transactions, queues, retention,
> > etc.
> > > > > These
> > > > > > > > > >>>>>> deviations
> > > > > > > > > >>>>>>> need
> > > > > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > > > > implemented
> > > > > > > and
> > > > > > > > > >>>>>>> tested. In
> > > > > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > > > > partition
> > > > > > > > > >>>> leaders
> > > > > > > > > >>>>>>> again
> > > > > > > > > >>>>>>>>> responsible for managing their partitions, even
> in
> > > > > diskless
> > > > > > > > > >>>> topics.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata
> > are
> > > > > > blended
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> the one partition log. The crux of the Diskless
> > idea
> > > is
> > > > > to
> > > > > > > > > >>>> decouple
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> and move data to the remote storage, while
> keeping
> > > > > metadata
> > > > > > > > > >>>> somewhere
> > > > > > > > > >>>>>>> else.
> > > > > > > > > >>>>>>>>> Using the central batch coordinator for managing
> > > batch
> > > > > > > metadata
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>> way,
> > > > > > > > > >>>>>>>>> but not the only.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s now think about managing metadata for each
> > user
> > > > > > > partition
> > > > > > > > > >>>>>>>>> independently. Generally partitions are
> independent
> > > and
> > > > > > don’t
> > > > > > > > > >>>> share
> > > > > > > > > >>>>>>>>> anything apart from that their data are mixed in
> > WAL
> > > > > files.
> > > > > > > If
> > > > > > > > we
> > > > > > > > > >>>>>>> figure
> > > > > > > > > >>>>>>>>> out how to commit and later delete WAL files
> > safely,
> > > we
> > > > > > will
> > > > > > > > > >>>> achieve
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of
> the
> > > > > central
> > > > > > > > batch
> > > > > > > > > >>>>>>>>> coordinator. Instead, *each diskless user
> partition
> > > will
> > > > > be
> > > > > > > > > >>>> managed
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> its
> > > > > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> > > classic
> > > > > > > > > >>>> topics, the
> > > > > > > > > >>>>>>>>> leader uses the partition log as the way to
> persist
> > > batch
> > > > > > > > > >>>> metadata,
> > > > > > > > > >>>>>>> i.e.
> > > > > > > > > >>>>>>>>> the regular batch header + the information about
> > how
> > > to
> > > > > > find
> > > > > > > > this
> > > > > > > > > >>>>>>> batch on
> > > > > > > > > >>>>>>>>> remote storage. In contrast to classic topics,
> > batch
> > > data
> > > > > > is
> > > > > > > in
> > > > > > > > > >>>>>> remote
> > > > > > > > > >>>>>>>>> storage.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > > > > >>>>>>>>> • Classic topics:
> > > > > > > > > >>>>>>>>>  • Data and metadata are co-located in the
> > partition
> > > log.
> > > > > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > > > > (metadata)|Batch
> > > > > > > > > >>>> data].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>> • Original Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is
> on
> > > > > remote
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>>>>  • The partition state is global in the batch
> > > > > coordinator.
> > > > > > > > > >>>>>>>>> • New Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on
> > > remote
> > > > > > > storage.
> > > > > > > > > >>>>>>>>>  • Partition log content: [Batch header
> > > (metadata)|Batch
> > > > > > > > > >>>>>> coordinates
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> remote storage].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the
> > reminder
> > > of
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless design:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The new approach could be depicted as the
> > following:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> As you can see, the main difference is that now
> > > instead
> > > > > of
> > > > > > a
> > > > > > > > > >>>> single
> > > > > > > > > >>>>>>> commit
> > > > > > > > > >>>>>>>>> request to the batch coordinator, we send
> multiple
> > > > > parallel
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to all the leaders of each partition involved in
> > the
> > > WAL
> > > > > > > file.
> > > > > > > > > >>>> Each
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> will commit its batches independently, without
> > > > > coordinating
> > > > > > > > with
> > > > > > > > > >>>>>> other
> > > > > > > > > >>>>>>>>> leaders and any other components. Batch data is
> > > addressed
> > > > > > by
> > > > > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> file
> > > > > > > > > >>>>>>>>> name, the byte offset and size, which allows
> > > partitions
> > > > > to
> > > > > > > know
> > > > > > > > > >>>>>> nothing
> > > > > > > > > >>>>>>>>> about other partitions to access their data in
> > > shared WAL
> > > > > > > > files.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The number of partitions involved in a single WAL
> > > file
> > > > > may
> > > > > > be
> > > > > > > > > >>>> quite
> > > > > > > > > >>>>>>> large,
> > > > > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to
> > commit
> > > one
> > > > > > WAL
> > > > > > > > > >>>> file is
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>> impractical. However, there are ways to reduce
> this
> > > > > number:
> > > > > > > > > >>>>>>>>> 1. Partition leaders are located on brokers.
> > > Requests to
> > > > > > > > > >>>> leaders on
> > > > > > > > > >>>>>>> one
> > > > > > > > > >>>>>>>>> broker could be grouped together into a single
> > > physical
> > > > > > > network
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>>>> (resembling the normal Produce request that may
> > carry
> > > > > > batches
> > > > > > > > for
> > > > > > > > > >>>>>> many
> > > > > > > > > >>>>>>>>> partitions inside). This will cap the number of
> > > network
> > > > > > > > requests
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make
> > producers
> > > > > send
> > > > > > > > their
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we
> may
> > > > > achieve
> > > > > > > the
> > > > > > > > > >>>> higher
> > > > > > > > > >>>>>>>>> concentration of logical commit requests in
> > physical
> > > > > > network
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>>>> reducing the number of the latter ones even
> > further,
> > > > > > ideally
> > > > > > > to
> > > > > > > > > >>>> one.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Obviously, out of multiple commit requests some
> may
> > > fail
> > > > > or
> > > > > > > > time
> > > > > > > > > >>>> out
> > > > > > > > > >>>>>>> for a
> > > > > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers
> > will
> > > > > > receive
> > > > > > > > > >>>> totally
> > > > > > > > > >>>>>>> or
> > > > > > > > > >>>>>>>>> partially failed responses to their Produce
> > requests,
> > > > > > similar
> > > > > > > > to
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> they
> > > > > > > > > >>>>>>>>> would have received when appending to a classic
> > topic
> > > > > fails
> > > > > > > or
> > > > > > > > > >>>> times
> > > > > > > > > >>>>>>> out.
> > > > > > > > > >>>>>>>>> If a partition experiences problems, other
> > partitions
> > > > > will
> > > > > > > not
> > > > > > > > be
> > > > > > > > > >>>>>>> affected
> > > > > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > > > > uncommitted
> > > > > > > > data
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> garbage in WAL files. But WAL files are
> short-lived
> > > > > > (batches
> > > > > > > > are
> > > > > > > > > >>>>>>> constantly
> > > > > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> > > storage),
> > > > > > so
> > > > > > > > this
> > > > > > > > > >>>>>>> garbage
> > > > > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For safely deleting WAL files we now need to
> > > centrally
> > > > > > manage
> > > > > > > > > >>>> them,
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>>>> this is the only state and logic that spans
> > multiple
> > > > > > > > partitions.
> > > > > > > > > >>>> On
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> diagram, you can see another commit request
> called
> > > > > “Commit
> > > > > > > file
> > > > > > > > > >>>> (best
> > > > > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This
> > manager
> > > will
> > > > > > be
> > > > > > > > > >>>>>>> responsible
> > > > > > > > > >>>>>>>>> for the following:
> > > > > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and
> > > persisting
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> committed WAL files.
> > > > > > > > > >>>>>>>>> 2. To handle potential failures in file
> information
> > > > > > delivery,
> > > > > > > > it
> > > > > > > > > >>>>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> doing prefix scan on the remote storage
> > periodically
> > > to
> > > > > > find
> > > > > > > > and
> > > > > > > > > >>>>>>> register
> > > > > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > > > > configurable
> > > > > > > and
> > > > > > > > > >>>>>> ideally
> > > > > > > > > >>>>>>>>> should be quite long.
> > > > > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders
> > > (after a
> > > > > > > grace
> > > > > > > > > >>>>>>> period) if
> > > > > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t
> > anymore
> > > > > > > referred
> > > > > > > > > >>>> to by
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> partition.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > > > > features
> > > > > > > such
> > > > > > > > > >>>> as
> > > > > > > > > >>>>>>>>> idempotence, transactions, queues, tiered
> storage,
> > > > > > retention.
> > > > > > > > > >>>> Now we
> > > > > > > > > >>>>>>> don’t
> > > > > > > > > >>>>>>>>> need to abstract away and reuse the code from
> > > partition
> > > > > > > leaders
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use
> > the
> > > > > same
> > > > > > > code
> > > > > > > > > >>>> paths
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from
> > > classic
> > > > > > > topics
> > > > > > > > > >>>> mostly
> > > > > > > > > >>>>>>>>> remain unchanged.
> > > > > > > > > >>>>>>>>> For example, it seems that
> > > > > > > > > >>>>>>>>>
> > > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > > > > and
> > > > > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > > > > transaction
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>> on
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> partition leader side could be used for diskless
> > > topics
> > > > > > with
> > > > > > > > > >>>> little
> > > > > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > > > > idempotent
> > > > > > > > > >>>> produce
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > > > > >>>>>>>>> Another example is share groups support, where
> the
> > > share
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader,
> > > > > > > > > >>>>>>>>> being co-located with the partition leader, would
> > > execute
> > > > > > the
> > > > > > > > > >>>> same
> > > > > > > > > >>>>>>> logic
> > > > > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > > > > >>>>>>>>> 2. It returns to the familiar partition-based
> > scaling
> > > > > > model,
> > > > > > > > > >>>> where
> > > > > > > > > >>>>>>>>> partitions are independent.
> > > > > > > > > >>>>>>>>> 3. It makes the operation and failure patterns
> > > closer to
> > > > > > the
> > > > > > > > > >>>>>> familiar
> > > > > > > > > >>>>>>>>> ones from classic topics.
> > > > > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> > > switching
> > > > > > the
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> modes
> > > > > > > > > >>>>>>>>> between diskless and classic.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The rest of the things remain unchanged compared
> to
> > > the
> > > > > > > > previous
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> design (after all previous discussions). Such
> > things
> > > as
> > > > > > local
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>>>> materialization by replicas, the consume path,
> > tiered
> > > > > > storage
> > > > > > > > > >>>>>>> integration,
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> If the community finds this design more suitable,
> > we
> > > will
> > > > > > > > update
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it.
> > > Please let
> > > > > > us
> > > > > > > > know
> > > > > > > > > >>>>>> what
> > > > > > > > > >>>>>>> you
> > > > > > > > > >>>>>>>>> think.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
> > wrote:
> > > > > > > > > >>>>>>>>>> Hi Justine,
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > > > > transactions
> > > > > > > > > >>>> for in
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> diskless coordinator for as long as the
> > corresponding
> > > > > > offsets
> > > > > > > > are
> > > > > > > > > >>>>>>> there.
> > > > > > > > > >>>>>>>>> With the tiered storage unification Greg
> mentioned
> > > > > earlier,
> > > > > > > > this
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks for the response. I think most of what
> you
> > > said
> > > > > > made
> > > > > > > > > >>>>>> sense,
> > > > > > > > > >>>>>>> but
> > > > > > > > > >>>>>>>>> I
> > > > > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it). The
> > > > > > > > > >>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > allowing
> > > > > > > > > >>>>>>> partition
> > > > > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if
> > > this is
> > > > > > > > > >>>> wrong. We
> > > > > > > > > >>>>>>> think
> > > > > > > > > >>>>>>>>>>> about relying on this with the batch
> coordinator
> > > and
> > > > > > delete
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s finished (as
> > there’s
> > > no
> > > > > > > > > >>>> replication
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> HWM
> > > > > > > > > >>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I didn't quite understand this. In classic
> > topics,
> > > we
> > > > > > have
> > > > > > > > > >>>> maps
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>> transactions which remove state when the
> > > transaction is
> > > > > > > > > >>>> completed
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> an
> > > > > > > > > >>>>>>>>>>> aborted transactions index which is retained
> for
> > > much
> > > > > > > longer.
> > > > > > > > > >>>>>> Once
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> > > longer
> > > > > > > > > >>>> involved
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>>>>> maintaining this partition side state, and it
> is
> > > > > subject
> > > > > > to
> > > > > > > > > >>>>>>> compaction
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>>> Looking back at the outline provided above, I
> > > didn't
> > > > > see
> > > > > > > much
> > > > > > > > > >>>>>>> about the
> > > > > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a
> bit
> > > > > > further.
> > > > > > > I
> > > > > > > > > >>>> saw
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> following in a response:
> > > > > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > > > > segment,
> > > > > > > > > >>>> all the
> > > > > > > > > >>>>>>>>> necessary
> > > > > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> > > including
> > > > > > the
> > > > > > > > > >>>>>>> transaction
> > > > > > > > > >>>>>>>>>>> index will be built to serve FetchRequests
> > exactly
> > > as
> > > > > > they
> > > > > > > > > >>>> are
> > > > > > > > > >>>>>>> today.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain
> > the
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>> Justine
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko
> <
> > > > > > > > > >>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> You’re right that we (probably
> unintentionally)
> > > > > focused
> > > > > > > > > >>>> only on
> > > > > > > > > >>>>>>>>> version 2.
> > > > > > > > > >>>>>>>>>>>> We can either limit the support to version 2
> or
> > > > > consider
> > > > > > > > > >>>> using
> > > > > > > > > >>>>>>> some
> > > > > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are?
> Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction
> is
> > > > > > ongoing,
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> course
> > > > > > > > > >>>>>>>>>>>> the normal idempotence checks. What the
> > partition
> > > > > leader
> > > > > > > > > >>>> in the
> > > > > > > > > >>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>> topics does before appending a batch to the
> > local
> > > log
> > > > > > > > > >>>> (e.g. in
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification
> and
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState).
> In
> > > > > > Diskless,
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>>>>> unfortunately
> > > > > > > > > >>>>>>>>>>>> cannot do these checks before appending the
> data
> > > to
> > > > > the
> > > > > > > WAL
> > > > > > > > > >>>>>>> segment
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these
> > > batches in
> > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not
> super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As
> > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader
> > > > > > > > > >>>>>>>>> tracks
> > > > > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the
> > batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> has
> > > > > > > > > >>>>>>>>> to as
> > > > > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends,
> the
> > > > > > > > > >>>> transaction
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> > > batches
> > > > > > and
> > > > > > > > > >>>> other
> > > > > > > > > >>>>>>>>>>>> information, but for the sake of performance
> > we’d
> > > > > better
> > > > > > > > > >>>> store
> > > > > > > > > >>>>>> it
> > > > > > > > > >>>>>>>>>>>> explicitly.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how
> it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it).
> > > > > > > > > >>>>>> The
> > > > > > > > > >>>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > > > >>>> allowing
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me
> if
> > > this
> > > > > is
> > > > > > > > > >>>> wrong.
> > > > > > > > > >>>>>> We
> > > > > > > > > >>>>>>>>> think
> > > > > > > > > >>>>>>>>>>>> about relying on this with the batch
> coordinator
> > > and
> > > > > > > > > >>>> delete the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as
> > > there’s no
> > > > > > > > > >>>>>> replication
> > > > > > > > > >>>>>>>>> and HWM
> > > > > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> > > transactions!
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are?
> Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is
> > > there
> > > > > > state
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not
> super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how
> it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> > > local
> > > > > > > > > >>>>>> segments,
> > > > > > > > > >>>>>>>>> making
> > > > > > > > > >>>>>>>>>>>> use of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> > > state for
> > > > > > > > > >>>> each
> > > > > > > > > >>>>>>>>> partition in
> > > > > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> > > fetches
> > > > > > > > > >>>> need
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> know
> > > > > > > > > >>>>>>>>>>>> aborted
> > > > > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > > > > >>>> transaction is
> > > > > > > > > >>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > > > > corresponding
> > > > > > > > > >>>> logs
> > > > > > > > > >>>>>>> of the
> > > > > > > > > >>>>>>>>>>>> involved
> > > > > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the
> > > metadata
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>>>> actual
> > > > > > > > > >>>>>>>>>>>> control
> > > > > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > > > > transactional
> > > > > > > > > >>>>>>> batches.
> > > > > > > > > >>>>>>>>> How
> > > > > > > > > >>>>>>>>>>>> does the
> > > > > > > > > >>>>>>>>>>>>>> broker obtain the information about the
> ending
> > > > > control
> > > > > > > > > >>>>>> batch
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response
> > > needs to
> > > > > > be
> > > > > > > > > >>>>>>> built by
> > > > > > > > > >>>>>>>>>>>>>> stitching record batches and generated
> control
> > > > > batches
> > > > > > > > > >>>>>>> together?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share
> > partition
> > > > > leader
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>>>>> consumers
> > > > > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to
> include
> > > this
> > > > > or
> > > > > > > > > >>>> it's
> > > > > > > > > >>>>>>> too
> > > > > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful
> to
> > > > > include
> > > > > > > > > >>>>>> enough
> > > > > > > > > >>>>>>>>> details
> > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>> understand correctness and performance
> impact.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from
> a
> > > given
> > > > > > > > > >>>>>> producer
> > > > > > > > > >>>>>>>>> contain a
> > > > > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a
> > producer
> > > > > sends
> > > > > > > > > >>>>>> every
> > > > > > > > > >>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those
> requests
> > > could
> > > > > > > > > >>>> reach
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> > > rejection
> > > > > > > > > >>>> of the
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> requests.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
> > Yurchenko <
> > > > > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details
> > > about
> > > > > > > > > >>>>>>> transactions
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> queues,
> > > > > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic
> > > topics* is
> > > > > > > > > >>>> based
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> precise
> > > > > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> > > (mostly
> > > > > > > > > >>>>>>> producers,
> > > > > > > > > >>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and
> other
> > > > > > > > > >>>> classes),
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition
> > > leaders
> > > > > with
> > > > > > > > > >>>>>> their
> > > > > > > > > >>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level (some details skipped)
> > workflow
> > > is
> > > > > the
> > > > > > > > > >>>>>>>>> following.
> > > > > > > > > >>>>>>>>>>>> When a
> > > > > > > > > >>>>>>>>>>>>>>> transactional Produce request is received
> by
> > > the
> > > > > > > > > >>>> broker:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> > > checks
> > > > > > > > > >>>> if a
> > > > > > > > > >>>>>>>>> non-empty
> > > > > > > > > >>>>>>>>>>>>>>> transaction is running for this partition.
> > > This is
> > > > > > > > > >>>> done
> > > > > > > > > >>>>>>> using
> > > > > > > > > >>>>>>>>> its
> > > > > > > > > >>>>>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > > > > >>>>>> (ProducerStateManager,
> > > > > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> > > about
> > > > > all
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> partitions
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include
> > them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > > > > transactional
> > > > > > > > > >>>>>>> checks.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the
> > > transactional
> > > > > > > > > >>>> data to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> update some of their state (for example,
> log
> > > the
> > > > > fact
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> > > offset).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or
> aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the
> > transaction
> > > > > > > > > >>>> markers to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders
> > and
> > > lots
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>>>> important
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is
> > > local.
> > > > > The
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>>>>> challenge
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the
> fact
> > > there
> > > > > > > > > >>>> are no
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of
> state
> > > need
> > > > > > > > > >>>> to be
> > > > > > > > > >>>>>>>>> globalized
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing
> this
> > to
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>>>>> idempotent
> > > > > > > > > >>>>>>>>>>>>>> produce.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless
> topics*
> > > would
> > > > > > > > > >>>> look
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>>>>> similar:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks
> if a
> > > > > > > > > >>>> non-empty
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> > > classic
> > > > > > > > > >>>>>> topics,
> > > > > > > > > >>>>>>>>> this is
> > > > > > > > > >>>>>>>>>>>>>> checked
> > > > > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single
> > > RPC.
> > > > > > > > > >>>> Since a
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>> could
> > > > > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> > > epoch,
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> positive
> > > > > > > > > >>>>>>>>>>>> result of
> > > > > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> > > double
> > > > > > > > > >>>>>>> configured
> > > > > > > > > >>>>>>>>>>>> duration
> > > > > > > > > >>>>>>>>>>>>>> of a
> > > > > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > > > > informed
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the
> > transaction
> > > to
> > > > > > > > > >>>> include
> > > > > > > > > >>>>>>> them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the
> > > broker
> > > > > > > > > >>>> side.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional
> data
> > > to the
> > > > > > > > > >>>>>> current
> > > > > > > > > >>>>>>>>> shared
> > > > > > > > > >>>>>>>>>>>> WAL
> > > > > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> > > transaction-related
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> Diskless
> > > > > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the
> batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> like
> > > > > > > > > >>>>>>>>> in the
> > > > > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the
> same
> > > > > errors
> > > > > > > > > >>>> like
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e.
> some
> > > > > batches
> > > > > > > > > >>>>>> could
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> rejected.
> > > > > > > > > >>>>>>>>>>>>>>> This means, there will potentially be
> garbage
> > > in
> > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>>>> segment
> > > > > > > > > >>>>>>>>>>>> file in
> > > > > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is
> > > preferable to
> > > > > > > > > >>>> doing
> > > > > > > > > >>>>>>> more
> > > > > > > > > >>>>>>>>>>>> network
> > > > > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > > > > segments
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> relatively
> > > > > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or
> aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
> > informs
> > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
> > the
> > > > > > > > > >>>>>>> transaction is
> > > > > > > > > >>>>>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > > > > >>>> corresponding
> > > > > > > > > >>>>>>> logs
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only
> > on
> > > the
> > > > > > > > > >>>>>> metadata
> > > > > > > > > >>>>>>>>> level, no
> > > > > > > > > >>>>>>>>>>>>>>> actual control batches are written to any
> > file.
> > > > > They
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> dynamically
> > > > > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations.
> > We
> > > > > could
> > > > > > > > > >>>>>>>>> technically
> > > > > > > > > >>>>>>>>>>>> write
> > > > > > > > > >>>>>>>>>>>>>>> these control batches for real, but this
> > would
> > > mean
> > > > > > > > > >>>> extra
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> latency,
> > > > > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the
> batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> save
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> now.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> > > write
> > > > > the
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> markers
> > > > > > > > > >>>>>>>>>>>>>>> to their logs and respond to the
> coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce
> > > flow, we
> > > > > > > > > >>>> get:
> > > > > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between
> > brokers
> > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch
> > > > > > > > > >>>>>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>>>>> when a new partition appear in the
> > > transaction. To
> > > > > > > > > >>>>>>> mitigate the
> > > > > > > > > >>>>>>>>>>>> impact of
> > > > > > > > > >>>>>>>>>>>>>>> them:
> > > > > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> > > Produce
> > > > > > > > > >>>>>>> request
> > > > > > > > > >>>>>>>>> will be
> > > > > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized
> > for
> > > > > > > > > >>>> fast
> > > > > > > > > >>>>>>>>> response to
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally
> > will
> > > > > > > > > >>>>>>> communicate
> > > > > > > > > >>>>>>>>> with a
> > > > > > > > > >>>>>>>>>>>>>>> single broker for the duration of the
> > > transaction
> > > > > > > > > >>>> further
> > > > > > > > > >>>>>>>>> reduces the
> > > > > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the
> > transaction
> > > > > > > > > >>>>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also
> be
> > > able
> > > > > to
> > > > > > > > > >>>>>> span
> > > > > > > > > >>>>>>> both
> > > > > > > > > >>>>>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The share group coordination and management
> > is
> > > a
> > > > > > > > > >>>> side job
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>> interfere with the topic itself
> (leadership,
> > > > > > > > > >>>> replicas,
> > > > > > > > > >>>>>>> physical
> > > > > > > > > >>>>>>>>>>>> storage
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > > > > consumers
> > > > > > > > > >>>>>>> (Fetch and
> > > > > > > > > >>>>>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > > > > affected.)
> > > > > > > > > >>>> We
> > > > > > > > > >>>>>>> don't
> > > > > > > > > >>>>>>>>> see any
> > > > > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> > > compatible
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>> share
> > > > > > > > > >>>>>>>>>>>> groups the
> > > > > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the
> > > code
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> we
> > > > > > > > > >>>>>>>>> don't
> > > > > > > > > >>>>>>>>>>>> expect
> > > > > > > > > >>>>>>>>>>>>>> any
> > > > > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading
> > routines
> > > are
> > > > > > > > > >>>> used
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>> used for
> > > > > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this
> > or
> > > it's
> > > > > > > > > >>>> too
> > > > > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and
> design
> > > input
> > > > > > > > > >>>> on
> > > > > > > > > >>>>>>>>> KIP-1150.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> > > with a
> > > > > > > > > >>>> new
> > > > > > > > > >>>>>>>>> design. To
> > > > > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> KIP-405
> > > > > > > > > >>>>>>>>>>>> Tiered
> > > > > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data
> produced
> > > to a
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> topic
> > > > > > > > > >>>>>>>>>>>> to be
> > > > > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements
> for
> > > the
> > > > > > > > > >>>> Batch
> > > > > > > > > >>>>>>>>> Coordinator
> > > > > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose
> > with
> > > > > > > > > >>>> Tiered
> > > > > > > > > >>>>>>> Storage
> > > > > > > > > >>>>>>>>>>>> plugin
> > > > > > > > > >>>>>>>>>>>>>>>> features such as encryption and
> alternative
> > > data
> > > > > > > > > >>>>>> formats.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
> > local
> > > > > > > > > >>>> segments,
> > > > > > > > > >>>>>>>>> making use
> > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory,
> and
> > > > > > > > > >>>> zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered
> > > cache
> > > > > > > > > >>>>>>> elements,
> > > > > > > > > >>>>>>>>> do not
> > > > > > > > > >>>>>>>>>>>> need
> > > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built
> without
> > > > > > > > > >>>> contacting
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified
> > > substantially,
> > > > > by
> > > > > > > > > >>>>>>> removing
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> previous
> > > > > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > > > > >>>> component, and
> > > > > > > > > >>>>>>>>> "object
> > > > > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces
> as
> > > > > > > > > >>>> enabled by
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> Batch
> > > > > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
> > as
> > > the
> > > > > > > > > >>>>>> earlier
> > > > > > > > > >>>>>>>>> design,
> > > > > > > > > >>>>>>>>>>>> while
> > > > > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> ecosystem.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> > > feedback on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> new
> > > > > > > > > >>>>>>>>> design.
> > > > > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> > > support
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>> all
> > > > > > > > > >>>>>>>>> existing
> > > > > > > > > >>>>>>>>>>>>>> client
> > > > > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > > > > >>>> supporting
> > > > > > > > > >>>>>> APIs
> > > > > > > > > >>>>>>>>> like
> > > > > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> > > Siekierski
> > > > > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid>
> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> > > current
> > > > > > > > > >>>>>>> status of
> > > > > > > > > >>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>> KIP?
> > > > > > > > > >>>>>>>>>>>>>> I
> > > > > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> > > past 2
> > > > > > > > > >>>>>>> months,
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>> vote got
> > > > > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have
> > been
> > > > > > > > > >>>>>> answered
> > > > > > > > > >>>>>>>>> since
> > > > > > > > > >>>>>>>>>>>> then.
> > > > > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > > > > >>>> activity
> > > > > > > > > >>>>>>> since
> > > > > > > > > >>>>>>>>> May.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> > > real
> > > > > > > > > >>>>>> choices
> > > > > > > > > >>>>>>>>> that can
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> > > being
> > > > > > > > > >>>> by
> > > > > > > > > >>>>>> far
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> dominant
> > > > > > > > > >>>>>>>>>>>>>> one
> > > > > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all
> > these
> > > are
> > > > > > > > > >>>>>>>>> standalone
> > > > > > > > > >>>>>>>>>>>>>> products.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
> > would
> > > > > > > > > >>>> hate to
> > > > > > > > > >>>>>>> see a
> > > > > > > > > >>>>>>>>>>>> stall. So
> > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
> > Colin
> > > > > > > > > >>>>>> McCabe -
> > > > > > > > > >>>>>>>>> have your
> > > > > > > > > >>>>>>>>>>>>>>> questions
> > > > > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there
> anything I
> > > can
> > > > > > > > > >>>> do
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>>>> help? I'm
> > > > > > > > > >>>>>>>>>>>>>> deeply
> > > > > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and
> have
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> both?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM
> Stanislav
> > > > > > > > > >>>>>> Kozlovski
> > > > > > > > > >>>>>>> <
> > > > > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in
> the
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>>>> (just a
> > > > > > > > > >>>>>>>>>>>> bunch
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > > > > >>>> example, I
> > > > > > > > > >>>>>>> was
> > > > > > > > > >>>>>>>>>>>> searching
> > > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a
> while;
> > As
> > > > > > > > > >>>> far
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>> I
> > > > > > > > > >>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>> tell, it
> > > > > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
> > of
> > > > > > > > > >>>> having
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> root
> > > > > > > > > >>>>>>>>>>>> KIP be
> > > > > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> > > links
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>> sub-KIPs
> > > > > > > > > >>>>>>>>>>>> where
> > > > > > > > > >>>>>>>>>>>>>> were
> > > > > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I
> think
> > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> good to
> > > > > > > > > >>>>>>>>>>>>>> create
> > > > > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat
> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> > > about
> > > > > > > > > >>>>>>>>> introducing a
> > > > > > > > > >>>>>>>>>>>> new
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > > > > >>>> Storage as
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> primary
> > > > > > > > > >>>>>>>>>>>>>>> source of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big
> we
> > > > > > > > > >>>> decided
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> split it
> > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > > > > >>>> should aim
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go
> onto
> > > > > > > > > >>>>>> details
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> how to
> > > > > > > > > >>>>>>>>>>>>>>> implement
> > > > > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used
> when
> > > we
> > > > > > > > > >>>>>>> discussed
> > > > > > > > > >>>>>>>>>>>> KRaft.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
> > really
> > > > > > > > > >>>> hard
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > >>>>>>>>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several
> sub-kips
> > > > > > > > > >>>> (linked
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > > > > >>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> > > DISCUSS
> > > > > > > > > >>>>>>> threads
> > > > > > > > > >>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > > > > >>>> whether
> > > > > > > > > >>>>>>> Apache
> > > > > > > > > >>>>>>>>> Kafka
> > > > > > > > > >>>>>>>>>>>> wants
> > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > > > > >>>> authors
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> this KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director,
> Aiven
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
> >  +491715557497 |
> > > > > > > > > >>>>>>> aiven.io
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa,
> Hannu
> > > > > > > > > >>>>>>> Valtonen,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB
> 209739 B
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Anatolii Popov
> > > > > Senior Software Developer, *Aiven OY*
> > > > > m: +358505126242
> > > > > w: aiven.io  e: anatolii.popov@aiven.io
> > > > > <https://www.facebook.com/aivencloud>
> > > > > <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Robin,

Here is my view on the "simpler" approach. First, because the focus is only
on saving cross-AZ network costs, it doesn't achieve other potential
benefits like better scalability and durability. Secondly, the "Simpler"
approach may not be that simple once we work out the details. For example,
during the discussion in kip-1176, we realized that to really save cost, we
need to rethink how replicas are assigned to brokers to avoid different
followers reading the same object hundreds of times. Thirdly, most variants
of the "simpler" approach are unnatural. They all try to piggyback the
existing fetch request protocol desgined for data to propagate just the
metadata. This could make evolving the protocol harder in the future. With
KIP-1150, I hope that we can gain more benefits from a cleaner design
that's more maintainable for the long term.

Hi, Ivan,

Thanks for the updated KIP. LGTM now.

Jun



On Thu, Jan 22, 2026 at 1:02 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi Jun,
>
> JR10. "Cluster scalability increases when less data is required to be
> stored on disks and rebalanced between brokers."
> This seems to contradict a bit with what KIP-1163 says: "Start building it
> from some configurable amount, e.g. 100 MiB prior to the current log end or
> earliest available offset, whichever is later.".  I just want to confirm
> that we plan to make data rebalancing much faster in this KIP.
>
>
> This 100 MiB is at the moment not a very well researched number. Probably
> is should be configurable per topic and the default should be smaller.
> Generally indeed we want to make moving Diskless partitions around, adding
> replicas, etc. much faster and less resource-consuming than classic topics.
>
> JR13. Since people keep asking about the "simpler" approach, it would be
> useful to include in the rejected section why we chose not to pursue the
> "simpler" approach like the one in KIP-1176. To me, there are two main
> reasons: (1) this KIP provides more benefits (saving all cross AZ network
> costs instead of just the replication part, better scalability and
> durability); (2) it provides an opportunity for a cleaner long-term design
> instead of retrofitting to the existing framework unnaturally.
>
>
> Thanks, we added [1] this to the rejected alternatives in KIP-1163.
>
> Best,
> Ivan
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=350783976#KIP1163:DisklessCore-Coordinator-lessapproaches
>
> On Wed, Jan 21, 2026, at 00:30, Jun Rao via dev wrote:
>
> Hi, Ivan,
>
> Thanks for the reply.
>
> JR10. "Cluster scalability increases when less data is required to be
> stored on disks and rebalanced between brokers."
> This seems to contradict a bit with what KIP-1163 says: "Start building it
> from some configurable amount, e.g. 100 MiB prior to the current log end or
> earliest available offset, whichever is later.".  I just want to confirm
> that we plan to make data rebalancing much faster in this KIP.
>
> JR11. Sounds good.
>
> JR13. Since people keep asking about the "simpler" approach, it would be
> useful to include in the rejected section why we chose not to pursue the
> "simpler" approach like the one in KIP-1176. To me, there are two main
> reasons: (1) this KIP provides more benefits (saving all cross AZ network
> costs instead of just the replication part, better scalability and
> durability); (2) it provides an opportunity for a cleaner long-term design
> instead of retrofitting to the existing framework unnaturally.
>
> Jun
>
>
>
> On Tue, Jan 20, 2026 at 11:22 AM Greg Harris via dev <dev@kafka.apache.org
> >
> wrote:
>
> > Hi Robin,
> >
> > Thanks for your design sketch.
> >
> > Based on your description, the GET operations on the leader and followers
> > must complete before offsets are assigned, adding Produce request latency
> > that is not present in the KIP-1163 design.
> >
> > KIP-1163 also assumes that object storage is more reliable than disk
> > storage, which is the case in the hyperscaler clouds. The parts of your
> > design that make it compatible with low-reliability object storage become
> > counterproductive when Diskless is used with a high-reliability object
> > storage, such as re-introducing direct replication and cross-az costs.
> >
> > We’ve already discussed and rejected KIP-1176 (another “simpler”
> proposal)
> > in favor of a complete solution. The folks who championed that proposal
> are
> > now working with us on KIP-1163, and are currently designing Compaction
> in
> > Tiered Storage to make Diskless a complete solution for compacted topics
> as
> > well. Similarly, we would appreciate your contributions to the design of
> > KIP-1163, and your vote on the motivational KIP-1150.
> >
> > Thanks!
> >
> > On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi Jun,
> > >
> > > Here are our answers to your questions and comments:
> > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing
> to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > >
> > > Thank you, good point, added this.
> > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > >
> > > Yes, there is a per-request cost for object storage which is not
> directly
> > > comparable to inter-zone transfer costs. We have found that the design
> in
> > > KIP-1163 amortizes those request costs such that they are insignificant
> > > compared to the $0.02/gb cost which is called out in KIP-1150.
> > > We currently have very precise cost models that were put together for
> an
> > > earlier iteration of the Diskless topics. These are somewhat stale and
> > > inaccurate for the latest design, so we will work in the background to
> > post
> > > concrete numbers in KIP-1163 before voting on that proposal. Would this
> > be
> > > OK?
> > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > >
> > > Thanks for calling out. We have updated KIP-1164 with the latest
> design.
> > > It's missing public interfaces, but we think it may be a bit too early
> to
> > > define them as some rounds of discussion is ahead, with potential to
> > change
> > > the design. But otherwise the KIP is good for the first review.
> > > We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
> > >
> > > Best,
> > > Ivan
> > >
> > > On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> > > > Hi, Anatolii,
> > > >
> > > > Thanks for the updated KIP. Looks good to me overall. A few more
> > > comments.
> > > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing
> to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > > >
> > > > Jun
> > > >
> > > > On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> > > dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thank you for the extensive feedback.
> > > > >
> > > > > We have substantially updated the KIP to address the points raised.
> > > Given
> > > > > the scope of these changes, we have started a new VOTE thread to
> > > restart
> > > > > the voting process cleanly.
> > > > >
> > > > > You can find the new thread here:
> > > > > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> > > > >
> > > > > Best,
> > > > > Anatolii
> > > > >
> > > > > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > > > > dev@kafka.apache.org> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We (the team at Slack) have been following the recent discussion
> > > > > regarding
> > > > > > the scope and timeline of KIP-1150. We agree with the community
> > > sentiment
> > > > > > that Diskless Topics represents the right long-term architecture
> > for
> > > > > Kafka
> > > > > > in the cloud, but we also recognize the valid concerns raised
> > > regarding
> > > > > the
> > > > > > engineering resources required to deliver such an ambitious
> change.
> > > > > >
> > > > > > To help address these concerns and accelerate the timeline, we
> are
> > > happy
> > > > > to
> > > > > > announce that we are partnering with the KIP-1150 authors to
> > > co-develop
> > > > > > this feature.
> > > > > >
> > > > > > We previously proposed KIP-1176: Tiered Storage for Active Log
> > > Segments
> > > > > to
> > > > > > solve similar problems. However, rather than fragmenting the
> > > community's
> > > > > > efforts across competing designs, we have decided to withdraw
> > > KIP-1176
> > > > > and
> > > > > > consolidate our engineering resources behind KIP-1150.
> > > > > >
> > > > > > To start, we plan to take ownership of Compaction for Tiered
> > Storage.
> > > > > This
> > > > > > has long been a missing feature in KIP-405, and it becomes
> critical
> > > in a
> > > > > > Diskless architecture where long-lived data is obligated to be
> > > tiered. By
> > > > > > driving this related prerequisite feature, we hope to allow for
> > > faster
> > > > > > delivery of KIP-1150.
> > > > > >
> > > > > > We are excited to collaborate on this to ensure a robust and
> timely
> > > > > > delivery for the community.
> > > > > >
> > > > > > Best,
> > > > > > Tom & Henry
> > > > > >
> > > > > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Thanks for sharing the meeting notes.
> > > > > > > I agree we should keep polishing the contents of 1150 & high
> > level
> > > > > design
> > > > > > > in 1163 to prepare for a vote.
> > > > > > >
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > There was a video call between myself, Ivan Yurchenko, Jun
> Rao,
> > > and
> > > > > > > Andrew
> > > > > > > > Schofield pertaining to KIP-1150. Here are the notes from
> that
> > > > > meeting:
> > > > > > > >
> > > > > > > > Ivan: What is the future state of Kafka in this area, in 5
> > years?
> > > > > > > > Jun: Do we want something more cloud native? Yes, started
> with
> > > Tiered
> > > > > > > > Storage. If there’s a better way, we should explore it. In
> the
> > > long
> > > > > > term
> > > > > > > > this will be useful
> > > > > > > > Because Kafka is used so widely, we need to make sure
> > everything
> > > we
> > > > > add
> > > > > > > is
> > > > > > > > for the long term and for everyone, not just for a single
> > > company.
> > > > > > > > When we add TS, it doesn’t just solve Uber’s use-case. We
> want
> > > > > > something
> > > > > > > > that’s high quality/lasts/maintainable, and can work with all
> > > > > existing
> > > > > > > > capabilities.
> > > > > > > > If both 1150 and 1176 proceed at the same time, it’s
> confusing.
> > > They
> > > > > > > > overlap, but Diskless is more ambitious.
> > > > > > > > If both KIPs are being seriously worked on, then we don’t
> > really
> > > need
> > > > > > > both,
> > > > > > > > because Diskless clearly is better. Having multiple will
> > confuse
> > > > > > people.
> > > > > > > It
> > > > > > > > will duplicate some of the effort.
> > > > > > > > If we want diskless ultimately, what is the short term
> > strategy,
> > > to
> > > > > get
> > > > > > > > some early wins first?
> > > > > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > > > > Andrew: Eventually the architecture will change
> substantially,
> > > it may
> > > > > > not
> > > > > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > > > > Greg: We all agree on having a high quality feature merged
> > > upstream,
> > > > > > and
> > > > > > > > supporting all APIs
> > > > > > > > Jun: We should try and keep things simple, but there is some
> > > minimum
> > > > > > > > complexity needed.
> > > > > > > > When doing the short term changes (1176), it doesn’t really
> > > progress
> > > > > in
> > > > > > > > changing to a more modern architecture.
> > > > > > > > Greg: Was TS+Compaction the only feature miss we’ve had so
> far?
> > > > > > > > Jun: The danger of only applying changes to some part of the
> > > API, you
> > > > > > set
> > > > > > > > the precedence that you only have to implement part of the
> API.
> > > > > > > Supporting
> > > > > > > > the full API set should be a minimum requirement.
> > > > > > > > Andrew: When we started Kraft, how much did we know the
> design?
> > > > > > > > Jun: For Kraft we didn’t really know much about the
> migration,
> > > but
> > > > > the
> > > > > > > > high-level was clear.
> > > > > > > > Greg: Is 1150 votable in its current state?
> > > > > > > > Jun: 1150 should promise to support all APIs. It doesn’t have
> > to
> > > have
> > > > > > all
> > > > > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > > > > We do need some high-level design enough to give confidence
> > that
> > > the
> > > > > > > > promise is able to be fulfilled.
> > > > > > > > Greg: Is the draft version in 1163 enough detail or is more
> > > needed?
> > > > > > > > Jun: We need to agree on the core design, such as leaderless
> > > etc. And
> > > > > > how
> > > > > > > > the APIs will be supported.
> > > > > > > > Greg: Okay we can include these things, and provide a sketch
> of
> > > how
> > > > > the
> > > > > > > > other leader-based features operate.
> > > > > > > > Jun: Yeah if at a high level the sketch appears to work, we
> can
> > > > > approve
> > > > > > > > that functionality.
> > > > > > > > Are you committed to doing the more involved and big project?
> > > > > > > > Greg: Yes, we’re committed to the 1163 design and can’t
> really
> > > accept
> > > > > > > 1176.
> > > > > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > > > > Greg: We’ll push internally for resources, and use the
> > community
> > > > > > > sentiment
> > > > > > > > to motivate Aiven.
> > > > > > > > How far into the future should we look? What sort of scale?
> > > > > > > > Jun: As long as there’s a path forward, and we’re not closing
> > off
> > > > > > future
> > > > > > > > improvements, we can figure out how to handle a larger scale
> > > when it
> > > > > > > > arises.
> > > > > > > > Greg: Random replica placement is very harmful, can we
> > recommend
> > > > > users
> > > > > > to
> > > > > > > > use an external tool like CruiseControl?
> > > > > > > > Jun: Not everyone uses CruiseControl, we would probably need
> > some
> > > > > > > solution
> > > > > > > > for this out of the box
> > > > > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > > > > Jun: Out-of-box experience should be good, good to allow
> other
> > > > > > > > implementations
> > > > > > > > Greg: But it could hurt Kafka feature/upgrade velocity when
> we
> > > wait
> > > > > for
> > > > > > > > plugin providers to implement it
> > > > > > > > Ivan: We imagined that maybe cloud hyperscalers could
> implement
> > > it
> > > > > with
> > > > > > > > e.g. dynamodb
> > > > > > > > Greg: Could we bake more details of the different providers
> > into
> > > > > Kafka,
> > > > > > > or
> > > > > > > > does it still make sense for it to be pluggable?
> > > > > > > > Jun: Make it whatever is easiest to roll out and add new
> > clients
> > > > > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > > > > Ivan: The vote is already open, we’re not too pressed for
> time.
> > > We’ll
> > > > > > go
> > > > > > > > improve the 1163 design and communication.
> > > > > > > > Is 1176 a competing design? Someone will ask.
> > > > > > > > Jun: If we are seriously working on something more ambitious,
> > > yeah we
> > > > > > > > shouldn’t do the stop-gap solution.
> > > > > > > > It’s diverting review resources. If we can get the short term
> > > thing
> > > > > in
> > > > > > > 1yr
> > > > > > > > but Diskless solution is 2y it makes sense to go for
> Diskless.
> > If
> > > > > it’s
> > > > > > > 5yr,
> > > > > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case.
> > Should
> > > we
> > > > > > > > explicitly exclude 1176?
> > > > > > > > Andrew: Put your arms around the feature set you actually
> want,
> > > and
> > > > > use
> > > > > > > > that to rule out 1176.
> > > > > > > > Probably don’t need -1 votes, most likely KIPs just don’t
> > receive
> > > > > > votes.
> > > > > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > > > > Jun: Satish posted meeting notes regularly, we should do the
> > > same.
> > > > > > > >
> > > > > > > > To summarize, we will be polishing the contents of 1150 &
> high
> > > level
> > > > > > > design
> > > > > > > > in 1163 to prepare for a vote.
> > > > > > > > We believe that the community should select the feature set
> of
> > > 1150
> > > > > to
> > > > > > > > fully eliminate producer cross-zone costs, and make the
> > > investment
> > > > > in a
> > > > > > > > high quality Diskless Topics implementation rather than in
> > > stop-gap
> > > > > > > > solutions.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
> > wrote:
> > > > > > > >
> > > > > > > > > This may be a tangent, but we needed to offload storage off
> > of
> > > > > Kafka
> > > > > > > into
> > > > > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> > > mostly
> > > > > > > > > ephemeral broker that can come and go as it pleases. Be
> that
> > > > > scaling
> > > > > > or
> > > > > > > > > outage. Disks can be destroyed and recreated at will, we
> > still
> > > > > retain
> > > > > > > > data
> > > > > > > > > and use broker for just that, brokering messages. Not only
> > > that, we
> > > > > > > > reduced
> > > > > > > > > the requirement on the actual Kafka resources by reducing
> the
> > > size
> > > > > > of a
> > > > > > > > > payload via a claim check pattern. Maybe this is an
> > > anti–pattern,
> > > > > but
> > > > > > > it
> > > > > > > > is
> > > > > > > > > super fast and highly cost efficient. We reworked
> > > ProducerRequest
> > > > > to
> > > > > > > > allow
> > > > > > > > > plugins. We added a custom http plugin that submits every
> > > request
> > > > > > via a
> > > > > > > > > persisted connection to a microservice. Microservice stores
> > the
> > > > > > payload
> > > > > > > > and
> > > > > > > > > returns a tiny json metadata object,a claim check, that can
> > be
> > > used
> > > > > > to
> > > > > > > > find
> > > > > > > > > the actual data. Think of it as zipping the payload. This
> > claim
> > > > > check
> > > > > > > > > metadata traverses the pipelines with consumers using the
> > urls
> > > in
> > > > > > > > metadata
> > > > > > > > > to pull what they need. Think unzipping. This allowed us to
> > > also
> > > > > pull
> > > > > > > > ONLY
> > > > > > > > > the data that we need in graphql like manner. So if you
> have
> > a
> > > 100K
> > > > > > > json
> > > > > > > > > payload and you need only a subsection, you can pull that
> by
> > > > > > jmespath.
> > > > > > > > When
> > > > > > > > > you have multiple consumer groups yanking down huge
> payloads
> > > it is
> > > > > > > > > cumbersome on the broker. When you have the same consumer
> > > groups
> > > > > > > yanking
> > > > > > > > > down a claim check, and then going out of band directly to
> > the
> > > > > source
> > > > > > > of
> > > > > > > > > truth, the broker has some breathing room. Obviously our
> > > > > microservice
> > > > > > > > does
> > > > > > > > > not go directly to the cloud storage, as that would be too
> > > slow. It
> > > > > > > > stores
> > > > > > > > > the payload in high speed memory cache and returns asap.
> That
> > > > > memory
> > > > > > is
> > > > > > > > > eventually persisted into S3. The retrieval goest against
> the
> > > cache
> > > > > > > > first,
> > > > > > > > > then against the S3. Overall a rather cheappy and zippy
> > > solution. I
> > > > > > > tried
> > > > > > > > > proposing the KIP for this, but there was no excitement.
> > Check
> > > this
> > > > > > > out:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi, Andrew,
> > > > > > > > > >
> > > > > > > > > > If we want to focus only on reducing cross-zone
> replication
> > > > > costs,
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > an alternative design in the KIP-1176 discussion thread
> > that
> > > > > seems
> > > > > > > > > simpler
> > > > > > > > > > than the proposal here. I am copying the outline of that
> > > approach
> > > > > > > > below.
> > > > > > > > > >
> > > > > > > > > > 1. A new leader is elected.
> > > > > > > > > > 2. Leader maintains a first tiered offset, which is
> > > initialized
> > > > > to
> > > > > > > log
> > > > > > > > > end
> > > > > > > > > > offset.
> > > > > > > > > > 3. Leader writes produced data from the client to local
> > log.
> > > > > > > > > > 4. Leader uploads produced data from all local logs as a
> > > combined
> > > > > > > > object
> > > > > > > > > > 5. Leader stores the metadata for the combined object in
> > > memory.
> > > > > > > > > > 6. If a follower fetch request has an offset >= first
> > tiered
> > > > > > offset,
> > > > > > > > the
> > > > > > > > > > metadata for the corresponding combined object is
> returned.
> > > > > > > Otherwise,
> > > > > > > > > the
> > > > > > > > > > local data is returned.
> > > > > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > > > > >
> > > > > > > > > > It's still a bit unnatural, but it could work.
> > > > > > > > > >
> > > > > > > > > > Hi, Ivan,
> > > > > > > > > >
> > > > > > > > > > Are you still committed to proceeding with the original
> > > design of
> > > > > > > > > KIP-1150?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >> I’ve been following KIP-1150 and friends for a while.
> I’m
> > > going
> > > > > to
> > > > > > > > jump
> > > > > > > > > >> into the discussions too.
> > > > > > > > > >>
> > > > > > > > > >> Looking back at Jack Vanlightly’s message, I am not
> quite
> > so
> > > > > > > convinced
> > > > > > > > > >> that it’s a kind of fork in the road. The primary aim of
> > the
> > > > > > effort
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > >> reduce cross-zone replication costs so Apache Kafka is
> not
> > > > > > > > prohibitively
> > > > > > > > > >> expensive to use on cloud storage. I think it would be
> > > entirely
> > > > > > > viable
> > > > > > > > > to
> > > > > > > > > >> prioritise code reuse for an initial implementation of
> > > diskless
> > > > > > > > topics,
> > > > > > > > > and
> > > > > > > > > >> we could still have a more cloud-native design in the
> > > future.
> > > > > It’s
> > > > > > > > hard
> > > > > > > > > to
> > > > > > > > > >> predict what the community will prioritise in the
> future.
> > > > > > > > > >>
> > > > > > > > > >> Of the three major revisions, I’m in the rev3 camp. We
> can
> > > > > support
> > > > > > > > > >> leaderless produce requests, first writing WAL segments
> > into
> > > > > > object
> > > > > > > > > >> storage, and then using the regular partition leaders to
> > > > > sequence
> > > > > > > the
> > > > > > > > > >> records. The active log segment for a diskless topic
> will
> > > > > > initially
> > > > > > > > > contain
> > > > > > > > > >> batch coordinates rather than record batches. The batch
> > > > > > coordinates
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > > >> resolved from WAL segments for consumers, and also in
> > order
> > > to
> > > > > > > prepare
> > > > > > > > > log
> > > > > > > > > >> segments for uploading to tiered storage. Jun is
> probably
> > > > > correct
> > > > > > > that
> > > > > > > > > we
> > > > > > > > > >> need a more frequent object merging process than tiered
> > > storage
> > > > > > > > > provides.
> > > > > > > > > >> This is just the transition from write-optimised WAL
> > > segments to
> > > > > > > > > >> read-optimised tiered segments, and all of the object
> > > > > > storage-based
> > > > > > > > > >> implementations of Kafka that I’m aware of do this
> > > > > rearrangement.
> > > > > > > But
> > > > > > > > > >> perhaps this more frequent object merging is a pre-GA
> > > > > improvement,
> > > > > > > > > rather
> > > > > > > > > >> than a strict requirement for an initial implementation
> > for
> > > > > early
> > > > > > > > access
> > > > > > > > > >> use.
> > > > > > > > > >>
> > > > > > > > > >> For zone-aligned share consumers, the share group
> assignor
> > > is
> > > > > > > intended
> > > > > > > > > to
> > > > > > > > > >> be rack-aware. Consumers should be assigned to
> partitions
> > > with
> > > > > > > leaders
> > > > > > > > > in
> > > > > > > > > >> their zone. The simple assignor is not rack-aware, but
> it
> > > easily
> > > > > > > could
> > > > > > > > > be
> > > > > > > > > >> or we could have a rack-aware assignor.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Andrew
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> Hi, Ivan,
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > >>>
> > > > > > > > > >>> "As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>> will indeed consume some IOPS. See them as a cache that
> > > could
> > > > > > > always
> > > > > > > > be
> > > > > > > > > >>> restored from the remote storage. While it’s not ideal,
> > > it's
> > > > > > still
> > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > of
> > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>> can avoid explicit flushing on local materialized
> > segments
> > > at
> > > > > all
> > > > > > > and
> > > > > > > > > let
> > > > > > > > > >>> the file system and page cache figure out when to flush
> > > > > > optimally.
> > > > > > > > This
> > > > > > > > > >>> would not eliminate the extra IOPS, but should reduce
> it
> > > > > > > > dramatically,
> > > > > > > > > >>> depending on throughput for each partition. We, of
> > course,
> > > > > > continue
> > > > > > > > > >>> flushing the metadata segments as before."
> > > > > > > > > >>>
> > > > > > > > > >>> If we have a mix of classic and diskless topics on the
> > same
> > > > > > broker,
> > > > > > > > > it's
> > > > > > > > > >>> important that the classic topics' data is flushed to
> > disk
> > > as
> > > > > > > quickly
> > > > > > > > > as
> > > > > > > > > >>> possible. To achieve this, users typically set
> > > > > > > dirty_expire_centisecs
> > > > > > > > > in
> > > > > > > > > >>> the kernel based on the number of available disk IOPS.
> > > Once you
> > > > > > set
> > > > > > > > > this
> > > > > > > > > >>> number, it applies to all dirty files, including the
> > cached
> > > > > data
> > > > > > in
> > > > > > > > > >>> diskless topics. So, if there are more files actively
> > > > > > accumulating
> > > > > > > > > data,
> > > > > > > > > >>> the flush frequency and therefore RPO is reduced for
> > > classic
> > > > > > > topics.
> > > > > > > > > >>>
> > > > > > > > > >>> "We should have mentioned this explicitly, but this
> step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>> in the form of segments offloading to tiered storage.
> > When
> > > we
> > > > > > > > assemble
> > > > > > > > > a
> > > > > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> > > effectively
> > > > > > > doing
> > > > > > > > > >>> metadata compaction: replacing a big number of pieces
> of
> > > > > metadata
> > > > > > > > about
> > > > > > > > > >>> individual batches with a single record in
> > > > > > __remote_log_metadata."
> > > > > > > > > >>>
> > > > > > > > > >>> The object merging in tier storage typically only kicks
> > in
> > > > > after
> > > > > > a
> > > > > > > > few
> > > > > > > > > >>> hours. The impact is (1) the amount of accumulated
> > > metadata is
> > > > > > > still
> > > > > > > > > >> quite
> > > > > > > > > >>> large; (2) there are many small objects, leading to
> poor
> > > read
> > > > > > > > > >> performance.
> > > > > > > > > >>> I think we need a more frequent object merging process
> > than
> > > > > tier
> > > > > > > > > storage
> > > > > > > > > >>> provides.
> > > > > > > > > >>>
> > > > > > > > > >>> Jun
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me>
> > > > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thank you for your messages.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> First, it degrades the durability.
> > > > > > > > > >>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at a
> > > > > > > > > >>>>> given point of time, one for the data and another for
> > the
> > > > > > > metadata.
> > > > > > > > > >>>>> Flushing each file requires a separate IO. If the
> disk
> > > has 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >> we
> > > > > > > > > >>>>> have 5K partitions in a broker, currently we can
> afford
> > > to
> > > > > > flush
> > > > > > > > each
> > > > > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> number of files per partition, we can only flush each
> > > > > partition
> > > > > > > > every
> > > > > > > > > >> 10
> > > > > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>>> will indeed consume some IOPS. See them as a cache
> that
> > > could
> > > > > > > always
> > > > > > > > > be
> > > > > > > > > >>>> restored from the remote storage. While it’s not
> ideal,
> > > it's
> > > > > > still
> > > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > > of
> > > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>>> can avoid explicit flushing on local materialized
> > > segments at
> > > > > > all
> > > > > > > > and
> > > > > > > > > >> let
> > > > > > > > > >>>> the file system and page cache figure out when to
> flush
> > > > > > optimally.
> > > > > > > > > This
> > > > > > > > > >>>> would not eliminate the extra IOPS, but should reduce
> it
> > > > > > > > dramatically,
> > > > > > > > > >>>> depending on throughput for each partition. We, of
> > course,
> > > > > > > continue
> > > > > > > > > >>>> flushing the metadata segments as before.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s worth making a note on caching. I think nobody
> will
> > > > > > disagree
> > > > > > > > that
> > > > > > > > > >>>> doing direct reads from remote storage every time a
> > batch
> > > is
> > > > > > > > requested
> > > > > > > > > >> by a
> > > > > > > > > >>>> consumer will not be practical neither from the
> > > performance
> > > > > nor
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > >>>> economy point of view. We need a way to keep the
> number
> > > of GET
> > > > > > > > > requests
> > > > > > > > > >>>> down. There are multiple options, for example:
> > > > > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > > > > >>>> 2. Local in-memory caching. Comes with less network
> > > chattiness
> > > > > > and
> > > > > > > > > >> works
> > > > > > > > > >>>> well if we have more or less stable brokers to consume
> > > from.
> > > > > > > > > >>>> 3. Materialization of diskless logs on local disk. Way
> > > lower
> > > > > > > impact
> > > > > > > > on
> > > > > > > > > >>>> RAM and also requires stable brokers for consumption
> > > (using
> > > > > just
> > > > > > > > > >> assigned
> > > > > > > > > >>>> replicas will probably work well).
> > > > > > > > > >>>>
> > > > > > > > > >>>> Materialization is one of possible options, but we can
> > > choose
> > > > > > > > another
> > > > > > > > > >> one.
> > > > > > > > > >>>> However, we will have this dilemma regardless of
> whether
> > > we
> > > > > have
> > > > > > > an
> > > > > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Second, if we ever need this
> > > > > > > > > >>>>> metadata somewhere else, say in the WAL file manager,
> > the
> > > > > > > consumer
> > > > > > > > > >> needs
> > > > > > > > > >>>> to
> > > > > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>> actual benefit of this approach is also questionable.
> > On
> > > the
> > > > > > > > surface,
> > > > > > > > > >> it
> > > > > > > > > >>>>> might seem that we could reduce the number of lines
> > that
> > > need
> > > > > > to
> > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>> for this KIP. However, the changes are quite
> intrusive
> > > to the
> > > > > > > > classic
> > > > > > > > > >>>>> partition's code path and will probably make the code
> > > base
> > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>> maintain in the long run. I like the original
> approach
> > > based
> > > > > on
> > > > > > > the
> > > > > > > > > >> batch
> > > > > > > > > >>>>> coordinator much better than this one. We could
> > probably
> > > > > > refactor
> > > > > > > > the
> > > > > > > > > >>>>> producer state code so that it could be reused in the
> > > batch
> > > > > > > > > >> coordinator.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s hard to disagree with this. The explicit
> > coordinator
> > > is
> > > > > > more
> > > > > > > a
> > > > > > > > > side
> > > > > > > > > >>>> thing, while coordinator-less approach is more about
> > > extending
> > > > > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Thanks for addressing the concerns on the number of
> > RPCs
> > > in
> > > > > the
> > > > > > > > > produce
> > > > > > > > > >>>>> path. I agree that with the metadata crafting
> > mechanism,
> > > we
> > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > be
> > > > > > > > > >>>>> collocated with the data on the same set of brokers,
> > it's
> > > > > weird
> > > > > > > > that
> > > > > > > > > >> they
> > > > > > > > > >>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > uses
> > > > > > > > > >> the
> > > > > > > > > >>>>> metadata crafting mechanism, but the metadata is
> stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>> partition using its own assignment strategy. It will
> be
> > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>> them collocated.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I would like to note that the metadata crafting is
> > needed
> > > only
> > > > > > to
> > > > > > > > tell
> > > > > > > > > >>>> producers which brokers they should send Produce
> > requests
> > > to,
> > > > > > but
> > > > > > > > data
> > > > > > > > > >> (as
> > > > > > > > > >>>> in “locally materialized log”) is located on partition
> > > > > replicas,
> > > > > > > > i.e.
> > > > > > > > > >>>> automatically co-located with metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As a side note, it would probably be better that
> instead
> > > of
> > > > > > > > implicitly
> > > > > > > > > >>>> crafting partition metadata, we extend the metadata
> > > protocol
> > > > > so
> > > > > > > that
> > > > > > > > > for
> > > > > > > > > >>>> diskless partitions we return not only the leader and
> > > > > replicas,
> > > > > > > but
> > > > > > > > > also
> > > > > > > > > >>>> some “recommended produce brokers”, selected for
> optimal
> > > > > > > performance
> > > > > > > > > and
> > > > > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>> step.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We should have mentioned this explicitly, but this
> step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>>> in the form of segments offloading to tiered storage.
> > > When we
> > > > > > > > > assemble a
> > > > > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > > > > effectively
> > > > > > > > doing
> > > > > > > > > >>>> metadata compaction: replacing a big number of pieces
> of
> > > > > > metadata
> > > > > > > > > about
> > > > > > > > > >>>> individual batches with a single record in
> > > > > > __remote_log_metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We could create a Diskless-specific merging mechanism
> > > instead
> > > > > if
> > > > > > > > > needed.
> > > > > > > > > >>>> It’s rather easy with the explicit coordinator
> approach.
> > > With
> > > > > > the
> > > > > > > > > >>>> coordinator-less approach, this would probably be a
> bit
> > > more
> > > > > > > tricky
> > > > > > > > > >>>> (rewriting the tail of the log by the leader +
> > replicating
> > > > > this
> > > > > > > > change
> > > > > > > > > >>>> reliably).
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >> changes
> > > > > > > > > >>>> in
> > > > > > > > > >>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >> last
> > > > > > > > > >>>>> for the long term.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Yes, totally agree.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Luke,
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler
> if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>
> > > > > > > > > >>>> After some evolution of the original proposal,
> > leaderless
> > > is
> > > > > now
> > > > > > > > > >> limited.
> > > > > > > > > >>>> We only need to be able to accept Produce requests on
> > more
> > > > > than
> > > > > > > one
> > > > > > > > > >> broker
> > > > > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I
> get
> > it
> > > > > right
> > > > > > > > that
> > > > > > > > > >> you
> > > > > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let’s now look at this problem from a higher level, as
> > > Jack
> > > > > > > > proposed.
> > > > > > > > > As
> > > > > > > > > >>>> it was said, the big choice we need to make is whether
> > we
> > > 1)
> > > > > > > create
> > > > > > > > an
> > > > > > > > > >>>> explicit batch coordinator; or 2) go for the
> > > coordinator-less
> > > > > > > > > approach,
> > > > > > > > > >>>> where each diskless partition is managed by its leader
> > as
> > > in
> > > > > > > classic
> > > > > > > > > >> topics.
> > > > > > > > > >>>>
> > > > > > > > > >>>> If we try to compare the two approaches:
> > > > > > > > > >>>>
> > > > > > > > > >>>> Pluggability:
> > > > > > > > > >>>> - Explicit coordinator: Possible. For example, some
> > > setups may
> > > > > > > > benefit
> > > > > > > > > >>>> from batch metadata being stored in a cloud database
> > > (such as
> > > > > > AWS
> > > > > > > > > >> DynamoDB
> > > > > > > > > >>>> or GCP Spanner).
> > > > > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Scalability and fault tolerance:
> > > > > > > > > >>>> - Explicit coordinator: Depends on the implementation
> > and
> > > it’s
> > > > > > > also
> > > > > > > > > >>>> necessary to actively work for it.
> > > > > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics.
> > > Scaling is
> > > > > > > done
> > > > > > > > by
> > > > > > > > > >>>> partition placement, partitions could fail
> > independently.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Separation of concerns:
> > > > > > > > > >>>> - Explicit coordinator: Very good. Diskless remains
> more
> > > > > > > independent
> > > > > > > > > >> from
> > > > > > > > > >>>> classic topics in terms of code and workflows. For
> > > example,
> > > > > the
> > > > > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > > > > mechanism
> > > > > > > > could
> > > > > > > > > >> be
> > > > > > > > > >>>> relatively simply implemented with it. As a flip side
> of
> > > this,
> > > > > > > some
> > > > > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > > > > bringing
> > > > > > > > > diskless
> > > > > > > > > >>>> closer to classic topics. Some code paths and
> workflows
> > > could
> > > > > be
> > > > > > > > more
> > > > > > > > > >>>> straightforwardly reused, but they will inevitably
> have
> > > to be
> > > > > > > > adapted
> > > > > > > > > to
> > > > > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Cloud-nativeness. This is a vague concept, also
> related
> > > to the
> > > > > > > > > previous,
> > > > > > > > > >>>> but let’s try:
> > > > > > > > > >>>> - Explicit coordinator: Storing and processing
> metadata
> > > > > > separately
> > > > > > > > > makes
> > > > > > > > > >>>> it easier for brokers to take different roles, be
> purely
> > > > > > stateless
> > > > > > > > if
> > > > > > > > > >>>> needed, etc.
> > > > > > > > > >>>> - Coordinator-less: Less so. Something could be
> achieved
> > > with
> > > > > > > > creative
> > > > > > > > > >>>> partition placement, but not much.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Both seem to have their pros and cons. However,
> > answering
> > > > > Jack’s
> > > > > > > > > >> question,
> > > > > > > > > >>>> the explicit coordinator approach may indeed lead to a
> > > more
> > > > > > > flexible
> > > > > > > > > >> design.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> The purpose of this deviation in the discussion was to
> > > > > receive a
> > > > > > > > > >>>> preliminary community evaluation of the
> coordinator-less
> > > > > > approach
> > > > > > > > > >> without
> > > > > > > > > >>>> taking on the task of writing a separate KIP and
> fitting
> > > it in
> > > > > > the
> > > > > > > > > >> system
> > > > > > > > > >>>> of KIP-1150 and its children. We’re open to stopping
> it
> > > and
> > > > > > > getting
> > > > > > > > > >> back to
> > > > > > > > > >>>> working out the coordinator design if the community
> > > doesn’t
> > > > > > favor
> > > > > > > > the
> > > > > > > > > >>>> proposed approach.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best,
> > > > > > > > > >>>> Ivan and Diskless team
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > > > > >>>>> Hi Ivan,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> As Jun pointed out, the updated design seems to have
> > some
> > > > > > > > > shortcomings
> > > > > > > > > >>>>> although it simplifies the implementation.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler
> if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thank you,
> > > > > > > > > >>>>> Luke
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for the explanation.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> "we write the reference to the WAL file with the
> batch
> > > data"
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I understand the approach now, but I think it is a
> > hacky
> > > > > one.
> > > > > > > > There
> > > > > > > > > >> are
> > > > > > > > > >>>>>> multiple short comings with this design. First, it
> > > degrades
> > > > > > the
> > > > > > > > > >>>> durability.
> > > > > > > > > >>>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at
> > > > > > > > > >> a
> > > > > > > > > >>>>>> given point of time, one for the data and another
> for
> > > the
> > > > > > > > metadata.
> > > > > > > > > >>>>>> Flushing each file requires a separate IO. If the
> disk
> > > has
> > > > > 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> have 5K partitions in a broker, currently we can
> > afford
> > > to
> > > > > > flush
> > > > > > > > > each
> > > > > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> number of files per partition, we can only flush
> each
> > > > > > partition
> > > > > > > > > every
> > > > > > > > > >>>> 10
> > > > > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we
> > > ever
> > > > > need
> > > > > > > > this
> > > > > > > > > >>>>>> metadata somewhere else, say in the WAL file
> manager,
> > > the
> > > > > > > consumer
> > > > > > > > > >>>> needs to
> > > > > > > > > >>>>>> subscribe to every partition in the cluster, which
> is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>>> actual benefit of this approach is also
> questionable.
> > > On the
> > > > > > > > > surface,
> > > > > > > > > >>>> it
> > > > > > > > > >>>>>> might seem that we could reduce the number of lines
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>>> for this KIP. However, the changes are quite
> intrusive
> > > to
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition's code path and will probably make the
> code
> > > base
> > > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>>> maintain in the long run. I like the original
> approach
> > > based
> > > > > > on
> > > > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>> coordinator much better than this one. We could
> > probably
> > > > > > > refactor
> > > > > > > > > the
> > > > > > > > > >>>>>> producer state code so that it could be reused in
> the
> > > batch
> > > > > > > > > >>>> coordinator.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for addressing the concerns on the number of
> > > RPCs in
> > > > > > the
> > > > > > > > > >> produce
> > > > > > > > > >>>>>> path. I agree that with the metadata crafting
> > > mechanism, we
> > > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > > be
> > > > > > > > > >>>>>> collocated with the data on the same set of brokers,
> > > it's
> > > > > > weird
> > > > > > > > that
> > > > > > > > > >>>> they
> > > > > > > > > >>>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > > uses
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> metadata crafting mechanism, but the metadata is
> > stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition using its own assignment strategy. It will
> > be
> > > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> them collocated.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>>> step. My first concern is on the amount of metadata
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> kept.
> > > > > > > > > >>>>>> Without object compcation, the metadata generated in
> > the
> > > > > > produce
> > > > > > > > > path
> > > > > > > > > >>>> can
> > > > > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's
> > > say for
> > > > > > > every
> > > > > > > > > >>>> 250ms we
> > > > > > > > > >>>>>> produce 100 byte of metadata per partition. Let's
> say
> > > > > remoting
> > > > > > > > > tiering
> > > > > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K
> > > partitions,
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> > > metadata,
> > > > > > > quite
> > > > > > > > > >>>>>> signficant. A second concern is on performance.
> Every
> > > time
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>>>> rebuild the caching data, we need to read a bunch of
> > > small
> > > > > > > objects
> > > > > > > > > >>>> from S3,
> > > > > > > > > >>>>>> slowing down the building process. If a consumer
> > > happens to
> > > > > > need
> > > > > > > > > such
> > > > > > > > > >>>> data,
> > > > > > > > > >>>>>> it could slow down the application.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >>>> changes in
> > > > > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >>>> last
> > > > > > > > > >>>>>> for the long term.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Jun
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hi Jun,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed
> > to
> > > > > > clearly
> > > > > > > > > >>>> explain
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it
> essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful
> to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if
> we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > partitions.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> No. There will be only one physical partition per
> > > diskless
> > > > > > > > > >>>> partition. Let
> > > > > > > > > >>>>>>> me explain this with an example. Let’s say we have
> a
> > > > > diskless
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> > > leader. We
> > > > > > > > produce
> > > > > > > > > >>>> some
> > > > > > > > > >>>>>>> batches to this partition. The content of the
> segment
> > > file
> > > > > > will
> > > > > > > > be
> > > > > > > > > >>>>>>> something like this (for each batch):
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > > > > >>>>>>> Magic: like in classic
> > > > > > > > > >>>>>>> CRC: like in classic
> > > > > > > > > >>>>>>> Attributes: like in classic
> > > > > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > > > > >>>>>>> ProducerId: like in classic
> > > > > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > > > > >>>>>>> Records:
> > > > > > > > > >>>>>>>
> > path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > > > > byte
> > > > > > > > offset
> > > > > > > > > >>>>>>> 123456
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It looks very much like classic log entries. The
> only
> > > > > > > difference
> > > > > > > > is
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> instead of writing real Records, we write the
> > > reference to
> > > > > > the
> > > > > > > > WAL
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> with the batch data (I guess we need only the name
> > and
> > > the
> > > > > > byte
> > > > > > > > > >>>> offset,
> > > > > > > > > >>>>>>> because the byte length is the standard field
> above).
> > > > > > > Otherwise,
> > > > > > > > > >>>> it’s a
> > > > > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> So we have as many partitions for diskless as for
> > > classic.
> > > > > As
> > > > > > > of
> > > > > > > > > open
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Are the metadata and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated
> on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The source of truth for the data is still in WAL
> > files
> > > on
> > > > > > > object
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>> The source of truth for the metadata is in segment
> > > files on
> > > > > > the
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>> the replica set. Two new mechanisms are planned,
> both
> > > > > > > independent
> > > > > > > > > of
> > > > > > > > > >>>> this
> > > > > > > > > >>>>>>> new proposal, but I want to present them to give
> the
> > > idea
> > > > > > that
> > > > > > > > > only a
> > > > > > > > > >>>>>>> limited amount of data files will be operated
> > locally:
> > > > > > > > > >>>>>>> - We want to assemble batches into segment files
> and
> > > > > offload
> > > > > > > them
> > > > > > > > > to
> > > > > > > > > >>>>>>> tiered storage in order to prevent the unbounded
> > > growth of
> > > > > > > batch
> > > > > > > > > >>>>>> metadata.
> > > > > > > > > >>>>>>> For this, we need to open only  a few file
> > descriptors
> > > (for
> > > > > > the
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>> file itself + the necessary indexes) before the
> > > segment is
> > > > > > > fully
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > > > > >>>>>>> - We want to assemble local segment files for
> caching
> > > > > > purposes
> > > > > > > as
> > > > > > > > > >>>> well,
> > > > > > > > > >>>>>>> i.e. to speed up fetching. This will not
> materialize
> > > the
> > > > > full
> > > > > > > > > >>>> content of
> > > > > > > > > >>>>>>> the log, but only the hot set according to some
> > policy
> > > (or
> > > > > > > > > >>>> configurable
> > > > > > > > > >>>>>>> policies), i.e. the number of segments and file
> > > descriptors
> > > > > > > will
> > > > > > > > > >>>> also be
> > > > > > > > > >>>>>>> limited.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This is a valid concern that we considered, but
> this
> > > issue
> > > > > > can
> > > > > > > be
> > > > > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > > > > >>>>>>> The situation with a single broker is trivial: all
> > the
> > > > > commit
> > > > > > > > > >>>> requests go
> > > > > > > > > >>>>>>> from the broker to itself.
> > > > > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but
> > > located in
> > > > > > the
> > > > > > > > > single
> > > > > > > > > >>>>>> rack
> > > > > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for
> > > diskless
> > > > > > > > > >>>> partitions, but
> > > > > > > > > >>>>>>> we can tell producers (through metadata) to always
> > send
> > > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader
> > > replicas
> > > > > for
> > > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive
> diskless
> > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>> these partitions in various combinations, but only
> > for
> > > > > them.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>                   Broker 0
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>             |    t1-0         |
> > > > > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > > > > >>>>>>> partitions   | +-------------+ |    for these
> > > partitions
> > > > > > > > > >>>>>>>             |                 |
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > > > > Effectively,
> > > > > > > > > each
> > > > > > > > > >>>>>>> broker will commit only to itself, which
> effectively
> > > means
> > > > > 1
> > > > > > > > commit
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>> per WAL buffer (this may be 0 physical network
> calls,
> > > if we
> > > > > > > wish,
> > > > > > > > > >>>> just a
> > > > > > > > > >>>>>>> local function call).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs).
> > > Obviously, we
> > > > > > > > cannot
> > > > > > > > > >>>> always
> > > > > > > > > >>>>>>> send Produce requests to the designated leaders of
> > > diskless
> > > > > > > > > >>>> partitions:
> > > > > > > > > >>>>>>> this would mean inter-AZ network traffic, which we
> > > would
> > > > > like
> > > > > > > to
> > > > > > > > > >>>> avoid.
> > > > > > > > > >>>>>> To
> > > > > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> > > produce
> > > > > > > > > >>>>>> representative”
> > > > > > > > > >>>>>>> in every AZ. If we continue our example: when a
> > Produce
> > > > > > request
> > > > > > > > for
> > > > > > > > > >>>> t1-0,
> > > > > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it
> lands
> > > on
> > > > > > > broker 0
> > > > > > > > > >>>> (in the
> > > > > > > > > >>>>>>> broker’s AZ the representative is the broker
> itself).
> > > > > > However,
> > > > > > > if
> > > > > > > > > it
> > > > > > > > > >>>>>> comes
> > > > > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s
> broker
> > > 2.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> |produce requests         |produce requests
> > > |produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
> > |for
> > > > > t1-0,
> > > > > > > > t2-1,
> > > > > > > > > >>>>>> t3-0
> > > > > > > > > >>>>>>> |from AZ 0                |from AZ 1
> > >  |from
> > > > > AZ 2
> > > > > > > > > >>>>>>> v                         v
> v
> > > > > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)
> Broker
> > 2
> > > (AZ
> > > > > 2)
> > > > > > > > > >>>>>>> +---------------+      +---------------+
> > > > > > +---------------+
> > > > > > > > > >>>>>>> |     t1-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t2-1      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t3-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> +---------------+      +--------+------+
> > > > > > +--------+------+
> > > > > > > > > >>>>>>>    ^     ^                    |
> > |
> > > > > > > > > >>>>>>>    |     +--------------------+
> > |
> > > > > > > > > >>>>>>>    |     commit requests for these partitions
> > |
> > > > > > > > > >>>>>>>    |
> >  |
> > > > > > > > > >>>>>>>
> > +-------------------------------------------------+
> > > > > > > > > >>>>>>>          commit requests for these partitions
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> All the partitions that broker 0 is the leader of
> > will
> > > be
> > > > > > > > > >>>> “represented”
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Of course, this relationship goes both ways between
> > AZs
> > > > > (not
> > > > > > > > > >>>> necessarily
> > > > > > > > > >>>>>>> between the same brokers). It means that provided
> the
> > > > > cluster
> > > > > > > is
> > > > > > > > > >>>> balanced
> > > > > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> > > represent
> > > > > > > > > >>>>>> (number_of_azs
> > > > > > > > > >>>>>>> - 1) other brokers. This will result in the
> situation
> > > that
> > > > > > for
> > > > > > > > the
> > > > > > > > > >>>>>> majority
> > > > > > > > > >>>>>>> of commits, each broker will do up to
> (number_of_azs
> > -
> > > 1)
> > > > > > > network
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests (plus one local). Cloud regions tend to
> > have 3
> > > > > AZs,
> > > > > > > very
> > > > > > > > > >>>> rarely
> > > > > > > > > >>>>>>> more. That means, brokers will be doing up to 2
> > network
> > > > > > commit
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> per
> > > > > > > > > >>>>>>> WAL file.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> There are the following exceptions:
> > > > > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example,
> > > when we
> > > > > > > have
> > > > > > > > 2
> > > > > > > > > >>>> AZs
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> one has three brokers and another AZ has one. This
> > one
> > > > > broker
> > > > > > > > will
> > > > > > > > > do
> > > > > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This
> is
> > > not
> > > > > an
> > > > > > > > > extreme
> > > > > > > > > >>>>>>> amplification. Such an imbalance is not healthy in
> > most
> > > > > > > practical
> > > > > > > > > >>>> setups
> > > > > > > > > >>>>>>> and should be avoided anyway.
> > > > > > > > > >>>>>>> 2. Leadership changes and metadata propagation
> > period.
> > > When
> > > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3,
> the
> > > > > > producers
> > > > > > > > > will
> > > > > > > > > >>>> not
> > > > > > > > > >>>>>>> know this immediately (unless we want to be strict
> > and
> > > > > > respond
> > > > > > > > with
> > > > > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> > > will
> > > > > come
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>> in a
> > > > > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two
> > commit
> > > > > > > requests:
> > > > > > > > > to
> > > > > > > > > >>>>>> broker
> > > > > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to
> commit
> > > t3-0.
> > > > > > This
> > > > > > > > > >>>> situation
> > > > > > > > > >>>>>>> is not permanent and as producers update the
> cluster
> > > > > > metadata,
> > > > > > > it
> > > > > > > > > >>>> will be
> > > > > > > > > >>>>>>> resolved.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This all could be built with the metadata crafting
> > > > > mechanism
> > > > > > > only
> > > > > > > > > >>>> (which
> > > > > > > > > >>>>>>> is anyway needed for Diskless in one way or another
> > to
> > > > > direct
> > > > > > > > > >>>> producers
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic),
> > > just
> > > > > with
> > > > > > > the
> > > > > > > > > >>>> right
> > > > > > > > > >>>>>>> policy for it (for example, some deterministic
> > > hash-based
> > > > > > > > formula).
> > > > > > > > > >>>> I.e.
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>> explicit support for “produce representative” or
> > > anything
> > > > > > like
> > > > > > > > this
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the
> source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> We do only two operations on WAL files that span
> > > multiple
> > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions: committing and deleting. Commits can be
> > > done
> > > > > > > > > >>>> independently as
> > > > > > > > > >>>>>>> described above. But deletes are different, because
> > > when a
> > > > > > file
> > > > > > > > is
> > > > > > > > > >>>>>> deleted,
> > > > > > > > > >>>>>>> this affects all the partitions that still have
> alive
> > > > > batches
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> (if any).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The WAL file manager is a necessary point of
> > > coordination
> > > > > to
> > > > > > > > delete
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> files safely. We can say it is the source of truth
> > > about
> > > > > > files
> > > > > > > > > >>>>>> themselves,
> > > > > > > > > >>>>>>> while the partition leaders and their logs hold the
> > > truth
> > > > > > about
> > > > > > > > > >>>> whether a
> > > > > > > > > >>>>>>> particular file contains live batches of this
> > > particular
> > > > > > > > partition.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager will do this important task: be
> able
> > > to
> > > > > say
> > > > > > > for
> > > > > > > > > sure
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> a file does not contain any live batch of any
> > existing
> > > > > > > partition.
> > > > > > > > > For
> > > > > > > > > >>>>>> this,
> > > > > > > > > >>>>>>> it will have to periodically check against the
> > > partition
> > > > > > > leaders.
> > > > > > > > > >>>>>>> Considering that batch deletion is irreversible,
> when
> > > we
> > > > > > > declare
> > > > > > > > a
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager has to know about files being
> > > committed to
> > > > > > > start
> > > > > > > > > >>>> track
> > > > > > > > > >>>>>>> them and periodically check if they are empty. We
> can
> > > > > > consider
> > > > > > > > > >>>> various
> > > > > > > > > >>>>>> ways
> > > > > > > > > >>>>>>> to achieve this:
> > > > > > > > > >>>>>>> 1. As was proposed in my previous message: best
> > effort
> > > > > commit
> > > > > > > by
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> +
> > > > > > > > > >>>>>>> periodic prefix scans of object storage to detect
> > files
> > > > > that
> > > > > > > went
> > > > > > > > > >>>> below
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> radar due to network issue or the file manager
> > > temporary
> > > > > > > > > >>>> unavailability.
> > > > > > > > > >>>>>>> We’re speaking about listing the file names only
> and
> > > > > opening
> > > > > > > only
> > > > > > > > > >>>>>>> previously unknown files in order to find the
> > > partitions
> > > > > > > involved
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>> them.
> > > > > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill
> > the
> > > > > list
> > > > > > of
> > > > > > > > > files
> > > > > > > > > >>>>>>> fully asynchronously and in the background. This
> may
> > > be not
> > > > > > > ideal
> > > > > > > > > >>>> due to
> > > > > > > > > >>>>>>> costs and performance of scanning tons of files.
> > > However,
> > > > > the
> > > > > > > > > number
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>> live WAL files should be limited due to tiered
> > storage
> > > > > > > > offloading +
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> can
> > > > > > > > > >>>>>>> optimize this if we give files some global soft
> order
> > > in
> > > > > > their
> > > > > > > > > names.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > logic.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > > > > UnifiedLog
> > > > > > in
> > > > > > > > > >>>> order to
> > > > > > > > > >>>>>>> support these changes. However, it seems that
> > > idempotence,
> > > > > > > > > >>>> transactions,
> > > > > > > > > >>>>>>> queues, tiered storage will have to be changed less
> > > than
> > > > > with
> > > > > > > the
> > > > > > > > > >>>>>> original
> > > > > > > > > >>>>>>> design. This is because the partition leader state
> > > would
> > > > > > remain
> > > > > > > > in
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> same
> > > > > > > > > >>>>>>> place (on brokers) and existing workflows that
> > involve
> > > it
> > > > > > would
> > > > > > > > > have
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> be
> > > > > > > > > >>>>>>> changed less compared to the situation where we
> > > globalize
> > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader state in the batch coordinator. I admit this
> > is
> > > hard
> > > > > > to
> > > > > > > > make
> > > > > > > > > >>>>>>> convincing without both real implementations to
> hand
> > :)
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more
> details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Let’s consider the scenario of turning a classic
> > topic
> > > into
> > > > > > > > > >>>> diskless. The
> > > > > > > > > >>>>>>> user sets diskless.enabled=true, the leader
> receives
> > > this
> > > > > > > > metadata
> > > > > > > > > >>>> update
> > > > > > > > > >>>>>>> and does the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Start a new segment that will be written in the
> > > diskless
> > > > > > > > format
> > > > > > > > > >>>> (i.e.
> > > > > > > > > >>>>>>> without data).
> > > > > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Since it’s the same log, the followers will know
> > about
> > > that
> > > > > > > > switch
> > > > > > > > > >>>>>>> consistently. They will finish replicating the
> > classic
> > > > > > segments
> > > > > > > > and
> > > > > > > > > >>>> start
> > > > > > > > > >>>>>>> replicating the diskless ones. They will always
> know
> > > where
> > > > > > each
> > > > > > > > > >>>> batch is
> > > > > > > > > >>>>>>> located (either inside a classic segment or
> > referenced
> > > by a
> > > > > > > > > diskless
> > > > > > > > > >>>>>> one).
> > > > > > > > > >>>>>>> Switching back should be similar.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Doing this with the coordinator is possible, but
> has
> > > some
> > > > > > > > caveats.
> > > > > > > > > >>>> The
> > > > > > > > > >>>>>>> leader must do the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Write a special control segment to persist and
> > > replicate
> > > > > > the
> > > > > > > > > fact
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> from offset N the partition is now in the diskless
> > > mode.
> > > > > > > > > >>>>>>> 4. Inform the coordinator about the first offset N
> of
> > > the
> > > > > > > > “diskless
> > > > > > > > > >>>> era”.
> > > > > > > > > >>>>>>> 5. Inform the controller quorum that the transition
> > has
> > > > > > > finished
> > > > > > > > > and
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> brokers now can process diskless writes for this
> > > partition.
> > > > > > > > > >>>>>>> This could fail at some points, so this will
> probably
> > > > > require
> > > > > > > > some
> > > > > > > > > >>>>>>> explicit state machine with replication either in
> the
> > > > > > partition
> > > > > > > > log
> > > > > > > > > >>>> or in
> > > > > > > > > >>>>>>> KRaft.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It seems that the coordinator-less approach makes
> > this
> > > > > > simpler
> > > > > > > > > >>>> because
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> “coordinator” for the partition and the partition
> > > leader
> > > > > are
> > > > > > > the
> > > > > > > > > >>>> same and
> > > > > > > > > >>>>>>> they store the partition metadata in the same log,
> > too.
> > > > > While
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>>>>>> coordinator approach we have to perform some kind
> of
> > a
> > > > > > > > distributed
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> to handover metadata management from the classic
> > > partition
> > > > > > > leader
> > > > > > > > > to
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch coordinator.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> > > Please
> > > > > > let
> > > > > > > me
> > > > > > > > > >>>> know if
> > > > > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Best,
> > > > > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > > > > >>>>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Thanks for the update.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure that I fully understand the new
> > design,
> > > but
> > > > > it
> > > > > > > > seems
> > > > > > > > > >>>> less
> > > > > > > > > >>>>>>>> clean than before.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it
> essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful
> to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if
> we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > > partitions. Are
> > > > > > the
> > > > > > > > > >>>> metadata
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated
> on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the
> source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > > logic. I
> > > > > > > > > >>>> am
> > > > > > > > > >>>>>>> also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more
> details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Jun
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to
> > > have a
> > > > > > > > separate
> > > > > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role
> > > would
> > > > > be
> > > > > > to
> > > > > > > > > >>>>>> centrally
> > > > > > > > > >>>>>>>>> manage the batch and WAL file metadata for
> diskless
> > > > > topics.
> > > > > > > > This
> > > > > > > > > >>>>>>> component
> > > > > > > > > >>>>>>>>> drew many reasonable comments from the community
> > > about
> > > > > how
> > > > > > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> support
> > > > > > > > > >>>>>>>>> various Kafka features (transactions, queues) and
> > its
> > > > > > > > > >>>> scalability.
> > > > > > > > > >>>>>>> While we
> > > > > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > > > > concerns,
> > > > > > > we
> > > > > > > > > >>>> took a
> > > > > > > > > >>>>>>> step
> > > > > > > > > >>>>>>>>> back and looked at the problem from a different
> > > > > > perspective.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> We would like to propose an alternative Diskless
> > > design
> > > > > > > > *without
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>>>>> centralized coordinator*. We believe this
> approach
> > > has
> > > > > > > > potential
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>>>>> propose to discuss it as it may be more appealing
> > to
> > > the
> > > > > > > > > >>>> community.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let us explain the idea. Most of the
> complications
> > > with
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless approach come from one necessary
> > > architecture
> > > > > > > change:
> > > > > > > > > >>>>>>> globalizing
> > > > > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > > > > coordinator.
> > > > > > > > > >>>> This
> > > > > > > > > >>>>>>> causes
> > > > > > > > > >>>>>>>>> deviations to the established workflows in
> various
> > > > > features
> > > > > > > > like
> > > > > > > > > >>>>>>> produce
> > > > > > > > > >>>>>>>>> idempotence and transactions, queues, retention,
> > etc.
> > > > > These
> > > > > > > > > >>>>>> deviations
> > > > > > > > > >>>>>>> need
> > > > > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > > > > implemented
> > > > > > > and
> > > > > > > > > >>>>>>> tested. In
> > > > > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > > > > partition
> > > > > > > > > >>>> leaders
> > > > > > > > > >>>>>>> again
> > > > > > > > > >>>>>>>>> responsible for managing their partitions, even
> in
> > > > > diskless
> > > > > > > > > >>>> topics.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata
> > are
> > > > > > blended
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> the one partition log. The crux of the Diskless
> > idea
> > > is
> > > > > to
> > > > > > > > > >>>> decouple
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> and move data to the remote storage, while
> keeping
> > > > > metadata
> > > > > > > > > >>>> somewhere
> > > > > > > > > >>>>>>> else.
> > > > > > > > > >>>>>>>>> Using the central batch coordinator for managing
> > > batch
> > > > > > > metadata
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>> way,
> > > > > > > > > >>>>>>>>> but not the only.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s now think about managing metadata for each
> > user
> > > > > > > partition
> > > > > > > > > >>>>>>>>> independently. Generally partitions are
> independent
> > > and
> > > > > > don’t
> > > > > > > > > >>>> share
> > > > > > > > > >>>>>>>>> anything apart from that their data are mixed in
> > WAL
> > > > > files.
> > > > > > > If
> > > > > > > > we
> > > > > > > > > >>>>>>> figure
> > > > > > > > > >>>>>>>>> out how to commit and later delete WAL files
> > safely,
> > > we
> > > > > > will
> > > > > > > > > >>>> achieve
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of
> the
> > > > > central
> > > > > > > > batch
> > > > > > > > > >>>>>>>>> coordinator. Instead, *each diskless user
> partition
> > > will
> > > > > be
> > > > > > > > > >>>> managed
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> its
> > > > > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> > > classic
> > > > > > > > > >>>> topics, the
> > > > > > > > > >>>>>>>>> leader uses the partition log as the way to
> persist
> > > batch
> > > > > > > > > >>>> metadata,
> > > > > > > > > >>>>>>> i.e.
> > > > > > > > > >>>>>>>>> the regular batch header + the information about
> > how
> > > to
> > > > > > find
> > > > > > > > this
> > > > > > > > > >>>>>>> batch on
> > > > > > > > > >>>>>>>>> remote storage. In contrast to classic topics,
> > batch
> > > data
> > > > > > is
> > > > > > > in
> > > > > > > > > >>>>>> remote
> > > > > > > > > >>>>>>>>> storage.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > > > > >>>>>>>>> • Classic topics:
> > > > > > > > > >>>>>>>>>  • Data and metadata are co-located in the
> > partition
> > > log.
> > > > > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > > > > (metadata)|Batch
> > > > > > > > > >>>> data].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>> • Original Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is
> on
> > > > > remote
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>>>>  • The partition state is global in the batch
> > > > > coordinator.
> > > > > > > > > >>>>>>>>> • New Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on
> > > remote
> > > > > > > storage.
> > > > > > > > > >>>>>>>>>  • Partition log content: [Batch header
> > > (metadata)|Batch
> > > > > > > > > >>>>>> coordinates
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> remote storage].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the
> > reminder
> > > of
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless design:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The new approach could be depicted as the
> > following:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> As you can see, the main difference is that now
> > > instead
> > > > > of
> > > > > > a
> > > > > > > > > >>>> single
> > > > > > > > > >>>>>>> commit
> > > > > > > > > >>>>>>>>> request to the batch coordinator, we send
> multiple
> > > > > parallel
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to all the leaders of each partition involved in
> > the
> > > WAL
> > > > > > > file.
> > > > > > > > > >>>> Each
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> will commit its batches independently, without
> > > > > coordinating
> > > > > > > > with
> > > > > > > > > >>>>>> other
> > > > > > > > > >>>>>>>>> leaders and any other components. Batch data is
> > > addressed
> > > > > > by
> > > > > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> file
> > > > > > > > > >>>>>>>>> name, the byte offset and size, which allows
> > > partitions
> > > > > to
> > > > > > > know
> > > > > > > > > >>>>>> nothing
> > > > > > > > > >>>>>>>>> about other partitions to access their data in
> > > shared WAL
> > > > > > > > files.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The number of partitions involved in a single WAL
> > > file
> > > > > may
> > > > > > be
> > > > > > > > > >>>> quite
> > > > > > > > > >>>>>>> large,
> > > > > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to
> > commit
> > > one
> > > > > > WAL
> > > > > > > > > >>>> file is
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>> impractical. However, there are ways to reduce
> this
> > > > > number:
> > > > > > > > > >>>>>>>>> 1. Partition leaders are located on brokers.
> > > Requests to
> > > > > > > > > >>>> leaders on
> > > > > > > > > >>>>>>> one
> > > > > > > > > >>>>>>>>> broker could be grouped together into a single
> > > physical
> > > > > > > network
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>>>> (resembling the normal Produce request that may
> > carry
> > > > > > batches
> > > > > > > > for
> > > > > > > > > >>>>>> many
> > > > > > > > > >>>>>>>>> partitions inside). This will cap the number of
> > > network
> > > > > > > > requests
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make
> > producers
> > > > > send
> > > > > > > > their
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we
> may
> > > > > achieve
> > > > > > > the
> > > > > > > > > >>>> higher
> > > > > > > > > >>>>>>>>> concentration of logical commit requests in
> > physical
> > > > > > network
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>>>> reducing the number of the latter ones even
> > further,
> > > > > > ideally
> > > > > > > to
> > > > > > > > > >>>> one.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Obviously, out of multiple commit requests some
> may
> > > fail
> > > > > or
> > > > > > > > time
> > > > > > > > > >>>> out
> > > > > > > > > >>>>>>> for a
> > > > > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers
> > will
> > > > > > receive
> > > > > > > > > >>>> totally
> > > > > > > > > >>>>>>> or
> > > > > > > > > >>>>>>>>> partially failed responses to their Produce
> > requests,
> > > > > > similar
> > > > > > > > to
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> they
> > > > > > > > > >>>>>>>>> would have received when appending to a classic
> > topic
> > > > > fails
> > > > > > > or
> > > > > > > > > >>>> times
> > > > > > > > > >>>>>>> out.
> > > > > > > > > >>>>>>>>> If a partition experiences problems, other
> > partitions
> > > > > will
> > > > > > > not
> > > > > > > > be
> > > > > > > > > >>>>>>> affected
> > > > > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > > > > uncommitted
> > > > > > > > data
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> garbage in WAL files. But WAL files are
> short-lived
> > > > > > (batches
> > > > > > > > are
> > > > > > > > > >>>>>>> constantly
> > > > > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> > > storage),
> > > > > > so
> > > > > > > > this
> > > > > > > > > >>>>>>> garbage
> > > > > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For safely deleting WAL files we now need to
> > > centrally
> > > > > > manage
> > > > > > > > > >>>> them,
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>>>> this is the only state and logic that spans
> > multiple
> > > > > > > > partitions.
> > > > > > > > > >>>> On
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> diagram, you can see another commit request
> called
> > > > > “Commit
> > > > > > > file
> > > > > > > > > >>>> (best
> > > > > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This
> > manager
> > > will
> > > > > > be
> > > > > > > > > >>>>>>> responsible
> > > > > > > > > >>>>>>>>> for the following:
> > > > > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and
> > > persisting
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> committed WAL files.
> > > > > > > > > >>>>>>>>> 2. To handle potential failures in file
> information
> > > > > > delivery,
> > > > > > > > it
> > > > > > > > > >>>>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> doing prefix scan on the remote storage
> > periodically
> > > to
> > > > > > find
> > > > > > > > and
> > > > > > > > > >>>>>>> register
> > > > > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > > > > configurable
> > > > > > > and
> > > > > > > > > >>>>>> ideally
> > > > > > > > > >>>>>>>>> should be quite long.
> > > > > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders
> > > (after a
> > > > > > > grace
> > > > > > > > > >>>>>>> period) if
> > > > > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t
> > anymore
> > > > > > > referred
> > > > > > > > > >>>> to by
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> partition.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > > > > features
> > > > > > > such
> > > > > > > > > >>>> as
> > > > > > > > > >>>>>>>>> idempotence, transactions, queues, tiered
> storage,
> > > > > > retention.
> > > > > > > > > >>>> Now we
> > > > > > > > > >>>>>>> don’t
> > > > > > > > > >>>>>>>>> need to abstract away and reuse the code from
> > > partition
> > > > > > > leaders
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use
> > the
> > > > > same
> > > > > > > code
> > > > > > > > > >>>> paths
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from
> > > classic
> > > > > > > topics
> > > > > > > > > >>>> mostly
> > > > > > > > > >>>>>>>>> remain unchanged.
> > > > > > > > > >>>>>>>>> For example, it seems that
> > > > > > > > > >>>>>>>>>
> > > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > > > > and
> > > > > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > > > > transaction
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>> on
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> partition leader side could be used for diskless
> > > topics
> > > > > > with
> > > > > > > > > >>>> little
> > > > > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > > > > idempotent
> > > > > > > > > >>>> produce
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > > > > >>>>>>>>> Another example is share groups support, where
> the
> > > share
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader,
> > > > > > > > > >>>>>>>>> being co-located with the partition leader, would
> > > execute
> > > > > > the
> > > > > > > > > >>>> same
> > > > > > > > > >>>>>>> logic
> > > > > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > > > > >>>>>>>>> 2. It returns to the familiar partition-based
> > scaling
> > > > > > model,
> > > > > > > > > >>>> where
> > > > > > > > > >>>>>>>>> partitions are independent.
> > > > > > > > > >>>>>>>>> 3. It makes the operation and failure patterns
> > > closer to
> > > > > > the
> > > > > > > > > >>>>>> familiar
> > > > > > > > > >>>>>>>>> ones from classic topics.
> > > > > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> > > switching
> > > > > > the
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> modes
> > > > > > > > > >>>>>>>>> between diskless and classic.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The rest of the things remain unchanged compared
> to
> > > the
> > > > > > > > previous
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> design (after all previous discussions). Such
> > things
> > > as
> > > > > > local
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>>>> materialization by replicas, the consume path,
> > tiered
> > > > > > storage
> > > > > > > > > >>>>>>> integration,
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> If the community finds this design more suitable,
> > we
> > > will
> > > > > > > > update
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it.
> > > Please let
> > > > > > us
> > > > > > > > know
> > > > > > > > > >>>>>> what
> > > > > > > > > >>>>>>> you
> > > > > > > > > >>>>>>>>> think.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
> > wrote:
> > > > > > > > > >>>>>>>>>> Hi Justine,
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > > > > transactions
> > > > > > > > > >>>> for in
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> diskless coordinator for as long as the
> > corresponding
> > > > > > offsets
> > > > > > > > are
> > > > > > > > > >>>>>>> there.
> > > > > > > > > >>>>>>>>> With the tiered storage unification Greg
> mentioned
> > > > > earlier,
> > > > > > > > this
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks for the response. I think most of what
> you
> > > said
> > > > > > made
> > > > > > > > > >>>>>> sense,
> > > > > > > > > >>>>>>> but
> > > > > > > > > >>>>>>>>> I
> > > > > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it). The
> > > > > > > > > >>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > allowing
> > > > > > > > > >>>>>>> partition
> > > > > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if
> > > this is
> > > > > > > > > >>>> wrong. We
> > > > > > > > > >>>>>>> think
> > > > > > > > > >>>>>>>>>>> about relying on this with the batch
> coordinator
> > > and
> > > > > > delete
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s finished (as
> > there’s
> > > no
> > > > > > > > > >>>> replication
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> HWM
> > > > > > > > > >>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I didn't quite understand this. In classic
> > topics,
> > > we
> > > > > > have
> > > > > > > > > >>>> maps
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>> transactions which remove state when the
> > > transaction is
> > > > > > > > > >>>> completed
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> an
> > > > > > > > > >>>>>>>>>>> aborted transactions index which is retained
> for
> > > much
> > > > > > > longer.
> > > > > > > > > >>>>>> Once
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> > > longer
> > > > > > > > > >>>> involved
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>>>>> maintaining this partition side state, and it
> is
> > > > > subject
> > > > > > to
> > > > > > > > > >>>>>>> compaction
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>>> Looking back at the outline provided above, I
> > > didn't
> > > > > see
> > > > > > > much
> > > > > > > > > >>>>>>> about the
> > > > > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a
> bit
> > > > > > further.
> > > > > > > I
> > > > > > > > > >>>> saw
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> following in a response:
> > > > > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > > > > segment,
> > > > > > > > > >>>> all the
> > > > > > > > > >>>>>>>>> necessary
> > > > > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> > > including
> > > > > > the
> > > > > > > > > >>>>>>> transaction
> > > > > > > > > >>>>>>>>>>> index will be built to serve FetchRequests
> > exactly
> > > as
> > > > > > they
> > > > > > > > > >>>> are
> > > > > > > > > >>>>>>> today.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain
> > the
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>> Justine
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko
> <
> > > > > > > > > >>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> You’re right that we (probably
> unintentionally)
> > > > > focused
> > > > > > > > > >>>> only on
> > > > > > > > > >>>>>>>>> version 2.
> > > > > > > > > >>>>>>>>>>>> We can either limit the support to version 2
> or
> > > > > consider
> > > > > > > > > >>>> using
> > > > > > > > > >>>>>>> some
> > > > > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are?
> Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction
> is
> > > > > > ongoing,
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> course
> > > > > > > > > >>>>>>>>>>>> the normal idempotence checks. What the
> > partition
> > > > > leader
> > > > > > > > > >>>> in the
> > > > > > > > > >>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>> topics does before appending a batch to the
> > local
> > > log
> > > > > > > > > >>>> (e.g. in
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification
> and
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState).
> In
> > > > > > Diskless,
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>>>>> unfortunately
> > > > > > > > > >>>>>>>>>>>> cannot do these checks before appending the
> data
> > > to
> > > > > the
> > > > > > > WAL
> > > > > > > > > >>>>>>> segment
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these
> > > batches in
> > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not
> super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As
> > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader
> > > > > > > > > >>>>>>>>> tracks
> > > > > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the
> > batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> has
> > > > > > > > > >>>>>>>>> to as
> > > > > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends,
> the
> > > > > > > > > >>>> transaction
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> > > batches
> > > > > > and
> > > > > > > > > >>>> other
> > > > > > > > > >>>>>>>>>>>> information, but for the sake of performance
> > we’d
> > > > > better
> > > > > > > > > >>>> store
> > > > > > > > > >>>>>> it
> > > > > > > > > >>>>>>>>>>>> explicitly.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how
> it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it).
> > > > > > > > > >>>>>> The
> > > > > > > > > >>>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > > > >>>> allowing
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me
> if
> > > this
> > > > > is
> > > > > > > > > >>>> wrong.
> > > > > > > > > >>>>>> We
> > > > > > > > > >>>>>>>>> think
> > > > > > > > > >>>>>>>>>>>> about relying on this with the batch
> coordinator
> > > and
> > > > > > > > > >>>> delete the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as
> > > there’s no
> > > > > > > > > >>>>>> replication
> > > > > > > > > >>>>>>>>> and HWM
> > > > > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> > > transactions!
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are?
> Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is
> > > there
> > > > > > state
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not
> super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how
> it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> > > local
> > > > > > > > > >>>>>> segments,
> > > > > > > > > >>>>>>>>> making
> > > > > > > > > >>>>>>>>>>>> use of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> > > state for
> > > > > > > > > >>>> each
> > > > > > > > > >>>>>>>>> partition in
> > > > > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> > > fetches
> > > > > > > > > >>>> need
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> know
> > > > > > > > > >>>>>>>>>>>> aborted
> > > > > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > > > > >>>> transaction is
> > > > > > > > > >>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > > > > corresponding
> > > > > > > > > >>>> logs
> > > > > > > > > >>>>>>> of the
> > > > > > > > > >>>>>>>>>>>> involved
> > > > > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the
> > > metadata
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>>>> actual
> > > > > > > > > >>>>>>>>>>>> control
> > > > > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > > > > transactional
> > > > > > > > > >>>>>>> batches.
> > > > > > > > > >>>>>>>>> How
> > > > > > > > > >>>>>>>>>>>> does the
> > > > > > > > > >>>>>>>>>>>>>> broker obtain the information about the
> ending
> > > > > control
> > > > > > > > > >>>>>> batch
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response
> > > needs to
> > > > > > be
> > > > > > > > > >>>>>>> built by
> > > > > > > > > >>>>>>>>>>>>>> stitching record batches and generated
> control
> > > > > batches
> > > > > > > > > >>>>>>> together?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share
> > partition
> > > > > leader
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>>>>> consumers
> > > > > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to
> include
> > > this
> > > > > or
> > > > > > > > > >>>> it's
> > > > > > > > > >>>>>>> too
> > > > > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful
> to
> > > > > include
> > > > > > > > > >>>>>> enough
> > > > > > > > > >>>>>>>>> details
> > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>> understand correctness and performance
> impact.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from
> a
> > > given
> > > > > > > > > >>>>>> producer
> > > > > > > > > >>>>>>>>> contain a
> > > > > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a
> > producer
> > > > > sends
> > > > > > > > > >>>>>> every
> > > > > > > > > >>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those
> requests
> > > could
> > > > > > > > > >>>> reach
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> > > rejection
> > > > > > > > > >>>> of the
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> requests.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
> > Yurchenko <
> > > > > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details
> > > about
> > > > > > > > > >>>>>>> transactions
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> queues,
> > > > > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic
> > > topics* is
> > > > > > > > > >>>> based
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> precise
> > > > > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> > > (mostly
> > > > > > > > > >>>>>>> producers,
> > > > > > > > > >>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and
> other
> > > > > > > > > >>>> classes),
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition
> > > leaders
> > > > > with
> > > > > > > > > >>>>>> their
> > > > > > > > > >>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level (some details skipped)
> > workflow
> > > is
> > > > > the
> > > > > > > > > >>>>>>>>> following.
> > > > > > > > > >>>>>>>>>>>> When a
> > > > > > > > > >>>>>>>>>>>>>>> transactional Produce request is received
> by
> > > the
> > > > > > > > > >>>> broker:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> > > checks
> > > > > > > > > >>>> if a
> > > > > > > > > >>>>>>>>> non-empty
> > > > > > > > > >>>>>>>>>>>>>>> transaction is running for this partition.
> > > This is
> > > > > > > > > >>>> done
> > > > > > > > > >>>>>>> using
> > > > > > > > > >>>>>>>>> its
> > > > > > > > > >>>>>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > > > > >>>>>> (ProducerStateManager,
> > > > > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> > > about
> > > > > all
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> partitions
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include
> > them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > > > > transactional
> > > > > > > > > >>>>>>> checks.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the
> > > transactional
> > > > > > > > > >>>> data to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> update some of their state (for example,
> log
> > > the
> > > > > fact
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> > > offset).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or
> aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the
> > transaction
> > > > > > > > > >>>> markers to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders
> > and
> > > lots
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>>>> important
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is
> > > local.
> > > > > The
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>>>>> challenge
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the
> fact
> > > there
> > > > > > > > > >>>> are no
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of
> state
> > > need
> > > > > > > > > >>>> to be
> > > > > > > > > >>>>>>>>> globalized
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing
> this
> > to
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>>>>> idempotent
> > > > > > > > > >>>>>>>>>>>>>> produce.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless
> topics*
> > > would
> > > > > > > > > >>>> look
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>>>>> similar:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks
> if a
> > > > > > > > > >>>> non-empty
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> > > classic
> > > > > > > > > >>>>>> topics,
> > > > > > > > > >>>>>>>>> this is
> > > > > > > > > >>>>>>>>>>>>>> checked
> > > > > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single
> > > RPC.
> > > > > > > > > >>>> Since a
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>> could
> > > > > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> > > epoch,
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> positive
> > > > > > > > > >>>>>>>>>>>> result of
> > > > > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> > > double
> > > > > > > > > >>>>>>> configured
> > > > > > > > > >>>>>>>>>>>> duration
> > > > > > > > > >>>>>>>>>>>>>> of a
> > > > > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > > > > informed
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the
> > transaction
> > > to
> > > > > > > > > >>>> include
> > > > > > > > > >>>>>>> them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the
> > > broker
> > > > > > > > > >>>> side.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional
> data
> > > to the
> > > > > > > > > >>>>>> current
> > > > > > > > > >>>>>>>>> shared
> > > > > > > > > >>>>>>>>>>>> WAL
> > > > > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> > > transaction-related
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> Diskless
> > > > > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the
> batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> like
> > > > > > > > > >>>>>>>>> in the
> > > > > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the
> same
> > > > > errors
> > > > > > > > > >>>> like
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e.
> some
> > > > > batches
> > > > > > > > > >>>>>> could
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> rejected.
> > > > > > > > > >>>>>>>>>>>>>>> This means, there will potentially be
> garbage
> > > in
> > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>>>> segment
> > > > > > > > > >>>>>>>>>>>> file in
> > > > > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is
> > > preferable to
> > > > > > > > > >>>> doing
> > > > > > > > > >>>>>>> more
> > > > > > > > > >>>>>>>>>>>> network
> > > > > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > > > > segments
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> relatively
> > > > > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or
> aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
> > informs
> > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
> > the
> > > > > > > > > >>>>>>> transaction is
> > > > > > > > > >>>>>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > > > > >>>> corresponding
> > > > > > > > > >>>>>>> logs
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only
> > on
> > > the
> > > > > > > > > >>>>>> metadata
> > > > > > > > > >>>>>>>>> level, no
> > > > > > > > > >>>>>>>>>>>>>>> actual control batches are written to any
> > file.
> > > > > They
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> dynamically
> > > > > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations.
> > We
> > > > > could
> > > > > > > > > >>>>>>>>> technically
> > > > > > > > > >>>>>>>>>>>> write
> > > > > > > > > >>>>>>>>>>>>>>> these control batches for real, but this
> > would
> > > mean
> > > > > > > > > >>>> extra
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> latency,
> > > > > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the
> batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> save
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> now.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> > > write
> > > > > the
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> markers
> > > > > > > > > >>>>>>>>>>>>>>> to their logs and respond to the
> coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce
> > > flow, we
> > > > > > > > > >>>> get:
> > > > > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between
> > brokers
> > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch
> > > > > > > > > >>>>>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>>>>> when a new partition appear in the
> > > transaction. To
> > > > > > > > > >>>>>>> mitigate the
> > > > > > > > > >>>>>>>>>>>> impact of
> > > > > > > > > >>>>>>>>>>>>>>> them:
> > > > > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> > > Produce
> > > > > > > > > >>>>>>> request
> > > > > > > > > >>>>>>>>> will be
> > > > > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized
> > for
> > > > > > > > > >>>> fast
> > > > > > > > > >>>>>>>>> response to
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally
> > will
> > > > > > > > > >>>>>>> communicate
> > > > > > > > > >>>>>>>>> with a
> > > > > > > > > >>>>>>>>>>>>>>> single broker for the duration of the
> > > transaction
> > > > > > > > > >>>> further
> > > > > > > > > >>>>>>>>> reduces the
> > > > > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the
> > transaction
> > > > > > > > > >>>>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also
> be
> > > able
> > > > > to
> > > > > > > > > >>>>>> span
> > > > > > > > > >>>>>>> both
> > > > > > > > > >>>>>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The share group coordination and management
> > is
> > > a
> > > > > > > > > >>>> side job
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>> interfere with the topic itself
> (leadership,
> > > > > > > > > >>>> replicas,
> > > > > > > > > >>>>>>> physical
> > > > > > > > > >>>>>>>>>>>> storage
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > > > > consumers
> > > > > > > > > >>>>>>> (Fetch and
> > > > > > > > > >>>>>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > > > > affected.)
> > > > > > > > > >>>> We
> > > > > > > > > >>>>>>> don't
> > > > > > > > > >>>>>>>>> see any
> > > > > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> > > compatible
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>> share
> > > > > > > > > >>>>>>>>>>>> groups the
> > > > > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the
> > > code
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> we
> > > > > > > > > >>>>>>>>> don't
> > > > > > > > > >>>>>>>>>>>> expect
> > > > > > > > > >>>>>>>>>>>>>> any
> > > > > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading
> > routines
> > > are
> > > > > > > > > >>>> used
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>> used for
> > > > > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog
> ).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this
> > or
> > > it's
> > > > > > > > > >>>> too
> > > > > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and
> design
> > > input
> > > > > > > > > >>>> on
> > > > > > > > > >>>>>>>>> KIP-1150.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> > > with a
> > > > > > > > > >>>> new
> > > > > > > > > >>>>>>>>> design. To
> > > > > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> KIP-405
> > > > > > > > > >>>>>>>>>>>> Tiered
> > > > > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data
> produced
> > > to a
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> topic
> > > > > > > > > >>>>>>>>>>>> to be
> > > > > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements
> for
> > > the
> > > > > > > > > >>>> Batch
> > > > > > > > > >>>>>>>>> Coordinator
> > > > > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose
> > with
> > > > > > > > > >>>> Tiered
> > > > > > > > > >>>>>>> Storage
> > > > > > > > > >>>>>>>>>>>> plugin
> > > > > > > > > >>>>>>>>>>>>>>>> features such as encryption and
> alternative
> > > data
> > > > > > > > > >>>>>> formats.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
> > local
> > > > > > > > > >>>> segments,
> > > > > > > > > >>>>>>>>> making use
> > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory,
> and
> > > > > > > > > >>>> zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered
> > > cache
> > > > > > > > > >>>>>>> elements,
> > > > > > > > > >>>>>>>>> do not
> > > > > > > > > >>>>>>>>>>>> need
> > > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built
> without
> > > > > > > > > >>>> contacting
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified
> > > substantially,
> > > > > by
> > > > > > > > > >>>>>>> removing
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> previous
> > > > > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > > > > >>>> component, and
> > > > > > > > > >>>>>>>>> "object
> > > > > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces
> as
> > > > > > > > > >>>> enabled by
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> Batch
> > > > > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
> > as
> > > the
> > > > > > > > > >>>>>> earlier
> > > > > > > > > >>>>>>>>> design,
> > > > > > > > > >>>>>>>>>>>> while
> > > > > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> ecosystem.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> > > feedback on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> new
> > > > > > > > > >>>>>>>>> design.
> > > > > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> > > support
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>> all
> > > > > > > > > >>>>>>>>> existing
> > > > > > > > > >>>>>>>>>>>>>> client
> > > > > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > > > > >>>> supporting
> > > > > > > > > >>>>>> APIs
> > > > > > > > > >>>>>>>>> like
> > > > > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> > > Siekierski
> > > > > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid>
> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> > > current
> > > > > > > > > >>>>>>> status of
> > > > > > > > > >>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>> KIP?
> > > > > > > > > >>>>>>>>>>>>>> I
> > > > > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> > > past 2
> > > > > > > > > >>>>>>> months,
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>> vote got
> > > > > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have
> > been
> > > > > > > > > >>>>>> answered
> > > > > > > > > >>>>>>>>> since
> > > > > > > > > >>>>>>>>>>>> then.
> > > > > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > > > > >>>> activity
> > > > > > > > > >>>>>>> since
> > > > > > > > > >>>>>>>>> May.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> > > real
> > > > > > > > > >>>>>> choices
> > > > > > > > > >>>>>>>>> that can
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> > > being
> > > > > > > > > >>>> by
> > > > > > > > > >>>>>> far
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> dominant
> > > > > > > > > >>>>>>>>>>>>>> one
> > > > > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all
> > these
> > > are
> > > > > > > > > >>>>>>>>> standalone
> > > > > > > > > >>>>>>>>>>>>>> products.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
> > would
> > > > > > > > > >>>> hate to
> > > > > > > > > >>>>>>> see a
> > > > > > > > > >>>>>>>>>>>> stall. So
> > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
> > Colin
> > > > > > > > > >>>>>> McCabe -
> > > > > > > > > >>>>>>>>> have your
> > > > > > > > > >>>>>>>>>>>>>>> questions
> > > > > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there
> anything I
> > > can
> > > > > > > > > >>>> do
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>>>> help? I'm
> > > > > > > > > >>>>>>>>>>>>>> deeply
> > > > > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and
> have
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> both?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM
> Stanislav
> > > > > > > > > >>>>>> Kozlovski
> > > > > > > > > >>>>>>> <
> > > > > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in
> the
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>>>> (just a
> > > > > > > > > >>>>>>>>>>>> bunch
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > > > > >>>> example, I
> > > > > > > > > >>>>>>> was
> > > > > > > > > >>>>>>>>>>>> searching
> > > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a
> while;
> > As
> > > > > > > > > >>>> far
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>> I
> > > > > > > > > >>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>> tell, it
> > > > > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
> > of
> > > > > > > > > >>>> having
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> root
> > > > > > > > > >>>>>>>>>>>> KIP be
> > > > > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> > > links
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>> sub-KIPs
> > > > > > > > > >>>>>>>>>>>> where
> > > > > > > > > >>>>>>>>>>>>>> were
> > > > > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I
> think
> > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> good to
> > > > > > > > > >>>>>>>>>>>>>> create
> > > > > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat
> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> > > about
> > > > > > > > > >>>>>>>>> introducing a
> > > > > > > > > >>>>>>>>>>>> new
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > > > > >>>> Storage as
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> primary
> > > > > > > > > >>>>>>>>>>>>>>> source of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big
> we
> > > > > > > > > >>>> decided
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> split it
> > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > > > > >>>> should aim
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go
> onto
> > > > > > > > > >>>>>> details
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> how to
> > > > > > > > > >>>>>>>>>>>>>>> implement
> > > > > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used
> when
> > > we
> > > > > > > > > >>>>>>> discussed
> > > > > > > > > >>>>>>>>>>>> KRaft.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
> > really
> > > > > > > > > >>>> hard
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > >>>>>>>>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several
> sub-kips
> > > > > > > > > >>>> (linked
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > > > > >>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> > > DISCUSS
> > > > > > > > > >>>>>>> threads
> > > > > > > > > >>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > > > > >>>> whether
> > > > > > > > > >>>>>>> Apache
> > > > > > > > > >>>>>>>>> Kafka
> > > > > > > > > >>>>>>>>>>>> wants
> > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > > > > >>>> authors
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> this KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director,
> Aiven
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
> >  +491715557497 |
> > > > > > > > > >>>>>>> aiven.io
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa,
> Hannu
> > > > > > > > > >>>>>>> Valtonen,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB
> 209739 B
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Anatolii Popov
> > > > > Senior Software Developer, *Aiven OY*
> > > > > m: +358505126242
> > > > > w: aiven.io  e: anatolii.popov@aiven.io
> > > > > <https://www.facebook.com/aivencloud>
> > > > > <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > > >
> > > >
> > >
> >
>
>
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Jun,

> JR10. "Cluster scalability increases when less data is required to be
> stored on disks and rebalanced between brokers."
> This seems to contradict a bit with what KIP-1163 says: "Start building it
> from some configurable amount, e.g. 100 MiB prior to the current log end or
> earliest available offset, whichever is later.".  I just want to confirm
> that we plan to make data rebalancing much faster in this KIP.

This 100 MiB is at the moment not a very well researched number. Probably is should be configurable per topic and the default should be smaller. Generally indeed we want to make moving Diskless partitions around, adding replicas, etc. much faster and less resource-consuming than classic topics.

> JR13. Since people keep asking about the "simpler" approach, it would be
> useful to include in the rejected section why we chose not to pursue the
> "simpler" approach like the one in KIP-1176. To me, there are two main
> reasons: (1) this KIP provides more benefits (saving all cross AZ network
> costs instead of just the replication part, better scalability and
> durability); (2) it provides an opportunity for a cleaner long-term design
> instead of retrofitting to the existing framework unnaturally.

Thanks, we added [1] this to the rejected alternatives in KIP-1163.

Best,
Ivan

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=350783976#KIP1163:DisklessCore-Coordinator-lessapproaches

On Wed, Jan 21, 2026, at 00:30, Jun Rao via dev wrote:
> Hi, Ivan,
> 
> Thanks for the reply.
> 
> JR10. "Cluster scalability increases when less data is required to be
> stored on disks and rebalanced between brokers."
> This seems to contradict a bit with what KIP-1163 says: "Start building it
> from some configurable amount, e.g. 100 MiB prior to the current log end or
> earliest available offset, whichever is later.".  I just want to confirm
> that we plan to make data rebalancing much faster in this KIP.
> 
> JR11. Sounds good.
> 
> JR13. Since people keep asking about the "simpler" approach, it would be
> useful to include in the rejected section why we chose not to pursue the
> "simpler" approach like the one in KIP-1176. To me, there are two main
> reasons: (1) this KIP provides more benefits (saving all cross AZ network
> costs instead of just the replication part, better scalability and
> durability); (2) it provides an opportunity for a cleaner long-term design
> instead of retrofitting to the existing framework unnaturally.
> 
> Jun
> 
> 
> 
> On Tue, Jan 20, 2026 at 11:22 AM Greg Harris via dev <de...@kafka.apache.org>
> wrote:
> 
> > Hi Robin,
> >
> > Thanks for your design sketch.
> >
> > Based on your description, the GET operations on the leader and followers
> > must complete before offsets are assigned, adding Produce request latency
> > that is not present in the KIP-1163 design.
> >
> > KIP-1163 also assumes that object storage is more reliable than disk
> > storage, which is the case in the hyperscaler clouds. The parts of your
> > design that make it compatible with low-reliability object storage become
> > counterproductive when Diskless is used with a high-reliability object
> > storage, such as re-introducing direct replication and cross-az costs.
> >
> > We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> > in favor of a complete solution. The folks who championed that proposal are
> > now working with us on KIP-1163, and are currently designing Compaction in
> > Tiered Storage to make Diskless a complete solution for compacted topics as
> > well. Similarly, we would appreciate your contributions to the design of
> > KIP-1163, and your vote on the motivational KIP-1150.
> >
> > Thanks!
> >
> > On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi Jun,
> > >
> > > Here are our answers to your questions and comments:
> > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > >
> > > Thank you, good point, added this.
> > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > >
> > > Yes, there is a per-request cost for object storage which is not directly
> > > comparable to inter-zone transfer costs. We have found that the design in
> > > KIP-1163 amortizes those request costs such that they are insignificant
> > > compared to the $0.02/gb cost which is called out in KIP-1150.
> > > We currently have very precise cost models that were put together for an
> > > earlier iteration of the Diskless topics. These are somewhat stale and
> > > inaccurate for the latest design, so we will work in the background to
> > post
> > > concrete numbers in KIP-1163 before voting on that proposal. Would this
> > be
> > > OK?
> > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > >
> > > Thanks for calling out. We have updated KIP-1164 with the latest design.
> > > It's missing public interfaces, but we think it may be a bit too early to
> > > define them as some rounds of discussion is ahead, with potential to
> > change
> > > the design. But otherwise the KIP is good for the first review.
> > > We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
> > >
> > > Best,
> > > Ivan
> > >
> > > On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> > > > Hi, Anatolii,
> > > >
> > > > Thanks for the updated KIP. Looks good to me overall. A few more
> > > comments.
> > > >
> > > > JR10. "Even in the case where network traffic is not accounted; we
> > > believe
> > > > that the operational benefits of Diskless topics are still appealing to
> > > > Kafka users."
> > > > Could you list other benefits such as better scalability and
> > durability?
> > > >
> > > > JR11. "AWS charges 0.02$ per GiB"
> > > > The object store is not free and has a different cost model based on
> > > > requests. It would be useful summarize the tradeoff on cost and
> > latency,
> > > > and when diskless topics are cost effective.
> > > >
> > > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > > been
> > > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > > included in KIP-1163.
> > > >
> > > > Jun
> > > >
> > > > On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> > > dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thank you for the extensive feedback.
> > > > >
> > > > > We have substantially updated the KIP to address the points raised.
> > > Given
> > > > > the scope of these changes, we have started a new VOTE thread to
> > > restart
> > > > > the voting process cleanly.
> > > > >
> > > > > You can find the new thread here:
> > > > > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> > > > >
> > > > > Best,
> > > > > Anatolii
> > > > >
> > > > > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > > > > dev@kafka.apache.org> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We (the team at Slack) have been following the recent discussion
> > > > > regarding
> > > > > > the scope and timeline of KIP-1150. We agree with the community
> > > sentiment
> > > > > > that Diskless Topics represents the right long-term architecture
> > for
> > > > > Kafka
> > > > > > in the cloud, but we also recognize the valid concerns raised
> > > regarding
> > > > > the
> > > > > > engineering resources required to deliver such an ambitious change.
> > > > > >
> > > > > > To help address these concerns and accelerate the timeline, we are
> > > happy
> > > > > to
> > > > > > announce that we are partnering with the KIP-1150 authors to
> > > co-develop
> > > > > > this feature.
> > > > > >
> > > > > > We previously proposed KIP-1176: Tiered Storage for Active Log
> > > Segments
> > > > > to
> > > > > > solve similar problems. However, rather than fragmenting the
> > > community's
> > > > > > efforts across competing designs, we have decided to withdraw
> > > KIP-1176
> > > > > and
> > > > > > consolidate our engineering resources behind KIP-1150.
> > > > > >
> > > > > > To start, we plan to take ownership of Compaction for Tiered
> > Storage.
> > > > > This
> > > > > > has long been a missing feature in KIP-405, and it becomes critical
> > > in a
> > > > > > Diskless architecture where long-lived data is obligated to be
> > > tiered. By
> > > > > > driving this related prerequisite feature, we hope to allow for
> > > faster
> > > > > > delivery of KIP-1150.
> > > > > >
> > > > > > We are excited to collaborate on this to ensure a robust and timely
> > > > > > delivery for the community.
> > > > > >
> > > > > > Best,
> > > > > > Tom & Henry
> > > > > >
> > > > > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Thanks for sharing the meeting notes.
> > > > > > > I agree we should keep polishing the contents of 1150 & high
> > level
> > > > > design
> > > > > > > in 1163 to prepare for a vote.
> > > > > > >
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > There was a video call between myself, Ivan Yurchenko, Jun Rao,
> > > and
> > > > > > > Andrew
> > > > > > > > Schofield pertaining to KIP-1150. Here are the notes from that
> > > > > meeting:
> > > > > > > >
> > > > > > > > Ivan: What is the future state of Kafka in this area, in 5
> > years?
> > > > > > > > Jun: Do we want something more cloud native? Yes, started with
> > > Tiered
> > > > > > > > Storage. If there’s a better way, we should explore it. In the
> > > long
> > > > > > term
> > > > > > > > this will be useful
> > > > > > > > Because Kafka is used so widely, we need to make sure
> > everything
> > > we
> > > > > add
> > > > > > > is
> > > > > > > > for the long term and for everyone, not just for a single
> > > company.
> > > > > > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > > > > > something
> > > > > > > > that’s high quality/lasts/maintainable, and can work with all
> > > > > existing
> > > > > > > > capabilities.
> > > > > > > > If both 1150 and 1176 proceed at the same time, it’s confusing.
> > > They
> > > > > > > > overlap, but Diskless is more ambitious.
> > > > > > > > If both KIPs are being seriously worked on, then we don’t
> > really
> > > need
> > > > > > > both,
> > > > > > > > because Diskless clearly is better. Having multiple will
> > confuse
> > > > > > people.
> > > > > > > It
> > > > > > > > will duplicate some of the effort.
> > > > > > > > If we want diskless ultimately, what is the short term
> > strategy,
> > > to
> > > > > get
> > > > > > > > some early wins first?
> > > > > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > > > > Andrew: Eventually the architecture will change substantially,
> > > it may
> > > > > > not
> > > > > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > > > > Greg: We all agree on having a high quality feature merged
> > > upstream,
> > > > > > and
> > > > > > > > supporting all APIs
> > > > > > > > Jun: We should try and keep things simple, but there is some
> > > minimum
> > > > > > > > complexity needed.
> > > > > > > > When doing the short term changes (1176), it doesn’t really
> > > progress
> > > > > in
> > > > > > > > changing to a more modern architecture.
> > > > > > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > > > > > Jun: The danger of only applying changes to some part of the
> > > API, you
> > > > > > set
> > > > > > > > the precedence that you only have to implement part of the API.
> > > > > > > Supporting
> > > > > > > > the full API set should be a minimum requirement.
> > > > > > > > Andrew: When we started Kraft, how much did we know the design?
> > > > > > > > Jun: For Kraft we didn’t really know much about the migration,
> > > but
> > > > > the
> > > > > > > > high-level was clear.
> > > > > > > > Greg: Is 1150 votable in its current state?
> > > > > > > > Jun: 1150 should promise to support all APIs. It doesn’t have
> > to
> > > have
> > > > > > all
> > > > > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > > > > We do need some high-level design enough to give confidence
> > that
> > > the
> > > > > > > > promise is able to be fulfilled.
> > > > > > > > Greg: Is the draft version in 1163 enough detail or is more
> > > needed?
> > > > > > > > Jun: We need to agree on the core design, such as leaderless
> > > etc. And
> > > > > > how
> > > > > > > > the APIs will be supported.
> > > > > > > > Greg: Okay we can include these things, and provide a sketch of
> > > how
> > > > > the
> > > > > > > > other leader-based features operate.
> > > > > > > > Jun: Yeah if at a high level the sketch appears to work, we can
> > > > > approve
> > > > > > > > that functionality.
> > > > > > > > Are you committed to doing the more involved and big project?
> > > > > > > > Greg: Yes, we’re committed to the 1163 design and can’t really
> > > accept
> > > > > > > 1176.
> > > > > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > > > > Greg: We’ll push internally for resources, and use the
> > community
> > > > > > > sentiment
> > > > > > > > to motivate Aiven.
> > > > > > > > How far into the future should we look? What sort of scale?
> > > > > > > > Jun: As long as there’s a path forward, and we’re not closing
> > off
> > > > > > future
> > > > > > > > improvements, we can figure out how to handle a larger scale
> > > when it
> > > > > > > > arises.
> > > > > > > > Greg: Random replica placement is very harmful, can we
> > recommend
> > > > > users
> > > > > > to
> > > > > > > > use an external tool like CruiseControl?
> > > > > > > > Jun: Not everyone uses CruiseControl, we would probably need
> > some
> > > > > > > solution
> > > > > > > > for this out of the box
> > > > > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > > > > Jun: Out-of-box experience should be good, good to allow other
> > > > > > > > implementations
> > > > > > > > Greg: But it could hurt Kafka feature/upgrade velocity when we
> > > wait
> > > > > for
> > > > > > > > plugin providers to implement it
> > > > > > > > Ivan: We imagined that maybe cloud hyperscalers could implement
> > > it
> > > > > with
> > > > > > > > e.g. dynamodb
> > > > > > > > Greg: Could we bake more details of the different providers
> > into
> > > > > Kafka,
> > > > > > > or
> > > > > > > > does it still make sense for it to be pluggable?
> > > > > > > > Jun: Make it whatever is easiest to roll out and add new
> > clients
> > > > > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > > > > Ivan: The vote is already open, we’re not too pressed for time.
> > > We’ll
> > > > > > go
> > > > > > > > improve the 1163 design and communication.
> > > > > > > > Is 1176 a competing design? Someone will ask.
> > > > > > > > Jun: If we are seriously working on something more ambitious,
> > > yeah we
> > > > > > > > shouldn’t do the stop-gap solution.
> > > > > > > > It’s diverting review resources. If we can get the short term
> > > thing
> > > > > in
> > > > > > > 1yr
> > > > > > > > but Diskless solution is 2y it makes sense to go for Diskless.
> > If
> > > > > it’s
> > > > > > > 5yr,
> > > > > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case.
> > Should
> > > we
> > > > > > > > explicitly exclude 1176?
> > > > > > > > Andrew: Put your arms around the feature set you actually want,
> > > and
> > > > > use
> > > > > > > > that to rule out 1176.
> > > > > > > > Probably don’t need -1 votes, most likely KIPs just don’t
> > receive
> > > > > > votes.
> > > > > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > > > > Jun: Satish posted meeting notes regularly, we should do the
> > > same.
> > > > > > > >
> > > > > > > > To summarize, we will be polishing the contents of 1150 & high
> > > level
> > > > > > > design
> > > > > > > > in 1163 to prepare for a vote.
> > > > > > > > We believe that the community should select the feature set of
> > > 1150
> > > > > to
> > > > > > > > fully eliminate producer cross-zone costs, and make the
> > > investment
> > > > > in a
> > > > > > > > high quality Diskless Topics implementation rather than in
> > > stop-gap
> > > > > > > > solutions.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
> > wrote:
> > > > > > > >
> > > > > > > > > This may be a tangent, but we needed to offload storage off
> > of
> > > > > Kafka
> > > > > > > into
> > > > > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> > > mostly
> > > > > > > > > ephemeral broker that can come and go as it pleases. Be that
> > > > > scaling
> > > > > > or
> > > > > > > > > outage. Disks can be destroyed and recreated at will, we
> > still
> > > > > retain
> > > > > > > > data
> > > > > > > > > and use broker for just that, brokering messages. Not only
> > > that, we
> > > > > > > > reduced
> > > > > > > > > the requirement on the actual Kafka resources by reducing the
> > > size
> > > > > > of a
> > > > > > > > > payload via a claim check pattern. Maybe this is an
> > > anti–pattern,
> > > > > but
> > > > > > > it
> > > > > > > > is
> > > > > > > > > super fast and highly cost efficient. We reworked
> > > ProducerRequest
> > > > > to
> > > > > > > > allow
> > > > > > > > > plugins. We added a custom http plugin that submits every
> > > request
> > > > > > via a
> > > > > > > > > persisted connection to a microservice. Microservice stores
> > the
> > > > > > payload
> > > > > > > > and
> > > > > > > > > returns a tiny json metadata object,a claim check, that can
> > be
> > > used
> > > > > > to
> > > > > > > > find
> > > > > > > > > the actual data. Think of it as zipping the payload. This
> > claim
> > > > > check
> > > > > > > > > metadata traverses the pipelines with consumers using the
> > urls
> > > in
> > > > > > > > metadata
> > > > > > > > > to pull what they need. Think unzipping. This allowed us to
> > > also
> > > > > pull
> > > > > > > > ONLY
> > > > > > > > > the data that we need in graphql like manner. So if you have
> > a
> > > 100K
> > > > > > > json
> > > > > > > > > payload and you need only a subsection, you can pull that by
> > > > > > jmespath.
> > > > > > > > When
> > > > > > > > > you have multiple consumer groups yanking down huge payloads
> > > it is
> > > > > > > > > cumbersome on the broker. When you have the same consumer
> > > groups
> > > > > > > yanking
> > > > > > > > > down a claim check, and then going out of band directly to
> > the
> > > > > source
> > > > > > > of
> > > > > > > > > truth, the broker has some breathing room. Obviously our
> > > > > microservice
> > > > > > > > does
> > > > > > > > > not go directly to the cloud storage, as that would be too
> > > slow. It
> > > > > > > > stores
> > > > > > > > > the payload in high speed memory cache and returns asap. That
> > > > > memory
> > > > > > is
> > > > > > > > > eventually persisted into S3. The retrieval goest against the
> > > cache
> > > > > > > > first,
> > > > > > > > > then against the S3. Overall a rather cheappy and zippy
> > > solution. I
> > > > > > > tried
> > > > > > > > > proposing the KIP for this, but there was no excitement.
> > Check
> > > this
> > > > > > > out:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi, Andrew,
> > > > > > > > > >
> > > > > > > > > > If we want to focus only on reducing cross-zone replication
> > > > > costs,
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > an alternative design in the KIP-1176 discussion thread
> > that
> > > > > seems
> > > > > > > > > simpler
> > > > > > > > > > than the proposal here. I am copying the outline of that
> > > approach
> > > > > > > > below.
> > > > > > > > > >
> > > > > > > > > > 1. A new leader is elected.
> > > > > > > > > > 2. Leader maintains a first tiered offset, which is
> > > initialized
> > > > > to
> > > > > > > log
> > > > > > > > > end
> > > > > > > > > > offset.
> > > > > > > > > > 3. Leader writes produced data from the client to local
> > log.
> > > > > > > > > > 4. Leader uploads produced data from all local logs as a
> > > combined
> > > > > > > > object
> > > > > > > > > > 5. Leader stores the metadata for the combined object in
> > > memory.
> > > > > > > > > > 6. If a follower fetch request has an offset >= first
> > tiered
> > > > > > offset,
> > > > > > > > the
> > > > > > > > > > metadata for the corresponding combined object is returned.
> > > > > > > Otherwise,
> > > > > > > > > the
> > > > > > > > > > local data is returned.
> > > > > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > > > > >
> > > > > > > > > > It's still a bit unnatural, but it could work.
> > > > > > > > > >
> > > > > > > > > > Hi, Ivan,
> > > > > > > > > >
> > > > > > > > > > Are you still committed to proceeding with the original
> > > design of
> > > > > > > > > KIP-1150?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >> I’ve been following KIP-1150 and friends for a while. I’m
> > > going
> > > > > to
> > > > > > > > jump
> > > > > > > > > >> into the discussions too.
> > > > > > > > > >>
> > > > > > > > > >> Looking back at Jack Vanlightly’s message, I am not quite
> > so
> > > > > > > convinced
> > > > > > > > > >> that it’s a kind of fork in the road. The primary aim of
> > the
> > > > > > effort
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > > > > > prohibitively
> > > > > > > > > >> expensive to use on cloud storage. I think it would be
> > > entirely
> > > > > > > viable
> > > > > > > > > to
> > > > > > > > > >> prioritise code reuse for an initial implementation of
> > > diskless
> > > > > > > > topics,
> > > > > > > > > and
> > > > > > > > > >> we could still have a more cloud-native design in the
> > > future.
> > > > > It’s
> > > > > > > > hard
> > > > > > > > > to
> > > > > > > > > >> predict what the community will prioritise in the future.
> > > > > > > > > >>
> > > > > > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> > > > > support
> > > > > > > > > >> leaderless produce requests, first writing WAL segments
> > into
> > > > > > object
> > > > > > > > > >> storage, and then using the regular partition leaders to
> > > > > sequence
> > > > > > > the
> > > > > > > > > >> records. The active log segment for a diskless topic will
> > > > > > initially
> > > > > > > > > contain
> > > > > > > > > >> batch coordinates rather than record batches. The batch
> > > > > > coordinates
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > > >> resolved from WAL segments for consumers, and also in
> > order
> > > to
> > > > > > > prepare
> > > > > > > > > log
> > > > > > > > > >> segments for uploading to tiered storage. Jun is probably
> > > > > correct
> > > > > > > that
> > > > > > > > > we
> > > > > > > > > >> need a more frequent object merging process than tiered
> > > storage
> > > > > > > > > provides.
> > > > > > > > > >> This is just the transition from write-optimised WAL
> > > segments to
> > > > > > > > > >> read-optimised tiered segments, and all of the object
> > > > > > storage-based
> > > > > > > > > >> implementations of Kafka that I’m aware of do this
> > > > > rearrangement.
> > > > > > > But
> > > > > > > > > >> perhaps this more frequent object merging is a pre-GA
> > > > > improvement,
> > > > > > > > > rather
> > > > > > > > > >> than a strict requirement for an initial implementation
> > for
> > > > > early
> > > > > > > > access
> > > > > > > > > >> use.
> > > > > > > > > >>
> > > > > > > > > >> For zone-aligned share consumers, the share group assignor
> > > is
> > > > > > > intended
> > > > > > > > > to
> > > > > > > > > >> be rack-aware. Consumers should be assigned to partitions
> > > with
> > > > > > > leaders
> > > > > > > > > in
> > > > > > > > > >> their zone. The simple assignor is not rack-aware, but it
> > > easily
> > > > > > > could
> > > > > > > > > be
> > > > > > > > > >> or we could have a rack-aware assignor.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Andrew
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao
> > <jun@confluent.io.INVALID
> > > >
> > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> Hi, Ivan,
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > >>>
> > > > > > > > > >>> "As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>> will indeed consume some IOPS. See them as a cache that
> > > could
> > > > > > > always
> > > > > > > > be
> > > > > > > > > >>> restored from the remote storage. While it’s not ideal,
> > > it's
> > > > > > still
> > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > of
> > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>> can avoid explicit flushing on local materialized
> > segments
> > > at
> > > > > all
> > > > > > > and
> > > > > > > > > let
> > > > > > > > > >>> the file system and page cache figure out when to flush
> > > > > > optimally.
> > > > > > > > This
> > > > > > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > > dramatically,
> > > > > > > > > >>> depending on throughput for each partition. We, of
> > course,
> > > > > > continue
> > > > > > > > > >>> flushing the metadata segments as before."
> > > > > > > > > >>>
> > > > > > > > > >>> If we have a mix of classic and diskless topics on the
> > same
> > > > > > broker,
> > > > > > > > > it's
> > > > > > > > > >>> important that the classic topics' data is flushed to
> > disk
> > > as
> > > > > > > quickly
> > > > > > > > > as
> > > > > > > > > >>> possible. To achieve this, users typically set
> > > > > > > dirty_expire_centisecs
> > > > > > > > > in
> > > > > > > > > >>> the kernel based on the number of available disk IOPS.
> > > Once you
> > > > > > set
> > > > > > > > > this
> > > > > > > > > >>> number, it applies to all dirty files, including the
> > cached
> > > > > data
> > > > > > in
> > > > > > > > > >>> diskless topics. So, if there are more files actively
> > > > > > accumulating
> > > > > > > > > data,
> > > > > > > > > >>> the flush frequency and therefore RPO is reduced for
> > > classic
> > > > > > > topics.
> > > > > > > > > >>>
> > > > > > > > > >>> "We should have mentioned this explicitly, but this step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>> in the form of segments offloading to tiered storage.
> > When
> > > we
> > > > > > > > assemble
> > > > > > > > > a
> > > > > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> > > effectively
> > > > > > > doing
> > > > > > > > > >>> metadata compaction: replacing a big number of pieces of
> > > > > metadata
> > > > > > > > about
> > > > > > > > > >>> individual batches with a single record in
> > > > > > __remote_log_metadata."
> > > > > > > > > >>>
> > > > > > > > > >>> The object merging in tier storage typically only kicks
> > in
> > > > > after
> > > > > > a
> > > > > > > > few
> > > > > > > > > >>> hours. The impact is (1) the amount of accumulated
> > > metadata is
> > > > > > > still
> > > > > > > > > >> quite
> > > > > > > > > >>> large; (2) there are many small objects, leading to poor
> > > read
> > > > > > > > > >> performance.
> > > > > > > > > >>> I think we need a more frequent object merging process
> > than
> > > > > tier
> > > > > > > > > storage
> > > > > > > > > >>> provides.
> > > > > > > > > >>>
> > > > > > > > > >>> Jun
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me>
> > > > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thank you for your messages.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> First, it degrades the durability.
> > > > > > > > > >>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at a
> > > > > > > > > >>>>> given point of time, one for the data and another for
> > the
> > > > > > > metadata.
> > > > > > > > > >>>>> Flushing each file requires a separate IO. If the disk
> > > has 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >> we
> > > > > > > > > >>>>> have 5K partitions in a broker, currently we can afford
> > > to
> > > > > > flush
> > > > > > > > each
> > > > > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> number of files per partition, we can only flush each
> > > > > partition
> > > > > > > > every
> > > > > > > > > >> 10
> > > > > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As I understand, you’re speaking about locally
> > > materialized
> > > > > > > > segments.
> > > > > > > > > >> They
> > > > > > > > > >>>> will indeed consume some IOPS. See them as a cache that
> > > could
> > > > > > > always
> > > > > > > > > be
> > > > > > > > > >>>> restored from the remote storage. While it’s not ideal,
> > > it's
> > > > > > still
> > > > > > > > OK
> > > > > > > > > to
> > > > > > > > > >>>> lose data in them due to a machine crash, for example.
> > > Because
> > > > > > of
> > > > > > > > > this,
> > > > > > > > > >> we
> > > > > > > > > >>>> can avoid explicit flushing on local materialized
> > > segments at
> > > > > > all
> > > > > > > > and
> > > > > > > > > >> let
> > > > > > > > > >>>> the file system and page cache figure out when to flush
> > > > > > optimally.
> > > > > > > > > This
> > > > > > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > > dramatically,
> > > > > > > > > >>>> depending on throughput for each partition. We, of
> > course,
> > > > > > > continue
> > > > > > > > > >>>> flushing the metadata segments as before.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s worth making a note on caching. I think nobody will
> > > > > > disagree
> > > > > > > > that
> > > > > > > > > >>>> doing direct reads from remote storage every time a
> > batch
> > > is
> > > > > > > > requested
> > > > > > > > > >> by a
> > > > > > > > > >>>> consumer will not be practical neither from the
> > > performance
> > > > > nor
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > >>>> economy point of view. We need a way to keep the number
> > > of GET
> > > > > > > > > requests
> > > > > > > > > >>>> down. There are multiple options, for example:
> > > > > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > > > > >>>> 2. Local in-memory caching. Comes with less network
> > > chattiness
> > > > > > and
> > > > > > > > > >> works
> > > > > > > > > >>>> well if we have more or less stable brokers to consume
> > > from.
> > > > > > > > > >>>> 3. Materialization of diskless logs on local disk. Way
> > > lower
> > > > > > > impact
> > > > > > > > on
> > > > > > > > > >>>> RAM and also requires stable brokers for consumption
> > > (using
> > > > > just
> > > > > > > > > >> assigned
> > > > > > > > > >>>> replicas will probably work well).
> > > > > > > > > >>>>
> > > > > > > > > >>>> Materialization is one of possible options, but we can
> > > choose
> > > > > > > > another
> > > > > > > > > >> one.
> > > > > > > > > >>>> However, we will have this dilemma regardless of whether
> > > we
> > > > > have
> > > > > > > an
> > > > > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Second, if we ever need this
> > > > > > > > > >>>>> metadata somewhere else, say in the WAL file manager,
> > the
> > > > > > > consumer
> > > > > > > > > >> needs
> > > > > > > > > >>>> to
> > > > > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>> actual benefit of this approach is also questionable.
> > On
> > > the
> > > > > > > > surface,
> > > > > > > > > >> it
> > > > > > > > > >>>>> might seem that we could reduce the number of lines
> > that
> > > need
> > > > > > to
> > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>> for this KIP. However, the changes are quite intrusive
> > > to the
> > > > > > > > classic
> > > > > > > > > >>>>> partition's code path and will probably make the code
> > > base
> > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>> maintain in the long run. I like the original approach
> > > based
> > > > > on
> > > > > > > the
> > > > > > > > > >> batch
> > > > > > > > > >>>>> coordinator much better than this one. We could
> > probably
> > > > > > refactor
> > > > > > > > the
> > > > > > > > > >>>>> producer state code so that it could be reused in the
> > > batch
> > > > > > > > > >> coordinator.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It’s hard to disagree with this. The explicit
> > coordinator
> > > is
> > > > > > more
> > > > > > > a
> > > > > > > > > side
> > > > > > > > > >>>> thing, while coordinator-less approach is more about
> > > extending
> > > > > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Thanks for addressing the concerns on the number of
> > RPCs
> > > in
> > > > > the
> > > > > > > > > produce
> > > > > > > > > >>>>> path. I agree that with the metadata crafting
> > mechanism,
> > > we
> > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > be
> > > > > > > > > >>>>> collocated with the data on the same set of brokers,
> > it's
> > > > > weird
> > > > > > > > that
> > > > > > > > > >> they
> > > > > > > > > >>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > uses
> > > > > > > > > >> the
> > > > > > > > > >>>>> metadata crafting mechanism, but the metadata is stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>> partition using its own assignment strategy. It will be
> > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>> them collocated.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I would like to note that the metadata crafting is
> > needed
> > > only
> > > > > > to
> > > > > > > > tell
> > > > > > > > > >>>> producers which brokers they should send Produce
> > requests
> > > to,
> > > > > > but
> > > > > > > > data
> > > > > > > > > >> (as
> > > > > > > > > >>>> in “locally materialized log”) is located on partition
> > > > > replicas,
> > > > > > > > i.e.
> > > > > > > > > >>>> automatically co-located with metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> As a side note, it would probably be better that instead
> > > of
> > > > > > > > implicitly
> > > > > > > > > >>>> crafting partition metadata, we extend the metadata
> > > protocol
> > > > > so
> > > > > > > that
> > > > > > > > > for
> > > > > > > > > >>>> diskless partitions we return not only the leader and
> > > > > replicas,
> > > > > > > but
> > > > > > > > > also
> > > > > > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > > > > > performance
> > > > > > > > > and
> > > > > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>> step.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We should have mentioned this explicitly, but this step,
> > > in
> > > > > > fact,
> > > > > > > > > >> remains
> > > > > > > > > >>>> in the form of segments offloading to tiered storage.
> > > When we
> > > > > > > > > assemble a
> > > > > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > > > > effectively
> > > > > > > > doing
> > > > > > > > > >>>> metadata compaction: replacing a big number of pieces of
> > > > > > metadata
> > > > > > > > > about
> > > > > > > > > >>>> individual batches with a single record in
> > > > > > __remote_log_metadata.
> > > > > > > > > >>>>
> > > > > > > > > >>>> We could create a Diskless-specific merging mechanism
> > > instead
> > > > > if
> > > > > > > > > needed.
> > > > > > > > > >>>> It’s rather easy with the explicit coordinator approach.
> > > With
> > > > > > the
> > > > > > > > > >>>> coordinator-less approach, this would probably be a bit
> > > more
> > > > > > > tricky
> > > > > > > > > >>>> (rewriting the tail of the log by the leader +
> > replicating
> > > > > this
> > > > > > > > change
> > > > > > > > > >>>> reliably).
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >> changes
> > > > > > > > > >>>> in
> > > > > > > > > >>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >> last
> > > > > > > > > >>>>> for the long term.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Yes, totally agree.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Luke,
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>
> > > > > > > > > >>>> After some evolution of the original proposal,
> > leaderless
> > > is
> > > > > now
> > > > > > > > > >> limited.
> > > > > > > > > >>>> We only need to be able to accept Produce requests on
> > more
> > > > > than
> > > > > > > one
> > > > > > > > > >> broker
> > > > > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get
> > it
> > > > > right
> > > > > > > > that
> > > > > > > > > >> you
> > > > > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Let’s now look at this problem from a higher level, as
> > > Jack
> > > > > > > > proposed.
> > > > > > > > > As
> > > > > > > > > >>>> it was said, the big choice we need to make is whether
> > we
> > > 1)
> > > > > > > create
> > > > > > > > an
> > > > > > > > > >>>> explicit batch coordinator; or 2) go for the
> > > coordinator-less
> > > > > > > > > approach,
> > > > > > > > > >>>> where each diskless partition is managed by its leader
> > as
> > > in
> > > > > > > classic
> > > > > > > > > >> topics.
> > > > > > > > > >>>>
> > > > > > > > > >>>> If we try to compare the two approaches:
> > > > > > > > > >>>>
> > > > > > > > > >>>> Pluggability:
> > > > > > > > > >>>> - Explicit coordinator: Possible. For example, some
> > > setups may
> > > > > > > > benefit
> > > > > > > > > >>>> from batch metadata being stored in a cloud database
> > > (such as
> > > > > > AWS
> > > > > > > > > >> DynamoDB
> > > > > > > > > >>>> or GCP Spanner).
> > > > > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Scalability and fault tolerance:
> > > > > > > > > >>>> - Explicit coordinator: Depends on the implementation
> > and
> > > it’s
> > > > > > > also
> > > > > > > > > >>>> necessary to actively work for it.
> > > > > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics.
> > > Scaling is
> > > > > > > done
> > > > > > > > by
> > > > > > > > > >>>> partition placement, partitions could fail
> > independently.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Separation of concerns:
> > > > > > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > > > > > independent
> > > > > > > > > >> from
> > > > > > > > > >>>> classic topics in terms of code and workflows. For
> > > example,
> > > > > the
> > > > > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > > > > mechanism
> > > > > > > > could
> > > > > > > > > >> be
> > > > > > > > > >>>> relatively simply implemented with it. As a flip side of
> > > this,
> > > > > > > some
> > > > > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > > > > bringing
> > > > > > > > > diskless
> > > > > > > > > >>>> closer to classic topics. Some code paths and workflows
> > > could
> > > > > be
> > > > > > > > more
> > > > > > > > > >>>> straightforwardly reused, but they will inevitably have
> > > to be
> > > > > > > > adapted
> > > > > > > > > to
> > > > > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Cloud-nativeness. This is a vague concept, also related
> > > to the
> > > > > > > > > previous,
> > > > > > > > > >>>> but let’s try:
> > > > > > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > > > > > separately
> > > > > > > > > makes
> > > > > > > > > >>>> it easier for brokers to take different roles, be purely
> > > > > > stateless
> > > > > > > > if
> > > > > > > > > >>>> needed, etc.
> > > > > > > > > >>>> - Coordinator-less: Less so. Something could be achieved
> > > with
> > > > > > > > creative
> > > > > > > > > >>>> partition placement, but not much.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Both seem to have their pros and cons. However,
> > answering
> > > > > Jack’s
> > > > > > > > > >> question,
> > > > > > > > > >>>> the explicit coordinator approach may indeed lead to a
> > > more
> > > > > > > flexible
> > > > > > > > > >> design.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> The purpose of this deviation in the discussion was to
> > > > > receive a
> > > > > > > > > >>>> preliminary community evaluation of the coordinator-less
> > > > > > approach
> > > > > > > > > >> without
> > > > > > > > > >>>> taking on the task of writing a separate KIP and fitting
> > > it in
> > > > > > the
> > > > > > > > > >> system
> > > > > > > > > >>>> of KIP-1150 and its children. We’re open to stopping it
> > > and
> > > > > > > getting
> > > > > > > > > >> back to
> > > > > > > > > >>>> working out the coordinator design if the community
> > > doesn’t
> > > > > > favor
> > > > > > > > the
> > > > > > > > > >>>> proposed approach.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best,
> > > > > > > > > >>>> Ivan and Diskless team
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > > > > >>>>> Hi Ivan,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> As Jun pointed out, the updated design seems to have
> > some
> > > > > > > > > shortcomings
> > > > > > > > > >>>>> although it simplifies the implementation.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > > queue is
> > > > > > > > because
> > > > > > > > > >> of
> > > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> > we
> > > > > only
> > > > > > > > focus
> > > > > > > > > on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > > to/from
> > > > > > the
> > > > > > > > > >> remote
> > > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thank you,
> > > > > > > > > >>>>> Luke
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for the explanation.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> "we write the reference to the WAL file with the batch
> > > data"
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I understand the approach now, but I think it is a
> > hacky
> > > > > one.
> > > > > > > > There
> > > > > > > > > >> are
> > > > > > > > > >>>>>> multiple short comings with this design. First, it
> > > degrades
> > > > > > the
> > > > > > > > > >>>> durability.
> > > > > > > > > >>>>>> For each partition, now there are two files being
> > > actively
> > > > > > > written
> > > > > > > > > at
> > > > > > > > > >> a
> > > > > > > > > >>>>>> given point of time, one for the data and another for
> > > the
> > > > > > > > metadata.
> > > > > > > > > >>>>>> Flushing each file requires a separate IO. If the disk
> > > has
> > > > > 1K
> > > > > > > IOPS
> > > > > > > > > and
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> have 5K partitions in a broker, currently we can
> > afford
> > > to
> > > > > > flush
> > > > > > > > > each
> > > > > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5
> > > seconds. If
> > > > > > we
> > > > > > > > > double
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> number of files per partition, we can only flush each
> > > > > > partition
> > > > > > > > > every
> > > > > > > > > >>>> 10
> > > > > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we
> > > ever
> > > > > need
> > > > > > > > this
> > > > > > > > > >>>>>> metadata somewhere else, say in the WAL file manager,
> > > the
> > > > > > > consumer
> > > > > > > > > >>>> needs to
> > > > > > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > > > > > inefficient.
> > > > > > > > > The
> > > > > > > > > >>>>>> actual benefit of this approach is also questionable.
> > > On the
> > > > > > > > > surface,
> > > > > > > > > >>>> it
> > > > > > > > > >>>>>> might seem that we could reduce the number of lines
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> changed
> > > > > > > > > >>>>>> for this KIP. However, the changes are quite intrusive
> > > to
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition's code path and will probably make the code
> > > base
> > > > > > > harder
> > > > > > > > to
> > > > > > > > > >>>>>> maintain in the long run. I like the original approach
> > > based
> > > > > > on
> > > > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>> coordinator much better than this one. We could
> > probably
> > > > > > > refactor
> > > > > > > > > the
> > > > > > > > > >>>>>> producer state code so that it could be reused in the
> > > batch
> > > > > > > > > >>>> coordinator.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks for addressing the concerns on the number of
> > > RPCs in
> > > > > > the
> > > > > > > > > >> produce
> > > > > > > > > >>>>>> path. I agree that with the metadata crafting
> > > mechanism, we
> > > > > > > could
> > > > > > > > > >>>> mitigate
> > > > > > > > > >>>>>> the PRC problem. However, since we now require the
> > > metadata
> > > > > to
> > > > > > > be
> > > > > > > > > >>>>>> collocated with the data on the same set of brokers,
> > > it's
> > > > > > weird
> > > > > > > > that
> > > > > > > > > >>>> they
> > > > > > > > > >>>>>> are now managed by different mechanisms. The data
> > > assignment
> > > > > > now
> > > > > > > > > uses
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> metadata crafting mechanism, but the metadata is
> > stored
> > > in
> > > > > the
> > > > > > > > > classic
> > > > > > > > > >>>>>> partition using its own assignment strategy. It will
> > be
> > > > > > > > complicated
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> them collocated.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > > > > compaction/merging
> > > > > > > > > >>>>>> step. My first concern is on the amount of metadata
> > that
> > > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > >>>> kept.
> > > > > > > > > >>>>>> Without object compcation, the metadata generated in
> > the
> > > > > > produce
> > > > > > > > > path
> > > > > > > > > >>>> can
> > > > > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's
> > > say for
> > > > > > > every
> > > > > > > > > >>>> 250ms we
> > > > > > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> > > > > remoting
> > > > > > > > > tiering
> > > > > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K
> > > partitions,
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>> keep
> > > > > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> > > metadata,
> > > > > > > quite
> > > > > > > > > >>>>>> signficant. A second concern is on performance. Every
> > > time
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >>>>>> rebuild the caching data, we need to read a bunch of
> > > small
> > > > > > > objects
> > > > > > > > > >>>> from S3,
> > > > > > > > > >>>>>> slowing down the building process. If a consumer
> > > happens to
> > > > > > need
> > > > > > > > > such
> > > > > > > > > >>>> data,
> > > > > > > > > >>>>>> it could slow down the application.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> > > fewest
> > > > > > code
> > > > > > > > > >>>> changes in
> > > > > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> > > design
> > > > > > that
> > > > > > > > can
> > > > > > > > > >>>> last
> > > > > > > > > >>>>>> for the long term.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Jun
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hi Jun,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed
> > to
> > > > > > clearly
> > > > > > > > > >>>> explain
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > partitions.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> No. There will be only one physical partition per
> > > diskless
> > > > > > > > > >>>> partition. Let
> > > > > > > > > >>>>>>> me explain this with an example. Let’s say we have a
> > > > > diskless
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> > > leader. We
> > > > > > > > produce
> > > > > > > > > >>>> some
> > > > > > > > > >>>>>>> batches to this partition. The content of the segment
> > > file
> > > > > > will
> > > > > > > > be
> > > > > > > > > >>>>>>> something like this (for each batch):
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > > > > >>>>>>> Magic: like in classic
> > > > > > > > > >>>>>>> CRC: like in classic
> > > > > > > > > >>>>>>> Attributes: like in classic
> > > > > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > > > > >>>>>>> ProducerId: like in classic
> > > > > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > > > > >>>>>>> Records:
> > > > > > > > > >>>>>>>
> > path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > > > > byte
> > > > > > > > offset
> > > > > > > > > >>>>>>> 123456
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It looks very much like classic log entries. The only
> > > > > > > difference
> > > > > > > > is
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> instead of writing real Records, we write the
> > > reference to
> > > > > > the
> > > > > > > > WAL
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> with the batch data (I guess we need only the name
> > and
> > > the
> > > > > > byte
> > > > > > > > > >>>> offset,
> > > > > > > > > >>>>>>> because the byte length is the standard field above).
> > > > > > > Otherwise,
> > > > > > > > > >>>> it’s a
> > > > > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> So we have as many partitions for diskless as for
> > > classic.
> > > > > As
> > > > > > > of
> > > > > > > > > open
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Are the metadata and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The source of truth for the data is still in WAL
> > files
> > > on
> > > > > > > object
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>> The source of truth for the metadata is in segment
> > > files on
> > > > > > the
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > > > > > independent
> > > > > > > > > of
> > > > > > > > > >>>> this
> > > > > > > > > >>>>>>> new proposal, but I want to present them to give the
> > > idea
> > > > > > that
> > > > > > > > > only a
> > > > > > > > > >>>>>>> limited amount of data files will be operated
> > locally:
> > > > > > > > > >>>>>>> - We want to assemble batches into segment files and
> > > > > offload
> > > > > > > them
> > > > > > > > > to
> > > > > > > > > >>>>>>> tiered storage in order to prevent the unbounded
> > > growth of
> > > > > > > batch
> > > > > > > > > >>>>>> metadata.
> > > > > > > > > >>>>>>> For this, we need to open only  a few file
> > descriptors
> > > (for
> > > > > > the
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>> file itself + the necessary indexes) before the
> > > segment is
> > > > > > > fully
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > > > > >>>>>>> - We want to assemble local segment files for caching
> > > > > > purposes
> > > > > > > as
> > > > > > > > > >>>> well,
> > > > > > > > > >>>>>>> i.e. to speed up fetching. This will not materialize
> > > the
> > > > > full
> > > > > > > > > >>>> content of
> > > > > > > > > >>>>>>> the log, but only the hot set according to some
> > policy
> > > (or
> > > > > > > > > >>>> configurable
> > > > > > > > > >>>>>>> policies), i.e. the number of segments and file
> > > descriptors
> > > > > > > will
> > > > > > > > > >>>> also be
> > > > > > > > > >>>>>>> limited.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This is a valid concern that we considered, but this
> > > issue
> > > > > > can
> > > > > > > be
> > > > > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > > > > >>>>>>> The situation with a single broker is trivial: all
> > the
> > > > > commit
> > > > > > > > > >>>> requests go
> > > > > > > > > >>>>>>> from the broker to itself.
> > > > > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but
> > > located in
> > > > > > the
> > > > > > > > > single
> > > > > > > > > >>>>>> rack
> > > > > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for
> > > diskless
> > > > > > > > > >>>> partitions, but
> > > > > > > > > >>>>>>> we can tell producers (through metadata) to always
> > send
> > > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader
> > > replicas
> > > > > for
> > > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> > > > > Produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>> these partitions in various combinations, but only
> > for
> > > > > them.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>                   Broker 0
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>             |    t1-0         |
> > > > > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > > > > >>>>>>> partitions   | +-------------+ |    for these
> > > partitions
> > > > > > > > > >>>>>>>             |                 |
> > > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > > > > Effectively,
> > > > > > > > > each
> > > > > > > > > >>>>>>> broker will commit only to itself, which effectively
> > > means
> > > > > 1
> > > > > > > > commit
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls,
> > > if we
> > > > > > > wish,
> > > > > > > > > >>>> just a
> > > > > > > > > >>>>>>> local function call).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs).
> > > Obviously, we
> > > > > > > > cannot
> > > > > > > > > >>>> always
> > > > > > > > > >>>>>>> send Produce requests to the designated leaders of
> > > diskless
> > > > > > > > > >>>> partitions:
> > > > > > > > > >>>>>>> this would mean inter-AZ network traffic, which we
> > > would
> > > > > like
> > > > > > > to
> > > > > > > > > >>>> avoid.
> > > > > > > > > >>>>>> To
> > > > > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> > > produce
> > > > > > > > > >>>>>> representative”
> > > > > > > > > >>>>>>> in every AZ. If we continue our example: when a
> > Produce
> > > > > > request
> > > > > > > > for
> > > > > > > > > >>>> t1-0,
> > > > > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands
> > > on
> > > > > > > broker 0
> > > > > > > > > >>>> (in the
> > > > > > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > > > > > However,
> > > > > > > if
> > > > > > > > > it
> > > > > > > > > >>>>>> comes
> > > > > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker
> > > 2.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> |produce requests         |produce requests
> > > |produce
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
> > |for
> > > > > t1-0,
> > > > > > > > t2-1,
> > > > > > > > > >>>>>> t3-0
> > > > > > > > > >>>>>>> |from AZ 0                |from AZ 1
> > >  |from
> > > > > AZ 2
> > > > > > > > > >>>>>>> v                         v                        v
> > > > > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker
> > 2
> > > (AZ
> > > > > 2)
> > > > > > > > > >>>>>>> +---------------+      +---------------+
> > > > > > +---------------+
> > > > > > > > > >>>>>>> |     t1-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t2-1      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> |     t3-0      |      |               |      |
> > > > > >  |
> > > > > > > > > >>>>>>> +---------------+      +--------+------+
> > > > > > +--------+------+
> > > > > > > > > >>>>>>>    ^     ^                    |
> > |
> > > > > > > > > >>>>>>>    |     +--------------------+
> > |
> > > > > > > > > >>>>>>>    |     commit requests for these partitions
> > |
> > > > > > > > > >>>>>>>    |
> >  |
> > > > > > > > > >>>>>>>
> > +-------------------------------------------------+
> > > > > > > > > >>>>>>>          commit requests for these partitions
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> All the partitions that broker 0 is the leader of
> > will
> > > be
> > > > > > > > > >>>> “represented”
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Of course, this relationship goes both ways between
> > AZs
> > > > > (not
> > > > > > > > > >>>> necessarily
> > > > > > > > > >>>>>>> between the same brokers). It means that provided the
> > > > > cluster
> > > > > > > is
> > > > > > > > > >>>> balanced
> > > > > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> > > represent
> > > > > > > > > >>>>>> (number_of_azs
> > > > > > > > > >>>>>>> - 1) other brokers. This will result in the situation
> > > that
> > > > > > for
> > > > > > > > the
> > > > > > > > > >>>>>> majority
> > > > > > > > > >>>>>>> of commits, each broker will do up to (number_of_azs
> > -
> > > 1)
> > > > > > > network
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests (plus one local). Cloud regions tend to
> > have 3
> > > > > AZs,
> > > > > > > very
> > > > > > > > > >>>> rarely
> > > > > > > > > >>>>>>> more. That means, brokers will be doing up to 2
> > network
> > > > > > commit
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>> per
> > > > > > > > > >>>>>>> WAL file.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> There are the following exceptions:
> > > > > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example,
> > > when we
> > > > > > > have
> > > > > > > > 2
> > > > > > > > > >>>> AZs
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> one has three brokers and another AZ has one. This
> > one
> > > > > broker
> > > > > > > > will
> > > > > > > > > do
> > > > > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is
> > > not
> > > > > an
> > > > > > > > > extreme
> > > > > > > > > >>>>>>> amplification. Such an imbalance is not healthy in
> > most
> > > > > > > practical
> > > > > > > > > >>>> setups
> > > > > > > > > >>>>>>> and should be avoided anyway.
> > > > > > > > > >>>>>>> 2. Leadership changes and metadata propagation
> > period.
> > > When
> > > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > > > > > producers
> > > > > > > > > will
> > > > > > > > > >>>> not
> > > > > > > > > >>>>>>> know this immediately (unless we want to be strict
> > and
> > > > > > respond
> > > > > > > > with
> > > > > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> > > will
> > > > > come
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>> in a
> > > > > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two
> > commit
> > > > > > > requests:
> > > > > > > > > to
> > > > > > > > > >>>>>> broker
> > > > > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
> > > t3-0.
> > > > > > This
> > > > > > > > > >>>> situation
> > > > > > > > > >>>>>>> is not permanent and as producers update the cluster
> > > > > > metadata,
> > > > > > > it
> > > > > > > > > >>>> will be
> > > > > > > > > >>>>>>> resolved.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> This all could be built with the metadata crafting
> > > > > mechanism
> > > > > > > only
> > > > > > > > > >>>> (which
> > > > > > > > > >>>>>>> is anyway needed for Diskless in one way or another
> > to
> > > > > direct
> > > > > > > > > >>>> producers
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic),
> > > just
> > > > > with
> > > > > > > the
> > > > > > > > > >>>> right
> > > > > > > > > >>>>>>> policy for it (for example, some deterministic
> > > hash-based
> > > > > > > > formula).
> > > > > > > > > >>>> I.e.
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>> explicit support for “produce representative” or
> > > anything
> > > > > > like
> > > > > > > > this
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> We do only two operations on WAL files that span
> > > multiple
> > > > > > > > diskless
> > > > > > > > > >>>>>>> partitions: committing and deleting. Commits can be
> > > done
> > > > > > > > > >>>> independently as
> > > > > > > > > >>>>>>> described above. But deletes are different, because
> > > when a
> > > > > > file
> > > > > > > > is
> > > > > > > > > >>>>>> deleted,
> > > > > > > > > >>>>>>> this affects all the partitions that still have alive
> > > > > batches
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > >>>>>> file
> > > > > > > > > >>>>>>> (if any).
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The WAL file manager is a necessary point of
> > > coordination
> > > > > to
> > > > > > > > delete
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> files safely. We can say it is the source of truth
> > > about
> > > > > > files
> > > > > > > > > >>>>>> themselves,
> > > > > > > > > >>>>>>> while the partition leaders and their logs hold the
> > > truth
> > > > > > about
> > > > > > > > > >>>> whether a
> > > > > > > > > >>>>>>> particular file contains live batches of this
> > > particular
> > > > > > > > partition.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager will do this important task: be able
> > > to
> > > > > say
> > > > > > > for
> > > > > > > > > sure
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> a file does not contain any live batch of any
> > existing
> > > > > > > partition.
> > > > > > > > > For
> > > > > > > > > >>>>>> this,
> > > > > > > > > >>>>>>> it will have to periodically check against the
> > > partition
> > > > > > > leaders.
> > > > > > > > > >>>>>>> Considering that batch deletion is irreversible, when
> > > we
> > > > > > > declare
> > > > > > > > a
> > > > > > > > > >>>> file
> > > > > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The file manager has to know about files being
> > > committed to
> > > > > > > start
> > > > > > > > > >>>> track
> > > > > > > > > >>>>>>> them and periodically check if they are empty. We can
> > > > > > consider
> > > > > > > > > >>>> various
> > > > > > > > > >>>>>> ways
> > > > > > > > > >>>>>>> to achieve this:
> > > > > > > > > >>>>>>> 1. As was proposed in my previous message: best
> > effort
> > > > > commit
> > > > > > > by
> > > > > > > > > >>>> brokers
> > > > > > > > > >>>>>> +
> > > > > > > > > >>>>>>> periodic prefix scans of object storage to detect
> > files
> > > > > that
> > > > > > > went
> > > > > > > > > >>>> below
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> radar due to network issue or the file manager
> > > temporary
> > > > > > > > > >>>> unavailability.
> > > > > > > > > >>>>>>> We’re speaking about listing the file names only and
> > > > > opening
> > > > > > > only
> > > > > > > > > >>>>>>> previously unknown files in order to find the
> > > partitions
> > > > > > > involved
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>> them.
> > > > > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill
> > the
> > > > > list
> > > > > > of
> > > > > > > > > files
> > > > > > > > > >>>>>>> fully asynchronously and in the background. This may
> > > be not
> > > > > > > ideal
> > > > > > > > > >>>> due to
> > > > > > > > > >>>>>>> costs and performance of scanning tons of files.
> > > However,
> > > > > the
> > > > > > > > > number
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>> live WAL files should be limited due to tiered
> > storage
> > > > > > > > offloading +
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> can
> > > > > > > > > >>>>>>> optimize this if we give files some global soft order
> > > in
> > > > > > their
> > > > > > > > > names.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > logic.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > > > > UnifiedLog
> > > > > > in
> > > > > > > > > >>>> order to
> > > > > > > > > >>>>>>> support these changes. However, it seems that
> > > idempotence,
> > > > > > > > > >>>> transactions,
> > > > > > > > > >>>>>>> queues, tiered storage will have to be changed less
> > > than
> > > > > with
> > > > > > > the
> > > > > > > > > >>>>>> original
> > > > > > > > > >>>>>>> design. This is because the partition leader state
> > > would
> > > > > > remain
> > > > > > > > in
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> same
> > > > > > > > > >>>>>>> place (on brokers) and existing workflows that
> > involve
> > > it
> > > > > > would
> > > > > > > > > have
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> be
> > > > > > > > > >>>>>>> changed less compared to the situation where we
> > > globalize
> > > > > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader state in the batch coordinator. I admit this
> > is
> > > hard
> > > > > > to
> > > > > > > > make
> > > > > > > > > >>>>>>> convincing without both real implementations to hand
> > :)
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I am also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Let’s consider the scenario of turning a classic
> > topic
> > > into
> > > > > > > > > >>>> diskless. The
> > > > > > > > > >>>>>>> user sets diskless.enabled=true, the leader receives
> > > this
> > > > > > > > metadata
> > > > > > > > > >>>> update
> > > > > > > > > >>>>>>> and does the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Start a new segment that will be written in the
> > > diskless
> > > > > > > > format
> > > > > > > > > >>>> (i.e.
> > > > > > > > > >>>>>>> without data).
> > > > > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Since it’s the same log, the followers will know
> > about
> > > that
> > > > > > > > switch
> > > > > > > > > >>>>>>> consistently. They will finish replicating the
> > classic
> > > > > > segments
> > > > > > > > and
> > > > > > > > > >>>> start
> > > > > > > > > >>>>>>> replicating the diskless ones. They will always know
> > > where
> > > > > > each
> > > > > > > > > >>>> batch is
> > > > > > > > > >>>>>>> located (either inside a classic segment or
> > referenced
> > > by a
> > > > > > > > > diskless
> > > > > > > > > >>>>>> one).
> > > > > > > > > >>>>>>> Switching back should be similar.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Doing this with the coordinator is possible, but has
> > > some
> > > > > > > > caveats.
> > > > > > > > > >>>> The
> > > > > > > > > >>>>>>> leader must do the following:
> > > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > > >>>>>>> 3. Write a special control segment to persist and
> > > replicate
> > > > > > the
> > > > > > > > > fact
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> from offset N the partition is now in the diskless
> > > mode.
> > > > > > > > > >>>>>>> 4. Inform the coordinator about the first offset N of
> > > the
> > > > > > > > “diskless
> > > > > > > > > >>>> era”.
> > > > > > > > > >>>>>>> 5. Inform the controller quorum that the transition
> > has
> > > > > > > finished
> > > > > > > > > and
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> brokers now can process diskless writes for this
> > > partition.
> > > > > > > > > >>>>>>> This could fail at some points, so this will probably
> > > > > require
> > > > > > > > some
> > > > > > > > > >>>>>>> explicit state machine with replication either in the
> > > > > > partition
> > > > > > > > log
> > > > > > > > > >>>> or in
> > > > > > > > > >>>>>>> KRaft.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> It seems that the coordinator-less approach makes
> > this
> > > > > > simpler
> > > > > > > > > >>>> because
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>> “coordinator” for the partition and the partition
> > > leader
> > > > > are
> > > > > > > the
> > > > > > > > > >>>> same and
> > > > > > > > > >>>>>>> they store the partition metadata in the same log,
> > too.
> > > > > While
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>>>>>> coordinator approach we have to perform some kind of
> > a
> > > > > > > > distributed
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> to handover metadata management from the classic
> > > partition
> > > > > > > leader
> > > > > > > > > to
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch coordinator.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> > > Please
> > > > > > let
> > > > > > > me
> > > > > > > > > >>>> know if
> > > > > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Best,
> > > > > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > > > > >>>>>>>> Hi, Ivan,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Thanks for the update.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure that I fully understand the new
> > design,
> > > but
> > > > > it
> > > > > > > > seems
> > > > > > > > > >>>> less
> > > > > > > > > >>>>>>>> clean than before.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> > and
> > > a
> > > > > > > separate
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > > doubles
> > > > > > > the
> > > > > > > > > >>>> number
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > > descriptors
> > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > > are
> > > > > > > creating
> > > > > > > > > an
> > > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > > partitions. Are
> > > > > > the
> > > > > > > > > >>>> metadata
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > > the
> > > > > > same
> > > > > > > > > >>>> broker?
> > > > > > > > > >>>>>> If
> > > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > > reassigned?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > > significantly
> > > > > > > higher.
> > > > > > > > > For
> > > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> > in a
> > > > > > cluster
> > > > > > > > > >>>> with 100
> > > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> > more
> > > RPC
> > > > > > > > > requests.
> > > > > > > > > >>>>>> This
> > > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> > two
> > > > > > places,
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > > of
> > > > > > truth,
> > > > > > > > and
> > > > > > > > > >>>> how
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > > implementation.
> > > > > > > The
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > > Adjusting
> > > > > > > both
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> write
> > > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > > follower
> > > > > > to
> > > > > > > > > >>>>>> understand
> > > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > > existing
> > > > > > > > logic. I
> > > > > > > > > >>>> am
> > > > > > > > > >>>>>>> also
> > > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> > topic
> > > > > modes
> > > > > > > > > >>>> between
> > > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > > on
> > > > > > those?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Jun
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > > > > ivan@ivanyu.me
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to
> > > have a
> > > > > > > > separate
> > > > > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role
> > > would
> > > > > be
> > > > > > to
> > > > > > > > > >>>>>> centrally
> > > > > > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> > > > > topics.
> > > > > > > > This
> > > > > > > > > >>>>>>> component
> > > > > > > > > >>>>>>>>> drew many reasonable comments from the community
> > > about
> > > > > how
> > > > > > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> support
> > > > > > > > > >>>>>>>>> various Kafka features (transactions, queues) and
> > its
> > > > > > > > > >>>> scalability.
> > > > > > > > > >>>>>>> While we
> > > > > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > > > > concerns,
> > > > > > > we
> > > > > > > > > >>>> took a
> > > > > > > > > >>>>>>> step
> > > > > > > > > >>>>>>>>> back and looked at the problem from a different
> > > > > > perspective.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> We would like to propose an alternative Diskless
> > > design
> > > > > > > > *without
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>>>>> centralized coordinator*. We believe this approach
> > > has
> > > > > > > > potential
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>>>>> propose to discuss it as it may be more appealing
> > to
> > > the
> > > > > > > > > >>>> community.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let us explain the idea. Most of the complications
> > > with
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless approach come from one necessary
> > > architecture
> > > > > > > change:
> > > > > > > > > >>>>>>> globalizing
> > > > > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > > > > coordinator.
> > > > > > > > > >>>> This
> > > > > > > > > >>>>>>> causes
> > > > > > > > > >>>>>>>>> deviations to the established workflows in various
> > > > > features
> > > > > > > > like
> > > > > > > > > >>>>>>> produce
> > > > > > > > > >>>>>>>>> idempotence and transactions, queues, retention,
> > etc.
> > > > > These
> > > > > > > > > >>>>>> deviations
> > > > > > > > > >>>>>>> need
> > > > > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > > > > implemented
> > > > > > > and
> > > > > > > > > >>>>>>> tested. In
> > > > > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > > > > partition
> > > > > > > > > >>>> leaders
> > > > > > > > > >>>>>>> again
> > > > > > > > > >>>>>>>>> responsible for managing their partitions, even in
> > > > > diskless
> > > > > > > > > >>>> topics.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata
> > are
> > > > > > blended
> > > > > > > > > >>>> together
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> the one partition log. The crux of the Diskless
> > idea
> > > is
> > > > > to
> > > > > > > > > >>>> decouple
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> and move data to the remote storage, while keeping
> > > > > metadata
> > > > > > > > > >>>> somewhere
> > > > > > > > > >>>>>>> else.
> > > > > > > > > >>>>>>>>> Using the central batch coordinator for managing
> > > batch
> > > > > > > metadata
> > > > > > > > > >>>> is
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>> way,
> > > > > > > > > >>>>>>>>> but not the only.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s now think about managing metadata for each
> > user
> > > > > > > partition
> > > > > > > > > >>>>>>>>> independently. Generally partitions are independent
> > > and
> > > > > > don’t
> > > > > > > > > >>>> share
> > > > > > > > > >>>>>>>>> anything apart from that their data are mixed in
> > WAL
> > > > > files.
> > > > > > > If
> > > > > > > > we
> > > > > > > > > >>>>>>> figure
> > > > > > > > > >>>>>>>>> out how to commit and later delete WAL files
> > safely,
> > > we
> > > > > > will
> > > > > > > > > >>>> achieve
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> > > > > central
> > > > > > > > batch
> > > > > > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition
> > > will
> > > > > be
> > > > > > > > > >>>> managed
> > > > > > > > > >>>>>> by
> > > > > > > > > >>>>>>> its
> > > > > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> > > classic
> > > > > > > > > >>>> topics, the
> > > > > > > > > >>>>>>>>> leader uses the partition log as the way to persist
> > > batch
> > > > > > > > > >>>> metadata,
> > > > > > > > > >>>>>>> i.e.
> > > > > > > > > >>>>>>>>> the regular batch header + the information about
> > how
> > > to
> > > > > > find
> > > > > > > > this
> > > > > > > > > >>>>>>> batch on
> > > > > > > > > >>>>>>>>> remote storage. In contrast to classic topics,
> > batch
> > > data
> > > > > > is
> > > > > > > in
> > > > > > > > > >>>>>> remote
> > > > > > > > > >>>>>>>>> storage.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > > > > >>>>>>>>> • Classic topics:
> > > > > > > > > >>>>>>>>>  • Data and metadata are co-located in the
> > partition
> > > log.
> > > > > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > > > > (metadata)|Batch
> > > > > > > > > >>>> data].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>> • Original Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> > > > > remote
> > > > > > > > > >>>> storage.
> > > > > > > > > >>>>>>>>>  • The partition state is global in the batch
> > > > > coordinator.
> > > > > > > > > >>>>>>>>> • New Diskless:
> > > > > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on
> > > remote
> > > > > > > storage.
> > > > > > > > > >>>>>>>>>  • Partition log content: [Batch header
> > > (metadata)|Batch
> > > > > > > > > >>>>>> coordinates
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> remote storage].
> > > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> > followers.
> > > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > > from
> > > > > > > > > >>>> metadata.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the
> > reminder
> > > of
> > > > > the
> > > > > > > > > >>>> original
> > > > > > > > > >>>>>>>>> Diskless design:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The new approach could be depicted as the
> > following:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> As you can see, the main difference is that now
> > > instead
> > > > > of
> > > > > > a
> > > > > > > > > >>>> single
> > > > > > > > > >>>>>>> commit
> > > > > > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> > > > > parallel
> > > > > > > > > >>>> commit
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to all the leaders of each partition involved in
> > the
> > > WAL
> > > > > > > file.
> > > > > > > > > >>>> Each
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>> them
> > > > > > > > > >>>>>>>>> will commit its batches independently, without
> > > > > coordinating
> > > > > > > > with
> > > > > > > > > >>>>>> other
> > > > > > > > > >>>>>>>>> leaders and any other components. Batch data is
> > > addressed
> > > > > > by
> > > > > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>> file
> > > > > > > > > >>>>>>>>> name, the byte offset and size, which allows
> > > partitions
> > > > > to
> > > > > > > know
> > > > > > > > > >>>>>> nothing
> > > > > > > > > >>>>>>>>> about other partitions to access their data in
> > > shared WAL
> > > > > > > > files.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The number of partitions involved in a single WAL
> > > file
> > > > > may
> > > > > > be
> > > > > > > > > >>>> quite
> > > > > > > > > >>>>>>> large,
> > > > > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to
> > commit
> > > one
> > > > > > WAL
> > > > > > > > > >>>> file is
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> > > > > number:
> > > > > > > > > >>>>>>>>> 1. Partition leaders are located on brokers.
> > > Requests to
> > > > > > > > > >>>> leaders on
> > > > > > > > > >>>>>>> one
> > > > > > > > > >>>>>>>>> broker could be grouped together into a single
> > > physical
> > > > > > > network
> > > > > > > > > >>>>>> request
> > > > > > > > > >>>>>>>>> (resembling the normal Produce request that may
> > carry
> > > > > > batches
> > > > > > > > for
> > > > > > > > > >>>>>> many
> > > > > > > > > >>>>>>>>> partitions inside). This will cap the number of
> > > network
> > > > > > > > requests
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make
> > producers
> > > > > send
> > > > > > > > their
> > > > > > > > > >>>>>>> requests
> > > > > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> > > > > achieve
> > > > > > > the
> > > > > > > > > >>>> higher
> > > > > > > > > >>>>>>>>> concentration of logical commit requests in
> > physical
> > > > > > network
> > > > > > > > > >>>> requests
> > > > > > > > > >>>>>>>>> reducing the number of the latter ones even
> > further,
> > > > > > ideally
> > > > > > > to
> > > > > > > > > >>>> one.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Obviously, out of multiple commit requests some may
> > > fail
> > > > > or
> > > > > > > > time
> > > > > > > > > >>>> out
> > > > > > > > > >>>>>>> for a
> > > > > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers
> > will
> > > > > > receive
> > > > > > > > > >>>> totally
> > > > > > > > > >>>>>>> or
> > > > > > > > > >>>>>>>>> partially failed responses to their Produce
> > requests,
> > > > > > similar
> > > > > > > > to
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> they
> > > > > > > > > >>>>>>>>> would have received when appending to a classic
> > topic
> > > > > fails
> > > > > > > or
> > > > > > > > > >>>> times
> > > > > > > > > >>>>>>> out.
> > > > > > > > > >>>>>>>>> If a partition experiences problems, other
> > partitions
> > > > > will
> > > > > > > not
> > > > > > > > be
> > > > > > > > > >>>>>>> affected
> > > > > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > > > > uncommitted
> > > > > > > > data
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > > > > > (batches
> > > > > > > > are
> > > > > > > > > >>>>>>> constantly
> > > > > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> > > storage),
> > > > > > so
> > > > > > > > this
> > > > > > > > > >>>>>>> garbage
> > > > > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> For safely deleting WAL files we now need to
> > > centrally
> > > > > > manage
> > > > > > > > > >>>> them,
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>>>> this is the only state and logic that spans
> > multiple
> > > > > > > > partitions.
> > > > > > > > > >>>> On
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> diagram, you can see another commit request called
> > > > > “Commit
> > > > > > > file
> > > > > > > > > >>>> (best
> > > > > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This
> > manager
> > > will
> > > > > > be
> > > > > > > > > >>>>>>> responsible
> > > > > > > > > >>>>>>>>> for the following:
> > > > > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and
> > > persisting
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> committed WAL files.
> > > > > > > > > >>>>>>>>> 2. To handle potential failures in file information
> > > > > > delivery,
> > > > > > > > it
> > > > > > > > > >>>>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> doing prefix scan on the remote storage
> > periodically
> > > to
> > > > > > find
> > > > > > > > and
> > > > > > > > > >>>>>>> register
> > > > > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > > > > configurable
> > > > > > > and
> > > > > > > > > >>>>>> ideally
> > > > > > > > > >>>>>>>>> should be quite long.
> > > > > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders
> > > (after a
> > > > > > > grace
> > > > > > > > > >>>>>>> period) if
> > > > > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t
> > anymore
> > > > > > > referred
> > > > > > > > > >>>> to by
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> partition.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > > > > features
> > > > > > > such
> > > > > > > > > >>>> as
> > > > > > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > > > > > retention.
> > > > > > > > > >>>> Now we
> > > > > > > > > >>>>>>> don’t
> > > > > > > > > >>>>>>>>> need to abstract away and reuse the code from
> > > partition
> > > > > > > leaders
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use
> > the
> > > > > same
> > > > > > > code
> > > > > > > > > >>>> paths
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from
> > > classic
> > > > > > > topics
> > > > > > > > > >>>> mostly
> > > > > > > > > >>>>>>>>> remain unchanged.
> > > > > > > > > >>>>>>>>> For example, it seems that
> > > > > > > > > >>>>>>>>>
> > > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > > > > and
> > > > > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > > > > transaction
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>> on
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> partition leader side could be used for diskless
> > > topics
> > > > > > with
> > > > > > > > > >>>> little
> > > > > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > > > > idempotent
> > > > > > > > > >>>> produce
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > > > > >>>>>>>>> Another example is share groups support, where the
> > > share
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader,
> > > > > > > > > >>>>>>>>> being co-located with the partition leader, would
> > > execute
> > > > > > the
> > > > > > > > > >>>> same
> > > > > > > > > >>>>>>> logic
> > > > > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > > > > >>>>>>>>> 2. It returns to the familiar partition-based
> > scaling
> > > > > > model,
> > > > > > > > > >>>> where
> > > > > > > > > >>>>>>>>> partitions are independent.
> > > > > > > > > >>>>>>>>> 3. It makes the operation and failure patterns
> > > closer to
> > > > > > the
> > > > > > > > > >>>>>> familiar
> > > > > > > > > >>>>>>>>> ones from classic topics.
> > > > > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> > > switching
> > > > > > the
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> modes
> > > > > > > > > >>>>>>>>> between diskless and classic.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The rest of the things remain unchanged compared to
> > > the
> > > > > > > > previous
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> design (after all previous discussions). Such
> > things
> > > as
> > > > > > local
> > > > > > > > > >>>> segment
> > > > > > > > > >>>>>>>>> materialization by replicas, the consume path,
> > tiered
> > > > > > storage
> > > > > > > > > >>>>>>> integration,
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> If the community finds this design more suitable,
> > we
> > > will
> > > > > > > > update
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it.
> > > Please let
> > > > > > us
> > > > > > > > know
> > > > > > > > > >>>>>> what
> > > > > > > > > >>>>>>> you
> > > > > > > > > >>>>>>>>> think.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
> > wrote:
> > > > > > > > > >>>>>>>>>> Hi Justine,
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > > > > transactions
> > > > > > > > > >>>> for in
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> diskless coordinator for as long as the
> > corresponding
> > > > > > offsets
> > > > > > > > are
> > > > > > > > > >>>>>>> there.
> > > > > > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> > > > > earlier,
> > > > > > > > this
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks for the response. I think most of what you
> > > said
> > > > > > made
> > > > > > > > > >>>>>> sense,
> > > > > > > > > >>>>>>> but
> > > > > > > > > >>>>>>>>> I
> > > > > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it). The
> > > > > > > > > >>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > allowing
> > > > > > > > > >>>>>>> partition
> > > > > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if
> > > this is
> > > > > > > > > >>>> wrong. We
> > > > > > > > > >>>>>>> think
> > > > > > > > > >>>>>>>>>>> about relying on this with the batch coordinator
> > > and
> > > > > > delete
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>> about a transaction once it’s finished (as
> > there’s
> > > no
> > > > > > > > > >>>> replication
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> HWM
> > > > > > > > > >>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I didn't quite understand this. In classic
> > topics,
> > > we
> > > > > > have
> > > > > > > > > >>>> maps
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>> transactions which remove state when the
> > > transaction is
> > > > > > > > > >>>> completed
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> an
> > > > > > > > > >>>>>>>>>>> aborted transactions index which is retained for
> > > much
> > > > > > > longer.
> > > > > > > > > >>>>>> Once
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> > > longer
> > > > > > > > > >>>> involved
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> > > > > subject
> > > > > > to
> > > > > > > > > >>>>>>> compaction
> > > > > > > > > >>>>>>>>> etc.
> > > > > > > > > >>>>>>>>>>> Looking back at the outline provided above, I
> > > didn't
> > > > > see
> > > > > > > much
> > > > > > > > > >>>>>>> about the
> > > > > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > > > > > further.
> > > > > > > I
> > > > > > > > > >>>> saw
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> following in a response:
> > > > > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > > > > segment,
> > > > > > > > > >>>> all the
> > > > > > > > > >>>>>>>>> necessary
> > > > > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> > > including
> > > > > > the
> > > > > > > > > >>>>>>> transaction
> > > > > > > > > >>>>>>>>>>> index will be built to serve FetchRequests
> > exactly
> > > as
> > > > > > they
> > > > > > > > > >>>> are
> > > > > > > > > >>>>>>> today.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain
> > the
> > > > > > > > > >>>> information
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>> Justine
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > > > > > >>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> > > > > focused
> > > > > > > > > >>>> only on
> > > > > > > > > >>>>>>>>> version 2.
> > > > > > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> > > > > consider
> > > > > > > > > >>>> using
> > > > > > > > > >>>>>>> some
> > > > > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > > > > > ongoing,
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> course
> > > > > > > > > >>>>>>>>>>>> the normal idempotence checks. What the
> > partition
> > > > > leader
> > > > > > > > > >>>> in the
> > > > > > > > > >>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>> topics does before appending a batch to the
> > local
> > > log
> > > > > > > > > >>>> (e.g. in
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > > > > > Diskless,
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>>>>> unfortunately
> > > > > > > > > >>>>>>>>>>>> cannot do these checks before appending the data
> > > to
> > > > > the
> > > > > > > WAL
> > > > > > > > > >>>>>>> segment
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these
> > > batches in
> > > > > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As
> > the
> > > > > > > > > >>>> partition
> > > > > > > > > >>>>>>> leader
> > > > > > > > > >>>>>>>>> tracks
> > > > > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the
> > batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> has
> > > > > > > > > >>>>>>>>> to as
> > > > > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > > > > > >>>> transaction
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> > > batches
> > > > > > and
> > > > > > > > > >>>> other
> > > > > > > > > >>>>>>>>>>>> information, but for the sake of performance
> > we’d
> > > > > better
> > > > > > > > > >>>> store
> > > > > > > > > >>>>>> it
> > > > > > > > > >>>>>>>>>>>> explicitly.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> forgets
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > > overpasses
> > > > > > > > > >>>> it).
> > > > > > > > > >>>>>> The
> > > > > > > > > >>>>>>>>>>>> transaction coordinator acts like the main
> > > guardian,
> > > > > > > > > >>>> allowing
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if
> > > this
> > > > > is
> > > > > > > > > >>>> wrong.
> > > > > > > > > >>>>>> We
> > > > > > > > > >>>>>>>>> think
> > > > > > > > > >>>>>>>>>>>> about relying on this with the batch coordinator
> > > and
> > > > > > > > > >>>> delete the
> > > > > > > > > >>>>>>>>> information
> > > > > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as
> > > there’s no
> > > > > > > > > >>>>>> replication
> > > > > > > > > >>>>>>>>> and HWM
> > > > > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> > > transactions!
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > > identified
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer ID
> > > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > > could be
> > > > > > > > > >>>> cached
> > > > > > > > > >>>>>>>>> locally
> > > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> > version 2
> > > > > > > > > >>>>>> transactions
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > > transactions
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>>>> producer id +
> > > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > > > >>>> like the
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > > you be
> > > > > > > > > >>>>>> checking
> > > > > > > > > >>>>>>> if
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is
> > > there
> > > > > > state
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>>>> ongoing
> > > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> > some
> > > > > other
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>>>> mentioned
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > > clear
> > > > > > > > > >>>> what
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > > LSO --
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > > transactional
> > > > > > > > > >>>>>>> state is
> > > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > > will
> > > > > be
> > > > > > > > > >>>>>> cleaned
> > > > > > > > > >>>>>>> up?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> > > local
> > > > > > > > > >>>>>> segments,
> > > > > > > > > >>>>>>>>> making
> > > > > > > > > >>>>>>>>>>>> use of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> > > state for
> > > > > > > > > >>>> each
> > > > > > > > > >>>>>>>>> partition in
> > > > > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> > > fetches
> > > > > > > > > >>>> need
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> know
> > > > > > > > > >>>>>>>>>>>> aborted
> > > > > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > > > > >>>> transaction is
> > > > > > > > > >>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > > > > corresponding
> > > > > > > > > >>>> logs
> > > > > > > > > >>>>>>> of the
> > > > > > > > > >>>>>>>>>>>> involved
> > > > > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the
> > > metadata
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>>>> actual
> > > > > > > > > >>>>>>>>>>>> control
> > > > > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > > > > transactional
> > > > > > > > > >>>>>>> batches.
> > > > > > > > > >>>>>>>>> How
> > > > > > > > > >>>>>>>>>>>> does the
> > > > > > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> > > > > control
> > > > > > > > > >>>>>> batch
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response
> > > needs to
> > > > > > be
> > > > > > > > > >>>>>>> built by
> > > > > > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> > > > > batches
> > > > > > > > > >>>>>>> together?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share
> > partition
> > > > > leader
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>>>>> consumers
> > > > > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
> > > this
> > > > > or
> > > > > > > > > >>>> it's
> > > > > > > > > >>>>>>> too
> > > > > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> > > > > include
> > > > > > > > > >>>>>> enough
> > > > > > > > > >>>>>>>>> details
> > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
> > > given
> > > > > > > > > >>>>>> producer
> > > > > > > > > >>>>>>>>> contain a
> > > > > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a
> > producer
> > > > > sends
> > > > > > > > > >>>>>> every
> > > > > > > > > >>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests
> > > could
> > > > > > > > > >>>> reach
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> > > rejection
> > > > > > > > > >>>> of the
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> requests.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
> > Yurchenko <
> > > > > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details
> > > about
> > > > > > > > > >>>>>>> transactions
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> queues,
> > > > > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic
> > > topics* is
> > > > > > > > > >>>> based
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> precise
> > > > > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> > > (mostly
> > > > > > > > > >>>>>>> producers,
> > > > > > > > > >>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > > > > > >>>> classes),
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition
> > > leaders
> > > > > with
> > > > > > > > > >>>>>> their
> > > > > > > > > >>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level (some details skipped)
> > workflow
> > > is
> > > > > the
> > > > > > > > > >>>>>>>>> following.
> > > > > > > > > >>>>>>>>>>>> When a
> > > > > > > > > >>>>>>>>>>>>>>> transactional Produce request is received by
> > > the
> > > > > > > > > >>>> broker:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> > > checks
> > > > > > > > > >>>> if a
> > > > > > > > > >>>>>>>>> non-empty
> > > > > > > > > >>>>>>>>>>>>>>> transaction is running for this partition.
> > > This is
> > > > > > > > > >>>> done
> > > > > > > > > >>>>>>> using
> > > > > > > > > >>>>>>>>> its
> > > > > > > > > >>>>>>>>>>>> local
> > > > > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > > > > >>>>>> (ProducerStateManager,
> > > > > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> > > about
> > > > > all
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>> partitions
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include
> > them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > > > > transactional
> > > > > > > > > >>>>>>> checks.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the
> > > transactional
> > > > > > > > > >>>> data to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> update some of their state (for example, log
> > > the
> > > > > fact
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> > > offset).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the
> > transaction
> > > > > > > > > >>>> markers to
> > > > > > > > > >>>>>>>>> their logs
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders
> > and
> > > lots
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>>>> important
> > > > > > > > > >>>>>>>>>>>> state
> > > > > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is
> > > local.
> > > > > The
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>>>>> challenge
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
> > > there
> > > > > > > > > >>>> are no
> > > > > > > > > >>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
> > > need
> > > > > > > > > >>>> to be
> > > > > > > > > >>>>>>>>> globalized
> > > > > > > > > >>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this
> > to
> > > > > > > > > >>>> support
> > > > > > > > > >>>>>>>>> idempotent
> > > > > > > > > >>>>>>>>>>>>>> produce.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics*
> > > would
> > > > > > > > > >>>> look
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>>>>> similar:
> > > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > > > > > >>>> non-empty
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> > > classic
> > > > > > > > > >>>>>> topics,
> > > > > > > > > >>>>>>>>> this is
> > > > > > > > > >>>>>>>>>>>>>> checked
> > > > > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single
> > > RPC.
> > > > > > > > > >>>> Since a
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>>>> could
> > > > > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> > > epoch,
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> positive
> > > > > > > > > >>>>>>>>>>>> result of
> > > > > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> > > double
> > > > > > > > > >>>>>>> configured
> > > > > > > > > >>>>>>>>>>>> duration
> > > > > > > > > >>>>>>>>>>>>>> of a
> > > > > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > > > > informed
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the
> > transaction
> > > to
> > > > > > > > > >>>> include
> > > > > > > > > >>>>>>> them.
> > > > > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the
> > > broker
> > > > > > > > > >>>> side.
> > > > > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data
> > > to the
> > > > > > > > > >>>>>> current
> > > > > > > > > >>>>>>>>> shared
> > > > > > > > > >>>>>>>>>>>> WAL
> > > > > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> > > transaction-related
> > > > > > > > > >>>> state
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> Diskless
> > > > > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> like
> > > > > > > > > >>>>>>>>> in the
> > > > > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > > > > transactional
> > > > > > > > > >>>>>>> checks
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> > > > > errors
> > > > > > > > > >>>> like
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> > > > > batches
> > > > > > > > > >>>>>> could
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> rejected.
> > > > > > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage
> > > in
> > > > > the
> > > > > > > > > >>>> WAL
> > > > > > > > > >>>>>>>>> segment
> > > > > > > > > >>>>>>>>>>>> file in
> > > > > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is
> > > preferable to
> > > > > > > > > >>>> doing
> > > > > > > > > >>>>>>> more
> > > > > > > > > >>>>>>>>>>>> network
> > > > > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > > > > segments
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> relatively
> > > > > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > > coordinator
> > > > > > > > > >>>>>>> directly
> > > > > > > > > >>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > > PREPARE_COMMIT
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
> > informs
> > > the
> > > > > > > > > >>>> batch
> > > > > > > > > >>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
> > the
> > > > > > > > > >>>>>>> transaction is
> > > > > > > > > >>>>>>>>>>>> finished
> > > > > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > > > > >>>> corresponding
> > > > > > > > > >>>>>>> logs
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only
> > on
> > > the
> > > > > > > > > >>>>>> metadata
> > > > > > > > > >>>>>>>>> level, no
> > > > > > > > > >>>>>>>>>>>>>>> actual control batches are written to any
> > file.
> > > > > They
> > > > > > > > > >>>> will
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>> dynamically
> > > > > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations.
> > We
> > > > > could
> > > > > > > > > >>>>>>>>> technically
> > > > > > > > > >>>>>>>>>>>> write
> > > > > > > > > >>>>>>>>>>>>>>> these control batches for real, but this
> > would
> > > mean
> > > > > > > > > >>>> extra
> > > > > > > > > >>>>>>>>> produce
> > > > > > > > > >>>>>>>>>>>>>> latency,
> > > > > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > > > > > >>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> save
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to
> > > classic
> > > > > > > > > >>>> topics
> > > > > > > > > >>>>>>> now.
> > > > > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> > > write
> > > > > the
> > > > > > > > > >>>>>>>>> transaction
> > > > > > > > > >>>>>>>>>>>> markers
> > > > > > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final
> > transaction
> > > > > state
> > > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce
> > > flow, we
> > > > > > > > > >>>> get:
> > > > > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between
> > brokers
> > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> batch
> > > > > > > > > >>>>>>>>>>>> coordinator
> > > > > > > > > >>>>>>>>>>>>>>> when a new partition appear in the
> > > transaction. To
> > > > > > > > > >>>>>>> mitigate the
> > > > > > > > > >>>>>>>>>>>> impact of
> > > > > > > > > >>>>>>>>>>>>>>> them:
> > > > > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> > > Produce
> > > > > > > > > >>>>>>> request
> > > > > > > > > >>>>>>>>> will be
> > > > > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized
> > for
> > > > > > > > > >>>> fast
> > > > > > > > > >>>>>>>>> response to
> > > > > > > > > >>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally
> > will
> > > > > > > > > >>>>>>> communicate
> > > > > > > > > >>>>>>>>> with a
> > > > > > > > > >>>>>>>>>>>>>>> single broker for the duration of the
> > > transaction
> > > > > > > > > >>>> further
> > > > > > > > > >>>>>>>>> reduces the
> > > > > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the
> > transaction
> > > > > > > > > >>>>>> coordinator
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> batch
> > > > > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be
> > > able
> > > > > to
> > > > > > > > > >>>>>> span
> > > > > > > > > >>>>>>> both
> > > > > > > > > >>>>>>>>>>>> classic
> > > > > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> The share group coordination and management
> > is
> > > a
> > > > > > > > > >>>> side job
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > > > > > >>>> replicas,
> > > > > > > > > >>>>>>> physical
> > > > > > > > > >>>>>>>>>>>> storage
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > > > > consumers
> > > > > > > > > >>>>>>> (Fetch and
> > > > > > > > > >>>>>>>>>>>> Produce
> > > > > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > > > > affected.)
> > > > > > > > > >>>> We
> > > > > > > > > >>>>>>> don't
> > > > > > > > > >>>>>>>>> see any
> > > > > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> > > compatible
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>> share
> > > > > > > > > >>>>>>>>>>>> groups the
> > > > > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the
> > > code
> > > > > > > > > >>>> level,
> > > > > > > > > >>>>>> we
> > > > > > > > > >>>>>>>>> don't
> > > > > > > > > >>>>>>>>>>>> expect
> > > > > > > > > >>>>>>>>>>>>>> any
> > > > > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading
> > routines
> > > are
> > > > > > > > > >>>> used
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>> used for
> > > > > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this
> > or
> > > it's
> > > > > > > > > >>>> too
> > > > > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design
> > > input
> > > > > > > > > >>>> on
> > > > > > > > > >>>>>>>>> KIP-1150.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> > > with a
> > > > > > > > > >>>> new
> > > > > > > > > >>>>>>>>> design. To
> > > > > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> KIP-405
> > > > > > > > > >>>>>>>>>>>> Tiered
> > > > > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced
> > > to a
> > > > > > > > > >>>>>>> Diskless
> > > > > > > > > >>>>>>>>> topic
> > > > > > > > > >>>>>>>>>>>> to be
> > > > > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for
> > > the
> > > > > > > > > >>>> Batch
> > > > > > > > > >>>>>>>>> Coordinator
> > > > > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose
> > with
> > > > > > > > > >>>> Tiered
> > > > > > > > > >>>>>>> Storage
> > > > > > > > > >>>>>>>>>>>> plugin
> > > > > > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative
> > > data
> > > > > > > > > >>>>>> formats.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
> > local
> > > > > > > > > >>>> segments,
> > > > > > > > > >>>>>>>>> making use
> > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > > > > > >>>> zero-copy
> > > > > > > > > >>>>>>>>> functionality
> > > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered
> > > cache
> > > > > > > > > >>>>>>> elements,
> > > > > > > > > >>>>>>>>> do not
> > > > > > > > > >>>>>>>>>>>> need
> > > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > > > > > >>>> contacting
> > > > > > > > > >>>>>>> any
> > > > > > > > > >>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified
> > > substantially,
> > > > > by
> > > > > > > > > >>>>>>> removing
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> previous
> > > > > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > > > > >>>> component, and
> > > > > > > > > >>>>>>>>> "object
> > > > > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > > > > > >>>> enabled by
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> Batch
> > > > > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
> > as
> > > the
> > > > > > > > > >>>>>> earlier
> > > > > > > > > >>>>>>>>> design,
> > > > > > > > > >>>>>>>>>>>> while
> > > > > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into
> > the
> > > > > > > > > >>>> existing
> > > > > > > > > >>>>>>>>> ecosystem.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> > > feedback on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> new
> > > > > > > > > >>>>>>>>> design.
> > > > > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> > > support
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>> all
> > > > > > > > > >>>>>>>>> existing
> > > > > > > > > >>>>>>>>>>>>>> client
> > > > > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > > > > >>>> supporting
> > > > > > > > > >>>>>> APIs
> > > > > > > > > >>>>>>>>> like
> > > > > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> > > Siekierski
> > > > > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> > > current
> > > > > > > > > >>>>>>> status of
> > > > > > > > > >>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>> KIP?
> > > > > > > > > >>>>>>>>>>>>>> I
> > > > > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> > > past 2
> > > > > > > > > >>>>>>> months,
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>> vote got
> > > > > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have
> > been
> > > > > > > > > >>>>>> answered
> > > > > > > > > >>>>>>>>> since
> > > > > > > > > >>>>>>>>>>>> then.
> > > > > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > > > > >>>> activity
> > > > > > > > > >>>>>>> since
> > > > > > > > > >>>>>>>>> May.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> > > real
> > > > > > > > > >>>>>> choices
> > > > > > > > > >>>>>>>>> that can
> > > > > > > > > >>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> > > being
> > > > > > > > > >>>> by
> > > > > > > > > >>>>>> far
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>> dominant
> > > > > > > > > >>>>>>>>>>>>>> one
> > > > > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all
> > these
> > > are
> > > > > > > > > >>>>>>>>> standalone
> > > > > > > > > >>>>>>>>>>>>>> products.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
> > would
> > > > > > > > > >>>> hate to
> > > > > > > > > >>>>>>> see a
> > > > > > > > > >>>>>>>>>>>> stall. So
> > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
> > Colin
> > > > > > > > > >>>>>> McCabe -
> > > > > > > > > >>>>>>>>> have your
> > > > > > > > > >>>>>>>>>>>>>>> questions
> > > > > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I
> > > can
> > > > > > > > > >>>> do
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>>>> help? I'm
> > > > > > > > > >>>>>>>>>>>>>> deeply
> > > > > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > > > > > >>>> written
> > > > > > > > > >>>>>>> about
> > > > > > > > > >>>>>>>>> both?
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > > > > > >>>>>> Kozlovski
> > > > > > > > > >>>>>>> <
> > > > > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > > > > > >>>> main
> > > > > > > > > >>>>>> one
> > > > > > > > > >>>>>>>>> (just a
> > > > > > > > > >>>>>>>>>>>> bunch
> > > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > > > > >>>> example, I
> > > > > > > > > >>>>>>> was
> > > > > > > > > >>>>>>>>>>>> searching
> > > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while;
> > As
> > > > > > > > > >>>> far
> > > > > > > > > >>>>>> as
> > > > > > > > > >>>>>>> I
> > > > > > > > > >>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>> tell, it
> > > > > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
> > of
> > > > > > > > > >>>> having
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> root
> > > > > > > > > >>>>>>>>>>>> KIP be
> > > > > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> > > links
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>> sub-KIPs
> > > > > > > > > >>>>>>>>>>>> where
> > > > > > > > > >>>>>>>>>>>>>> were
> > > > > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think
> > it
> > > > > > > > > >>>> would
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>> good to
> > > > > > > > > >>>>>>>>>>>>>> create
> > > > > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> > > about
> > > > > > > > > >>>>>>>>> introducing a
> > > > > > > > > >>>>>>>>>>>> new
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > > > > >>>> Storage as
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> primary
> > > > > > > > > >>>>>>>>>>>>>>> source of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > > > > > >>>> decided
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> split it
> > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > > > > >>>> should aim
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > > > > > >>>>>> details
> > > > > > > > > >>>>>>> on
> > > > > > > > > >>>>>>>>> how to
> > > > > > > > > >>>>>>>>>>>>>>> implement
> > > > > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when
> > > we
> > > > > > > > > >>>>>>> discussed
> > > > > > > > > >>>>>>>>>>>> KRaft.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
> > really
> > > > > > > > > >>>> hard
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > >>>>>>>>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > > > > > >>>> (linked
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > > > > >>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> > > DISCUSS
> > > > > > > > > >>>>>>> threads
> > > > > > > > > >>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>> each
> > > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > > > > >>>> whether
> > > > > > > > > >>>>>>> Apache
> > > > > > > > > >>>>>>>>> Kafka
> > > > > > > > > >>>>>>>>>>>> wants
> > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > > > > >>>> authors
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> this KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
> >  +491715557497 |
> > > > > > > > > >>>>>>> aiven.io
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > > > > > >>>>>>> Valtonen,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Anatolii Popov
> > > > > Senior Software Developer, *Aiven OY*
> > > > > m: +358505126242
> > > > > w: aiven.io  e: anatolii.popov@aiven.io
> > > > > <https://www.facebook.com/aivencloud>
> > > > > <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Ivan,

Thanks for the reply.

JR10. "Cluster scalability increases when less data is required to be
stored on disks and rebalanced between brokers."
This seems to contradict a bit with what KIP-1163 says: "Start building it
from some configurable amount, e.g. 100 MiB prior to the current log end or
earliest available offset, whichever is later.".  I just want to confirm
that we plan to make data rebalancing much faster in this KIP.

JR11. Sounds good.

JR13. Since people keep asking about the "simpler" approach, it would be
useful to include in the rejected section why we chose not to pursue the
"simpler" approach like the one in KIP-1176. To me, there are two main
reasons: (1) this KIP provides more benefits (saving all cross AZ network
costs instead of just the replication part, better scalability and
durability); (2) it provides an opportunity for a cleaner long-term design
instead of retrofitting to the existing framework unnaturally.

Jun



On Tue, Jan 20, 2026 at 11:22 AM Greg Harris via dev <de...@kafka.apache.org>
wrote:

> Hi Robin,
>
> Thanks for your design sketch.
>
> Based on your description, the GET operations on the leader and followers
> must complete before offsets are assigned, adding Produce request latency
> that is not present in the KIP-1163 design.
>
> KIP-1163 also assumes that object storage is more reliable than disk
> storage, which is the case in the hyperscaler clouds. The parts of your
> design that make it compatible with low-reliability object storage become
> counterproductive when Diskless is used with a high-reliability object
> storage, such as re-introducing direct replication and cross-az costs.
>
> We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
> in favor of a complete solution. The folks who championed that proposal are
> now working with us on KIP-1163, and are currently designing Compaction in
> Tiered Storage to make Diskless a complete solution for compacted topics as
> well. Similarly, we would appreciate your contributions to the design of
> KIP-1163, and your vote on the motivational KIP-1150.
>
> Thanks!
>
> On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>
> > Hi Jun,
> >
> > Here are our answers to your questions and comments:
> >
> > > JR10. "Even in the case where network traffic is not accounted; we
> > believe
> > > that the operational benefits of Diskless topics are still appealing to
> > > Kafka users."
> > > Could you list other benefits such as better scalability and
> durability?
> >
> > Thank you, good point, added this.
> >
> > > JR11. "AWS charges 0.02$ per GiB"
> > > The object store is not free and has a different cost model based on
> > > requests. It would be useful summarize the tradeoff on cost and
> latency,
> > > and when diskless topics are cost effective.
> >
> > Yes, there is a per-request cost for object storage which is not directly
> > comparable to inter-zone transfer costs. We have found that the design in
> > KIP-1163 amortizes those request costs such that they are insignificant
> > compared to the $0.02/gb cost which is called out in KIP-1150.
> > We currently have very precise cost models that were put together for an
> > earlier iteration of the Diskless topics. These are somewhat stale and
> > inaccurate for the latest design, so we will work in the background to
> post
> > concrete numbers in KIP-1163 before voting on that proposal. Would this
> be
> > OK?
> >
> > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > been
> > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > included in KIP-1163.
> >
> > Thanks for calling out. We have updated KIP-1164 with the latest design.
> > It's missing public interfaces, but we think it may be a bit too early to
> > define them as some rounds of discussion is ahead, with potential to
> change
> > the design. But otherwise the KIP is good for the first review.
> > We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
> >
> > Best,
> > Ivan
> >
> > On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> > > Hi, Anatolii,
> > >
> > > Thanks for the updated KIP. Looks good to me overall. A few more
> > comments.
> > >
> > > JR10. "Even in the case where network traffic is not accounted; we
> > believe
> > > that the operational benefits of Diskless topics are still appealing to
> > > Kafka users."
> > > Could you list other benefits such as better scalability and
> durability?
> > >
> > > JR11. "AWS charges 0.02$ per GiB"
> > > The object store is not free and has a different cost model based on
> > > requests. It would be useful summarize the tradeoff on cost and
> latency,
> > > and when diskless topics are cost effective.
> > >
> > > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> > been
> > > updated in the last 6 months. The metadata change in KIP-1181 is now
> > > included in KIP-1163.
> > >
> > > Jun
> > >
> > > On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> > dev@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thank you for the extensive feedback.
> > > >
> > > > We have substantially updated the KIP to address the points raised.
> > Given
> > > > the scope of these changes, we have started a new VOTE thread to
> > restart
> > > > the voting process cleanly.
> > > >
> > > > You can find the new thread here:
> > > > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> > > >
> > > > Best,
> > > > Anatolii
> > > >
> > > > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > > > dev@kafka.apache.org> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We (the team at Slack) have been following the recent discussion
> > > > regarding
> > > > > the scope and timeline of KIP-1150. We agree with the community
> > sentiment
> > > > > that Diskless Topics represents the right long-term architecture
> for
> > > > Kafka
> > > > > in the cloud, but we also recognize the valid concerns raised
> > regarding
> > > > the
> > > > > engineering resources required to deliver such an ambitious change.
> > > > >
> > > > > To help address these concerns and accelerate the timeline, we are
> > happy
> > > > to
> > > > > announce that we are partnering with the KIP-1150 authors to
> > co-develop
> > > > > this feature.
> > > > >
> > > > > We previously proposed KIP-1176: Tiered Storage for Active Log
> > Segments
> > > > to
> > > > > solve similar problems. However, rather than fragmenting the
> > community's
> > > > > efforts across competing designs, we have decided to withdraw
> > KIP-1176
> > > > and
> > > > > consolidate our engineering resources behind KIP-1150.
> > > > >
> > > > > To start, we plan to take ownership of Compaction for Tiered
> Storage.
> > > > This
> > > > > has long been a missing feature in KIP-405, and it becomes critical
> > in a
> > > > > Diskless architecture where long-lived data is obligated to be
> > tiered. By
> > > > > driving this related prerequisite feature, we hope to allow for
> > faster
> > > > > delivery of KIP-1150.
> > > > >
> > > > > We are excited to collaborate on this to ensure a robust and timely
> > > > > delivery for the community.
> > > > >
> > > > > Best,
> > > > > Tom & Henry
> > > > >
> > > > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > Thanks for sharing the meeting notes.
> > > > > > I agree we should keep polishing the contents of 1150 & high
> level
> > > > design
> > > > > > in 1163 to prepare for a vote.
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > > > <greg.harris@aiven.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > There was a video call between myself, Ivan Yurchenko, Jun Rao,
> > and
> > > > > > Andrew
> > > > > > > Schofield pertaining to KIP-1150. Here are the notes from that
> > > > meeting:
> > > > > > >
> > > > > > > Ivan: What is the future state of Kafka in this area, in 5
> years?
> > > > > > > Jun: Do we want something more cloud native? Yes, started with
> > Tiered
> > > > > > > Storage. If there’s a better way, we should explore it. In the
> > long
> > > > > term
> > > > > > > this will be useful
> > > > > > > Because Kafka is used so widely, we need to make sure
> everything
> > we
> > > > add
> > > > > > is
> > > > > > > for the long term and for everyone, not just for a single
> > company.
> > > > > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > > > > something
> > > > > > > that’s high quality/lasts/maintainable, and can work with all
> > > > existing
> > > > > > > capabilities.
> > > > > > > If both 1150 and 1176 proceed at the same time, it’s confusing.
> > They
> > > > > > > overlap, but Diskless is more ambitious.
> > > > > > > If both KIPs are being seriously worked on, then we don’t
> really
> > need
> > > > > > both,
> > > > > > > because Diskless clearly is better. Having multiple will
> confuse
> > > > > people.
> > > > > > It
> > > > > > > will duplicate some of the effort.
> > > > > > > If we want diskless ultimately, what is the short term
> strategy,
> > to
> > > > get
> > > > > > > some early wins first?
> > > > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > > > Andrew: Eventually the architecture will change substantially,
> > it may
> > > > > not
> > > > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > > > Greg: We all agree on having a high quality feature merged
> > upstream,
> > > > > and
> > > > > > > supporting all APIs
> > > > > > > Jun: We should try and keep things simple, but there is some
> > minimum
> > > > > > > complexity needed.
> > > > > > > When doing the short term changes (1176), it doesn’t really
> > progress
> > > > in
> > > > > > > changing to a more modern architecture.
> > > > > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > > > > Jun: The danger of only applying changes to some part of the
> > API, you
> > > > > set
> > > > > > > the precedence that you only have to implement part of the API.
> > > > > > Supporting
> > > > > > > the full API set should be a minimum requirement.
> > > > > > > Andrew: When we started Kraft, how much did we know the design?
> > > > > > > Jun: For Kraft we didn’t really know much about the migration,
> > but
> > > > the
> > > > > > > high-level was clear.
> > > > > > > Greg: Is 1150 votable in its current state?
> > > > > > > Jun: 1150 should promise to support all APIs. It doesn’t have
> to
> > have
> > > > > all
> > > > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > > > We do need some high-level design enough to give confidence
> that
> > the
> > > > > > > promise is able to be fulfilled.
> > > > > > > Greg: Is the draft version in 1163 enough detail or is more
> > needed?
> > > > > > > Jun: We need to agree on the core design, such as leaderless
> > etc. And
> > > > > how
> > > > > > > the APIs will be supported.
> > > > > > > Greg: Okay we can include these things, and provide a sketch of
> > how
> > > > the
> > > > > > > other leader-based features operate.
> > > > > > > Jun: Yeah if at a high level the sketch appears to work, we can
> > > > approve
> > > > > > > that functionality.
> > > > > > > Are you committed to doing the more involved and big project?
> > > > > > > Greg: Yes, we’re committed to the 1163 design and can’t really
> > accept
> > > > > > 1176.
> > > > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > > > Greg: We’ll push internally for resources, and use the
> community
> > > > > > sentiment
> > > > > > > to motivate Aiven.
> > > > > > > How far into the future should we look? What sort of scale?
> > > > > > > Jun: As long as there’s a path forward, and we’re not closing
> off
> > > > > future
> > > > > > > improvements, we can figure out how to handle a larger scale
> > when it
> > > > > > > arises.
> > > > > > > Greg: Random replica placement is very harmful, can we
> recommend
> > > > users
> > > > > to
> > > > > > > use an external tool like CruiseControl?
> > > > > > > Jun: Not everyone uses CruiseControl, we would probably need
> some
> > > > > > solution
> > > > > > > for this out of the box
> > > > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > > > Jun: Out-of-box experience should be good, good to allow other
> > > > > > > implementations
> > > > > > > Greg: But it could hurt Kafka feature/upgrade velocity when we
> > wait
> > > > for
> > > > > > > plugin providers to implement it
> > > > > > > Ivan: We imagined that maybe cloud hyperscalers could implement
> > it
> > > > with
> > > > > > > e.g. dynamodb
> > > > > > > Greg: Could we bake more details of the different providers
> into
> > > > Kafka,
> > > > > > or
> > > > > > > does it still make sense for it to be pluggable?
> > > > > > > Jun: Make it whatever is easiest to roll out and add new
> clients
> > > > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > > > Ivan: The vote is already open, we’re not too pressed for time.
> > We’ll
> > > > > go
> > > > > > > improve the 1163 design and communication.
> > > > > > > Is 1176 a competing design? Someone will ask.
> > > > > > > Jun: If we are seriously working on something more ambitious,
> > yeah we
> > > > > > > shouldn’t do the stop-gap solution.
> > > > > > > It’s diverting review resources. If we can get the short term
> > thing
> > > > in
> > > > > > 1yr
> > > > > > > but Diskless solution is 2y it makes sense to go for Diskless.
> If
> > > > it’s
> > > > > > 5yr,
> > > > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case.
> Should
> > we
> > > > > > > explicitly exclude 1176?
> > > > > > > Andrew: Put your arms around the feature set you actually want,
> > and
> > > > use
> > > > > > > that to rule out 1176.
> > > > > > > Probably don’t need -1 votes, most likely KIPs just don’t
> receive
> > > > > votes.
> > > > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > > > Jun: Satish posted meeting notes regularly, we should do the
> > same.
> > > > > > >
> > > > > > > To summarize, we will be polishing the contents of 1150 & high
> > level
> > > > > > design
> > > > > > > in 1163 to prepare for a vote.
> > > > > > > We believe that the community should select the feature set of
> > 1150
> > > > to
> > > > > > > fully eliminate producer cross-zone costs, and make the
> > investment
> > > > in a
> > > > > > > high quality Diskless Topics implementation rather than in
> > stop-gap
> > > > > > > solutions.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net>
> wrote:
> > > > > > >
> > > > > > > > This may be a tangent, but we needed to offload storage off
> of
> > > > Kafka
> > > > > > into
> > > > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> > mostly
> > > > > > > > ephemeral broker that can come and go as it pleases. Be that
> > > > scaling
> > > > > or
> > > > > > > > outage. Disks can be destroyed and recreated at will, we
> still
> > > > retain
> > > > > > > data
> > > > > > > > and use broker for just that, brokering messages. Not only
> > that, we
> > > > > > > reduced
> > > > > > > > the requirement on the actual Kafka resources by reducing the
> > size
> > > > > of a
> > > > > > > > payload via a claim check pattern. Maybe this is an
> > anti–pattern,
> > > > but
> > > > > > it
> > > > > > > is
> > > > > > > > super fast and highly cost efficient. We reworked
> > ProducerRequest
> > > > to
> > > > > > > allow
> > > > > > > > plugins. We added a custom http plugin that submits every
> > request
> > > > > via a
> > > > > > > > persisted connection to a microservice. Microservice stores
> the
> > > > > payload
> > > > > > > and
> > > > > > > > returns a tiny json metadata object,a claim check, that can
> be
> > used
> > > > > to
> > > > > > > find
> > > > > > > > the actual data. Think of it as zipping the payload. This
> claim
> > > > check
> > > > > > > > metadata traverses the pipelines with consumers using the
> urls
> > in
> > > > > > > metadata
> > > > > > > > to pull what they need. Think unzipping. This allowed us to
> > also
> > > > pull
> > > > > > > ONLY
> > > > > > > > the data that we need in graphql like manner. So if you have
> a
> > 100K
> > > > > > json
> > > > > > > > payload and you need only a subsection, you can pull that by
> > > > > jmespath.
> > > > > > > When
> > > > > > > > you have multiple consumer groups yanking down huge payloads
> > it is
> > > > > > > > cumbersome on the broker. When you have the same consumer
> > groups
> > > > > > yanking
> > > > > > > > down a claim check, and then going out of band directly to
> the
> > > > source
> > > > > > of
> > > > > > > > truth, the broker has some breathing room. Obviously our
> > > > microservice
> > > > > > > does
> > > > > > > > not go directly to the cloud storage, as that would be too
> > slow. It
> > > > > > > stores
> > > > > > > > the payload in high speed memory cache and returns asap. That
> > > > memory
> > > > > is
> > > > > > > > eventually persisted into S3. The retrieval goest against the
> > cache
> > > > > > > first,
> > > > > > > > then against the S3. Overall a rather cheappy and zippy
> > solution. I
> > > > > > tried
> > > > > > > > proposing the KIP for this, but there was no excitement.
> Check
> > this
> > > > > > out:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > > > >
> > > > > > > >
> > > > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao
> <jun@confluent.io.INVALID
> > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi, Andrew,
> > > > > > > > >
> > > > > > > > > If we want to focus only on reducing cross-zone replication
> > > > costs,
> > > > > > > there
> > > > > > > > is
> > > > > > > > > an alternative design in the KIP-1176 discussion thread
> that
> > > > seems
> > > > > > > > simpler
> > > > > > > > > than the proposal here. I am copying the outline of that
> > approach
> > > > > > > below.
> > > > > > > > >
> > > > > > > > > 1. A new leader is elected.
> > > > > > > > > 2. Leader maintains a first tiered offset, which is
> > initialized
> > > > to
> > > > > > log
> > > > > > > > end
> > > > > > > > > offset.
> > > > > > > > > 3. Leader writes produced data from the client to local
> log.
> > > > > > > > > 4. Leader uploads produced data from all local logs as a
> > combined
> > > > > > > object
> > > > > > > > > 5. Leader stores the metadata for the combined object in
> > memory.
> > > > > > > > > 6. If a follower fetch request has an offset >= first
> tiered
> > > > > offset,
> > > > > > > the
> > > > > > > > > metadata for the corresponding combined object is returned.
> > > > > > Otherwise,
> > > > > > > > the
> > > > > > > > > local data is returned.
> > > > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > > > >
> > > > > > > > > It's still a bit unnatural, but it could work.
> > > > > > > > >
> > > > > > > > > Hi, Ivan,
> > > > > > > > >
> > > > > > > > > Are you still committed to proceeding with the original
> > design of
> > > > > > > > KIP-1150?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > > > andrew_schofield@live.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >> I’ve been following KIP-1150 and friends for a while. I’m
> > going
> > > > to
> > > > > > > jump
> > > > > > > > >> into the discussions too.
> > > > > > > > >>
> > > > > > > > >> Looking back at Jack Vanlightly’s message, I am not quite
> so
> > > > > > convinced
> > > > > > > > >> that it’s a kind of fork in the road. The primary aim of
> the
> > > > > effort
> > > > > > is
> > > > > > > > to
> > > > > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > > > > prohibitively
> > > > > > > > >> expensive to use on cloud storage. I think it would be
> > entirely
> > > > > > viable
> > > > > > > > to
> > > > > > > > >> prioritise code reuse for an initial implementation of
> > diskless
> > > > > > > topics,
> > > > > > > > and
> > > > > > > > >> we could still have a more cloud-native design in the
> > future.
> > > > It’s
> > > > > > > hard
> > > > > > > > to
> > > > > > > > >> predict what the community will prioritise in the future.
> > > > > > > > >>
> > > > > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> > > > support
> > > > > > > > >> leaderless produce requests, first writing WAL segments
> into
> > > > > object
> > > > > > > > >> storage, and then using the regular partition leaders to
> > > > sequence
> > > > > > the
> > > > > > > > >> records. The active log segment for a diskless topic will
> > > > > initially
> > > > > > > > contain
> > > > > > > > >> batch coordinates rather than record batches. The batch
> > > > > coordinates
> > > > > > > can
> > > > > > > > be
> > > > > > > > >> resolved from WAL segments for consumers, and also in
> order
> > to
> > > > > > prepare
> > > > > > > > log
> > > > > > > > >> segments for uploading to tiered storage. Jun is probably
> > > > correct
> > > > > > that
> > > > > > > > we
> > > > > > > > >> need a more frequent object merging process than tiered
> > storage
> > > > > > > > provides.
> > > > > > > > >> This is just the transition from write-optimised WAL
> > segments to
> > > > > > > > >> read-optimised tiered segments, and all of the object
> > > > > storage-based
> > > > > > > > >> implementations of Kafka that I’m aware of do this
> > > > rearrangement.
> > > > > > But
> > > > > > > > >> perhaps this more frequent object merging is a pre-GA
> > > > improvement,
> > > > > > > > rather
> > > > > > > > >> than a strict requirement for an initial implementation
> for
> > > > early
> > > > > > > access
> > > > > > > > >> use.
> > > > > > > > >>
> > > > > > > > >> For zone-aligned share consumers, the share group assignor
> > is
> > > > > > intended
> > > > > > > > to
> > > > > > > > >> be rack-aware. Consumers should be assigned to partitions
> > with
> > > > > > leaders
> > > > > > > > in
> > > > > > > > >> their zone. The simple assignor is not rack-aware, but it
> > easily
> > > > > > could
> > > > > > > > be
> > > > > > > > >> or we could have a rack-aware assignor.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Andrew
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao
> <jun@confluent.io.INVALID
> > >
> > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> Hi, Ivan,
> > > > > > > > >>>
> > > > > > > > >>> Thanks for the reply.
> > > > > > > > >>>
> > > > > > > > >>> "As I understand, you’re speaking about locally
> > materialized
> > > > > > > segments.
> > > > > > > > >> They
> > > > > > > > >>> will indeed consume some IOPS. See them as a cache that
> > could
> > > > > > always
> > > > > > > be
> > > > > > > > >>> restored from the remote storage. While it’s not ideal,
> > it's
> > > > > still
> > > > > > OK
> > > > > > > > to
> > > > > > > > >>> lose data in them due to a machine crash, for example.
> > Because
> > > > of
> > > > > > > this,
> > > > > > > > >> we
> > > > > > > > >>> can avoid explicit flushing on local materialized
> segments
> > at
> > > > all
> > > > > > and
> > > > > > > > let
> > > > > > > > >>> the file system and page cache figure out when to flush
> > > > > optimally.
> > > > > > > This
> > > > > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > dramatically,
> > > > > > > > >>> depending on throughput for each partition. We, of
> course,
> > > > > continue
> > > > > > > > >>> flushing the metadata segments as before."
> > > > > > > > >>>
> > > > > > > > >>> If we have a mix of classic and diskless topics on the
> same
> > > > > broker,
> > > > > > > > it's
> > > > > > > > >>> important that the classic topics' data is flushed to
> disk
> > as
> > > > > > quickly
> > > > > > > > as
> > > > > > > > >>> possible. To achieve this, users typically set
> > > > > > dirty_expire_centisecs
> > > > > > > > in
> > > > > > > > >>> the kernel based on the number of available disk IOPS.
> > Once you
> > > > > set
> > > > > > > > this
> > > > > > > > >>> number, it applies to all dirty files, including the
> cached
> > > > data
> > > > > in
> > > > > > > > >>> diskless topics. So, if there are more files actively
> > > > > accumulating
> > > > > > > > data,
> > > > > > > > >>> the flush frequency and therefore RPO is reduced for
> > classic
> > > > > > topics.
> > > > > > > > >>>
> > > > > > > > >>> "We should have mentioned this explicitly, but this step,
> > in
> > > > > fact,
> > > > > > > > >> remains
> > > > > > > > >>> in the form of segments offloading to tiered storage.
> When
> > we
> > > > > > > assemble
> > > > > > > > a
> > > > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> > effectively
> > > > > > doing
> > > > > > > > >>> metadata compaction: replacing a big number of pieces of
> > > > metadata
> > > > > > > about
> > > > > > > > >>> individual batches with a single record in
> > > > > __remote_log_metadata."
> > > > > > > > >>>
> > > > > > > > >>> The object merging in tier storage typically only kicks
> in
> > > > after
> > > > > a
> > > > > > > few
> > > > > > > > >>> hours. The impact is (1) the amount of accumulated
> > metadata is
> > > > > > still
> > > > > > > > >> quite
> > > > > > > > >>> large; (2) there are many small objects, leading to poor
> > read
> > > > > > > > >> performance.
> > > > > > > > >>> I think we need a more frequent object merging process
> than
> > > > tier
> > > > > > > > storage
> > > > > > > > >>> provides.
> > > > > > > > >>>
> > > > > > > > >>> Jun
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > > > ivan@ivanyu.me>
> > > > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > > > >>>>
> > > > > > > > >>>> Thank you for your messages.
> > > > > > > > >>>>
> > > > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > > > >>>>
> > > > > > > > >>>>> First, it degrades the durability.
> > > > > > > > >>>>> For each partition, now there are two files being
> > actively
> > > > > > written
> > > > > > > > at a
> > > > > > > > >>>>> given point of time, one for the data and another for
> the
> > > > > > metadata.
> > > > > > > > >>>>> Flushing each file requires a separate IO. If the disk
> > has 1K
> > > > > > IOPS
> > > > > > > > and
> > > > > > > > >> we
> > > > > > > > >>>>> have 5K partitions in a broker, currently we can afford
> > to
> > > > > flush
> > > > > > > each
> > > > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5
> > seconds. If
> > > > we
> > > > > > > > double
> > > > > > > > >>>> the
> > > > > > > > >>>>> number of files per partition, we can only flush each
> > > > partition
> > > > > > > every
> > > > > > > > >> 10
> > > > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > > > >>>>
> > > > > > > > >>>> As I understand, you’re speaking about locally
> > materialized
> > > > > > > segments.
> > > > > > > > >> They
> > > > > > > > >>>> will indeed consume some IOPS. See them as a cache that
> > could
> > > > > > always
> > > > > > > > be
> > > > > > > > >>>> restored from the remote storage. While it’s not ideal,
> > it's
> > > > > still
> > > > > > > OK
> > > > > > > > to
> > > > > > > > >>>> lose data in them due to a machine crash, for example.
> > Because
> > > > > of
> > > > > > > > this,
> > > > > > > > >> we
> > > > > > > > >>>> can avoid explicit flushing on local materialized
> > segments at
> > > > > all
> > > > > > > and
> > > > > > > > >> let
> > > > > > > > >>>> the file system and page cache figure out when to flush
> > > > > optimally.
> > > > > > > > This
> > > > > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > > > > dramatically,
> > > > > > > > >>>> depending on throughput for each partition. We, of
> course,
> > > > > > continue
> > > > > > > > >>>> flushing the metadata segments as before.
> > > > > > > > >>>>
> > > > > > > > >>>> It’s worth making a note on caching. I think nobody will
> > > > > disagree
> > > > > > > that
> > > > > > > > >>>> doing direct reads from remote storage every time a
> batch
> > is
> > > > > > > requested
> > > > > > > > >> by a
> > > > > > > > >>>> consumer will not be practical neither from the
> > performance
> > > > nor
> > > > > > from
> > > > > > > > the
> > > > > > > > >>>> economy point of view. We need a way to keep the number
> > of GET
> > > > > > > > requests
> > > > > > > > >>>> down. There are multiple options, for example:
> > > > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > > > >>>> 2. Local in-memory caching. Comes with less network
> > chattiness
> > > > > and
> > > > > > > > >> works
> > > > > > > > >>>> well if we have more or less stable brokers to consume
> > from.
> > > > > > > > >>>> 3. Materialization of diskless logs on local disk. Way
> > lower
> > > > > > impact
> > > > > > > on
> > > > > > > > >>>> RAM and also requires stable brokers for consumption
> > (using
> > > > just
> > > > > > > > >> assigned
> > > > > > > > >>>> replicas will probably work well).
> > > > > > > > >>>>
> > > > > > > > >>>> Materialization is one of possible options, but we can
> > choose
> > > > > > > another
> > > > > > > > >> one.
> > > > > > > > >>>> However, we will have this dilemma regardless of whether
> > we
> > > > have
> > > > > > an
> > > > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > > > >>>>
> > > > > > > > >>>>> Second, if we ever need this
> > > > > > > > >>>>> metadata somewhere else, say in the WAL file manager,
> the
> > > > > > consumer
> > > > > > > > >> needs
> > > > > > > > >>>> to
> > > > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > > > inefficient.
> > > > > > > > The
> > > > > > > > >>>>> actual benefit of this approach is also questionable.
> On
> > the
> > > > > > > surface,
> > > > > > > > >> it
> > > > > > > > >>>>> might seem that we could reduce the number of lines
> that
> > need
> > > > > to
> > > > > > be
> > > > > > > > >>>> changed
> > > > > > > > >>>>> for this KIP. However, the changes are quite intrusive
> > to the
> > > > > > > classic
> > > > > > > > >>>>> partition's code path and will probably make the code
> > base
> > > > > harder
> > > > > > > to
> > > > > > > > >>>>> maintain in the long run. I like the original approach
> > based
> > > > on
> > > > > > the
> > > > > > > > >> batch
> > > > > > > > >>>>> coordinator much better than this one. We could
> probably
> > > > > refactor
> > > > > > > the
> > > > > > > > >>>>> producer state code so that it could be reused in the
> > batch
> > > > > > > > >> coordinator.
> > > > > > > > >>>>
> > > > > > > > >>>> It’s hard to disagree with this. The explicit
> coordinator
> > is
> > > > > more
> > > > > > a
> > > > > > > > side
> > > > > > > > >>>> thing, while coordinator-less approach is more about
> > extending
> > > > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > > > >>>>
> > > > > > > > >>>>> Thanks for addressing the concerns on the number of
> RPCs
> > in
> > > > the
> > > > > > > > produce
> > > > > > > > >>>>> path. I agree that with the metadata crafting
> mechanism,
> > we
> > > > > could
> > > > > > > > >>>> mitigate
> > > > > > > > >>>>> the PRC problem. However, since we now require the
> > metadata
> > > > to
> > > > > be
> > > > > > > > >>>>> collocated with the data on the same set of brokers,
> it's
> > > > weird
> > > > > > > that
> > > > > > > > >> they
> > > > > > > > >>>>> are now managed by different mechanisms. The data
> > assignment
> > > > > now
> > > > > > > uses
> > > > > > > > >> the
> > > > > > > > >>>>> metadata crafting mechanism, but the metadata is stored
> > in
> > > > the
> > > > > > > > classic
> > > > > > > > >>>>> partition using its own assignment strategy. It will be
> > > > > > complicated
> > > > > > > > to
> > > > > > > > >>>> keep
> > > > > > > > >>>>> them collocated.
> > > > > > > > >>>>
> > > > > > > > >>>> I would like to note that the metadata crafting is
> needed
> > only
> > > > > to
> > > > > > > tell
> > > > > > > > >>>> producers which brokers they should send Produce
> requests
> > to,
> > > > > but
> > > > > > > data
> > > > > > > > >> (as
> > > > > > > > >>>> in “locally materialized log”) is located on partition
> > > > replicas,
> > > > > > > i.e.
> > > > > > > > >>>> automatically co-located with metadata.
> > > > > > > > >>>>
> > > > > > > > >>>> As a side note, it would probably be better that instead
> > of
> > > > > > > implicitly
> > > > > > > > >>>> crafting partition metadata, we extend the metadata
> > protocol
> > > > so
> > > > > > that
> > > > > > > > for
> > > > > > > > >>>> diskless partitions we return not only the leader and
> > > > replicas,
> > > > > > but
> > > > > > > > also
> > > > > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > > > > performance
> > > > > > > > and
> > > > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > > > >>>>
> > > > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > > > compaction/merging
> > > > > > > > >>>>> step.
> > > > > > > > >>>>
> > > > > > > > >>>> We should have mentioned this explicitly, but this step,
> > in
> > > > > fact,
> > > > > > > > >> remains
> > > > > > > > >>>> in the form of segments offloading to tiered storage.
> > When we
> > > > > > > > assemble a
> > > > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > > > effectively
> > > > > > > doing
> > > > > > > > >>>> metadata compaction: replacing a big number of pieces of
> > > > > metadata
> > > > > > > > about
> > > > > > > > >>>> individual batches with a single record in
> > > > > __remote_log_metadata.
> > > > > > > > >>>>
> > > > > > > > >>>> We could create a Diskless-specific merging mechanism
> > instead
> > > > if
> > > > > > > > needed.
> > > > > > > > >>>> It’s rather easy with the explicit coordinator approach.
> > With
> > > > > the
> > > > > > > > >>>> coordinator-less approach, this would probably be a bit
> > more
> > > > > > tricky
> > > > > > > > >>>> (rewriting the tail of the log by the leader +
> replicating
> > > > this
> > > > > > > change
> > > > > > > > >>>> reliably).
> > > > > > > > >>>>
> > > > > > > > >>>>> I see a tendency toward primarily optimizing for the
> > fewest
> > > > > code
> > > > > > > > >> changes
> > > > > > > > >>>> in
> > > > > > > > >>>>> the KIP. Instead, our primary goal should be a clean
> > design
> > > > > that
> > > > > > > can
> > > > > > > > >> last
> > > > > > > > >>>>> for the long term.
> > > > > > > > >>>>
> > > > > > > > >>>> Yes, totally agree.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Luke,
> > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > queue is
> > > > > > > because
> > > > > > > > >> of
> > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> we
> > > > only
> > > > > > > focus
> > > > > > > > on
> > > > > > > > >>>> the
> > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > to/from
> > > > > the
> > > > > > > > >> remote
> > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > >>>>
> > > > > > > > >>>> After some evolution of the original proposal,
> leaderless
> > is
> > > > now
> > > > > > > > >> limited.
> > > > > > > > >>>> We only need to be able to accept Produce requests on
> more
> > > > than
> > > > > > one
> > > > > > > > >> broker
> > > > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get
> it
> > > > right
> > > > > > > that
> > > > > > > > >> you
> > > > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Let’s now look at this problem from a higher level, as
> > Jack
> > > > > > > proposed.
> > > > > > > > As
> > > > > > > > >>>> it was said, the big choice we need to make is whether
> we
> > 1)
> > > > > > create
> > > > > > > an
> > > > > > > > >>>> explicit batch coordinator; or 2) go for the
> > coordinator-less
> > > > > > > > approach,
> > > > > > > > >>>> where each diskless partition is managed by its leader
> as
> > in
> > > > > > classic
> > > > > > > > >> topics.
> > > > > > > > >>>>
> > > > > > > > >>>> If we try to compare the two approaches:
> > > > > > > > >>>>
> > > > > > > > >>>> Pluggability:
> > > > > > > > >>>> - Explicit coordinator: Possible. For example, some
> > setups may
> > > > > > > benefit
> > > > > > > > >>>> from batch metadata being stored in a cloud database
> > (such as
> > > > > AWS
> > > > > > > > >> DynamoDB
> > > > > > > > >>>> or GCP Spanner).
> > > > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > > > >>>>
> > > > > > > > >>>> Scalability and fault tolerance:
> > > > > > > > >>>> - Explicit coordinator: Depends on the implementation
> and
> > it’s
> > > > > > also
> > > > > > > > >>>> necessary to actively work for it.
> > > > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics.
> > Scaling is
> > > > > > done
> > > > > > > by
> > > > > > > > >>>> partition placement, partitions could fail
> independently.
> > > > > > > > >>>>
> > > > > > > > >>>> Separation of concerns:
> > > > > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > > > > independent
> > > > > > > > >> from
> > > > > > > > >>>> classic topics in terms of code and workflows. For
> > example,
> > > > the
> > > > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > > > mechanism
> > > > > > > could
> > > > > > > > >> be
> > > > > > > > >>>> relatively simply implemented with it. As a flip side of
> > this,
> > > > > > some
> > > > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > > > bringing
> > > > > > > > diskless
> > > > > > > > >>>> closer to classic topics. Some code paths and workflows
> > could
> > > > be
> > > > > > > more
> > > > > > > > >>>> straightforwardly reused, but they will inevitably have
> > to be
> > > > > > > adapted
> > > > > > > > to
> > > > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > > > >>>>
> > > > > > > > >>>> Cloud-nativeness. This is a vague concept, also related
> > to the
> > > > > > > > previous,
> > > > > > > > >>>> but let’s try:
> > > > > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > > > > separately
> > > > > > > > makes
> > > > > > > > >>>> it easier for brokers to take different roles, be purely
> > > > > stateless
> > > > > > > if
> > > > > > > > >>>> needed, etc.
> > > > > > > > >>>> - Coordinator-less: Less so. Something could be achieved
> > with
> > > > > > > creative
> > > > > > > > >>>> partition placement, but not much.
> > > > > > > > >>>>
> > > > > > > > >>>> Both seem to have their pros and cons. However,
> answering
> > > > Jack’s
> > > > > > > > >> question,
> > > > > > > > >>>> the explicit coordinator approach may indeed lead to a
> > more
> > > > > > flexible
> > > > > > > > >> design.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> The purpose of this deviation in the discussion was to
> > > > receive a
> > > > > > > > >>>> preliminary community evaluation of the coordinator-less
> > > > > approach
> > > > > > > > >> without
> > > > > > > > >>>> taking on the task of writing a separate KIP and fitting
> > it in
> > > > > the
> > > > > > > > >> system
> > > > > > > > >>>> of KIP-1150 and its children. We’re open to stopping it
> > and
> > > > > > getting
> > > > > > > > >> back to
> > > > > > > > >>>> working out the coordinator design if the community
> > doesn’t
> > > > > favor
> > > > > > > the
> > > > > > > > >>>> proposed approach.
> > > > > > > > >>>>
> > > > > > > > >>>> Best,
> > > > > > > > >>>> Ivan and Diskless team
> > > > > > > > >>>>
> > > > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > > > >>>>> Hi Ivan,
> > > > > > > > >>>>>
> > > > > > > > >>>>> As Jun pointed out, the updated design seems to have
> some
> > > > > > > > shortcomings
> > > > > > > > >>>>> although it simplifies the implementation.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> > queue is
> > > > > > > because
> > > > > > > > >> of
> > > > > > > > >>>>> leaderless cluster, do you think it will be simpler if
> we
> > > > only
> > > > > > > focus
> > > > > > > > on
> > > > > > > > >>>> the
> > > > > > > > >>>>> "diskless" design to handle object compaction/merging
> > to/from
> > > > > the
> > > > > > > > >> remote
> > > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thank you,
> > > > > > > > >>>>> Luke
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Hi, Ivan,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks for the explanation.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> "we write the reference to the WAL file with the batch
> > data"
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I understand the approach now, but I think it is a
> hacky
> > > > one.
> > > > > > > There
> > > > > > > > >> are
> > > > > > > > >>>>>> multiple short comings with this design. First, it
> > degrades
> > > > > the
> > > > > > > > >>>> durability.
> > > > > > > > >>>>>> For each partition, now there are two files being
> > actively
> > > > > > written
> > > > > > > > at
> > > > > > > > >> a
> > > > > > > > >>>>>> given point of time, one for the data and another for
> > the
> > > > > > > metadata.
> > > > > > > > >>>>>> Flushing each file requires a separate IO. If the disk
> > has
> > > > 1K
> > > > > > IOPS
> > > > > > > > and
> > > > > > > > >>>> we
> > > > > > > > >>>>>> have 5K partitions in a broker, currently we can
> afford
> > to
> > > > > flush
> > > > > > > > each
> > > > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5
> > seconds. If
> > > > > we
> > > > > > > > double
> > > > > > > > >>>> the
> > > > > > > > >>>>>> number of files per partition, we can only flush each
> > > > > partition
> > > > > > > > every
> > > > > > > > >>>> 10
> > > > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we
> > ever
> > > > need
> > > > > > > this
> > > > > > > > >>>>>> metadata somewhere else, say in the WAL file manager,
> > the
> > > > > > consumer
> > > > > > > > >>>> needs to
> > > > > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > > > > inefficient.
> > > > > > > > The
> > > > > > > > >>>>>> actual benefit of this approach is also questionable.
> > On the
> > > > > > > > surface,
> > > > > > > > >>>> it
> > > > > > > > >>>>>> might seem that we could reduce the number of lines
> that
> > > > need
> > > > > to
> > > > > > > be
> > > > > > > > >>>> changed
> > > > > > > > >>>>>> for this KIP. However, the changes are quite intrusive
> > to
> > > > the
> > > > > > > > classic
> > > > > > > > >>>>>> partition's code path and will probably make the code
> > base
> > > > > > harder
> > > > > > > to
> > > > > > > > >>>>>> maintain in the long run. I like the original approach
> > based
> > > > > on
> > > > > > > the
> > > > > > > > >>>> batch
> > > > > > > > >>>>>> coordinator much better than this one. We could
> probably
> > > > > > refactor
> > > > > > > > the
> > > > > > > > >>>>>> producer state code so that it could be reused in the
> > batch
> > > > > > > > >>>> coordinator.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks for addressing the concerns on the number of
> > RPCs in
> > > > > the
> > > > > > > > >> produce
> > > > > > > > >>>>>> path. I agree that with the metadata crafting
> > mechanism, we
> > > > > > could
> > > > > > > > >>>> mitigate
> > > > > > > > >>>>>> the PRC problem. However, since we now require the
> > metadata
> > > > to
> > > > > > be
> > > > > > > > >>>>>> collocated with the data on the same set of brokers,
> > it's
> > > > > weird
> > > > > > > that
> > > > > > > > >>>> they
> > > > > > > > >>>>>> are now managed by different mechanisms. The data
> > assignment
> > > > > now
> > > > > > > > uses
> > > > > > > > >>>> the
> > > > > > > > >>>>>> metadata crafting mechanism, but the metadata is
> stored
> > in
> > > > the
> > > > > > > > classic
> > > > > > > > >>>>>> partition using its own assignment strategy. It will
> be
> > > > > > > complicated
> > > > > > > > to
> > > > > > > > >>>> keep
> > > > > > > > >>>>>> them collocated.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > > > compaction/merging
> > > > > > > > >>>>>> step. My first concern is on the amount of metadata
> that
> > > > need
> > > > > to
> > > > > > > be
> > > > > > > > >>>> kept.
> > > > > > > > >>>>>> Without object compcation, the metadata generated in
> the
> > > > > produce
> > > > > > > > path
> > > > > > > > >>>> can
> > > > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's
> > say for
> > > > > > every
> > > > > > > > >>>> 250ms we
> > > > > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> > > > remoting
> > > > > > > > tiering
> > > > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K
> > partitions,
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > >>>> keep
> > > > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> > metadata,
> > > > > > quite
> > > > > > > > >>>>>> signficant. A second concern is on performance. Every
> > time
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > >>>>>> rebuild the caching data, we need to read a bunch of
> > small
> > > > > > objects
> > > > > > > > >>>> from S3,
> > > > > > > > >>>>>> slowing down the building process. If a consumer
> > happens to
> > > > > need
> > > > > > > > such
> > > > > > > > >>>> data,
> > > > > > > > >>>>>> it could slow down the application.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> > fewest
> > > > > code
> > > > > > > > >>>> changes in
> > > > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> > design
> > > > > that
> > > > > > > can
> > > > > > > > >>>> last
> > > > > > > > >>>>>> for the long term.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Jun
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me
> > > > > > >
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Hi Jun,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed
> to
> > > > > clearly
> > > > > > > > >>>> explain
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> and
> > a
> > > > > > separate
> > > > > > > > >>>> data
> > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > doubles
> > > > > > the
> > > > > > > > >>>> number
> > > > > > > > >>>>>>> of
> > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > descriptors
> > > > > > > and
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > have a
> > > > > > > > separate
> > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > are
> > > > > > creating
> > > > > > > > an
> > > > > > > > >>>>>>>> internal topic with an unbounded number of
> partitions.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> No. There will be only one physical partition per
> > diskless
> > > > > > > > >>>> partition. Let
> > > > > > > > >>>>>>> me explain this with an example. Let’s say we have a
> > > > diskless
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> > leader. We
> > > > > > > produce
> > > > > > > > >>>> some
> > > > > > > > >>>>>>> batches to this partition. The content of the segment
> > file
> > > > > will
> > > > > > > be
> > > > > > > > >>>>>>> something like this (for each batch):
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > > > >>>>>>> Magic: like in classic
> > > > > > > > >>>>>>> CRC: like in classic
> > > > > > > > >>>>>>> Attributes: like in classic
> > > > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > > > >>>>>>> ProducerId: like in classic
> > > > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > > > >>>>>>> Records:
> > > > > > > > >>>>>>>
> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > > > byte
> > > > > > > offset
> > > > > > > > >>>>>>> 123456
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> It looks very much like classic log entries. The only
> > > > > > difference
> > > > > > > is
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> instead of writing real Records, we write the
> > reference to
> > > > > the
> > > > > > > WAL
> > > > > > > > >>>> file
> > > > > > > > >>>>>>> with the batch data (I guess we need only the name
> and
> > the
> > > > > byte
> > > > > > > > >>>> offset,
> > > > > > > > >>>>>>> because the byte length is the standard field above).
> > > > > > Otherwise,
> > > > > > > > >>>> it’s a
> > > > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> So we have as many partitions for diskless as for
> > classic.
> > > > As
> > > > > > of
> > > > > > > > open
> > > > > > > > >>>>>> file
> > > > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Are the metadata and
> > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > the
> > > > > same
> > > > > > > > >>>> broker?
> > > > > > > > >>>>>> If
> > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > reassigned?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The source of truth for the data is still in WAL
> files
> > on
> > > > > > object
> > > > > > > > >>>> storage.
> > > > > > > > >>>>>>> The source of truth for the metadata is in segment
> > files on
> > > > > the
> > > > > > > > >>>> brokers
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > > > > independent
> > > > > > > > of
> > > > > > > > >>>> this
> > > > > > > > >>>>>>> new proposal, but I want to present them to give the
> > idea
> > > > > that
> > > > > > > > only a
> > > > > > > > >>>>>>> limited amount of data files will be operated
> locally:
> > > > > > > > >>>>>>> - We want to assemble batches into segment files and
> > > > offload
> > > > > > them
> > > > > > > > to
> > > > > > > > >>>>>>> tiered storage in order to prevent the unbounded
> > growth of
> > > > > > batch
> > > > > > > > >>>>>> metadata.
> > > > > > > > >>>>>>> For this, we need to open only  a few file
> descriptors
> > (for
> > > > > the
> > > > > > > > >>>> segment
> > > > > > > > >>>>>>> file itself + the necessary indexes) before the
> > segment is
> > > > > > fully
> > > > > > > > >>>> written
> > > > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > > > >>>>>>> - We want to assemble local segment files for caching
> > > > > purposes
> > > > > > as
> > > > > > > > >>>> well,
> > > > > > > > >>>>>>> i.e. to speed up fetching. This will not materialize
> > the
> > > > full
> > > > > > > > >>>> content of
> > > > > > > > >>>>>>> the log, but only the hot set according to some
> policy
> > (or
> > > > > > > > >>>> configurable
> > > > > > > > >>>>>>> policies), i.e. the number of segments and file
> > descriptors
> > > > > > will
> > > > > > > > >>>> also be
> > > > > > > > >>>>>>> limited.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > significantly
> > > > > > higher.
> > > > > > > > For
> > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> in a
> > > > > cluster
> > > > > > > > >>>> with 100
> > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> more
> > RPC
> > > > > > > > requests.
> > > > > > > > >>>>>> This
> > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> This is a valid concern that we considered, but this
> > issue
> > > > > can
> > > > > > be
> > > > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > > > >>>>>>> The situation with a single broker is trivial: all
> the
> > > > commit
> > > > > > > > >>>> requests go
> > > > > > > > >>>>>>> from the broker to itself.
> > > > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but
> > located in
> > > > > the
> > > > > > > > single
> > > > > > > > >>>>>> rack
> > > > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for
> > diskless
> > > > > > > > >>>> partitions, but
> > > > > > > > >>>>>>> we can tell producers (through metadata) to always
> send
> > > > > Produce
> > > > > > > > >>>> requests
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader
> > replicas
> > > > for
> > > > > > > > diskless
> > > > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> > > > Produce
> > > > > > > > >>>> requests
> > > > > > > > >>>>>> for
> > > > > > > > >>>>>>> these partitions in various combinations, but only
> for
> > > > them.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>                   Broker 0
> > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > >>>>>>>             |    t1-0         |
> > > > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > > > >>>>>>> partitions   | +-------------+ |    for these
> > partitions
> > > > > > > > >>>>>>>             |                 |
> > > > > > > > >>>>>>>             +-----------------+
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > > > Effectively,
> > > > > > > > each
> > > > > > > > >>>>>>> broker will commit only to itself, which effectively
> > means
> > > > 1
> > > > > > > commit
> > > > > > > > >>>>>> request
> > > > > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls,
> > if we
> > > > > > wish,
> > > > > > > > >>>> just a
> > > > > > > > >>>>>>> local function call).
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs).
> > Obviously, we
> > > > > > > cannot
> > > > > > > > >>>> always
> > > > > > > > >>>>>>> send Produce requests to the designated leaders of
> > diskless
> > > > > > > > >>>> partitions:
> > > > > > > > >>>>>>> this would mean inter-AZ network traffic, which we
> > would
> > > > like
> > > > > > to
> > > > > > > > >>>> avoid.
> > > > > > > > >>>>>> To
> > > > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> > produce
> > > > > > > > >>>>>> representative”
> > > > > > > > >>>>>>> in every AZ. If we continue our example: when a
> Produce
> > > > > request
> > > > > > > for
> > > > > > > > >>>> t1-0,
> > > > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands
> > on
> > > > > > broker 0
> > > > > > > > >>>> (in the
> > > > > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > > > > However,
> > > > > > if
> > > > > > > > it
> > > > > > > > >>>>>> comes
> > > > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker
> > 2.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> |produce requests         |produce requests
> > |produce
> > > > > > > > >>>> requests
> > > > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0
> |for
> > > > t1-0,
> > > > > > > t2-1,
> > > > > > > > >>>>>> t3-0
> > > > > > > > >>>>>>> |from AZ 0                |from AZ 1
> >  |from
> > > > AZ 2
> > > > > > > > >>>>>>> v                         v                        v
> > > > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker
> 2
> > (AZ
> > > > 2)
> > > > > > > > >>>>>>> +---------------+      +---------------+
> > > > > +---------------+
> > > > > > > > >>>>>>> |     t1-0      |      |               |      |
> > > > >  |
> > > > > > > > >>>>>>> |     t2-1      |      |               |      |
> > > > >  |
> > > > > > > > >>>>>>> |     t3-0      |      |               |      |
> > > > >  |
> > > > > > > > >>>>>>> +---------------+      +--------+------+
> > > > > +--------+------+
> > > > > > > > >>>>>>>    ^     ^                    |
> |
> > > > > > > > >>>>>>>    |     +--------------------+
> |
> > > > > > > > >>>>>>>    |     commit requests for these partitions
> |
> > > > > > > > >>>>>>>    |
>  |
> > > > > > > > >>>>>>>
> +-------------------------------------------------+
> > > > > > > > >>>>>>>          commit requests for these partitions
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> All the partitions that broker 0 is the leader of
> will
> > be
> > > > > > > > >>>> “represented”
> > > > > > > > >>>>>> by
> > > > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Of course, this relationship goes both ways between
> AZs
> > > > (not
> > > > > > > > >>>> necessarily
> > > > > > > > >>>>>>> between the same brokers). It means that provided the
> > > > cluster
> > > > > > is
> > > > > > > > >>>> balanced
> > > > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> > represent
> > > > > > > > >>>>>> (number_of_azs
> > > > > > > > >>>>>>> - 1) other brokers. This will result in the situation
> > that
> > > > > for
> > > > > > > the
> > > > > > > > >>>>>> majority
> > > > > > > > >>>>>>> of commits, each broker will do up to (number_of_azs
> -
> > 1)
> > > > > > network
> > > > > > > > >>>> commit
> > > > > > > > >>>>>>> requests (plus one local). Cloud regions tend to
> have 3
> > > > AZs,
> > > > > > very
> > > > > > > > >>>> rarely
> > > > > > > > >>>>>>> more. That means, brokers will be doing up to 2
> network
> > > > > commit
> > > > > > > > >>>> requests
> > > > > > > > >>>>>> per
> > > > > > > > >>>>>>> WAL file.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> There are the following exceptions:
> > > > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example,
> > when we
> > > > > > have
> > > > > > > 2
> > > > > > > > >>>> AZs
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>> one has three brokers and another AZ has one. This
> one
> > > > broker
> > > > > > > will
> > > > > > > > do
> > > > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is
> > not
> > > > an
> > > > > > > > extreme
> > > > > > > > >>>>>>> amplification. Such an imbalance is not healthy in
> most
> > > > > > practical
> > > > > > > > >>>> setups
> > > > > > > > >>>>>>> and should be avoided anyway.
> > > > > > > > >>>>>>> 2. Leadership changes and metadata propagation
> period.
> > When
> > > > > the
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > > > > producers
> > > > > > > > will
> > > > > > > > >>>> not
> > > > > > > > >>>>>>> know this immediately (unless we want to be strict
> and
> > > > > respond
> > > > > > > with
> > > > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> > will
> > > > come
> > > > > > > > >>>> together
> > > > > > > > >>>>>> in a
> > > > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two
> commit
> > > > > > requests:
> > > > > > > > to
> > > > > > > > >>>>>> broker
> > > > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
> > t3-0.
> > > > > This
> > > > > > > > >>>> situation
> > > > > > > > >>>>>>> is not permanent and as producers update the cluster
> > > > > metadata,
> > > > > > it
> > > > > > > > >>>> will be
> > > > > > > > >>>>>>> resolved.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> This all could be built with the metadata crafting
> > > > mechanism
> > > > > > only
> > > > > > > > >>>> (which
> > > > > > > > >>>>>>> is anyway needed for Diskless in one way or another
> to
> > > > direct
> > > > > > > > >>>> producers
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic),
> > just
> > > > with
> > > > > > the
> > > > > > > > >>>> right
> > > > > > > > >>>>>>> policy for it (for example, some deterministic
> > hash-based
> > > > > > > formula).
> > > > > > > > >>>> I.e.
> > > > > > > > >>>>>> no
> > > > > > > > >>>>>>> explicit support for “produce representative” or
> > anything
> > > > > like
> > > > > > > this
> > > > > > > > >>>> is
> > > > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> two
> > > > > places,
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > of
> > > > > truth,
> > > > > > > and
> > > > > > > > >>>> how
> > > > > > > > >>>>>> do
> > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> We do only two operations on WAL files that span
> > multiple
> > > > > > > diskless
> > > > > > > > >>>>>>> partitions: committing and deleting. Commits can be
> > done
> > > > > > > > >>>> independently as
> > > > > > > > >>>>>>> described above. But deletes are different, because
> > when a
> > > > > file
> > > > > > > is
> > > > > > > > >>>>>> deleted,
> > > > > > > > >>>>>>> this affects all the partitions that still have alive
> > > > batches
> > > > > > in
> > > > > > > > this
> > > > > > > > >>>>>> file
> > > > > > > > >>>>>>> (if any).
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The WAL file manager is a necessary point of
> > coordination
> > > > to
> > > > > > > delete
> > > > > > > > >>>> WAL
> > > > > > > > >>>>>>> files safely. We can say it is the source of truth
> > about
> > > > > files
> > > > > > > > >>>>>> themselves,
> > > > > > > > >>>>>>> while the partition leaders and their logs hold the
> > truth
> > > > > about
> > > > > > > > >>>> whether a
> > > > > > > > >>>>>>> particular file contains live batches of this
> > particular
> > > > > > > partition.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The file manager will do this important task: be able
> > to
> > > > say
> > > > > > for
> > > > > > > > sure
> > > > > > > > >>>>>> that
> > > > > > > > >>>>>>> a file does not contain any live batch of any
> existing
> > > > > > partition.
> > > > > > > > For
> > > > > > > > >>>>>> this,
> > > > > > > > >>>>>>> it will have to periodically check against the
> > partition
> > > > > > leaders.
> > > > > > > > >>>>>>> Considering that batch deletion is irreversible, when
> > we
> > > > > > declare
> > > > > > > a
> > > > > > > > >>>> file
> > > > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The file manager has to know about files being
> > committed to
> > > > > > start
> > > > > > > > >>>> track
> > > > > > > > >>>>>>> them and periodically check if they are empty. We can
> > > > > consider
> > > > > > > > >>>> various
> > > > > > > > >>>>>> ways
> > > > > > > > >>>>>>> to achieve this:
> > > > > > > > >>>>>>> 1. As was proposed in my previous message: best
> effort
> > > > commit
> > > > > > by
> > > > > > > > >>>> brokers
> > > > > > > > >>>>>> +
> > > > > > > > >>>>>>> periodic prefix scans of object storage to detect
> files
> > > > that
> > > > > > went
> > > > > > > > >>>> below
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>> radar due to network issue or the file manager
> > temporary
> > > > > > > > >>>> unavailability.
> > > > > > > > >>>>>>> We’re speaking about listing the file names only and
> > > > opening
> > > > > > only
> > > > > > > > >>>>>>> previously unknown files in order to find the
> > partitions
> > > > > > involved
> > > > > > > > >>>> with
> > > > > > > > >>>>>> them.
> > > > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill
> the
> > > > list
> > > > > of
> > > > > > > > files
> > > > > > > > >>>>>>> fully asynchronously and in the background. This may
> > be not
> > > > > > ideal
> > > > > > > > >>>> due to
> > > > > > > > >>>>>>> costs and performance of scanning tons of files.
> > However,
> > > > the
> > > > > > > > number
> > > > > > > > >>>> of
> > > > > > > > >>>>>>> live WAL files should be limited due to tiered
> storage
> > > > > > > offloading +
> > > > > > > > >>>> we
> > > > > > > > >>>>>> can
> > > > > > > > >>>>>>> optimize this if we give files some global soft order
> > in
> > > > > their
> > > > > > > > names.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > implementation.
> > > > > > The
> > > > > > > > >>>>>> existing
> > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > Adjusting
> > > > > > both
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> write
> > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > follower
> > > > > to
> > > > > > > > >>>>>> understand
> > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > existing
> > > > > > logic.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > > > UnifiedLog
> > > > > in
> > > > > > > > >>>> order to
> > > > > > > > >>>>>>> support these changes. However, it seems that
> > idempotence,
> > > > > > > > >>>> transactions,
> > > > > > > > >>>>>>> queues, tiered storage will have to be changed less
> > than
> > > > with
> > > > > > the
> > > > > > > > >>>>>> original
> > > > > > > > >>>>>>> design. This is because the partition leader state
> > would
> > > > > remain
> > > > > > > in
> > > > > > > > >>>> the
> > > > > > > > >>>>>> same
> > > > > > > > >>>>>>> place (on brokers) and existing workflows that
> involve
> > it
> > > > > would
> > > > > > > > have
> > > > > > > > >>>> to
> > > > > > > > >>>>>> be
> > > > > > > > >>>>>>> changed less compared to the situation where we
> > globalize
> > > > the
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> leader state in the batch coordinator. I admit this
> is
> > hard
> > > > > to
> > > > > > > make
> > > > > > > > >>>>>>> convincing without both real implementations to hand
> :)
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> I am also
> > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> topic
> > > > modes
> > > > > > > > >>>> between
> > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > on
> > > > > those?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Let’s consider the scenario of turning a classic
> topic
> > into
> > > > > > > > >>>> diskless. The
> > > > > > > > >>>>>>> user sets diskless.enabled=true, the leader receives
> > this
> > > > > > > metadata
> > > > > > > > >>>> update
> > > > > > > > >>>>>>> and does the following:
> > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > >>>>>>> 3. Start a new segment that will be written in the
> > diskless
> > > > > > > format
> > > > > > > > >>>> (i.e.
> > > > > > > > >>>>>>> without data).
> > > > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Since it’s the same log, the followers will know
> about
> > that
> > > > > > > switch
> > > > > > > > >>>>>>> consistently. They will finish replicating the
> classic
> > > > > segments
> > > > > > > and
> > > > > > > > >>>> start
> > > > > > > > >>>>>>> replicating the diskless ones. They will always know
> > where
> > > > > each
> > > > > > > > >>>> batch is
> > > > > > > > >>>>>>> located (either inside a classic segment or
> referenced
> > by a
> > > > > > > > diskless
> > > > > > > > >>>>>> one).
> > > > > > > > >>>>>>> Switching back should be similar.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Doing this with the coordinator is possible, but has
> > some
> > > > > > > caveats.
> > > > > > > > >>>> The
> > > > > > > > >>>>>>> leader must do the following:
> > > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > > >>>>>>> 3. Write a special control segment to persist and
> > replicate
> > > > > the
> > > > > > > > fact
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> from offset N the partition is now in the diskless
> > mode.
> > > > > > > > >>>>>>> 4. Inform the coordinator about the first offset N of
> > the
> > > > > > > “diskless
> > > > > > > > >>>> era”.
> > > > > > > > >>>>>>> 5. Inform the controller quorum that the transition
> has
> > > > > > finished
> > > > > > > > and
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> brokers now can process diskless writes for this
> > partition.
> > > > > > > > >>>>>>> This could fail at some points, so this will probably
> > > > require
> > > > > > > some
> > > > > > > > >>>>>>> explicit state machine with replication either in the
> > > > > partition
> > > > > > > log
> > > > > > > > >>>> or in
> > > > > > > > >>>>>>> KRaft.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> It seems that the coordinator-less approach makes
> this
> > > > > simpler
> > > > > > > > >>>> because
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>> “coordinator” for the partition and the partition
> > leader
> > > > are
> > > > > > the
> > > > > > > > >>>> same and
> > > > > > > > >>>>>>> they store the partition metadata in the same log,
> too.
> > > > While
> > > > > > in
> > > > > > > > the
> > > > > > > > >>>>>>> coordinator approach we have to perform some kind of
> a
> > > > > > > distributed
> > > > > > > > >>>> commit
> > > > > > > > >>>>>>> to handover metadata management from the classic
> > partition
> > > > > > leader
> > > > > > > > to
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> batch coordinator.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> > Please
> > > > > let
> > > > > > me
> > > > > > > > >>>> know if
> > > > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Best,
> > > > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > > > >>>>>>>> Hi, Ivan,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thanks for the update.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> I am not sure that I fully understand the new
> design,
> > but
> > > > it
> > > > > > > seems
> > > > > > > > >>>> less
> > > > > > > > >>>>>>>> clean than before.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Does each partition now have a metadata partition
> and
> > a
> > > > > > separate
> > > > > > > > >>>> data
> > > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > > doubles
> > > > > > the
> > > > > > > > >>>> number
> > > > > > > > >>>>>>> of
> > > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > > descriptors
> > > > > > > and
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> > have a
> > > > > > > > separate
> > > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> > are
> > > > > > creating
> > > > > > > > an
> > > > > > > > >>>>>>>> internal topic with an unbounded number of
> > partitions. Are
> > > > > the
> > > > > > > > >>>> metadata
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>> the data for the same partition always collocated on
> > the
> > > > > same
> > > > > > > > >>>> broker?
> > > > > > > > >>>>>> If
> > > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> > reassigned?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> The number of RPCs in the produce path is
> > significantly
> > > > > > higher.
> > > > > > > > For
> > > > > > > > >>>>>>>> example, if a produce request has 100 partitions,
> in a
> > > > > cluster
> > > > > > > > >>>> with 100
> > > > > > > > >>>>>>>> brokers, each produce request could generate 100
> more
> > RPC
> > > > > > > > requests.
> > > > > > > > >>>>>> This
> > > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into
> two
> > > > > places,
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> > of
> > > > > truth,
> > > > > > > and
> > > > > > > > >>>> how
> > > > > > > > >>>>>> do
> > > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > > implementation.
> > > > > > The
> > > > > > > > >>>>>> existing
> > > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > > Adjusting
> > > > > > both
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> write
> > > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > > follower
> > > > > to
> > > > > > > > >>>>>> understand
> > > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> > existing
> > > > > > > logic. I
> > > > > > > > >>>> am
> > > > > > > > >>>>>>> also
> > > > > > > > >>>>>>>> not sure how this enables seamless switching the
> topic
> > > > modes
> > > > > > > > >>>> between
> > > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> > on
> > > > > those?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Jun
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > > > ivan@ivanyu.me
> > > > > > >
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to
> > have a
> > > > > > > separate
> > > > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role
> > would
> > > > be
> > > > > to
> > > > > > > > >>>>>> centrally
> > > > > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> > > > topics.
> > > > > > > This
> > > > > > > > >>>>>>> component
> > > > > > > > >>>>>>>>> drew many reasonable comments from the community
> > about
> > > > how
> > > > > it
> > > > > > > > >>>> would
> > > > > > > > >>>>>>> support
> > > > > > > > >>>>>>>>> various Kafka features (transactions, queues) and
> its
> > > > > > > > >>>> scalability.
> > > > > > > > >>>>>>> While we
> > > > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > > > concerns,
> > > > > > we
> > > > > > > > >>>> took a
> > > > > > > > >>>>>>> step
> > > > > > > > >>>>>>>>> back and looked at the problem from a different
> > > > > perspective.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> We would like to propose an alternative Diskless
> > design
> > > > > > > *without
> > > > > > > > >>>> a
> > > > > > > > >>>>>>>>> centralized coordinator*. We believe this approach
> > has
> > > > > > > potential
> > > > > > > > >>>> and
> > > > > > > > >>>>>>>>> propose to discuss it as it may be more appealing
> to
> > the
> > > > > > > > >>>> community.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Let us explain the idea. Most of the complications
> > with
> > > > the
> > > > > > > > >>>> original
> > > > > > > > >>>>>>>>> Diskless approach come from one necessary
> > architecture
> > > > > > change:
> > > > > > > > >>>>>>> globalizing
> > > > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > > > coordinator.
> > > > > > > > >>>> This
> > > > > > > > >>>>>>> causes
> > > > > > > > >>>>>>>>> deviations to the established workflows in various
> > > > features
> > > > > > > like
> > > > > > > > >>>>>>> produce
> > > > > > > > >>>>>>>>> idempotence and transactions, queues, retention,
> etc.
> > > > These
> > > > > > > > >>>>>> deviations
> > > > > > > > >>>>>>> need
> > > > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > > > implemented
> > > > > > and
> > > > > > > > >>>>>>> tested. In
> > > > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > > > partition
> > > > > > > > >>>> leaders
> > > > > > > > >>>>>>> again
> > > > > > > > >>>>>>>>> responsible for managing their partitions, even in
> > > > diskless
> > > > > > > > >>>> topics.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata
> are
> > > > > blended
> > > > > > > > >>>> together
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>> the one partition log. The crux of the Diskless
> idea
> > is
> > > > to
> > > > > > > > >>>> decouple
> > > > > > > > >>>>>>> them
> > > > > > > > >>>>>>>>> and move data to the remote storage, while keeping
> > > > metadata
> > > > > > > > >>>> somewhere
> > > > > > > > >>>>>>> else.
> > > > > > > > >>>>>>>>> Using the central batch coordinator for managing
> > batch
> > > > > > metadata
> > > > > > > > >>>> is
> > > > > > > > >>>>>> one
> > > > > > > > >>>>>>> way,
> > > > > > > > >>>>>>>>> but not the only.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Let’s now think about managing metadata for each
> user
> > > > > > partition
> > > > > > > > >>>>>>>>> independently. Generally partitions are independent
> > and
> > > > > don’t
> > > > > > > > >>>> share
> > > > > > > > >>>>>>>>> anything apart from that their data are mixed in
> WAL
> > > > files.
> > > > > > If
> > > > > > > we
> > > > > > > > >>>>>>> figure
> > > > > > > > >>>>>>>>> out how to commit and later delete WAL files
> safely,
> > we
> > > > > will
> > > > > > > > >>>> achieve
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> > > > central
> > > > > > > batch
> > > > > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition
> > will
> > > > be
> > > > > > > > >>>> managed
> > > > > > > > >>>>>> by
> > > > > > > > >>>>>>> its
> > > > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> > classic
> > > > > > > > >>>> topics, the
> > > > > > > > >>>>>>>>> leader uses the partition log as the way to persist
> > batch
> > > > > > > > >>>> metadata,
> > > > > > > > >>>>>>> i.e.
> > > > > > > > >>>>>>>>> the regular batch header + the information about
> how
> > to
> > > > > find
> > > > > > > this
> > > > > > > > >>>>>>> batch on
> > > > > > > > >>>>>>>>> remote storage. In contrast to classic topics,
> batch
> > data
> > > > > is
> > > > > > in
> > > > > > > > >>>>>> remote
> > > > > > > > >>>>>>>>> storage.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > > > >>>>>>>>> • Classic topics:
> > > > > > > > >>>>>>>>>  • Data and metadata are co-located in the
> partition
> > log.
> > > > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > > > (metadata)|Batch
> > > > > > > > >>>> data].
> > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> followers.
> > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > from
> > > > > > > > >>>> metadata.
> > > > > > > > >>>>>>>>> • Original Diskless:
> > > > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> > > > remote
> > > > > > > > >>>> storage.
> > > > > > > > >>>>>>>>>  • The partition state is global in the batch
> > > > coordinator.
> > > > > > > > >>>>>>>>> • New Diskless:
> > > > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on
> > remote
> > > > > > storage.
> > > > > > > > >>>>>>>>>  • Partition log content: [Batch header
> > (metadata)|Batch
> > > > > > > > >>>>>> coordinates
> > > > > > > > >>>>>>> on
> > > > > > > > >>>>>>>>> remote storage].
> > > > > > > > >>>>>>>>>  • The partition log is replicated to the
> followers.
> > > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> > from
> > > > > > > > >>>> metadata.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the
> reminder
> > of
> > > > the
> > > > > > > > >>>> original
> > > > > > > > >>>>>>>>> Diskless design:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The new approach could be depicted as the
> following:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> As you can see, the main difference is that now
> > instead
> > > > of
> > > > > a
> > > > > > > > >>>> single
> > > > > > > > >>>>>>> commit
> > > > > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> > > > parallel
> > > > > > > > >>>> commit
> > > > > > > > >>>>>>> requests
> > > > > > > > >>>>>>>>> to all the leaders of each partition involved in
> the
> > WAL
> > > > > > file.
> > > > > > > > >>>> Each
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>> them
> > > > > > > > >>>>>>>>> will commit its batches independently, without
> > > > coordinating
> > > > > > > with
> > > > > > > > >>>>>> other
> > > > > > > > >>>>>>>>> leaders and any other components. Batch data is
> > addressed
> > > > > by
> > > > > > > the
> > > > > > > > >>>> WAL
> > > > > > > > >>>>>>> file
> > > > > > > > >>>>>>>>> name, the byte offset and size, which allows
> > partitions
> > > > to
> > > > > > know
> > > > > > > > >>>>>> nothing
> > > > > > > > >>>>>>>>> about other partitions to access their data in
> > shared WAL
> > > > > > > files.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The number of partitions involved in a single WAL
> > file
> > > > may
> > > > > be
> > > > > > > > >>>> quite
> > > > > > > > >>>>>>> large,
> > > > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to
> commit
> > one
> > > > > WAL
> > > > > > > > >>>> file is
> > > > > > > > >>>>>>> very
> > > > > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> > > > number:
> > > > > > > > >>>>>>>>> 1. Partition leaders are located on brokers.
> > Requests to
> > > > > > > > >>>> leaders on
> > > > > > > > >>>>>>> one
> > > > > > > > >>>>>>>>> broker could be grouped together into a single
> > physical
> > > > > > network
> > > > > > > > >>>>>> request
> > > > > > > > >>>>>>>>> (resembling the normal Produce request that may
> carry
> > > > > batches
> > > > > > > for
> > > > > > > > >>>>>> many
> > > > > > > > >>>>>>>>> partitions inside). This will cap the number of
> > network
> > > > > > > requests
> > > > > > > > >>>> to
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make
> producers
> > > > send
> > > > > > > their
> > > > > > > > >>>>>>> requests
> > > > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> > > > achieve
> > > > > > the
> > > > > > > > >>>> higher
> > > > > > > > >>>>>>>>> concentration of logical commit requests in
> physical
> > > > > network
> > > > > > > > >>>> requests
> > > > > > > > >>>>>>>>> reducing the number of the latter ones even
> further,
> > > > > ideally
> > > > > > to
> > > > > > > > >>>> one.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Obviously, out of multiple commit requests some may
> > fail
> > > > or
> > > > > > > time
> > > > > > > > >>>> out
> > > > > > > > >>>>>>> for a
> > > > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers
> will
> > > > > receive
> > > > > > > > >>>> totally
> > > > > > > > >>>>>>> or
> > > > > > > > >>>>>>>>> partially failed responses to their Produce
> requests,
> > > > > similar
> > > > > > > to
> > > > > > > > >>>> what
> > > > > > > > >>>>>>> they
> > > > > > > > >>>>>>>>> would have received when appending to a classic
> topic
> > > > fails
> > > > > > or
> > > > > > > > >>>> times
> > > > > > > > >>>>>>> out.
> > > > > > > > >>>>>>>>> If a partition experiences problems, other
> partitions
> > > > will
> > > > > > not
> > > > > > > be
> > > > > > > > >>>>>>> affected
> > > > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > > > uncommitted
> > > > > > > data
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > > > > (batches
> > > > > > > are
> > > > > > > > >>>>>>> constantly
> > > > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> > storage),
> > > > > so
> > > > > > > this
> > > > > > > > >>>>>>> garbage
> > > > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> For safely deleting WAL files we now need to
> > centrally
> > > > > manage
> > > > > > > > >>>> them,
> > > > > > > > >>>>>> as
> > > > > > > > >>>>>>>>> this is the only state and logic that spans
> multiple
> > > > > > > partitions.
> > > > > > > > >>>> On
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>> diagram, you can see another commit request called
> > > > “Commit
> > > > > > file
> > > > > > > > >>>> (best
> > > > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This
> manager
> > will
> > > > > be
> > > > > > > > >>>>>>> responsible
> > > > > > > > >>>>>>>>> for the following:
> > > > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and
> > persisting
> > > > > > > > >>>> information
> > > > > > > > >>>>>>> about
> > > > > > > > >>>>>>>>> committed WAL files.
> > > > > > > > >>>>>>>>> 2. To handle potential failures in file information
> > > > > delivery,
> > > > > > > it
> > > > > > > > >>>>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> doing prefix scan on the remote storage
> periodically
> > to
> > > > > find
> > > > > > > and
> > > > > > > > >>>>>>> register
> > > > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > > > configurable
> > > > > > and
> > > > > > > > >>>>>> ideally
> > > > > > > > >>>>>>>>> should be quite long.
> > > > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders
> > (after a
> > > > > > grace
> > > > > > > > >>>>>>> period) if
> > > > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t
> anymore
> > > > > > referred
> > > > > > > > >>>> to by
> > > > > > > > >>>>>>> any
> > > > > > > > >>>>>>>>> partition.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > > > features
> > > > > > such
> > > > > > > > >>>> as
> > > > > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > > > > retention.
> > > > > > > > >>>> Now we
> > > > > > > > >>>>>>> don’t
> > > > > > > > >>>>>>>>> need to abstract away and reuse the code from
> > partition
> > > > > > leaders
> > > > > > > > >>>> in
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use
> the
> > > > same
> > > > > > code
> > > > > > > > >>>> paths
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from
> > classic
> > > > > > topics
> > > > > > > > >>>> mostly
> > > > > > > > >>>>>>>>> remain unchanged.
> > > > > > > > >>>>>>>>> For example, it seems that
> > > > > > > > >>>>>>>>>
> > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > > > and
> > > > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > > > transaction
> > > > > > > > >>>> support
> > > > > > > > >>>>>> on
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> partition leader side could be used for diskless
> > topics
> > > > > with
> > > > > > > > >>>> little
> > > > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > > > idempotent
> > > > > > > > >>>> produce
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > > > >>>>>>>>> Another example is share groups support, where the
> > share
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> leader,
> > > > > > > > >>>>>>>>> being co-located with the partition leader, would
> > execute
> > > > > the
> > > > > > > > >>>> same
> > > > > > > > >>>>>>> logic
> > > > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > > > >>>>>>>>> 2. It returns to the familiar partition-based
> scaling
> > > > > model,
> > > > > > > > >>>> where
> > > > > > > > >>>>>>>>> partitions are independent.
> > > > > > > > >>>>>>>>> 3. It makes the operation and failure patterns
> > closer to
> > > > > the
> > > > > > > > >>>>>> familiar
> > > > > > > > >>>>>>>>> ones from classic topics.
> > > > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> > switching
> > > > > the
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> modes
> > > > > > > > >>>>>>>>> between diskless and classic.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The rest of the things remain unchanged compared to
> > the
> > > > > > > previous
> > > > > > > > >>>>>>> Diskless
> > > > > > > > >>>>>>>>> design (after all previous discussions). Such
> things
> > as
> > > > > local
> > > > > > > > >>>> segment
> > > > > > > > >>>>>>>>> materialization by replicas, the consume path,
> tiered
> > > > > storage
> > > > > > > > >>>>>>> integration,
> > > > > > > > >>>>>>>>> etc.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> If the community finds this design more suitable,
> we
> > will
> > > > > > > update
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it.
> > Please let
> > > > > us
> > > > > > > know
> > > > > > > > >>>>>> what
> > > > > > > > >>>>>>> you
> > > > > > > > >>>>>>>>> think.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Best regards,
> > > > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko
> wrote:
> > > > > > > > >>>>>>>>>> Hi Justine,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > > > transactions
> > > > > > > > >>>> for in
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> diskless coordinator for as long as the
> corresponding
> > > > > offsets
> > > > > > > are
> > > > > > > > >>>>>>> there.
> > > > > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> > > > earlier,
> > > > > > > this
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>> Ivan
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> > wrote:
> > > > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks for the response. I think most of what you
> > said
> > > > > made
> > > > > > > > >>>>>> sense,
> > > > > > > > >>>>>>> but
> > > > > > > > >>>>>>>>> I
> > > > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > classic
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> forgets
> > > > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > overpasses
> > > > > > > > >>>> it). The
> > > > > > > > >>>>>>>>>>> transaction coordinator acts like the main
> > guardian,
> > > > > > allowing
> > > > > > > > >>>>>>> partition
> > > > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if
> > this is
> > > > > > > > >>>> wrong. We
> > > > > > > > >>>>>>> think
> > > > > > > > >>>>>>>>>>> about relying on this with the batch coordinator
> > and
> > > > > delete
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>> information
> > > > > > > > >>>>>>>>>>> about a transaction once it’s finished (as
> there’s
> > no
> > > > > > > > >>>> replication
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> HWM
> > > > > > > > >>>>>>>>>>> advances immediately).
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I didn't quite understand this. In classic
> topics,
> > we
> > > > > have
> > > > > > > > >>>> maps
> > > > > > > > >>>>>> for
> > > > > > > > >>>>>>>>> ongoing
> > > > > > > > >>>>>>>>>>> transactions which remove state when the
> > transaction is
> > > > > > > > >>>> completed
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> an
> > > > > > > > >>>>>>>>>>> aborted transactions index which is retained for
> > much
> > > > > > longer.
> > > > > > > > >>>>>> Once
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> > longer
> > > > > > > > >>>> involved
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> > > > subject
> > > > > to
> > > > > > > > >>>>>>> compaction
> > > > > > > > >>>>>>>>> etc.
> > > > > > > > >>>>>>>>>>> Looking back at the outline provided above, I
> > didn't
> > > > see
> > > > > > much
> > > > > > > > >>>>>>> about the
> > > > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > > > > further.
> > > > > > I
> > > > > > > > >>>> saw
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> following in a response:
> > > > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > > > segment,
> > > > > > > > >>>> all the
> > > > > > > > >>>>>>>>> necessary
> > > > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> > including
> > > > > the
> > > > > > > > >>>>>>> transaction
> > > > > > > > >>>>>>>>>>> index will be built to serve FetchRequests
> exactly
> > as
> > > > > they
> > > > > > > > >>>> are
> > > > > > > > >>>>>>> today.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain
> the
> > > > > > > > >>>> information
> > > > > > > > >>>>>>> about
> > > > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>> Justine
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > > > > >>>> ivan@ivanyu.me>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > identified
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer ID
> > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > could be
> > > > > > > > >>>> cached
> > > > > > > > >>>>>>>>> locally
> > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> version 2
> > > > > > > > >>>>>> transactions
> > > > > > > > >>>>>>> can
> > > > > > > > >>>>>>>>> be
> > > > > > > > >>>>>>>>>>>> used
> > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > transactions
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer id +
> > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> > > > focused
> > > > > > > > >>>> only on
> > > > > > > > >>>>>>>>> version 2.
> > > > > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> > > > consider
> > > > > > > > >>>> using
> > > > > > > > >>>>>>> some
> > > > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > transactional
> > > > > > > > >>>>>>> checks
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > errors
> > > > > > > > >>>> like the
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > you be
> > > > > > > > >>>>>> checking
> > > > > > > > >>>>>>> if
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > > > > ongoing,
> > > > > > > > >>>> and
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>> course
> > > > > > > > >>>>>>>>>>>> the normal idempotence checks. What the
> partition
> > > > leader
> > > > > > > > >>>> in the
> > > > > > > > >>>>>>>>> classic
> > > > > > > > >>>>>>>>>>>> topics does before appending a batch to the
> local
> > log
> > > > > > > > >>>> (e.g. in
> > > > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > > > > Diskless,
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>> unfortunately
> > > > > > > > >>>>>>>>>>>> cannot do these checks before appending the data
> > to
> > > > the
> > > > > > WAL
> > > > > > > > >>>>>>> segment
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these
> > batches in
> > > > > the
> > > > > > > > >>>> batch
> > > > > > > > >>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> some
> > > > other
> > > > > > > > >>>> state
> > > > > > > > >>>>>>>>> mentioned
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > clear
> > > > > > > > >>>> what
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As
> the
> > > > > > > > >>>> partition
> > > > > > > > >>>>>>> leader
> > > > > > > > >>>>>>>>> tracks
> > > > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the
> batch
> > > > > > > > >>>> coordinator
> > > > > > > > >>>>>>> has
> > > > > > > > >>>>>>>>> to as
> > > > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > > > > >>>> transaction
> > > > > > > > >>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > LSO --
> > > > > > > > >>>> perhaps
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>> would
> > > > > > > > >>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> > batches
> > > > > and
> > > > > > > > >>>> other
> > > > > > > > >>>>>>>>>>>> information, but for the sake of performance
> we’d
> > > > better
> > > > > > > > >>>> store
> > > > > > > > >>>>>> it
> > > > > > > > >>>>>>>>>>>> explicitly.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > transactional
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > will
> > > > be
> > > > > > > > >>>>>> cleaned
> > > > > > > > >>>>>>> up?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> > classic
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> forgets
> > > > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > > overpasses
> > > > > > > > >>>> it).
> > > > > > > > >>>>>> The
> > > > > > > > >>>>>>>>>>>> transaction coordinator acts like the main
> > guardian,
> > > > > > > > >>>> allowing
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if
> > this
> > > > is
> > > > > > > > >>>> wrong.
> > > > > > > > >>>>>> We
> > > > > > > > >>>>>>>>> think
> > > > > > > > >>>>>>>>>>>> about relying on this with the batch coordinator
> > and
> > > > > > > > >>>> delete the
> > > > > > > > >>>>>>>>> information
> > > > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as
> > there’s no
> > > > > > > > >>>>>> replication
> > > > > > > > >>>>>>>>> and HWM
> > > > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>> Ivan
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> > wrote:
> > > > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> > transactions!
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > > identified
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer ID
> > > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> > could be
> > > > > > > > >>>> cached
> > > > > > > > >>>>>>>>> locally
> > > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction
> version 2
> > > > > > > > >>>>>> transactions
> > > > > > > > >>>>>>> can
> > > > > > > > >>>>>>>>> be
> > > > > > > > >>>>>>>>>>>> used
> > > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> > transactions
> > > > > > > > >>>> with
> > > > > > > > >>>>>>>>> producer id +
> > > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > > transactional
> > > > > > > > >>>>>>> checks
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> > errors
> > > > > > > > >>>> like the
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> > you be
> > > > > > > > >>>>>> checking
> > > > > > > > >>>>>>> if
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is
> > there
> > > > > state
> > > > > > > > >>>>>> about
> > > > > > > > >>>>>>>>> ongoing
> > > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see
> some
> > > > other
> > > > > > > > >>>> state
> > > > > > > > >>>>>>>>> mentioned
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> > clear
> > > > > > > > >>>> what
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>> stored
> > > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> > LSO --
> > > > > > > > >>>> perhaps
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>> would
> > > > > > > > >>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > > transactional
> > > > > > > > >>>>>>> state is
> > > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> > will
> > > > be
> > > > > > > > >>>>>> cleaned
> > > > > > > > >>>>>>> up?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> > local
> > > > > > > > >>>>>> segments,
> > > > > > > > >>>>>>>>> making
> > > > > > > > >>>>>>>>>>>> use of
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > zero-copy
> > > > > > > > >>>>>>>>> functionality
> > > > > > > > >>>>>>>>>>>> already
> > > > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> > state for
> > > > > > > > >>>> each
> > > > > > > > >>>>>>>>> partition in
> > > > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> > fetches
> > > > > > > > >>>> need
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> know
> > > > > > > > >>>>>>>>>>>> aborted
> > > > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > > > >>>> transaction is
> > > > > > > > >>>>>>>>> finished
> > > > > > > > >>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > > > corresponding
> > > > > > > > >>>> logs
> > > > > > > > >>>>>>> of the
> > > > > > > > >>>>>>>>>>>> involved
> > > > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the
> > metadata
> > > > > > > > >>>> level,
> > > > > > > > >>>>>> no
> > > > > > > > >>>>>>>>> actual
> > > > > > > > >>>>>>>>>>>> control
> > > > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > > > transactional
> > > > > > > > >>>>>>> batches.
> > > > > > > > >>>>>>>>> How
> > > > > > > > >>>>>>>>>>>> does the
> > > > > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> > > > control
> > > > > > > > >>>>>> batch
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> each
> > > > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response
> > needs to
> > > > > be
> > > > > > > > >>>>>>> built by
> > > > > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> > > > batches
> > > > > > > > >>>>>>> together?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share
> partition
> > > > leader
> > > > > > > > >>>> that
> > > > > > > > >>>>>>> all
> > > > > > > > >>>>>>>>>>>> consumers
> > > > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
> > this
> > > > or
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>> too
> > > > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> > > > include
> > > > > > > > >>>>>> enough
> > > > > > > > >>>>>>>>> details
> > > > > > > > >>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
> > given
> > > > > > > > >>>>>> producer
> > > > > > > > >>>>>>>>> contain a
> > > > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a
> producer
> > > > sends
> > > > > > > > >>>>>> every
> > > > > > > > >>>>>>>>> Produce
> > > > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests
> > could
> > > > > > > > >>>> reach
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> batch
> > > > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> > rejection
> > > > > > > > >>>> of the
> > > > > > > > >>>>>>>>> produce
> > > > > > > > >>>>>>>>>>>>>> requests.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan
> Yurchenko <
> > > > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details
> > about
> > > > > > > > >>>>>>> transactions
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>>>> queues,
> > > > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic
> > topics* is
> > > > > > > > >>>> based
> > > > > > > > >>>>>>> on
> > > > > > > > >>>>>>>>> precise
> > > > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> > (mostly
> > > > > > > > >>>>>>> producers,
> > > > > > > > >>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > > > > >>>> classes),
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition
> > leaders
> > > > with
> > > > > > > > >>>>>> their
> > > > > > > > >>>>>>>>> local
> > > > > > > > >>>>>>>>>>>> state
> > > > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The high level (some details skipped)
> workflow
> > is
> > > > the
> > > > > > > > >>>>>>>>> following.
> > > > > > > > >>>>>>>>>>>> When a
> > > > > > > > >>>>>>>>>>>>>>> transactional Produce request is received by
> > the
> > > > > > > > >>>> broker:
> > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> > checks
> > > > > > > > >>>> if a
> > > > > > > > >>>>>>>>> non-empty
> > > > > > > > >>>>>>>>>>>>>>> transaction is running for this partition.
> > This is
> > > > > > > > >>>> done
> > > > > > > > >>>>>>> using
> > > > > > > > >>>>>>>>> its
> > > > > > > > >>>>>>>>>>>> local
> > > > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > > > >>>>>> (ProducerStateManager,
> > > > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> > about
> > > > all
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>> partitions
> > > > > > > > >>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include
> them.
> > > > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > > > transactional
> > > > > > > > >>>>>>> checks.
> > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the
> > transactional
> > > > > > > > >>>> data to
> > > > > > > > >>>>>>>>> their logs
> > > > > > > > >>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>> update some of their state (for example, log
> > the
> > > > fact
> > > > > > > > >>>>>> that
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> > offset).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > coordinator
> > > > > > > > >>>>>>> directly
> > > > > > > > >>>>>>>>> with
> > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > PREPARE_COMMIT
> > > > > > > > >>>> or
> > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the
> transaction
> > > > > > > > >>>> markers to
> > > > > > > > >>>>>>>>> their logs
> > > > > > > > >>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final
> transaction
> > > > state
> > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders
> and
> > lots
> > > > > > > > >>>> of
> > > > > > > > >>>>>>>>> important
> > > > > > > > >>>>>>>>>>>> state
> > > > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is
> > local.
> > > > The
> > > > > > > > >>>> main
> > > > > > > > >>>>>>>>> challenge
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
> > there
> > > > > > > > >>>> are no
> > > > > > > > >>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
> > need
> > > > > > > > >>>> to be
> > > > > > > > >>>>>>>>> globalized
> > > > > > > > >>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this
> to
> > > > > > > > >>>> support
> > > > > > > > >>>>>>>>> idempotent
> > > > > > > > >>>>>>>>>>>>>> produce.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics*
> > would
> > > > > > > > >>>> look
> > > > > > > > >>>>>>> very
> > > > > > > > >>>>>>>>>>>> similar:
> > > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > > > > >>>> non-empty
> > > > > > > > >>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> > classic
> > > > > > > > >>>>>> topics,
> > > > > > > > >>>>>>>>> this is
> > > > > > > > >>>>>>>>>>>>>> checked
> > > > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single
> > RPC.
> > > > > > > > >>>> Since a
> > > > > > > > >>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>>>> could
> > > > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> > epoch,
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> positive
> > > > > > > > >>>>>>>>>>>> result of
> > > > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> > double
> > > > > > > > >>>>>>> configured
> > > > > > > > >>>>>>>>>>>> duration
> > > > > > > > >>>>>>>>>>>>>> of a
> > > > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > > > informed
> > > > > > > > >>>>>> about
> > > > > > > > >>>>>>> all
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the
> transaction
> > to
> > > > > > > > >>>> include
> > > > > > > > >>>>>>> them.
> > > > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the
> > broker
> > > > > > > > >>>> side.
> > > > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data
> > to the
> > > > > > > > >>>>>> current
> > > > > > > > >>>>>>>>> shared
> > > > > > > > >>>>>>>>>>>> WAL
> > > > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> > transaction-related
> > > > > > > > >>>> state
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> Diskless
> > > > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > > > > >>>> coordinator
> > > > > > > > >>>>>>> like
> > > > > > > > >>>>>>>>> in the
> > > > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > > > transactional
> > > > > > > > >>>>>>> checks
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> > > > errors
> > > > > > > > >>>> like
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> > > > batches
> > > > > > > > >>>>>> could
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>>>>> rejected.
> > > > > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage
> > in
> > > > the
> > > > > > > > >>>> WAL
> > > > > > > > >>>>>>>>> segment
> > > > > > > > >>>>>>>>>>>> file in
> > > > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is
> > preferable to
> > > > > > > > >>>> doing
> > > > > > > > >>>>>>> more
> > > > > > > > >>>>>>>>>>>> network
> > > > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > > > segments
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>>>>> relatively
> > > > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > > coordinator
> > > > > > > > >>>>>>> directly
> > > > > > > > >>>>>>>>> with
> > > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > > PREPARE_COMMIT
> > > > > > > > >>>> or
> > > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator
> informs
> > the
> > > > > > > > >>>> batch
> > > > > > > > >>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that
> the
> > > > > > > > >>>>>>> transaction is
> > > > > > > > >>>>>>>>>>>> finished
> > > > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > > > >>>> corresponding
> > > > > > > > >>>>>>> logs
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only
> on
> > the
> > > > > > > > >>>>>> metadata
> > > > > > > > >>>>>>>>> level, no
> > > > > > > > >>>>>>>>>>>>>>> actual control batches are written to any
> file.
> > > > They
> > > > > > > > >>>> will
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>>>>> dynamically
> > > > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations.
> We
> > > > could
> > > > > > > > >>>>>>>>> technically
> > > > > > > > >>>>>>>>>>>> write
> > > > > > > > >>>>>>>>>>>>>>> these control batches for real, but this
> would
> > mean
> > > > > > > > >>>> extra
> > > > > > > > >>>>>>>>> produce
> > > > > > > > >>>>>>>>>>>>>> latency,
> > > > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > > > > >>>> coordinator
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> save
> > > > > > > > >>>>>>>>>>>> these
> > > > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to
> > classic
> > > > > > > > >>>> topics
> > > > > > > > >>>>>>> now.
> > > > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> > write
> > > > the
> > > > > > > > >>>>>>>>> transaction
> > > > > > > > >>>>>>>>>>>> markers
> > > > > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final
> transaction
> > > > state
> > > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce
> > flow, we
> > > > > > > > >>>> get:
> > > > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between
> brokers
> > and
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> batch
> > > > > > > > >>>>>>>>>>>> coordinator
> > > > > > > > >>>>>>>>>>>>>>> when a new partition appear in the
> > transaction. To
> > > > > > > > >>>>>>> mitigate the
> > > > > > > > >>>>>>>>>>>> impact of
> > > > > > > > >>>>>>>>>>>>>>> them:
> > > > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> > Produce
> > > > > > > > >>>>>>> request
> > > > > > > > >>>>>>>>> will be
> > > > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized
> for
> > > > > > > > >>>> fast
> > > > > > > > >>>>>>>>> response to
> > > > > > > > >>>>>>>>>>>> these
> > > > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally
> will
> > > > > > > > >>>>>>> communicate
> > > > > > > > >>>>>>>>> with a
> > > > > > > > >>>>>>>>>>>>>>> single broker for the duration of the
> > transaction
> > > > > > > > >>>> further
> > > > > > > > >>>>>>>>> reduces the
> > > > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the
> transaction
> > > > > > > > >>>>>> coordinator
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> batch
> > > > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be
> > able
> > > > to
> > > > > > > > >>>>>> span
> > > > > > > > >>>>>>> both
> > > > > > > > >>>>>>>>>>>> classic
> > > > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> The share group coordination and management
> is
> > a
> > > > > > > > >>>> side job
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > > > > >>>> replicas,
> > > > > > > > >>>>>>> physical
> > > > > > > > >>>>>>>>>>>> storage
> > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > > > consumers
> > > > > > > > >>>>>>> (Fetch and
> > > > > > > > >>>>>>>>>>>> Produce
> > > > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > > > affected.)
> > > > > > > > >>>> We
> > > > > > > > >>>>>>> don't
> > > > > > > > >>>>>>>>> see any
> > > > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> > compatible
> > > > > > > > >>>> with
> > > > > > > > >>>>>>> share
> > > > > > > > >>>>>>>>>>>> groups the
> > > > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the
> > code
> > > > > > > > >>>> level,
> > > > > > > > >>>>>> we
> > > > > > > > >>>>>>>>> don't
> > > > > > > > >>>>>>>>>>>> expect
> > > > > > > > >>>>>>>>>>>>>> any
> > > > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading
> routines
> > are
> > > > > > > > >>>> used
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>>> are
> > > > > > > > >>>>>>>>>>>> used for
> > > > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this
> or
> > it's
> > > > > > > > >>>> too
> > > > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> > wrote:
> > > > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design
> > input
> > > > > > > > >>>> on
> > > > > > > > >>>>>>>>> KIP-1150.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> > with a
> > > > > > > > >>>> new
> > > > > > > > >>>>>>>>> design. To
> > > > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with
> the
> > > > > > > > >>>> existing
> > > > > > > > >>>>>>>>> KIP-405
> > > > > > > > >>>>>>>>>>>> Tiered
> > > > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced
> > to a
> > > > > > > > >>>>>>> Diskless
> > > > > > > > >>>>>>>>> topic
> > > > > > > > >>>>>>>>>>>> to be
> > > > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for
> > the
> > > > > > > > >>>> Batch
> > > > > > > > >>>>>>>>> Coordinator
> > > > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose
> with
> > > > > > > > >>>> Tiered
> > > > > > > > >>>>>>> Storage
> > > > > > > > >>>>>>>>>>>> plugin
> > > > > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative
> > data
> > > > > > > > >>>>>> formats.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from
> local
> > > > > > > > >>>> segments,
> > > > > > > > >>>>>>>>> making use
> > > > > > > > >>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > > > > >>>> zero-copy
> > > > > > > > >>>>>>>>> functionality
> > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered
> > cache
> > > > > > > > >>>>>>> elements,
> > > > > > > > >>>>>>>>> do not
> > > > > > > > >>>>>>>>>>>> need
> > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > > > > >>>> contacting
> > > > > > > > >>>>>>> any
> > > > > > > > >>>>>>>>> other
> > > > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified
> > substantially,
> > > > by
> > > > > > > > >>>>>>> removing
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> previous
> > > > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > > > >>>> component, and
> > > > > > > > >>>>>>>>> "object
> > > > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > > > > >>>> enabled by
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> Batch
> > > > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles
> as
> > the
> > > > > > > > >>>>>> earlier
> > > > > > > > >>>>>>>>> design,
> > > > > > > > >>>>>>>>>>>> while
> > > > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into
> the
> > > > > > > > >>>> existing
> > > > > > > > >>>>>>>>> ecosystem.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> > feedback on
> > > > > > > > >>>> the
> > > > > > > > >>>>>> new
> > > > > > > > >>>>>>>>> design.
> > > > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> > support
> > > > > > > > >>>> of
> > > > > > > > >>>>>> all
> > > > > > > > >>>>>>>>> existing
> > > > > > > > >>>>>>>>>>>>>> client
> > > > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > > > >>>> supporting
> > > > > > > > >>>>>> APIs
> > > > > > > > >>>>>>>>> like
> > > > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> > Siekierski
> > > > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> > current
> > > > > > > > >>>>>>> status of
> > > > > > > > >>>>>>>>> this
> > > > > > > > >>>>>>>>>>>> KIP?
> > > > > > > > >>>>>>>>>>>>>> I
> > > > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> > past 2
> > > > > > > > >>>>>>> months,
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>> vote got
> > > > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have
> been
> > > > > > > > >>>>>> answered
> > > > > > > > >>>>>>>>> since
> > > > > > > > >>>>>>>>>>>> then.
> > > > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > > > >>>> activity
> > > > > > > > >>>>>>> since
> > > > > > > > >>>>>>>>> May.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> > real
> > > > > > > > >>>>>> choices
> > > > > > > > >>>>>>>>> that can
> > > > > > > > >>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> > being
> > > > > > > > >>>> by
> > > > > > > > >>>>>> far
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>> dominant
> > > > > > > > >>>>>>>>>>>>>> one
> > > > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all
> these
> > are
> > > > > > > > >>>>>>>>> standalone
> > > > > > > > >>>>>>>>>>>>>> products.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but
> would
> > > > > > > > >>>> hate to
> > > > > > > > >>>>>>> see a
> > > > > > > > >>>>>>>>>>>> stall. So
> > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote?
> Colin
> > > > > > > > >>>>>> McCabe -
> > > > > > > > >>>>>>>>> have your
> > > > > > > > >>>>>>>>>>>>>>> questions
> > > > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I
> > can
> > > > > > > > >>>> do
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>>>> help? I'm
> > > > > > > > >>>>>>>>>>>>>> deeply
> > > > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > > > > >>>> written
> > > > > > > > >>>>>>> about
> > > > > > > > >>>>>>>>> both?
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > > > > >>>>>> Kozlovski
> > > > > > > > >>>>>>> <
> > > > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > > > > >>>> main
> > > > > > > > >>>>>> one
> > > > > > > > >>>>>>>>> (just a
> > > > > > > > >>>>>>>>>>>> bunch
> > > > > > > > >>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > > > >>>> example, I
> > > > > > > > >>>>>>> was
> > > > > > > > >>>>>>>>>>>> searching
> > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while;
> As
> > > > > > > > >>>> far
> > > > > > > > >>>>>> as
> > > > > > > > >>>>>>> I
> > > > > > > > >>>>>>>>> can
> > > > > > > > >>>>>>>>>>>> tell, it
> > > > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue
> of
> > > > > > > > >>>> having
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> root
> > > > > > > > >>>>>>>>>>>> KIP be
> > > > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> > links
> > > > > > > > >>>> to
> > > > > > > > >>>>>>>>> sub-KIPs
> > > > > > > > >>>>>>>>>>>> where
> > > > > > > > >>>>>>>>>>>>>> were
> > > > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think
> it
> > > > > > > > >>>> would
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>> good to
> > > > > > > > >>>>>>>>>>>>>> create
> > > > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> > about
> > > > > > > > >>>>>>>>> introducing a
> > > > > > > > >>>>>>>>>>>> new
> > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > > > >>>> Storage as
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> primary
> > > > > > > > >>>>>>>>>>>>>>> source of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > > > > >>>> decided
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>> split it
> > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > > > >>>> should aim
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>> have
> > > > > > > > >>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > > > > >>>>>> details
> > > > > > > > >>>>>>> on
> > > > > > > > >>>>>>>>> how to
> > > > > > > > >>>>>>>>>>>>>>> implement
> > > > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when
> > we
> > > > > > > > >>>>>>> discussed
> > > > > > > > >>>>>>>>>>>> KRaft.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes
> really
> > > > > > > > >>>> hard
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>> discuss
> > > > > > > > >>>>>>>>>>>> on
> > > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > > > > >>>> (linked
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > > > >>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> > DISCUSS
> > > > > > > > >>>>>>> threads
> > > > > > > > >>>>>>>>> for
> > > > > > > > >>>>>>>>>>>> each
> > > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > > > >>>> whether
> > > > > > > > >>>>>>> Apache
> > > > > > > > >>>>>>>>> Kafka
> > > > > > > > >>>>>>>>>>>> wants
> > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > > > >>>> authors
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>> this KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |
>  +491715557497 |
> > > > > > > > >>>>>>> aiven.io
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > > > > >>>>>>> Valtonen,
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Anatolii Popov
> > > > Senior Software Developer, *Aiven OY*
> > > > m: +358505126242
> > > > w: aiven.io  e: anatolii.popov@aiven.io
> > > > <https://www.facebook.com/aivencloud>
> > > > <https://www.linkedin.com/company/aiven/>   <
> > https://twitter.com/aiven_io>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Greg Harris via dev <de...@kafka.apache.org>]

Hi Robin,

Thanks for your design sketch.

Based on your description, the GET operations on the leader and followers
must complete before offsets are assigned, adding Produce request latency
that is not present in the KIP-1163 design.

KIP-1163 also assumes that object storage is more reliable than disk
storage, which is the case in the hyperscaler clouds. The parts of your
design that make it compatible with low-reliability object storage become
counterproductive when Diskless is used with a high-reliability object
storage, such as re-introducing direct replication and cross-az costs.

We’ve already discussed and rejected KIP-1176 (another “simpler” proposal)
in favor of a complete solution. The folks who championed that proposal are
now working with us on KIP-1163, and are currently designing Compaction in
Tiered Storage to make Diskless a complete solution for compacted topics as
well. Similarly, we would appreciate your contributions to the design of
KIP-1163, and your vote on the motivational KIP-1150.

Thanks!

On Mon, Jan 19, 2026 at 10:18 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi Jun,
>
> Here are our answers to your questions and comments:
>
> > JR10. "Even in the case where network traffic is not accounted; we
> believe
> > that the operational benefits of Diskless topics are still appealing to
> > Kafka users."
> > Could you list other benefits such as better scalability and durability?
>
> Thank you, good point, added this.
>
> > JR11. "AWS charges 0.02$ per GiB"
> > The object store is not free and has a different cost model based on
> > requests. It would be useful summarize the tradeoff on cost and latency,
> > and when diskless topics are cost effective.
>
> Yes, there is a per-request cost for object storage which is not directly
> comparable to inter-zone transfer costs. We have found that the design in
> KIP-1163 amortizes those request costs such that they are insignificant
> compared to the $0.02/gb cost which is called out in KIP-1150.
> We currently have very precise cost models that were put together for an
> earlier iteration of the Diskless topics. These are somewhat stale and
> inaccurate for the latest design, so we will work in the background to post
> concrete numbers in KIP-1163 before voting on that proposal. Would this be
> OK?
>
> > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> been
> > updated in the last 6 months. The metadata change in KIP-1181 is now
> > included in KIP-1163.
>
> Thanks for calling out. We have updated KIP-1164 with the latest design.
> It's missing public interfaces, but we think it may be a bit too early to
> define them as some rounds of discussion is ahead, with potential to change
> the design. But otherwise the KIP is good for the first review.
> We have deprecated KIP-1181 as it's now incorporated into KIP-1163.
>
> Best,
> Ivan
>
> On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> > Hi, Anatolii,
> >
> > Thanks for the updated KIP. Looks good to me overall. A few more
> comments.
> >
> > JR10. "Even in the case where network traffic is not accounted; we
> believe
> > that the operational benefits of Diskless topics are still appealing to
> > Kafka users."
> > Could you list other benefits such as better scalability and durability?
> >
> > JR11. "AWS charges 0.02$ per GiB"
> > The object store is not free and has a different cost model based on
> > requests. It would be useful summarize the tradeoff on cost and latency,
> > and when diskless topics are cost effective.
> >
> > JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't
> been
> > updated in the last 6 months. The metadata change in KIP-1181 is now
> > included in KIP-1163.
> >
> > Jun
> >
> > On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> dev@kafka.apache.org>
> > wrote:
> >
> > > Hi all,
> > >
> > > Thank you for the extensive feedback.
> > >
> > > We have substantially updated the KIP to address the points raised.
> Given
> > > the scope of these changes, we have started a new VOTE thread to
> restart
> > > the voting process cleanly.
> > >
> > > You can find the new thread here:
> > > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> > >
> > > Best,
> > > Anatolii
> > >
> > > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > > dev@kafka.apache.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We (the team at Slack) have been following the recent discussion
> > > regarding
> > > > the scope and timeline of KIP-1150. We agree with the community
> sentiment
> > > > that Diskless Topics represents the right long-term architecture for
> > > Kafka
> > > > in the cloud, but we also recognize the valid concerns raised
> regarding
> > > the
> > > > engineering resources required to deliver such an ambitious change.
> > > >
> > > > To help address these concerns and accelerate the timeline, we are
> happy
> > > to
> > > > announce that we are partnering with the KIP-1150 authors to
> co-develop
> > > > this feature.
> > > >
> > > > We previously proposed KIP-1176: Tiered Storage for Active Log
> Segments
> > > to
> > > > solve similar problems. However, rather than fragmenting the
> community's
> > > > efforts across competing designs, we have decided to withdraw
> KIP-1176
> > > and
> > > > consolidate our engineering resources behind KIP-1150.
> > > >
> > > > To start, we plan to take ownership of Compaction for Tiered Storage.
> > > This
> > > > has long been a missing feature in KIP-405, and it becomes critical
> in a
> > > > Diskless architecture where long-lived data is obligated to be
> tiered. By
> > > > driving this related prerequisite feature, we hope to allow for
> faster
> > > > delivery of KIP-1150.
> > > >
> > > > We are excited to collaborate on this to ensure a robust and timely
> > > > delivery for the community.
> > > >
> > > > Best,
> > > > Tom & Henry
> > > >
> > > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com> wrote:
> > > >
> > > > > Hi Greg,
> > > > >
> > > > > Thanks for sharing the meeting notes.
> > > > > I agree we should keep polishing the contents of 1150 & high level
> > > design
> > > > > in 1163 to prepare for a vote.
> > > > >
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > > <greg.harris@aiven.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > There was a video call between myself, Ivan Yurchenko, Jun Rao,
> and
> > > > > Andrew
> > > > > > Schofield pertaining to KIP-1150. Here are the notes from that
> > > meeting:
> > > > > >
> > > > > > Ivan: What is the future state of Kafka in this area, in 5 years?
> > > > > > Jun: Do we want something more cloud native? Yes, started with
> Tiered
> > > > > > Storage. If there’s a better way, we should explore it. In the
> long
> > > > term
> > > > > > this will be useful
> > > > > > Because Kafka is used so widely, we need to make sure everything
> we
> > > add
> > > > > is
> > > > > > for the long term and for everyone, not just for a single
> company.
> > > > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > > > something
> > > > > > that’s high quality/lasts/maintainable, and can work with all
> > > existing
> > > > > > capabilities.
> > > > > > If both 1150 and 1176 proceed at the same time, it’s confusing.
> They
> > > > > > overlap, but Diskless is more ambitious.
> > > > > > If both KIPs are being seriously worked on, then we don’t really
> need
> > > > > both,
> > > > > > because Diskless clearly is better. Having multiple will confuse
> > > > people.
> > > > > It
> > > > > > will duplicate some of the effort.
> > > > > > If we want diskless ultimately, what is the short term strategy,
> to
> > > get
> > > > > > some early wins first?
> > > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > > Andrew: Eventually the architecture will change substantially,
> it may
> > > > not
> > > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > > Greg: We all agree on having a high quality feature merged
> upstream,
> > > > and
> > > > > > supporting all APIs
> > > > > > Jun: We should try and keep things simple, but there is some
> minimum
> > > > > > complexity needed.
> > > > > > When doing the short term changes (1176), it doesn’t really
> progress
> > > in
> > > > > > changing to a more modern architecture.
> > > > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > > > Jun: The danger of only applying changes to some part of the
> API, you
> > > > set
> > > > > > the precedence that you only have to implement part of the API.
> > > > > Supporting
> > > > > > the full API set should be a minimum requirement.
> > > > > > Andrew: When we started Kraft, how much did we know the design?
> > > > > > Jun: For Kraft we didn’t really know much about the migration,
> but
> > > the
> > > > > > high-level was clear.
> > > > > > Greg: Is 1150 votable in its current state?
> > > > > > Jun: 1150 should promise to support all APIs. It doesn’t have to
> have
> > > > all
> > > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > > We do need some high-level design enough to give confidence that
> the
> > > > > > promise is able to be fulfilled.
> > > > > > Greg: Is the draft version in 1163 enough detail or is more
> needed?
> > > > > > Jun: We need to agree on the core design, such as leaderless
> etc. And
> > > > how
> > > > > > the APIs will be supported.
> > > > > > Greg: Okay we can include these things, and provide a sketch of
> how
> > > the
> > > > > > other leader-based features operate.
> > > > > > Jun: Yeah if at a high level the sketch appears to work, we can
> > > approve
> > > > > > that functionality.
> > > > > > Are you committed to doing the more involved and big project?
> > > > > > Greg: Yes, we’re committed to the 1163 design and can’t really
> accept
> > > > > 1176.
> > > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > > Greg: We’ll push internally for resources, and use the community
> > > > > sentiment
> > > > > > to motivate Aiven.
> > > > > > How far into the future should we look? What sort of scale?
> > > > > > Jun: As long as there’s a path forward, and we’re not closing off
> > > > future
> > > > > > improvements, we can figure out how to handle a larger scale
> when it
> > > > > > arises.
> > > > > > Greg: Random replica placement is very harmful, can we recommend
> > > users
> > > > to
> > > > > > use an external tool like CruiseControl?
> > > > > > Jun: Not everyone uses CruiseControl, we would probably need some
> > > > > solution
> > > > > > for this out of the box
> > > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > > Jun: Out-of-box experience should be good, good to allow other
> > > > > > implementations
> > > > > > Greg: But it could hurt Kafka feature/upgrade velocity when we
> wait
> > > for
> > > > > > plugin providers to implement it
> > > > > > Ivan: We imagined that maybe cloud hyperscalers could implement
> it
> > > with
> > > > > > e.g. dynamodb
> > > > > > Greg: Could we bake more details of the different providers into
> > > Kafka,
> > > > > or
> > > > > > does it still make sense for it to be pluggable?
> > > > > > Jun: Make it whatever is easiest to roll out and add new clients
> > > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > > Ivan: The vote is already open, we’re not too pressed for time.
> We’ll
> > > > go
> > > > > > improve the 1163 design and communication.
> > > > > > Is 1176 a competing design? Someone will ask.
> > > > > > Jun: If we are seriously working on something more ambitious,
> yeah we
> > > > > > shouldn’t do the stop-gap solution.
> > > > > > It’s diverting review resources. If we can get the short term
> thing
> > > in
> > > > > 1yr
> > > > > > but Diskless solution is 2y it makes sense to go for Diskless. If
> > > it’s
> > > > > 5yr,
> > > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should
> we
> > > > > > explicitly exclude 1176?
> > > > > > Andrew: Put your arms around the feature set you actually want,
> and
> > > use
> > > > > > that to rule out 1176.
> > > > > > Probably don’t need -1 votes, most likely KIPs just don’t receive
> > > > votes.
> > > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > > Jun: Satish posted meeting notes regularly, we should do the
> same.
> > > > > >
> > > > > > To summarize, we will be polishing the contents of 1150 & high
> level
> > > > > design
> > > > > > in 1163 to prepare for a vote.
> > > > > > We believe that the community should select the feature set of
> 1150
> > > to
> > > > > > fully eliminate producer cross-zone costs, and make the
> investment
> > > in a
> > > > > > high quality Diskless Topics implementation rather than in
> stop-gap
> > > > > > solutions.
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:
> > > > > >
> > > > > > > This may be a tangent, but we needed to offload storage off of
> > > Kafka
> > > > > into
> > > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> mostly
> > > > > > > ephemeral broker that can come and go as it pleases. Be that
> > > scaling
> > > > or
> > > > > > > outage. Disks can be destroyed and recreated at will, we still
> > > retain
> > > > > > data
> > > > > > > and use broker for just that, brokering messages. Not only
> that, we
> > > > > > reduced
> > > > > > > the requirement on the actual Kafka resources by reducing the
> size
> > > > of a
> > > > > > > payload via a claim check pattern. Maybe this is an
> anti–pattern,
> > > but
> > > > > it
> > > > > > is
> > > > > > > super fast and highly cost efficient. We reworked
> ProducerRequest
> > > to
> > > > > > allow
> > > > > > > plugins. We added a custom http plugin that submits every
> request
> > > > via a
> > > > > > > persisted connection to a microservice. Microservice stores the
> > > > payload
> > > > > > and
> > > > > > > returns a tiny json metadata object,a claim check, that can be
> used
> > > > to
> > > > > > find
> > > > > > > the actual data. Think of it as zipping the payload. This claim
> > > check
> > > > > > > metadata traverses the pipelines with consumers using the urls
> in
> > > > > > metadata
> > > > > > > to pull what they need. Think unzipping. This allowed us to
> also
> > > pull
> > > > > > ONLY
> > > > > > > the data that we need in graphql like manner. So if you have a
> 100K
> > > > > json
> > > > > > > payload and you need only a subsection, you can pull that by
> > > > jmespath.
> > > > > > When
> > > > > > > you have multiple consumer groups yanking down huge payloads
> it is
> > > > > > > cumbersome on the broker. When you have the same consumer
> groups
> > > > > yanking
> > > > > > > down a claim check, and then going out of band directly to the
> > > source
> > > > > of
> > > > > > > truth, the broker has some breathing room. Obviously our
> > > microservice
> > > > > > does
> > > > > > > not go directly to the cloud storage, as that would be too
> slow. It
> > > > > > stores
> > > > > > > the payload in high speed memory cache and returns asap. That
> > > memory
> > > > is
> > > > > > > eventually persisted into S3. The retrieval goest against the
> cache
> > > > > > first,
> > > > > > > then against the S3. Overall a rather cheappy and zippy
> solution. I
> > > > > tried
> > > > > > > proposing the KIP for this, but there was no excitement. Check
> this
> > > > > out:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > > >
> > > > > > >
> > > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao <jun@confluent.io.INVALID
> >
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi, Andrew,
> > > > > > > >
> > > > > > > > If we want to focus only on reducing cross-zone replication
> > > costs,
> > > > > > there
> > > > > > > is
> > > > > > > > an alternative design in the KIP-1176 discussion thread that
> > > seems
> > > > > > > simpler
> > > > > > > > than the proposal here. I am copying the outline of that
> approach
> > > > > > below.
> > > > > > > >
> > > > > > > > 1. A new leader is elected.
> > > > > > > > 2. Leader maintains a first tiered offset, which is
> initialized
> > > to
> > > > > log
> > > > > > > end
> > > > > > > > offset.
> > > > > > > > 3. Leader writes produced data from the client to local log.
> > > > > > > > 4. Leader uploads produced data from all local logs as a
> combined
> > > > > > object
> > > > > > > > 5. Leader stores the metadata for the combined object in
> memory.
> > > > > > > > 6. If a follower fetch request has an offset >= first tiered
> > > > offset,
> > > > > > the
> > > > > > > > metadata for the corresponding combined object is returned.
> > > > > Otherwise,
> > > > > > > the
> > > > > > > > local data is returned.
> > > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > > >
> > > > > > > > It's still a bit unnatural, but it could work.
> > > > > > > >
> > > > > > > > Hi, Ivan,
> > > > > > > >
> > > > > > > > Are you still committed to proceeding with the original
> design of
> > > > > > > KIP-1150?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > > andrew_schofield@live.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >> I’ve been following KIP-1150 and friends for a while. I’m
> going
> > > to
> > > > > > jump
> > > > > > > >> into the discussions too.
> > > > > > > >>
> > > > > > > >> Looking back at Jack Vanlightly’s message, I am not quite so
> > > > > convinced
> > > > > > > >> that it’s a kind of fork in the road. The primary aim of the
> > > > effort
> > > > > is
> > > > > > > to
> > > > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > > > prohibitively
> > > > > > > >> expensive to use on cloud storage. I think it would be
> entirely
> > > > > viable
> > > > > > > to
> > > > > > > >> prioritise code reuse for an initial implementation of
> diskless
> > > > > > topics,
> > > > > > > and
> > > > > > > >> we could still have a more cloud-native design in the
> future.
> > > It’s
> > > > > > hard
> > > > > > > to
> > > > > > > >> predict what the community will prioritise in the future.
> > > > > > > >>
> > > > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> > > support
> > > > > > > >> leaderless produce requests, first writing WAL segments into
> > > > object
> > > > > > > >> storage, and then using the regular partition leaders to
> > > sequence
> > > > > the
> > > > > > > >> records. The active log segment for a diskless topic will
> > > > initially
> > > > > > > contain
> > > > > > > >> batch coordinates rather than record batches. The batch
> > > > coordinates
> > > > > > can
> > > > > > > be
> > > > > > > >> resolved from WAL segments for consumers, and also in order
> to
> > > > > prepare
> > > > > > > log
> > > > > > > >> segments for uploading to tiered storage. Jun is probably
> > > correct
> > > > > that
> > > > > > > we
> > > > > > > >> need a more frequent object merging process than tiered
> storage
> > > > > > > provides.
> > > > > > > >> This is just the transition from write-optimised WAL
> segments to
> > > > > > > >> read-optimised tiered segments, and all of the object
> > > > storage-based
> > > > > > > >> implementations of Kafka that I’m aware of do this
> > > rearrangement.
> > > > > But
> > > > > > > >> perhaps this more frequent object merging is a pre-GA
> > > improvement,
> > > > > > > rather
> > > > > > > >> than a strict requirement for an initial implementation for
> > > early
> > > > > > access
> > > > > > > >> use.
> > > > > > > >>
> > > > > > > >> For zone-aligned share consumers, the share group assignor
> is
> > > > > intended
> > > > > > > to
> > > > > > > >> be rack-aware. Consumers should be assigned to partitions
> with
> > > > > leaders
> > > > > > > in
> > > > > > > >> their zone. The simple assignor is not rack-aware, but it
> easily
> > > > > could
> > > > > > > be
> > > > > > > >> or we could have a rack-aware assignor.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Andrew
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao <jun@confluent.io.INVALID
> >
> > > > > wrote:
> > > > > > > >>>
> > > > > > > >>> Hi, Ivan,
> > > > > > > >>>
> > > > > > > >>> Thanks for the reply.
> > > > > > > >>>
> > > > > > > >>> "As I understand, you’re speaking about locally
> materialized
> > > > > > segments.
> > > > > > > >> They
> > > > > > > >>> will indeed consume some IOPS. See them as a cache that
> could
> > > > > always
> > > > > > be
> > > > > > > >>> restored from the remote storage. While it’s not ideal,
> it's
> > > > still
> > > > > OK
> > > > > > > to
> > > > > > > >>> lose data in them due to a machine crash, for example.
> Because
> > > of
> > > > > > this,
> > > > > > > >> we
> > > > > > > >>> can avoid explicit flushing on local materialized segments
> at
> > > all
> > > > > and
> > > > > > > let
> > > > > > > >>> the file system and page cache figure out when to flush
> > > > optimally.
> > > > > > This
> > > > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > > > dramatically,
> > > > > > > >>> depending on throughput for each partition. We, of course,
> > > > continue
> > > > > > > >>> flushing the metadata segments as before."
> > > > > > > >>>
> > > > > > > >>> If we have a mix of classic and diskless topics on the same
> > > > broker,
> > > > > > > it's
> > > > > > > >>> important that the classic topics' data is flushed to disk
> as
> > > > > quickly
> > > > > > > as
> > > > > > > >>> possible. To achieve this, users typically set
> > > > > dirty_expire_centisecs
> > > > > > > in
> > > > > > > >>> the kernel based on the number of available disk IOPS.
> Once you
> > > > set
> > > > > > > this
> > > > > > > >>> number, it applies to all dirty files, including the cached
> > > data
> > > > in
> > > > > > > >>> diskless topics. So, if there are more files actively
> > > > accumulating
> > > > > > > data,
> > > > > > > >>> the flush frequency and therefore RPO is reduced for
> classic
> > > > > topics.
> > > > > > > >>>
> > > > > > > >>> "We should have mentioned this explicitly, but this step,
> in
> > > > fact,
> > > > > > > >> remains
> > > > > > > >>> in the form of segments offloading to tiered storage. When
> we
> > > > > > assemble
> > > > > > > a
> > > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> effectively
> > > > > doing
> > > > > > > >>> metadata compaction: replacing a big number of pieces of
> > > metadata
> > > > > > about
> > > > > > > >>> individual batches with a single record in
> > > > __remote_log_metadata."
> > > > > > > >>>
> > > > > > > >>> The object merging in tier storage typically only kicks in
> > > after
> > > > a
> > > > > > few
> > > > > > > >>> hours. The impact is (1) the amount of accumulated
> metadata is
> > > > > still
> > > > > > > >> quite
> > > > > > > >>> large; (2) there are many small objects, leading to poor
> read
> > > > > > > >> performance.
> > > > > > > >>> I think we need a more frequent object merging process than
> > > tier
> > > > > > > storage
> > > > > > > >>> provides.
> > > > > > > >>>
> > > > > > > >>> Jun
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > > ivan@ivanyu.me>
> > > > > > > wrote:
> > > > > > > >>>
> > > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > > >>>>
> > > > > > > >>>> Thank you for your messages.
> > > > > > > >>>>
> > > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > > >>>>
> > > > > > > >>>>> First, it degrades the durability.
> > > > > > > >>>>> For each partition, now there are two files being
> actively
> > > > > written
> > > > > > > at a
> > > > > > > >>>>> given point of time, one for the data and another for the
> > > > > metadata.
> > > > > > > >>>>> Flushing each file requires a separate IO. If the disk
> has 1K
> > > > > IOPS
> > > > > > > and
> > > > > > > >> we
> > > > > > > >>>>> have 5K partitions in a broker, currently we can afford
> to
> > > > flush
> > > > > > each
> > > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5
> seconds. If
> > > we
> > > > > > > double
> > > > > > > >>>> the
> > > > > > > >>>>> number of files per partition, we can only flush each
> > > partition
> > > > > > every
> > > > > > > >> 10
> > > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > > >>>>
> > > > > > > >>>> As I understand, you’re speaking about locally
> materialized
> > > > > > segments.
> > > > > > > >> They
> > > > > > > >>>> will indeed consume some IOPS. See them as a cache that
> could
> > > > > always
> > > > > > > be
> > > > > > > >>>> restored from the remote storage. While it’s not ideal,
> it's
> > > > still
> > > > > > OK
> > > > > > > to
> > > > > > > >>>> lose data in them due to a machine crash, for example.
> Because
> > > > of
> > > > > > > this,
> > > > > > > >> we
> > > > > > > >>>> can avoid explicit flushing on local materialized
> segments at
> > > > all
> > > > > > and
> > > > > > > >> let
> > > > > > > >>>> the file system and page cache figure out when to flush
> > > > optimally.
> > > > > > > This
> > > > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > > > dramatically,
> > > > > > > >>>> depending on throughput for each partition. We, of course,
> > > > > continue
> > > > > > > >>>> flushing the metadata segments as before.
> > > > > > > >>>>
> > > > > > > >>>> It’s worth making a note on caching. I think nobody will
> > > > disagree
> > > > > > that
> > > > > > > >>>> doing direct reads from remote storage every time a batch
> is
> > > > > > requested
> > > > > > > >> by a
> > > > > > > >>>> consumer will not be practical neither from the
> performance
> > > nor
> > > > > from
> > > > > > > the
> > > > > > > >>>> economy point of view. We need a way to keep the number
> of GET
> > > > > > > requests
> > > > > > > >>>> down. There are multiple options, for example:
> > > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > > >>>> 2. Local in-memory caching. Comes with less network
> chattiness
> > > > and
> > > > > > > >> works
> > > > > > > >>>> well if we have more or less stable brokers to consume
> from.
> > > > > > > >>>> 3. Materialization of diskless logs on local disk. Way
> lower
> > > > > impact
> > > > > > on
> > > > > > > >>>> RAM and also requires stable brokers for consumption
> (using
> > > just
> > > > > > > >> assigned
> > > > > > > >>>> replicas will probably work well).
> > > > > > > >>>>
> > > > > > > >>>> Materialization is one of possible options, but we can
> choose
> > > > > > another
> > > > > > > >> one.
> > > > > > > >>>> However, we will have this dilemma regardless of whether
> we
> > > have
> > > > > an
> > > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > > >>>>
> > > > > > > >>>>> Second, if we ever need this
> > > > > > > >>>>> metadata somewhere else, say in the WAL file manager, the
> > > > > consumer
> > > > > > > >> needs
> > > > > > > >>>> to
> > > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > > inefficient.
> > > > > > > The
> > > > > > > >>>>> actual benefit of this approach is also questionable. On
> the
> > > > > > surface,
> > > > > > > >> it
> > > > > > > >>>>> might seem that we could reduce the number of lines that
> need
> > > > to
> > > > > be
> > > > > > > >>>> changed
> > > > > > > >>>>> for this KIP. However, the changes are quite intrusive
> to the
> > > > > > classic
> > > > > > > >>>>> partition's code path and will probably make the code
> base
> > > > harder
> > > > > > to
> > > > > > > >>>>> maintain in the long run. I like the original approach
> based
> > > on
> > > > > the
> > > > > > > >> batch
> > > > > > > >>>>> coordinator much better than this one. We could probably
> > > > refactor
> > > > > > the
> > > > > > > >>>>> producer state code so that it could be reused in the
> batch
> > > > > > > >> coordinator.
> > > > > > > >>>>
> > > > > > > >>>> It’s hard to disagree with this. The explicit coordinator
> is
> > > > more
> > > > > a
> > > > > > > side
> > > > > > > >>>> thing, while coordinator-less approach is more about
> extending
> > > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > > >>>>
> > > > > > > >>>>> Thanks for addressing the concerns on the number of RPCs
> in
> > > the
> > > > > > > produce
> > > > > > > >>>>> path. I agree that with the metadata crafting mechanism,
> we
> > > > could
> > > > > > > >>>> mitigate
> > > > > > > >>>>> the PRC problem. However, since we now require the
> metadata
> > > to
> > > > be
> > > > > > > >>>>> collocated with the data on the same set of brokers, it's
> > > weird
> > > > > > that
> > > > > > > >> they
> > > > > > > >>>>> are now managed by different mechanisms. The data
> assignment
> > > > now
> > > > > > uses
> > > > > > > >> the
> > > > > > > >>>>> metadata crafting mechanism, but the metadata is stored
> in
> > > the
> > > > > > > classic
> > > > > > > >>>>> partition using its own assignment strategy. It will be
> > > > > complicated
> > > > > > > to
> > > > > > > >>>> keep
> > > > > > > >>>>> them collocated.
> > > > > > > >>>>
> > > > > > > >>>> I would like to note that the metadata crafting is needed
> only
> > > > to
> > > > > > tell
> > > > > > > >>>> producers which brokers they should send Produce requests
> to,
> > > > but
> > > > > > data
> > > > > > > >> (as
> > > > > > > >>>> in “locally materialized log”) is located on partition
> > > replicas,
> > > > > > i.e.
> > > > > > > >>>> automatically co-located with metadata.
> > > > > > > >>>>
> > > > > > > >>>> As a side note, it would probably be better that instead
> of
> > > > > > implicitly
> > > > > > > >>>> crafting partition metadata, we extend the metadata
> protocol
> > > so
> > > > > that
> > > > > > > for
> > > > > > > >>>> diskless partitions we return not only the leader and
> > > replicas,
> > > > > but
> > > > > > > also
> > > > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > > > performance
> > > > > > > and
> > > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > > >>>>
> > > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > > compaction/merging
> > > > > > > >>>>> step.
> > > > > > > >>>>
> > > > > > > >>>> We should have mentioned this explicitly, but this step,
> in
> > > > fact,
> > > > > > > >> remains
> > > > > > > >>>> in the form of segments offloading to tiered storage.
> When we
> > > > > > > assemble a
> > > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > > effectively
> > > > > > doing
> > > > > > > >>>> metadata compaction: replacing a big number of pieces of
> > > > metadata
> > > > > > > about
> > > > > > > >>>> individual batches with a single record in
> > > > __remote_log_metadata.
> > > > > > > >>>>
> > > > > > > >>>> We could create a Diskless-specific merging mechanism
> instead
> > > if
> > > > > > > needed.
> > > > > > > >>>> It’s rather easy with the explicit coordinator approach.
> With
> > > > the
> > > > > > > >>>> coordinator-less approach, this would probably be a bit
> more
> > > > > tricky
> > > > > > > >>>> (rewriting the tail of the log by the leader + replicating
> > > this
> > > > > > change
> > > > > > > >>>> reliably).
> > > > > > > >>>>
> > > > > > > >>>>> I see a tendency toward primarily optimizing for the
> fewest
> > > > code
> > > > > > > >> changes
> > > > > > > >>>> in
> > > > > > > >>>>> the KIP. Instead, our primary goal should be a clean
> design
> > > > that
> > > > > > can
> > > > > > > >> last
> > > > > > > >>>>> for the long term.
> > > > > > > >>>>
> > > > > > > >>>> Yes, totally agree.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> Luke,
> > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> queue is
> > > > > > because
> > > > > > > >> of
> > > > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> > > only
> > > > > > focus
> > > > > > > on
> > > > > > > >>>> the
> > > > > > > >>>>> "diskless" design to handle object compaction/merging
> to/from
> > > > the
> > > > > > > >> remote
> > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > >>>>
> > > > > > > >>>> After some evolution of the original proposal, leaderless
> is
> > > now
> > > > > > > >> limited.
> > > > > > > >>>> We only need to be able to accept Produce requests on more
> > > than
> > > > > one
> > > > > > > >> broker
> > > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get it
> > > right
> > > > > > that
> > > > > > > >> you
> > > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> Let’s now look at this problem from a higher level, as
> Jack
> > > > > > proposed.
> > > > > > > As
> > > > > > > >>>> it was said, the big choice we need to make is whether we
> 1)
> > > > > create
> > > > > > an
> > > > > > > >>>> explicit batch coordinator; or 2) go for the
> coordinator-less
> > > > > > > approach,
> > > > > > > >>>> where each diskless partition is managed by its leader as
> in
> > > > > classic
> > > > > > > >> topics.
> > > > > > > >>>>
> > > > > > > >>>> If we try to compare the two approaches:
> > > > > > > >>>>
> > > > > > > >>>> Pluggability:
> > > > > > > >>>> - Explicit coordinator: Possible. For example, some
> setups may
> > > > > > benefit
> > > > > > > >>>> from batch metadata being stored in a cloud database
> (such as
> > > > AWS
> > > > > > > >> DynamoDB
> > > > > > > >>>> or GCP Spanner).
> > > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > > >>>>
> > > > > > > >>>> Scalability and fault tolerance:
> > > > > > > >>>> - Explicit coordinator: Depends on the implementation and
> it’s
> > > > > also
> > > > > > > >>>> necessary to actively work for it.
> > > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics.
> Scaling is
> > > > > done
> > > > > > by
> > > > > > > >>>> partition placement, partitions could fail independently.
> > > > > > > >>>>
> > > > > > > >>>> Separation of concerns:
> > > > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > > > independent
> > > > > > > >> from
> > > > > > > >>>> classic topics in terms of code and workflows. For
> example,
> > > the
> > > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > > mechanism
> > > > > > could
> > > > > > > >> be
> > > > > > > >>>> relatively simply implemented with it. As a flip side of
> this,
> > > > > some
> > > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > > bringing
> > > > > > > diskless
> > > > > > > >>>> closer to classic topics. Some code paths and workflows
> could
> > > be
> > > > > > more
> > > > > > > >>>> straightforwardly reused, but they will inevitably have
> to be
> > > > > > adapted
> > > > > > > to
> > > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > > >>>>
> > > > > > > >>>> Cloud-nativeness. This is a vague concept, also related
> to the
> > > > > > > previous,
> > > > > > > >>>> but let’s try:
> > > > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > > > separately
> > > > > > > makes
> > > > > > > >>>> it easier for brokers to take different roles, be purely
> > > > stateless
> > > > > > if
> > > > > > > >>>> needed, etc.
> > > > > > > >>>> - Coordinator-less: Less so. Something could be achieved
> with
> > > > > > creative
> > > > > > > >>>> partition placement, but not much.
> > > > > > > >>>>
> > > > > > > >>>> Both seem to have their pros and cons. However, answering
> > > Jack’s
> > > > > > > >> question,
> > > > > > > >>>> the explicit coordinator approach may indeed lead to a
> more
> > > > > flexible
> > > > > > > >> design.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> The purpose of this deviation in the discussion was to
> > > receive a
> > > > > > > >>>> preliminary community evaluation of the coordinator-less
> > > > approach
> > > > > > > >> without
> > > > > > > >>>> taking on the task of writing a separate KIP and fitting
> it in
> > > > the
> > > > > > > >> system
> > > > > > > >>>> of KIP-1150 and its children. We’re open to stopping it
> and
> > > > > getting
> > > > > > > >> back to
> > > > > > > >>>> working out the coordinator design if the community
> doesn’t
> > > > favor
> > > > > > the
> > > > > > > >>>> proposed approach.
> > > > > > > >>>>
> > > > > > > >>>> Best,
> > > > > > > >>>> Ivan and Diskless team
> > > > > > > >>>>
> > > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > > >>>>> Hi Ivan,
> > > > > > > >>>>>
> > > > > > > >>>>> As Jun pointed out, the updated design seems to have some
> > > > > > > shortcomings
> > > > > > > >>>>> although it simplifies the implementation.
> > > > > > > >>>>>
> > > > > > > >>>>> I'm wondering if the complexity of designing txn and
> queue is
> > > > > > because
> > > > > > > >> of
> > > > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> > > only
> > > > > > focus
> > > > > > > on
> > > > > > > >>>> the
> > > > > > > >>>>> "diskless" design to handle object compaction/merging
> to/from
> > > > the
> > > > > > > >> remote
> > > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> Thank you,
> > > > > > > >>>>> Luke
> > > > > > > >>>>>
> > > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > > <jun@confluent.io.invalid
> > > > > >
> > > > > > > >>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Hi, Ivan,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks for the explanation.
> > > > > > > >>>>>>
> > > > > > > >>>>>> "we write the reference to the WAL file with the batch
> data"
> > > > > > > >>>>>>
> > > > > > > >>>>>> I understand the approach now, but I think it is a hacky
> > > one.
> > > > > > There
> > > > > > > >> are
> > > > > > > >>>>>> multiple short comings with this design. First, it
> degrades
> > > > the
> > > > > > > >>>> durability.
> > > > > > > >>>>>> For each partition, now there are two files being
> actively
> > > > > written
> > > > > > > at
> > > > > > > >> a
> > > > > > > >>>>>> given point of time, one for the data and another for
> the
> > > > > > metadata.
> > > > > > > >>>>>> Flushing each file requires a separate IO. If the disk
> has
> > > 1K
> > > > > IOPS
> > > > > > > and
> > > > > > > >>>> we
> > > > > > > >>>>>> have 5K partitions in a broker, currently we can afford
> to
> > > > flush
> > > > > > > each
> > > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5
> seconds. If
> > > > we
> > > > > > > double
> > > > > > > >>>> the
> > > > > > > >>>>>> number of files per partition, we can only flush each
> > > > partition
> > > > > > > every
> > > > > > > >>>> 10
> > > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we
> ever
> > > need
> > > > > > this
> > > > > > > >>>>>> metadata somewhere else, say in the WAL file manager,
> the
> > > > > consumer
> > > > > > > >>>> needs to
> > > > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > > > inefficient.
> > > > > > > The
> > > > > > > >>>>>> actual benefit of this approach is also questionable.
> On the
> > > > > > > surface,
> > > > > > > >>>> it
> > > > > > > >>>>>> might seem that we could reduce the number of lines that
> > > need
> > > > to
> > > > > > be
> > > > > > > >>>> changed
> > > > > > > >>>>>> for this KIP. However, the changes are quite intrusive
> to
> > > the
> > > > > > > classic
> > > > > > > >>>>>> partition's code path and will probably make the code
> base
> > > > > harder
> > > > > > to
> > > > > > > >>>>>> maintain in the long run. I like the original approach
> based
> > > > on
> > > > > > the
> > > > > > > >>>> batch
> > > > > > > >>>>>> coordinator much better than this one. We could probably
> > > > > refactor
> > > > > > > the
> > > > > > > >>>>>> producer state code so that it could be reused in the
> batch
> > > > > > > >>>> coordinator.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks for addressing the concerns on the number of
> RPCs in
> > > > the
> > > > > > > >> produce
> > > > > > > >>>>>> path. I agree that with the metadata crafting
> mechanism, we
> > > > > could
> > > > > > > >>>> mitigate
> > > > > > > >>>>>> the PRC problem. However, since we now require the
> metadata
> > > to
> > > > > be
> > > > > > > >>>>>> collocated with the data on the same set of brokers,
> it's
> > > > weird
> > > > > > that
> > > > > > > >>>> they
> > > > > > > >>>>>> are now managed by different mechanisms. The data
> assignment
> > > > now
> > > > > > > uses
> > > > > > > >>>> the
> > > > > > > >>>>>> metadata crafting mechanism, but the metadata is stored
> in
> > > the
> > > > > > > classic
> > > > > > > >>>>>> partition using its own assignment strategy. It will be
> > > > > > complicated
> > > > > > > to
> > > > > > > >>>> keep
> > > > > > > >>>>>> them collocated.
> > > > > > > >>>>>>
> > > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > > compaction/merging
> > > > > > > >>>>>> step. My first concern is on the amount of metadata that
> > > need
> > > > to
> > > > > > be
> > > > > > > >>>> kept.
> > > > > > > >>>>>> Without object compcation, the metadata generated in the
> > > > produce
> > > > > > > path
> > > > > > > >>>> can
> > > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's
> say for
> > > > > every
> > > > > > > >>>> 250ms we
> > > > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> > > remoting
> > > > > > > tiering
> > > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K
> partitions,
> > > we
> > > > > need
> > > > > > > to
> > > > > > > >>>> keep
> > > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> metadata,
> > > > > quite
> > > > > > > >>>>>> signficant. A second concern is on performance. Every
> time
> > > we
> > > > > need
> > > > > > > to
> > > > > > > >>>>>> rebuild the caching data, we need to read a bunch of
> small
> > > > > objects
> > > > > > > >>>> from S3,
> > > > > > > >>>>>> slowing down the building process. If a consumer
> happens to
> > > > need
> > > > > > > such
> > > > > > > >>>> data,
> > > > > > > >>>>>> it could slow down the application.
> > > > > > > >>>>>>
> > > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> fewest
> > > > code
> > > > > > > >>>> changes in
> > > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> design
> > > > that
> > > > > > can
> > > > > > > >>>> last
> > > > > > > >>>>>> for the long term.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Jun
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > > ivan@ivanyu.me
> > > > > >
> > > > > > > >>>> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Hi Jun,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed to
> > > > clearly
> > > > > > > >>>> explain
> > > > > > > >>>>>> the
> > > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Does each partition now have a metadata partition and
> a
> > > > > separate
> > > > > > > >>>> data
> > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > doubles
> > > > > the
> > > > > > > >>>> number
> > > > > > > >>>>>>> of
> > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > descriptors
> > > > > > and
> > > > > > > >>>> the
> > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> have a
> > > > > > > separate
> > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> are
> > > > > creating
> > > > > > > an
> > > > > > > >>>>>>>> internal topic with an unbounded number of partitions.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> No. There will be only one physical partition per
> diskless
> > > > > > > >>>> partition. Let
> > > > > > > >>>>>>> me explain this with an example. Let’s say we have a
> > > diskless
> > > > > > > >>>> partition
> > > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the
> leader. We
> > > > > > produce
> > > > > > > >>>> some
> > > > > > > >>>>>>> batches to this partition. The content of the segment
> file
> > > > will
> > > > > > be
> > > > > > > >>>>>>> something like this (for each batch):
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > > >>>>>>> Magic: like in classic
> > > > > > > >>>>>>> CRC: like in classic
> > > > > > > >>>>>>> Attributes: like in classic
> > > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > > >>>>>>> ProducerId: like in classic
> > > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > > >>>>>>> Records:
> > > > > > > >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > > byte
> > > > > > offset
> > > > > > > >>>>>>> 123456
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> It looks very much like classic log entries. The only
> > > > > difference
> > > > > > is
> > > > > > > >>>> that
> > > > > > > >>>>>>> instead of writing real Records, we write the
> reference to
> > > > the
> > > > > > WAL
> > > > > > > >>>> file
> > > > > > > >>>>>>> with the batch data (I guess we need only the name and
> the
> > > > byte
> > > > > > > >>>> offset,
> > > > > > > >>>>>>> because the byte length is the standard field above).
> > > > > Otherwise,
> > > > > > > >>>> it’s a
> > > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> So we have as many partitions for diskless as for
> classic.
> > > As
> > > > > of
> > > > > > > open
> > > > > > > >>>>>> file
> > > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Are the metadata and
> > > > > > > >>>>>>>> the data for the same partition always collocated on
> the
> > > > same
> > > > > > > >>>> broker?
> > > > > > > >>>>>> If
> > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> reassigned?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> The source of truth for the data is still in WAL files
> on
> > > > > object
> > > > > > > >>>> storage.
> > > > > > > >>>>>>> The source of truth for the metadata is in segment
> files on
> > > > the
> > > > > > > >>>> brokers
> > > > > > > >>>>>> in
> > > > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > > > independent
> > > > > > > of
> > > > > > > >>>> this
> > > > > > > >>>>>>> new proposal, but I want to present them to give the
> idea
> > > > that
> > > > > > > only a
> > > > > > > >>>>>>> limited amount of data files will be operated locally:
> > > > > > > >>>>>>> - We want to assemble batches into segment files and
> > > offload
> > > > > them
> > > > > > > to
> > > > > > > >>>>>>> tiered storage in order to prevent the unbounded
> growth of
> > > > > batch
> > > > > > > >>>>>> metadata.
> > > > > > > >>>>>>> For this, we need to open only  a few file descriptors
> (for
> > > > the
> > > > > > > >>>> segment
> > > > > > > >>>>>>> file itself + the necessary indexes) before the
> segment is
> > > > > fully
> > > > > > > >>>> written
> > > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > > >>>>>>> - We want to assemble local segment files for caching
> > > > purposes
> > > > > as
> > > > > > > >>>> well,
> > > > > > > >>>>>>> i.e. to speed up fetching. This will not materialize
> the
> > > full
> > > > > > > >>>> content of
> > > > > > > >>>>>>> the log, but only the hot set according to some policy
> (or
> > > > > > > >>>> configurable
> > > > > > > >>>>>>> policies), i.e. the number of segments and file
> descriptors
> > > > > will
> > > > > > > >>>> also be
> > > > > > > >>>>>>> limited.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> The number of RPCs in the produce path is
> significantly
> > > > > higher.
> > > > > > > For
> > > > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > > > cluster
> > > > > > > >>>> with 100
> > > > > > > >>>>>>>> brokers, each produce request could generate 100 more
> RPC
> > > > > > > requests.
> > > > > > > >>>>>> This
> > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> This is a valid concern that we considered, but this
> issue
> > > > can
> > > > > be
> > > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > > >>>>>>> The situation with a single broker is trivial: all the
> > > commit
> > > > > > > >>>> requests go
> > > > > > > >>>>>>> from the broker to itself.
> > > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but
> located in
> > > > the
> > > > > > > single
> > > > > > > >>>>>> rack
> > > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for
> diskless
> > > > > > > >>>> partitions, but
> > > > > > > >>>>>>> we can tell producers (through metadata) to always send
> > > > Produce
> > > > > > > >>>> requests
> > > > > > > >>>>>> to
> > > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader
> replicas
> > > for
> > > > > > > diskless
> > > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> > > Produce
> > > > > > > >>>> requests
> > > > > > > >>>>>> for
> > > > > > > >>>>>>> these partitions in various combinations, but only for
> > > them.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>                   Broker 0
> > > > > > > >>>>>>>             +-----------------+
> > > > > > > >>>>>>>             |    t1-0         |
> > > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > > >>>>>>> partitions   | +-------------+ |    for these
> partitions
> > > > > > > >>>>>>>             |                 |
> > > > > > > >>>>>>>             +-----------------+
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > > Effectively,
> > > > > > > each
> > > > > > > >>>>>>> broker will commit only to itself, which effectively
> means
> > > 1
> > > > > > commit
> > > > > > > >>>>>> request
> > > > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls,
> if we
> > > > > wish,
> > > > > > > >>>> just a
> > > > > > > >>>>>>> local function call).
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs).
> Obviously, we
> > > > > > cannot
> > > > > > > >>>> always
> > > > > > > >>>>>>> send Produce requests to the designated leaders of
> diskless
> > > > > > > >>>> partitions:
> > > > > > > >>>>>>> this would mean inter-AZ network traffic, which we
> would
> > > like
> > > > > to
> > > > > > > >>>> avoid.
> > > > > > > >>>>>> To
> > > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> produce
> > > > > > > >>>>>> representative”
> > > > > > > >>>>>>> in every AZ. If we continue our example: when a Produce
> > > > request
> > > > > > for
> > > > > > > >>>> t1-0,
> > > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands
> on
> > > > > broker 0
> > > > > > > >>>> (in the
> > > > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > > > However,
> > > > > if
> > > > > > > it
> > > > > > > >>>>>> comes
> > > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker
> 2.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> |produce requests         |produce requests
> |produce
> > > > > > > >>>> requests
> > > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for
> > > t1-0,
> > > > > > t2-1,
> > > > > > > >>>>>> t3-0
> > > > > > > >>>>>>> |from AZ 0                |from AZ 1
>  |from
> > > AZ 2
> > > > > > > >>>>>>> v                         v                        v
> > > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2
> (AZ
> > > 2)
> > > > > > > >>>>>>> +---------------+      +---------------+
> > > > +---------------+
> > > > > > > >>>>>>> |     t1-0      |      |               |      |
> > > >  |
> > > > > > > >>>>>>> |     t2-1      |      |               |      |
> > > >  |
> > > > > > > >>>>>>> |     t3-0      |      |               |      |
> > > >  |
> > > > > > > >>>>>>> +---------------+      +--------+------+
> > > > +--------+------+
> > > > > > > >>>>>>>    ^     ^                    |                      |
> > > > > > > >>>>>>>    |     +--------------------+                      |
> > > > > > > >>>>>>>    |     commit requests for these partitions        |
> > > > > > > >>>>>>>    |                                                 |
> > > > > > > >>>>>>>    +-------------------------------------------------+
> > > > > > > >>>>>>>          commit requests for these partitions
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> All the partitions that broker 0 is the leader of will
> be
> > > > > > > >>>> “represented”
> > > > > > > >>>>>> by
> > > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Of course, this relationship goes both ways between AZs
> > > (not
> > > > > > > >>>> necessarily
> > > > > > > >>>>>>> between the same brokers). It means that provided the
> > > cluster
> > > > > is
> > > > > > > >>>> balanced
> > > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> represent
> > > > > > > >>>>>> (number_of_azs
> > > > > > > >>>>>>> - 1) other brokers. This will result in the situation
> that
> > > > for
> > > > > > the
> > > > > > > >>>>>> majority
> > > > > > > >>>>>>> of commits, each broker will do up to (number_of_azs -
> 1)
> > > > > network
> > > > > > > >>>> commit
> > > > > > > >>>>>>> requests (plus one local). Cloud regions tend to have 3
> > > AZs,
> > > > > very
> > > > > > > >>>> rarely
> > > > > > > >>>>>>> more. That means, brokers will be doing up to 2 network
> > > > commit
> > > > > > > >>>> requests
> > > > > > > >>>>>> per
> > > > > > > >>>>>>> WAL file.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> There are the following exceptions:
> > > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example,
> when we
> > > > > have
> > > > > > 2
> > > > > > > >>>> AZs
> > > > > > > >>>>>> and
> > > > > > > >>>>>>> one has three brokers and another AZ has one. This one
> > > broker
> > > > > > will
> > > > > > > do
> > > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is
> not
> > > an
> > > > > > > extreme
> > > > > > > >>>>>>> amplification. Such an imbalance is not healthy in most
> > > > > practical
> > > > > > > >>>> setups
> > > > > > > >>>>>>> and should be avoided anyway.
> > > > > > > >>>>>>> 2. Leadership changes and metadata propagation period.
> When
> > > > the
> > > > > > > >>>> partition
> > > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > > > producers
> > > > > > > will
> > > > > > > >>>> not
> > > > > > > >>>>>>> know this immediately (unless we want to be strict and
> > > > respond
> > > > > > with
> > > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0
> will
> > > come
> > > > > > > >>>> together
> > > > > > > >>>>>> in a
> > > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two commit
> > > > > requests:
> > > > > > > to
> > > > > > > >>>>>> broker
> > > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
> t3-0.
> > > > This
> > > > > > > >>>> situation
> > > > > > > >>>>>>> is not permanent and as producers update the cluster
> > > > metadata,
> > > > > it
> > > > > > > >>>> will be
> > > > > > > >>>>>>> resolved.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> This all could be built with the metadata crafting
> > > mechanism
> > > > > only
> > > > > > > >>>> (which
> > > > > > > >>>>>>> is anyway needed for Diskless in one way or another to
> > > direct
> > > > > > > >>>> producers
> > > > > > > >>>>>> and
> > > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic),
> just
> > > with
> > > > > the
> > > > > > > >>>> right
> > > > > > > >>>>>>> policy for it (for example, some deterministic
> hash-based
> > > > > > formula).
> > > > > > > >>>> I.e.
> > > > > > > >>>>>> no
> > > > > > > >>>>>>> explicit support for “produce representative” or
> anything
> > > > like
> > > > > > this
> > > > > > > >>>> is
> > > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > > > places,
> > > > > > > >>>> partition
> > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> of
> > > > truth,
> > > > > > and
> > > > > > > >>>> how
> > > > > > > >>>>>> do
> > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> We do only two operations on WAL files that span
> multiple
> > > > > > diskless
> > > > > > > >>>>>>> partitions: committing and deleting. Commits can be
> done
> > > > > > > >>>> independently as
> > > > > > > >>>>>>> described above. But deletes are different, because
> when a
> > > > file
> > > > > > is
> > > > > > > >>>>>> deleted,
> > > > > > > >>>>>>> this affects all the partitions that still have alive
> > > batches
> > > > > in
> > > > > > > this
> > > > > > > >>>>>> file
> > > > > > > >>>>>>> (if any).
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> The WAL file manager is a necessary point of
> coordination
> > > to
> > > > > > delete
> > > > > > > >>>> WAL
> > > > > > > >>>>>>> files safely. We can say it is the source of truth
> about
> > > > files
> > > > > > > >>>>>> themselves,
> > > > > > > >>>>>>> while the partition leaders and their logs hold the
> truth
> > > > about
> > > > > > > >>>> whether a
> > > > > > > >>>>>>> particular file contains live batches of this
> particular
> > > > > > partition.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> The file manager will do this important task: be able
> to
> > > say
> > > > > for
> > > > > > > sure
> > > > > > > >>>>>> that
> > > > > > > >>>>>>> a file does not contain any live batch of any existing
> > > > > partition.
> > > > > > > For
> > > > > > > >>>>>> this,
> > > > > > > >>>>>>> it will have to periodically check against the
> partition
> > > > > leaders.
> > > > > > > >>>>>>> Considering that batch deletion is irreversible, when
> we
> > > > > declare
> > > > > > a
> > > > > > > >>>> file
> > > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> The file manager has to know about files being
> committed to
> > > > > start
> > > > > > > >>>> track
> > > > > > > >>>>>>> them and periodically check if they are empty. We can
> > > > consider
> > > > > > > >>>> various
> > > > > > > >>>>>> ways
> > > > > > > >>>>>>> to achieve this:
> > > > > > > >>>>>>> 1. As was proposed in my previous message: best effort
> > > commit
> > > > > by
> > > > > > > >>>> brokers
> > > > > > > >>>>>> +
> > > > > > > >>>>>>> periodic prefix scans of object storage to detect files
> > > that
> > > > > went
> > > > > > > >>>> below
> > > > > > > >>>>>> the
> > > > > > > >>>>>>> radar due to network issue or the file manager
> temporary
> > > > > > > >>>> unavailability.
> > > > > > > >>>>>>> We’re speaking about listing the file names only and
> > > opening
> > > > > only
> > > > > > > >>>>>>> previously unknown files in order to find the
> partitions
> > > > > involved
> > > > > > > >>>> with
> > > > > > > >>>>>> them.
> > > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill the
> > > list
> > > > of
> > > > > > > files
> > > > > > > >>>>>>> fully asynchronously and in the background. This may
> be not
> > > > > ideal
> > > > > > > >>>> due to
> > > > > > > >>>>>>> costs and performance of scanning tons of files.
> However,
> > > the
> > > > > > > number
> > > > > > > >>>> of
> > > > > > > >>>>>>> live WAL files should be limited due to tiered storage
> > > > > > offloading +
> > > > > > > >>>> we
> > > > > > > >>>>>> can
> > > > > > > >>>>>>> optimize this if we give files some global soft order
> in
> > > > their
> > > > > > > names.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > implementation.
> > > > > The
> > > > > > > >>>>>> existing
> > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > Adjusting
> > > > > both
> > > > > > > >>>> the
> > > > > > > >>>>>>> write
> > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > follower
> > > > to
> > > > > > > >>>>>> understand
> > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> existing
> > > > > logic.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > > UnifiedLog
> > > > in
> > > > > > > >>>> order to
> > > > > > > >>>>>>> support these changes. However, it seems that
> idempotence,
> > > > > > > >>>> transactions,
> > > > > > > >>>>>>> queues, tiered storage will have to be changed less
> than
> > > with
> > > > > the
> > > > > > > >>>>>> original
> > > > > > > >>>>>>> design. This is because the partition leader state
> would
> > > > remain
> > > > > > in
> > > > > > > >>>> the
> > > > > > > >>>>>> same
> > > > > > > >>>>>>> place (on brokers) and existing workflows that involve
> it
> > > > would
> > > > > > > have
> > > > > > > >>>> to
> > > > > > > >>>>>> be
> > > > > > > >>>>>>> changed less compared to the situation where we
> globalize
> > > the
> > > > > > > >>>> partition
> > > > > > > >>>>>>> leader state in the batch coordinator. I admit this is
> hard
> > > > to
> > > > > > make
> > > > > > > >>>>>>> convincing without both real implementations to hand :)
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> I am also
> > > > > > > >>>>>>>> not sure how this enables seamless switching the topic
> > > modes
> > > > > > > >>>> between
> > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> on
> > > > those?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Let’s consider the scenario of turning a classic topic
> into
> > > > > > > >>>> diskless. The
> > > > > > > >>>>>>> user sets diskless.enabled=true, the leader receives
> this
> > > > > > metadata
> > > > > > > >>>> update
> > > > > > > >>>>>>> and does the following:
> > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > >>>>>>> 3. Start a new segment that will be written in the
> diskless
> > > > > > format
> > > > > > > >>>> (i.e.
> > > > > > > >>>>>>> without data).
> > > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Since it’s the same log, the followers will know about
> that
> > > > > > switch
> > > > > > > >>>>>>> consistently. They will finish replicating the classic
> > > > segments
> > > > > > and
> > > > > > > >>>> start
> > > > > > > >>>>>>> replicating the diskless ones. They will always know
> where
> > > > each
> > > > > > > >>>> batch is
> > > > > > > >>>>>>> located (either inside a classic segment or referenced
> by a
> > > > > > > diskless
> > > > > > > >>>>>> one).
> > > > > > > >>>>>>> Switching back should be similar.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Doing this with the coordinator is possible, but has
> some
> > > > > > caveats.
> > > > > > > >>>> The
> > > > > > > >>>>>>> leader must do the following:
> > > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > > >>>>>>> 3. Write a special control segment to persist and
> replicate
> > > > the
> > > > > > > fact
> > > > > > > >>>> that
> > > > > > > >>>>>>> from offset N the partition is now in the diskless
> mode.
> > > > > > > >>>>>>> 4. Inform the coordinator about the first offset N of
> the
> > > > > > “diskless
> > > > > > > >>>> era”.
> > > > > > > >>>>>>> 5. Inform the controller quorum that the transition has
> > > > > finished
> > > > > > > and
> > > > > > > >>>> that
> > > > > > > >>>>>>> brokers now can process diskless writes for this
> partition.
> > > > > > > >>>>>>> This could fail at some points, so this will probably
> > > require
> > > > > > some
> > > > > > > >>>>>>> explicit state machine with replication either in the
> > > > partition
> > > > > > log
> > > > > > > >>>> or in
> > > > > > > >>>>>>> KRaft.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> It seems that the coordinator-less approach makes this
> > > > simpler
> > > > > > > >>>> because
> > > > > > > >>>>>> the
> > > > > > > >>>>>>> “coordinator” for the partition and the partition
> leader
> > > are
> > > > > the
> > > > > > > >>>> same and
> > > > > > > >>>>>>> they store the partition metadata in the same log, too.
> > > While
> > > > > in
> > > > > > > the
> > > > > > > >>>>>>> coordinator approach we have to perform some kind of a
> > > > > > distributed
> > > > > > > >>>> commit
> > > > > > > >>>>>>> to handover metadata management from the classic
> partition
> > > > > leader
> > > > > > > to
> > > > > > > >>>> the
> > > > > > > >>>>>>> batch coordinator.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> Please
> > > > let
> > > > > me
> > > > > > > >>>> know if
> > > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Best,
> > > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > > >>>>>>>> Hi, Ivan,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks for the update.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> I am not sure that I fully understand the new design,
> but
> > > it
> > > > > > seems
> > > > > > > >>>> less
> > > > > > > >>>>>>>> clean than before.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Does each partition now have a metadata partition and
> a
> > > > > separate
> > > > > > > >>>> data
> > > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > > doubles
> > > > > the
> > > > > > > >>>> number
> > > > > > > >>>>>>> of
> > > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > > descriptors
> > > > > > and
> > > > > > > >>>> the
> > > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> have a
> > > > > > > separate
> > > > > > > >>>>>>>> partition just to store the metadata. It's as if we
> are
> > > > > creating
> > > > > > > an
> > > > > > > >>>>>>>> internal topic with an unbounded number of
> partitions. Are
> > > > the
> > > > > > > >>>> metadata
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>> the data for the same partition always collocated on
> the
> > > > same
> > > > > > > >>>> broker?
> > > > > > > >>>>>> If
> > > > > > > >>>>>>>> so, how do we enforce that when replicas are
> reassigned?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> The number of RPCs in the produce path is
> significantly
> > > > > higher.
> > > > > > > For
> > > > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > > > cluster
> > > > > > > >>>> with 100
> > > > > > > >>>>>>>> brokers, each produce request could generate 100 more
> RPC
> > > > > > > requests.
> > > > > > > >>>>>> This
> > > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > > > places,
> > > > > > > >>>> partition
> > > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source
> of
> > > > truth,
> > > > > > and
> > > > > > > >>>> how
> > > > > > > >>>>>> do
> > > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> I am not sure how this design simplifies the
> > > implementation.
> > > > > The
> > > > > > > >>>>>> existing
> > > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > > Adjusting
> > > > > both
> > > > > > > >>>> the
> > > > > > > >>>>>>> write
> > > > > > > >>>>>>>> path in the leader and the replication path in the
> > > follower
> > > > to
> > > > > > > >>>>>> understand
> > > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> existing
> > > > > > logic. I
> > > > > > > >>>> am
> > > > > > > >>>>>>> also
> > > > > > > >>>>>>>> not sure how this enables seamless switching the topic
> > > modes
> > > > > > > >>>> between
> > > > > > > >>>>>>>> diskless and classic. Could you provide more details
> on
> > > > those?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Jun
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > > ivan@ivanyu.me
> > > > > >
> > > > > > > >>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to
> have a
> > > > > > separate
> > > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role
> would
> > > be
> > > > to
> > > > > > > >>>>>> centrally
> > > > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> > > topics.
> > > > > > This
> > > > > > > >>>>>>> component
> > > > > > > >>>>>>>>> drew many reasonable comments from the community
> about
> > > how
> > > > it
> > > > > > > >>>> would
> > > > > > > >>>>>>> support
> > > > > > > >>>>>>>>> various Kafka features (transactions, queues) and its
> > > > > > > >>>> scalability.
> > > > > > > >>>>>>> While we
> > > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > > concerns,
> > > > > we
> > > > > > > >>>> took a
> > > > > > > >>>>>>> step
> > > > > > > >>>>>>>>> back and looked at the problem from a different
> > > > perspective.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> We would like to propose an alternative Diskless
> design
> > > > > > *without
> > > > > > > >>>> a
> > > > > > > >>>>>>>>> centralized coordinator*. We believe this approach
> has
> > > > > > potential
> > > > > > > >>>> and
> > > > > > > >>>>>>>>> propose to discuss it as it may be more appealing to
> the
> > > > > > > >>>> community.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Let us explain the idea. Most of the complications
> with
> > > the
> > > > > > > >>>> original
> > > > > > > >>>>>>>>> Diskless approach come from one necessary
> architecture
> > > > > change:
> > > > > > > >>>>>>> globalizing
> > > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > > coordinator.
> > > > > > > >>>> This
> > > > > > > >>>>>>> causes
> > > > > > > >>>>>>>>> deviations to the established workflows in various
> > > features
> > > > > > like
> > > > > > > >>>>>>> produce
> > > > > > > >>>>>>>>> idempotence and transactions, queues, retention, etc.
> > > These
> > > > > > > >>>>>> deviations
> > > > > > > >>>>>>> need
> > > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > > implemented
> > > > > and
> > > > > > > >>>>>>> tested. In
> > > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > > partition
> > > > > > > >>>> leaders
> > > > > > > >>>>>>> again
> > > > > > > >>>>>>>>> responsible for managing their partitions, even in
> > > diskless
> > > > > > > >>>> topics.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata are
> > > > blended
> > > > > > > >>>> together
> > > > > > > >>>>>>> in
> > > > > > > >>>>>>>>> the one partition log. The crux of the Diskless idea
> is
> > > to
> > > > > > > >>>> decouple
> > > > > > > >>>>>>> them
> > > > > > > >>>>>>>>> and move data to the remote storage, while keeping
> > > metadata
> > > > > > > >>>> somewhere
> > > > > > > >>>>>>> else.
> > > > > > > >>>>>>>>> Using the central batch coordinator for managing
> batch
> > > > > metadata
> > > > > > > >>>> is
> > > > > > > >>>>>> one
> > > > > > > >>>>>>> way,
> > > > > > > >>>>>>>>> but not the only.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Let’s now think about managing metadata for each user
> > > > > partition
> > > > > > > >>>>>>>>> independently. Generally partitions are independent
> and
> > > > don’t
> > > > > > > >>>> share
> > > > > > > >>>>>>>>> anything apart from that their data are mixed in WAL
> > > files.
> > > > > If
> > > > > > we
> > > > > > > >>>>>>> figure
> > > > > > > >>>>>>>>> out how to commit and later delete WAL files safely,
> we
> > > > will
> > > > > > > >>>> achieve
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> > > central
> > > > > > batch
> > > > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition
> will
> > > be
> > > > > > > >>>> managed
> > > > > > > >>>>>> by
> > > > > > > >>>>>>> its
> > > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> classic
> > > > > > > >>>> topics, the
> > > > > > > >>>>>>>>> leader uses the partition log as the way to persist
> batch
> > > > > > > >>>> metadata,
> > > > > > > >>>>>>> i.e.
> > > > > > > >>>>>>>>> the regular batch header + the information about how
> to
> > > > find
> > > > > > this
> > > > > > > >>>>>>> batch on
> > > > > > > >>>>>>>>> remote storage. In contrast to classic topics, batch
> data
> > > > is
> > > > > in
> > > > > > > >>>>>> remote
> > > > > > > >>>>>>>>> storage.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > > >>>>>>>>> • Classic topics:
> > > > > > > >>>>>>>>>  • Data and metadata are co-located in the partition
> log.
> > > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > > (metadata)|Batch
> > > > > > > >>>> data].
> > > > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> from
> > > > > > > >>>> metadata.
> > > > > > > >>>>>>>>> • Original Diskless:
> > > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> > > remote
> > > > > > > >>>> storage.
> > > > > > > >>>>>>>>>  • The partition state is global in the batch
> > > coordinator.
> > > > > > > >>>>>>>>> • New Diskless:
> > > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on
> remote
> > > > > storage.
> > > > > > > >>>>>>>>>  • Partition log content: [Batch header
> (metadata)|Batch
> > > > > > > >>>>>> coordinates
> > > > > > > >>>>>>> on
> > > > > > > >>>>>>>>> remote storage].
> > > > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > > > >>>>>>>>>  • The replicas and leader have local state built
> from
> > > > > > > >>>> metadata.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the reminder
> of
> > > the
> > > > > > > >>>> original
> > > > > > > >>>>>>>>> Diskless design:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> The new approach could be depicted as the following:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> As you can see, the main difference is that now
> instead
> > > of
> > > > a
> > > > > > > >>>> single
> > > > > > > >>>>>>> commit
> > > > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> > > parallel
> > > > > > > >>>> commit
> > > > > > > >>>>>>> requests
> > > > > > > >>>>>>>>> to all the leaders of each partition involved in the
> WAL
> > > > > file.
> > > > > > > >>>> Each
> > > > > > > >>>>>> of
> > > > > > > >>>>>>> them
> > > > > > > >>>>>>>>> will commit its batches independently, without
> > > coordinating
> > > > > > with
> > > > > > > >>>>>> other
> > > > > > > >>>>>>>>> leaders and any other components. Batch data is
> addressed
> > > > by
> > > > > > the
> > > > > > > >>>> WAL
> > > > > > > >>>>>>> file
> > > > > > > >>>>>>>>> name, the byte offset and size, which allows
> partitions
> > > to
> > > > > know
> > > > > > > >>>>>> nothing
> > > > > > > >>>>>>>>> about other partitions to access their data in
> shared WAL
> > > > > > files.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> The number of partitions involved in a single WAL
> file
> > > may
> > > > be
> > > > > > > >>>> quite
> > > > > > > >>>>>>> large,
> > > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to commit
> one
> > > > WAL
> > > > > > > >>>> file is
> > > > > > > >>>>>>> very
> > > > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> > > number:
> > > > > > > >>>>>>>>> 1. Partition leaders are located on brokers.
> Requests to
> > > > > > > >>>> leaders on
> > > > > > > >>>>>>> one
> > > > > > > >>>>>>>>> broker could be grouped together into a single
> physical
> > > > > network
> > > > > > > >>>>>> request
> > > > > > > >>>>>>>>> (resembling the normal Produce request that may carry
> > > > batches
> > > > > > for
> > > > > > > >>>>>> many
> > > > > > > >>>>>>>>> partitions inside). This will cap the number of
> network
> > > > > > requests
> > > > > > > >>>> to
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make producers
> > > send
> > > > > > their
> > > > > > > >>>>>>> requests
> > > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> > > achieve
> > > > > the
> > > > > > > >>>> higher
> > > > > > > >>>>>>>>> concentration of logical commit requests in physical
> > > > network
> > > > > > > >>>> requests
> > > > > > > >>>>>>>>> reducing the number of the latter ones even further,
> > > > ideally
> > > > > to
> > > > > > > >>>> one.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Obviously, out of multiple commit requests some may
> fail
> > > or
> > > > > > time
> > > > > > > >>>> out
> > > > > > > >>>>>>> for a
> > > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers will
> > > > receive
> > > > > > > >>>> totally
> > > > > > > >>>>>>> or
> > > > > > > >>>>>>>>> partially failed responses to their Produce requests,
> > > > similar
> > > > > > to
> > > > > > > >>>> what
> > > > > > > >>>>>>> they
> > > > > > > >>>>>>>>> would have received when appending to a classic topic
> > > fails
> > > > > or
> > > > > > > >>>> times
> > > > > > > >>>>>>> out.
> > > > > > > >>>>>>>>> If a partition experiences problems, other partitions
> > > will
> > > > > not
> > > > > > be
> > > > > > > >>>>>>> affected
> > > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > > uncommitted
> > > > > > data
> > > > > > > >>>> will
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > > > (batches
> > > > > > are
> > > > > > > >>>>>>> constantly
> > > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> storage),
> > > > so
> > > > > > this
> > > > > > > >>>>>>> garbage
> > > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> For safely deleting WAL files we now need to
> centrally
> > > > manage
> > > > > > > >>>> them,
> > > > > > > >>>>>> as
> > > > > > > >>>>>>>>> this is the only state and logic that spans multiple
> > > > > > partitions.
> > > > > > > >>>> On
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>> diagram, you can see another commit request called
> > > “Commit
> > > > > file
> > > > > > > >>>> (best
> > > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This manager
> will
> > > > be
> > > > > > > >>>>>>> responsible
> > > > > > > >>>>>>>>> for the following:
> > > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and
> persisting
> > > > > > > >>>> information
> > > > > > > >>>>>>> about
> > > > > > > >>>>>>>>> committed WAL files.
> > > > > > > >>>>>>>>> 2. To handle potential failures in file information
> > > > delivery,
> > > > > > it
> > > > > > > >>>>>> will
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>> doing prefix scan on the remote storage periodically
> to
> > > > find
> > > > > > and
> > > > > > > >>>>>>> register
> > > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > > configurable
> > > > > and
> > > > > > > >>>>>> ideally
> > > > > > > >>>>>>>>> should be quite long.
> > > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders
> (after a
> > > > > grace
> > > > > > > >>>>>>> period) if
> > > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t anymore
> > > > > referred
> > > > > > > >>>> to by
> > > > > > > >>>>>>> any
> > > > > > > >>>>>>>>> partition.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > > features
> > > > > such
> > > > > > > >>>> as
> > > > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > > > retention.
> > > > > > > >>>> Now we
> > > > > > > >>>>>>> don’t
> > > > > > > >>>>>>>>> need to abstract away and reuse the code from
> partition
> > > > > leaders
> > > > > > > >>>> in
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use the
> > > same
> > > > > code
> > > > > > > >>>> paths
> > > > > > > >>>>>>> in
> > > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from
> classic
> > > > > topics
> > > > > > > >>>> mostly
> > > > > > > >>>>>>>>> remain unchanged.
> > > > > > > >>>>>>>>> For example, it seems that
> > > > > > > >>>>>>>>>
> > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > > and
> > > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > > transaction
> > > > > > > >>>> support
> > > > > > > >>>>>> on
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> partition leader side could be used for diskless
> topics
> > > > with
> > > > > > > >>>> little
> > > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > > idempotent
> > > > > > > >>>> produce
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > > >>>>>>>>> Another example is share groups support, where the
> share
> > > > > > > >>>> partition
> > > > > > > >>>>>>> leader,
> > > > > > > >>>>>>>>> being co-located with the partition leader, would
> execute
> > > > the
> > > > > > > >>>> same
> > > > > > > >>>>>>> logic
> > > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > > >>>>>>>>> 2. It returns to the familiar partition-based scaling
> > > > model,
> > > > > > > >>>> where
> > > > > > > >>>>>>>>> partitions are independent.
> > > > > > > >>>>>>>>> 3. It makes the operation and failure patterns
> closer to
> > > > the
> > > > > > > >>>>>> familiar
> > > > > > > >>>>>>>>> ones from classic topics.
> > > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> switching
> > > > the
> > > > > > > >>>> topics
> > > > > > > >>>>>>> modes
> > > > > > > >>>>>>>>> between diskless and classic.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> The rest of the things remain unchanged compared to
> the
> > > > > > previous
> > > > > > > >>>>>>> Diskless
> > > > > > > >>>>>>>>> design (after all previous discussions). Such things
> as
> > > > local
> > > > > > > >>>> segment
> > > > > > > >>>>>>>>> materialization by replicas, the consume path, tiered
> > > > storage
> > > > > > > >>>>>>> integration,
> > > > > > > >>>>>>>>> etc.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> If the community finds this design more suitable, we
> will
> > > > > > update
> > > > > > > >>>> the
> > > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it.
> Please let
> > > > us
> > > > > > know
> > > > > > > >>>>>> what
> > > > > > > >>>>>>> you
> > > > > > > >>>>>>>>> think.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Best regards,
> > > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > > > >>>>>>>>>> Hi Justine,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > > transactions
> > > > > > > >>>> for in
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> diskless coordinator for as long as the corresponding
> > > > offsets
> > > > > > are
> > > > > > > >>>>>>> there.
> > > > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> > > earlier,
> > > > > > this
> > > > > > > >>>> will
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Best,
> > > > > > > >>>>>>>>>> Ivan
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan
> wrote:
> > > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Thanks for the response. I think most of what you
> said
> > > > made
> > > > > > > >>>>>> sense,
> > > > > > > >>>>>>> but
> > > > > > > >>>>>>>>> I
> > > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> classic
> > > > > > > >>>> topics
> > > > > > > >>>>>>> forgets
> > > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > overpasses
> > > > > > > >>>> it). The
> > > > > > > >>>>>>>>>>> transaction coordinator acts like the main
> guardian,
> > > > > allowing
> > > > > > > >>>>>>> partition
> > > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if
> this is
> > > > > > > >>>> wrong. We
> > > > > > > >>>>>>> think
> > > > > > > >>>>>>>>>>> about relying on this with the batch coordinator
> and
> > > > delete
> > > > > > > >>>> the
> > > > > > > >>>>>>>>> information
> > > > > > > >>>>>>>>>>> about a transaction once it’s finished (as there’s
> no
> > > > > > > >>>> replication
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>> HWM
> > > > > > > >>>>>>>>>>> advances immediately).
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I didn't quite understand this. In classic topics,
> we
> > > > have
> > > > > > > >>>> maps
> > > > > > > >>>>>> for
> > > > > > > >>>>>>>>> ongoing
> > > > > > > >>>>>>>>>>> transactions which remove state when the
> transaction is
> > > > > > > >>>> completed
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>> an
> > > > > > > >>>>>>>>>>> aborted transactions index which is retained for
> much
> > > > > longer.
> > > > > > > >>>>>> Once
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> longer
> > > > > > > >>>> involved
> > > > > > > >>>>>> in
> > > > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> > > subject
> > > > to
> > > > > > > >>>>>>> compaction
> > > > > > > >>>>>>>>> etc.
> > > > > > > >>>>>>>>>>> Looking back at the outline provided above, I
> didn't
> > > see
> > > > > much
> > > > > > > >>>>>>> about the
> > > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > > > further.
> > > > > I
> > > > > > > >>>> saw
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>> following in a response:
> > > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > > segment,
> > > > > > > >>>> all the
> > > > > > > >>>>>>>>> necessary
> > > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> including
> > > > the
> > > > > > > >>>>>>> transaction
> > > > > > > >>>>>>>>>>> index will be built to serve FetchRequests exactly
> as
> > > > they
> > > > > > > >>>> are
> > > > > > > >>>>>>> today.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain the
> > > > > > > >>>> information
> > > > > > > >>>>>>> about
> > > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>> Justine
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > > > >>>> ivan@ivanyu.me>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > identified
> > > > > > > >>>> with
> > > > > > > >>>>>>>>> producer ID
> > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> could be
> > > > > > > >>>> cached
> > > > > > > >>>>>>>>> locally
> > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > > > >>>>>> transactions
> > > > > > > >>>>>>> can
> > > > > > > >>>>>>>>> be
> > > > > > > >>>>>>>>>>>> used
> > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> transactions
> > > > > > > >>>> with
> > > > > > > >>>>>>>>> producer id +
> > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> > > focused
> > > > > > > >>>> only on
> > > > > > > >>>>>>>>> version 2.
> > > > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> > > consider
> > > > > > > >>>> using
> > > > > > > >>>>>>> some
> > > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > transactional
> > > > > > > >>>>>>> checks
> > > > > > > >>>>>>>>> of the
> > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> errors
> > > > > > > >>>> like the
> > > > > > > >>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> you be
> > > > > > > >>>>>> checking
> > > > > > > >>>>>>> if
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > > > ongoing,
> > > > > > > >>>> and
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>>> course
> > > > > > > >>>>>>>>>>>> the normal idempotence checks. What the partition
> > > leader
> > > > > > > >>>> in the
> > > > > > > >>>>>>>>> classic
> > > > > > > >>>>>>>>>>>> topics does before appending a batch to the local
> log
> > > > > > > >>>> (e.g. in
> > > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > > > Diskless,
> > > > > > > >>>> we
> > > > > > > >>>>>>>>> unfortunately
> > > > > > > >>>>>>>>>>>> cannot do these checks before appending the data
> to
> > > the
> > > > > WAL
> > > > > > > >>>>>>> segment
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these
> batches in
> > > > the
> > > > > > > >>>> batch
> > > > > > > >>>>>>>>> coordinator
> > > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> > > other
> > > > > > > >>>> state
> > > > > > > >>>>>>>>> mentioned
> > > > > > > >>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> clear
> > > > > > > >>>> what
> > > > > > > >>>>>>> state is
> > > > > > > >>>>>>>>>>>> stored
> > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As the
> > > > > > > >>>> partition
> > > > > > > >>>>>>> leader
> > > > > > > >>>>>>>>> tracks
> > > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> > > > > > > >>>> coordinator
> > > > > > > >>>>>>> has
> > > > > > > >>>>>>>>> to as
> > > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > > > >>>> transaction
> > > > > > > >>>>>>>>> coordinator
> > > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> LSO --
> > > > > > > >>>> perhaps
> > > > > > > >>>>>>> that
> > > > > > > >>>>>>>>> would
> > > > > > > >>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> batches
> > > > and
> > > > > > > >>>> other
> > > > > > > >>>>>>>>>>>> information, but for the sake of performance we’d
> > > better
> > > > > > > >>>> store
> > > > > > > >>>>>> it
> > > > > > > >>>>>>>>>>>> explicitly.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > transactional
> > > > > > > >>>>>>> state is
> > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> will
> > > be
> > > > > > > >>>>>> cleaned
> > > > > > > >>>>>>> up?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> classic
> > > > > > > >>>> topics
> > > > > > > >>>>>>> forgets
> > > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > > overpasses
> > > > > > > >>>> it).
> > > > > > > >>>>>> The
> > > > > > > >>>>>>>>>>>> transaction coordinator acts like the main
> guardian,
> > > > > > > >>>> allowing
> > > > > > > >>>>>>>>> partition
> > > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if
> this
> > > is
> > > > > > > >>>> wrong.
> > > > > > > >>>>>> We
> > > > > > > >>>>>>>>> think
> > > > > > > >>>>>>>>>>>> about relying on this with the batch coordinator
> and
> > > > > > > >>>> delete the
> > > > > > > >>>>>>>>> information
> > > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as
> there’s no
> > > > > > > >>>>>> replication
> > > > > > > >>>>>>>>> and HWM
> > > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>> Ivan
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan
> wrote:
> > > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> transactions!
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > > identified
> > > > > > > >>>> with
> > > > > > > >>>>>>>>> producer ID
> > > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check
> could be
> > > > > > > >>>> cached
> > > > > > > >>>>>>>>> locally
> > > > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > > > >>>>>> transactions
> > > > > > > >>>>>>> can
> > > > > > > >>>>>>>>> be
> > > > > > > >>>>>>>>>>>> used
> > > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> transactions
> > > > > > > >>>> with
> > > > > > > >>>>>>>>> producer id +
> > > > > > > >>>>>>>>>>>>> epoch
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > > transactional
> > > > > > > >>>>>>> checks
> > > > > > > >>>>>>>>> of the
> > > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> errors
> > > > > > > >>>> like the
> > > > > > > >>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would
> you be
> > > > > > > >>>>>> checking
> > > > > > > >>>>>>> if
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is
> there
> > > > state
> > > > > > > >>>>>> about
> > > > > > > >>>>>>>>> ongoing
> > > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> > > other
> > > > > > > >>>> state
> > > > > > > >>>>>>>>> mentioned
> > > > > > > >>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> clear
> > > > > > > >>>> what
> > > > > > > >>>>>>> state is
> > > > > > > >>>>>>>>>>>> stored
> > > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining
> LSO --
> > > > > > > >>>> perhaps
> > > > > > > >>>>>>> that
> > > > > > > >>>>>>>>> would
> > > > > > > >>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > > transactional
> > > > > > > >>>>>>> state is
> > > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it
> will
> > > be
> > > > > > > >>>>>> cleaned
> > > > > > > >>>>>>> up?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from
> local
> > > > > > > >>>>>> segments,
> > > > > > > >>>>>>>>> making
> > > > > > > >>>>>>>>>>>> use of
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > zero-copy
> > > > > > > >>>>>>>>> functionality
> > > > > > > >>>>>>>>>>>> already
> > > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer
> state for
> > > > > > > >>>> each
> > > > > > > >>>>>>>>> partition in
> > > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> fetches
> > > > > > > >>>> need
> > > > > > > >>>>>> to
> > > > > > > >>>>>>> know
> > > > > > > >>>>>>>>>>>> aborted
> > > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > > >>>> transaction is
> > > > > > > >>>>>>>>> finished
> > > > > > > >>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > > corresponding
> > > > > > > >>>> logs
> > > > > > > >>>>>>> of the
> > > > > > > >>>>>>>>>>>> involved
> > > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the
> metadata
> > > > > > > >>>> level,
> > > > > > > >>>>>> no
> > > > > > > >>>>>>>>> actual
> > > > > > > >>>>>>>>>>>> control
> > > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > > transactional
> > > > > > > >>>>>>> batches.
> > > > > > > >>>>>>>>> How
> > > > > > > >>>>>>>>>>>> does the
> > > > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> > > control
> > > > > > > >>>>>> batch
> > > > > > > >>>>>>> for
> > > > > > > >>>>>>>>> each
> > > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response
> needs to
> > > > be
> > > > > > > >>>>>>> built by
> > > > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> > > batches
> > > > > > > >>>>>>> together?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition
> > > leader
> > > > > > > >>>> that
> > > > > > > >>>>>>> all
> > > > > > > >>>>>>>>>>>> consumers
> > > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
> this
> > > or
> > > > > > > >>>> it's
> > > > > > > >>>>>>> too
> > > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> > > include
> > > > > > > >>>>>> enough
> > > > > > > >>>>>>>>> details
> > > > > > > >>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
> given
> > > > > > > >>>>>> producer
> > > > > > > >>>>>>>>> contain a
> > > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a producer
> > > sends
> > > > > > > >>>>>> every
> > > > > > > >>>>>>>>> Produce
> > > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests
> could
> > > > > > > >>>> reach
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> batch
> > > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> rejection
> > > > > > > >>>> of the
> > > > > > > >>>>>>>>> produce
> > > > > > > >>>>>>>>>>>>>> requests.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > > >>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details
> about
> > > > > > > >>>>>>> transactions
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>>>>> queues,
> > > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic
> topics* is
> > > > > > > >>>> based
> > > > > > > >>>>>>> on
> > > > > > > >>>>>>>>> precise
> > > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> (mostly
> > > > > > > >>>>>>> producers,
> > > > > > > >>>>>>>>> but
> > > > > > > >>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > > > >>>> classes),
> > > > > > > >>>>>> and
> > > > > > > >>>>>>>>>>>> transaction
> > > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition
> leaders
> > > with
> > > > > > > >>>>>> their
> > > > > > > >>>>>>>>> local
> > > > > > > >>>>>>>>>>>> state
> > > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> The high level (some details skipped) workflow
> is
> > > the
> > > > > > > >>>>>>>>> following.
> > > > > > > >>>>>>>>>>>> When a
> > > > > > > >>>>>>>>>>>>>>> transactional Produce request is received by
> the
> > > > > > > >>>> broker:
> > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> checks
> > > > > > > >>>> if a
> > > > > > > >>>>>>>>> non-empty
> > > > > > > >>>>>>>>>>>>>>> transaction is running for this partition.
> This is
> > > > > > > >>>> done
> > > > > > > >>>>>>> using
> > > > > > > >>>>>>>>> its
> > > > > > > >>>>>>>>>>>> local
> > > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > > >>>>>> (ProducerStateManager,
> > > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed
> about
> > > all
> > > > > > > >>>> the
> > > > > > > >>>>>>>>> partitions
> > > > > > > >>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> > > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > > transactional
> > > > > > > >>>>>>> checks.
> > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the
> transactional
> > > > > > > >>>> data to
> > > > > > > >>>>>>>>> their logs
> > > > > > > >>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>> update some of their state (for example, log
> the
> > > fact
> > > > > > > >>>>>> that
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>>>>> transaction
> > > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> offset).
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > coordinator
> > > > > > > >>>>>>> directly
> > > > > > > >>>>>>>>> with
> > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > PREPARE_COMMIT
> > > > > > > >>>> or
> > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> > > > > > > >>>> markers to
> > > > > > > >>>>>>>>> their logs
> > > > > > > >>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction
> > > state
> > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders and
> lots
> > > > > > > >>>> of
> > > > > > > >>>>>>>>> important
> > > > > > > >>>>>>>>>>>> state
> > > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is
> local.
> > > The
> > > > > > > >>>> main
> > > > > > > >>>>>>>>> challenge
> > > > > > > >>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
> there
> > > > > > > >>>> are no
> > > > > > > >>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
> need
> > > > > > > >>>> to be
> > > > > > > >>>>>>>>> globalized
> > > > > > > >>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> > > > > > > >>>> support
> > > > > > > >>>>>>>>> idempotent
> > > > > > > >>>>>>>>>>>>>> produce.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics*
> would
> > > > > > > >>>> look
> > > > > > > >>>>>>> very
> > > > > > > >>>>>>>>>>>> similar:
> > > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > > > >>>> non-empty
> > > > > > > >>>>>>>>> transaction
> > > > > > > >>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> classic
> > > > > > > >>>>>> topics,
> > > > > > > >>>>>>>>> this is
> > > > > > > >>>>>>>>>>>>>> checked
> > > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single
> RPC.
> > > > > > > >>>> Since a
> > > > > > > >>>>>>>>> transaction
> > > > > > > >>>>>>>>>>>>>> could
> > > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> epoch,
> > > > > > > >>>> the
> > > > > > > >>>>>>> positive
> > > > > > > >>>>>>>>>>>> result of
> > > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> double
> > > > > > > >>>>>>> configured
> > > > > > > >>>>>>>>>>>> duration
> > > > > > > >>>>>>>>>>>>>> of a
> > > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > > informed
> > > > > > > >>>>>> about
> > > > > > > >>>>>>> all
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the transaction
> to
> > > > > > > >>>> include
> > > > > > > >>>>>>> them.
> > > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the
> broker
> > > > > > > >>>> side.
> > > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data
> to the
> > > > > > > >>>>>> current
> > > > > > > >>>>>>>>> shared
> > > > > > > >>>>>>>>>>>> WAL
> > > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> transaction-related
> > > > > > > >>>> state
> > > > > > > >>>>>>> for
> > > > > > > >>>>>>>>> Diskless
> > > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > > > >>>> coordinator
> > > > > > > >>>>>>> like
> > > > > > > >>>>>>>>> in the
> > > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > > transactional
> > > > > > > >>>>>>> checks
> > > > > > > >>>>>>>>> of the
> > > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> > > errors
> > > > > > > >>>> like
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> > > batches
> > > > > > > >>>>>> could
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>>>>> rejected.
> > > > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage
> in
> > > the
> > > > > > > >>>> WAL
> > > > > > > >>>>>>>>> segment
> > > > > > > >>>>>>>>>>>> file in
> > > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is
> preferable to
> > > > > > > >>>> doing
> > > > > > > >>>>>>> more
> > > > > > > >>>>>>>>>>>> network
> > > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > > segments
> > > > > > > >>>> will
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>>>>> relatively
> > > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > > coordinator
> > > > > > > >>>>>>> directly
> > > > > > > >>>>>>>>> with
> > > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > > PREPARE_COMMIT
> > > > > > > >>>> or
> > > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs
> the
> > > > > > > >>>> batch
> > > > > > > >>>>>>>>> coordinator
> > > > > > > >>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> > > > > > > >>>>>>> transaction is
> > > > > > > >>>>>>>>>>>> finished
> > > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > > >>>> corresponding
> > > > > > > >>>>>>> logs
> > > > > > > >>>>>>>>> of the
> > > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only on
> the
> > > > > > > >>>>>> metadata
> > > > > > > >>>>>>>>> level, no
> > > > > > > >>>>>>>>>>>>>>> actual control batches are written to any file.
> > > They
> > > > > > > >>>> will
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>>>>> dynamically
> > > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations. We
> > > could
> > > > > > > >>>>>>>>> technically
> > > > > > > >>>>>>>>>>>> write
> > > > > > > >>>>>>>>>>>>>>> these control batches for real, but this would
> mean
> > > > > > > >>>> extra
> > > > > > > >>>>>>>>> produce
> > > > > > > >>>>>>>>>>>>>> latency,
> > > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > > > >>>> coordinator
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>> save
> > > > > > > >>>>>>>>>>>> these
> > > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> leaders
> > > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to
> classic
> > > > > > > >>>> topics
> > > > > > > >>>>>>> now.
> > > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics
> write
> > > the
> > > > > > > >>>>>>>>> transaction
> > > > > > > >>>>>>>>>>>> markers
> > > > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction
> > > state
> > > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce
> flow, we
> > > > > > > >>>> get:
> > > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between brokers
> and
> > > > > > > >>>> the
> > > > > > > >>>>>>> batch
> > > > > > > >>>>>>>>>>>> coordinator
> > > > > > > >>>>>>>>>>>>>>> when a new partition appear in the
> transaction. To
> > > > > > > >>>>>>> mitigate the
> > > > > > > >>>>>>>>>>>> impact of
> > > > > > > >>>>>>>>>>>>>>> them:
> > > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> Produce
> > > > > > > >>>>>>> request
> > > > > > > >>>>>>>>> will be
> > > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> > > > > > > >>>> fast
> > > > > > > >>>>>>>>> response to
> > > > > > > >>>>>>>>>>>> these
> > > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally will
> > > > > > > >>>>>>> communicate
> > > > > > > >>>>>>>>> with a
> > > > > > > >>>>>>>>>>>>>>> single broker for the duration of the
> transaction
> > > > > > > >>>> further
> > > > > > > >>>>>>>>> reduces the
> > > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> > > > > > > >>>>>> coordinator
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>> batch
> > > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be
> able
> > > to
> > > > > > > >>>>>> span
> > > > > > > >>>>>>> both
> > > > > > > >>>>>>>>>>>> classic
> > > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> The share group coordination and management is
> a
> > > > > > > >>>> side job
> > > > > > > >>>>>>> that
> > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > > > >>>> replicas,
> > > > > > > >>>>>>> physical
> > > > > > > >>>>>>>>>>>> storage
> > > > > > > >>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > > consumers
> > > > > > > >>>>>>> (Fetch and
> > > > > > > >>>>>>>>>>>> Produce
> > > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > > affected.)
> > > > > > > >>>> We
> > > > > > > >>>>>>> don't
> > > > > > > >>>>>>>>> see any
> > > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> compatible
> > > > > > > >>>> with
> > > > > > > >>>>>>> share
> > > > > > > >>>>>>>>>>>> groups the
> > > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the
> code
> > > > > > > >>>> level,
> > > > > > > >>>>>> we
> > > > > > > >>>>>>>>> don't
> > > > > > > >>>>>>>>>>>> expect
> > > > > > > >>>>>>>>>>>>>> any
> > > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading routines
> are
> > > > > > > >>>> used
> > > > > > > >>>>>>> that
> > > > > > > >>>>>>>>> are
> > > > > > > >>>>>>>>>>>> used for
> > > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this or
> it's
> > > > > > > >>>> too
> > > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris
> wrote:
> > > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design
> input
> > > > > > > >>>> on
> > > > > > > >>>>>>>>> KIP-1150.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163
> with a
> > > > > > > >>>> new
> > > > > > > >>>>>>>>> design. To
> > > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> > > > > > > >>>> existing
> > > > > > > >>>>>>>>> KIP-405
> > > > > > > >>>>>>>>>>>> Tiered
> > > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced
> to a
> > > > > > > >>>>>>> Diskless
> > > > > > > >>>>>>>>> topic
> > > > > > > >>>>>>>>>>>> to be
> > > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for
> the
> > > > > > > >>>> Batch
> > > > > > > >>>>>>>>> Coordinator
> > > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> > > > > > > >>>> Tiered
> > > > > > > >>>>>>> Storage
> > > > > > > >>>>>>>>>>>> plugin
> > > > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative
> data
> > > > > > > >>>>>> formats.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> > > > > > > >>>> segments,
> > > > > > > >>>>>>>>> making use
> > > > > > > >>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > > > >>>> zero-copy
> > > > > > > >>>>>>>>> functionality
> > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered
> cache
> > > > > > > >>>>>>> elements,
> > > > > > > >>>>>>>>> do not
> > > > > > > >>>>>>>>>>>> need
> > > > > > > >>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > > > >>>> contacting
> > > > > > > >>>>>>> any
> > > > > > > >>>>>>>>> other
> > > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified
> substantially,
> > > by
> > > > > > > >>>>>>> removing
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> previous
> > > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > > >>>> component, and
> > > > > > > >>>>>>>>> "object
> > > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > > > >>>> enabled by
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> Batch
> > > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as
> the
> > > > > > > >>>>>> earlier
> > > > > > > >>>>>>>>> design,
> > > > > > > >>>>>>>>>>>> while
> > > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into the
> > > > > > > >>>> existing
> > > > > > > >>>>>>>>> ecosystem.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your
> feedback on
> > > > > > > >>>> the
> > > > > > > >>>>>> new
> > > > > > > >>>>>>>>> design.
> > > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the
> support
> > > > > > > >>>> of
> > > > > > > >>>>>> all
> > > > > > > >>>>>>>>> existing
> > > > > > > >>>>>>>>>>>>>> client
> > > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > > >>>> supporting
> > > > > > > >>>>>> APIs
> > > > > > > >>>>>>>>> like
> > > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan
> Siekierski
> > > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the
> current
> > > > > > > >>>>>>> status of
> > > > > > > >>>>>>>>> this
> > > > > > > >>>>>>>>>>>> KIP?
> > > > > > > >>>>>>>>>>>>>> I
> > > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the
> past 2
> > > > > > > >>>>>>> months,
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>> vote got
> > > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> > > > > > > >>>>>> answered
> > > > > > > >>>>>>>>> since
> > > > > > > >>>>>>>>>>>> then.
> > > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > > >>>> activity
> > > > > > > >>>>>>> since
> > > > > > > >>>>>>>>> May.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two
> real
> > > > > > > >>>>>> choices
> > > > > > > >>>>>>>>> that can
> > > > > > > >>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach
> being
> > > > > > > >>>> by
> > > > > > > >>>>>> far
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>>> dominant
> > > > > > > >>>>>>>>>>>>>> one
> > > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these
> are
> > > > > > > >>>>>>>>> standalone
> > > > > > > >>>>>>>>>>>>>> products.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> > > > > > > >>>> hate to
> > > > > > > >>>>>>> see a
> > > > > > > >>>>>>>>>>>> stall. So
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> > > > > > > >>>>>> McCabe -
> > > > > > > >>>>>>>>> have your
> > > > > > > >>>>>>>>>>>>>>> questions
> > > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I
> can
> > > > > > > >>>> do
> > > > > > > >>>>>> to
> > > > > > > >>>>>>>>> help? I'm
> > > > > > > >>>>>>>>>>>>>> deeply
> > > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > > > >>>> written
> > > > > > > >>>>>>> about
> > > > > > > >>>>>>>>> both?
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > > > >>>>>> Kozlovski
> > > > > > > >>>>>>> <
> > > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > > > >>>> main
> > > > > > > >>>>>> one
> > > > > > > >>>>>>>>> (just a
> > > > > > > >>>>>>>>>>>> bunch
> > > > > > > >>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > > >>>> example, I
> > > > > > > >>>>>>> was
> > > > > > > >>>>>>>>>>>> searching
> > > > > > > >>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> > > > > > > >>>> far
> > > > > > > >>>>>> as
> > > > > > > >>>>>>> I
> > > > > > > >>>>>>>>> can
> > > > > > > >>>>>>>>>>>> tell, it
> > > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> > > > > > > >>>> having
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> root
> > > > > > > >>>>>>>>>>>> KIP be
> > > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and
> links
> > > > > > > >>>> to
> > > > > > > >>>>>>>>> sub-KIPs
> > > > > > > >>>>>>>>>>>> where
> > > > > > > >>>>>>>>>>>>>> were
> > > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> > > > > > > >>>> would
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>> good to
> > > > > > > >>>>>>>>>>>>>> create
> > > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion
> about
> > > > > > > >>>>>>>>> introducing a
> > > > > > > >>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > > >>>> Storage as
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> primary
> > > > > > > >>>>>>>>>>>>>>> source of
> > > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > > > >>>> decided
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>> split it
> > > > > > > >>>>>>>>>>>>>> into
> > > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > > >>>> should aim
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>> have
> > > > > > > >>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > > > >>>>>> details
> > > > > > > >>>>>>> on
> > > > > > > >>>>>>>>> how to
> > > > > > > >>>>>>>>>>>>>>> implement
> > > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when
> we
> > > > > > > >>>>>>> discussed
> > > > > > > >>>>>>>>>>>> KRaft.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> > > > > > > >>>> hard
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>> discuss
> > > > > > > >>>>>>>>>>>> on
> > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > > > >>>> (linked
> > > > > > > >>>>>>> in
> > > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > > >>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper
> DISCUSS
> > > > > > > >>>>>>> threads
> > > > > > > >>>>>>>>> for
> > > > > > > >>>>>>>>>>>> each
> > > > > > > >>>>>>>>>>>>>>> type of
> > > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > > >>>> whether
> > > > > > > >>>>>>> Apache
> > > > > > > >>>>>>>>> Kafka
> > > > > > > >>>>>>>>>>>> wants
> > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > > >>>> authors
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>>> this KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> > > > > > > >>>>>>> aiven.io
> > > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > > > >>>>>>> Valtonen,
> > > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Anatolii Popov
> > > Senior Software Developer, *Aiven OY*
> > > m: +358505126242
> > > w: aiven.io  e: anatolii.popov@aiven.io
> > > <https://www.facebook.com/aivencloud>
> > > <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Jun,

Here are our answers to your questions and comments:

> JR10. "Even in the case where network traffic is not accounted; we believe
> that the operational benefits of Diskless topics are still appealing to
> Kafka users."
> Could you list other benefits such as better scalability and durability?

Thank you, good point, added this.

> JR11. "AWS charges 0.02$ per GiB"
> The object store is not free and has a different cost model based on
> requests. It would be useful summarize the tradeoff on cost and latency,
> and when diskless topics are cost effective.

Yes, there is a per-request cost for object storage which is not directly comparable to inter-zone transfer costs. We have found that the design in KIP-1163 amortizes those request costs such that they are insignificant compared to the $0.02/gb cost which is called out in KIP-1150.
We currently have very precise cost models that were put together for an earlier iteration of the Diskless topics. These are somewhat stale and inaccurate for the latest design, so we will work in the background to post concrete numbers in KIP-1163 before voting on that proposal. Would this be OK?

> JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't been
> updated in the last 6 months. The metadata change in KIP-1181 is now
> included in KIP-1163.

Thanks for calling out. We have updated KIP-1164 with the latest design. It's missing public interfaces, but we think it may be a bit too early to define them as some rounds of discussion is ahead, with potential to change the design. But otherwise the KIP is good for the first review.
We have deprecated KIP-1181 as it's now incorporated into KIP-1163.

Best,
Ivan

On Mon, Jan 12, 2026, at 19:38, Jun Rao via dev wrote:
> Hi, Anatolii,
> 
> Thanks for the updated KIP. Looks good to me overall. A few more comments.
> 
> JR10. "Even in the case where network traffic is not accounted; we believe
> that the operational benefits of Diskless topics are still appealing to
> Kafka users."
> Could you list other benefits such as better scalability and durability?
> 
> JR11. "AWS charges 0.02$ per GiB"
> The object store is not free and has a different cost model based on
> requests. It would be useful summarize the tradeoff on cost and latency,
> and when diskless topics are cost effective.
> 
> JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't been
> updated in the last 6 months. The metadata change in KIP-1181 is now
> included in KIP-1163.
> 
> Jun
> 
> On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <de...@kafka.apache.org>
> wrote:
> 
> > Hi all,
> >
> > Thank you for the extensive feedback.
> >
> > We have substantially updated the KIP to address the points raised. Given
> > the scope of these changes, we have started a new VOTE thread to restart
> > the voting process cleanly.
> >
> > You can find the new thread here:
> > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> >
> > Best,
> > Anatolii
> >
> > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > dev@kafka.apache.org> wrote:
> >
> > > Hi all,
> > >
> > > We (the team at Slack) have been following the recent discussion
> > regarding
> > > the scope and timeline of KIP-1150. We agree with the community sentiment
> > > that Diskless Topics represents the right long-term architecture for
> > Kafka
> > > in the cloud, but we also recognize the valid concerns raised regarding
> > the
> > > engineering resources required to deliver such an ambitious change.
> > >
> > > To help address these concerns and accelerate the timeline, we are happy
> > to
> > > announce that we are partnering with the KIP-1150 authors to co-develop
> > > this feature.
> > >
> > > We previously proposed KIP-1176: Tiered Storage for Active Log Segments
> > to
> > > solve similar problems. However, rather than fragmenting the community's
> > > efforts across competing designs, we have decided to withdraw KIP-1176
> > and
> > > consolidate our engineering resources behind KIP-1150.
> > >
> > > To start, we plan to take ownership of Compaction for Tiered Storage.
> > This
> > > has long been a missing feature in KIP-405, and it becomes critical in a
> > > Diskless architecture where long-lived data is obligated to be tiered. By
> > > driving this related prerequisite feature, we hope to allow for faster
> > > delivery of KIP-1150.
> > >
> > > We are excited to collaborate on this to ensure a robust and timely
> > > delivery for the community.
> > >
> > > Best,
> > > Tom & Henry
> > >
> > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi Greg,
> > > >
> > > > Thanks for sharing the meeting notes.
> > > > I agree we should keep polishing the contents of 1150 & high level
> > design
> > > > in 1163 to prepare for a vote.
> > > >
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > There was a video call between myself, Ivan Yurchenko, Jun Rao, and
> > > > Andrew
> > > > > Schofield pertaining to KIP-1150. Here are the notes from that
> > meeting:
> > > > >
> > > > > Ivan: What is the future state of Kafka in this area, in 5 years?
> > > > > Jun: Do we want something more cloud native? Yes, started with Tiered
> > > > > Storage. If there’s a better way, we should explore it. In the long
> > > term
> > > > > this will be useful
> > > > > Because Kafka is used so widely, we need to make sure everything we
> > add
> > > > is
> > > > > for the long term and for everyone, not just for a single company.
> > > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > > something
> > > > > that’s high quality/lasts/maintainable, and can work with all
> > existing
> > > > > capabilities.
> > > > > If both 1150 and 1176 proceed at the same time, it’s confusing. They
> > > > > overlap, but Diskless is more ambitious.
> > > > > If both KIPs are being seriously worked on, then we don’t really need
> > > > both,
> > > > > because Diskless clearly is better. Having multiple will confuse
> > > people.
> > > > It
> > > > > will duplicate some of the effort.
> > > > > If we want diskless ultimately, what is the short term strategy, to
> > get
> > > > > some early wins first?
> > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > Andrew: Eventually the architecture will change substantially, it may
> > > not
> > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > Greg: We all agree on having a high quality feature merged upstream,
> > > and
> > > > > supporting all APIs
> > > > > Jun: We should try and keep things simple, but there is some minimum
> > > > > complexity needed.
> > > > > When doing the short term changes (1176), it doesn’t really progress
> > in
> > > > > changing to a more modern architecture.
> > > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > > Jun: The danger of only applying changes to some part of the API, you
> > > set
> > > > > the precedence that you only have to implement part of the API.
> > > > Supporting
> > > > > the full API set should be a minimum requirement.
> > > > > Andrew: When we started Kraft, how much did we know the design?
> > > > > Jun: For Kraft we didn’t really know much about the migration, but
> > the
> > > > > high-level was clear.
> > > > > Greg: Is 1150 votable in its current state?
> > > > > Jun: 1150 should promise to support all APIs. It doesn’t have to have
> > > all
> > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > We do need some high-level design enough to give confidence that the
> > > > > promise is able to be fulfilled.
> > > > > Greg: Is the draft version in 1163 enough detail or is more needed?
> > > > > Jun: We need to agree on the core design, such as leaderless etc. And
> > > how
> > > > > the APIs will be supported.
> > > > > Greg: Okay we can include these things, and provide a sketch of how
> > the
> > > > > other leader-based features operate.
> > > > > Jun: Yeah if at a high level the sketch appears to work, we can
> > approve
> > > > > that functionality.
> > > > > Are you committed to doing the more involved and big project?
> > > > > Greg: Yes, we’re committed to the 1163 design and can’t really accept
> > > > 1176.
> > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > Greg: We’ll push internally for resources, and use the community
> > > > sentiment
> > > > > to motivate Aiven.
> > > > > How far into the future should we look? What sort of scale?
> > > > > Jun: As long as there’s a path forward, and we’re not closing off
> > > future
> > > > > improvements, we can figure out how to handle a larger scale when it
> > > > > arises.
> > > > > Greg: Random replica placement is very harmful, can we recommend
> > users
> > > to
> > > > > use an external tool like CruiseControl?
> > > > > Jun: Not everyone uses CruiseControl, we would probably need some
> > > > solution
> > > > > for this out of the box
> > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > Jun: Out-of-box experience should be good, good to allow other
> > > > > implementations
> > > > > Greg: But it could hurt Kafka feature/upgrade velocity when we wait
> > for
> > > > > plugin providers to implement it
> > > > > Ivan: We imagined that maybe cloud hyperscalers could implement it
> > with
> > > > > e.g. dynamodb
> > > > > Greg: Could we bake more details of the different providers into
> > Kafka,
> > > > or
> > > > > does it still make sense for it to be pluggable?
> > > > > Jun: Make it whatever is easiest to roll out and add new clients
> > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > Ivan: The vote is already open, we’re not too pressed for time. We’ll
> > > go
> > > > > improve the 1163 design and communication.
> > > > > Is 1176 a competing design? Someone will ask.
> > > > > Jun: If we are seriously working on something more ambitious, yeah we
> > > > > shouldn’t do the stop-gap solution.
> > > > > It’s diverting review resources. If we can get the short term thing
> > in
> > > > 1yr
> > > > > but Diskless solution is 2y it makes sense to go for Diskless. If
> > it’s
> > > > 5yr,
> > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should we
> > > > > explicitly exclude 1176?
> > > > > Andrew: Put your arms around the feature set you actually want, and
> > use
> > > > > that to rule out 1176.
> > > > > Probably don’t need -1 votes, most likely KIPs just don’t receive
> > > votes.
> > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > Jun: Satish posted meeting notes regularly, we should do the same.
> > > > >
> > > > > To summarize, we will be polishing the contents of 1150 & high level
> > > > design
> > > > > in 1163 to prepare for a vote.
> > > > > We believe that the community should select the feature set of 1150
> > to
> > > > > fully eliminate producer cross-zone costs, and make the investment
> > in a
> > > > > high quality Diskless Topics implementation rather than in stop-gap
> > > > > solutions.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:
> > > > >
> > > > > > This may be a tangent, but we needed to offload storage off of
> > Kafka
> > > > into
> > > > > > S3. We are keeping Kafka not as a source of truth, but as a mostly
> > > > > > ephemeral broker that can come and go as it pleases. Be that
> > scaling
> > > or
> > > > > > outage. Disks can be destroyed and recreated at will, we still
> > retain
> > > > > data
> > > > > > and use broker for just that, brokering messages. Not only that, we
> > > > > reduced
> > > > > > the requirement on the actual Kafka resources by reducing the size
> > > of a
> > > > > > payload via a claim check pattern. Maybe this is an anti–pattern,
> > but
> > > > it
> > > > > is
> > > > > > super fast and highly cost efficient. We reworked ProducerRequest
> > to
> > > > > allow
> > > > > > plugins. We added a custom http plugin that submits every request
> > > via a
> > > > > > persisted connection to a microservice. Microservice stores the
> > > payload
> > > > > and
> > > > > > returns a tiny json metadata object,a claim check, that can be used
> > > to
> > > > > find
> > > > > > the actual data. Think of it as zipping the payload. This claim
> > check
> > > > > > metadata traverses the pipelines with consumers using the urls in
> > > > > metadata
> > > > > > to pull what they need. Think unzipping. This allowed us to also
> > pull
> > > > > ONLY
> > > > > > the data that we need in graphql like manner. So if you have a 100K
> > > > json
> > > > > > payload and you need only a subsection, you can pull that by
> > > jmespath.
> > > > > When
> > > > > > you have multiple consumer groups yanking down huge payloads it is
> > > > > > cumbersome on the broker. When you have the same consumer groups
> > > > yanking
> > > > > > down a claim check, and then going out of band directly to the
> > source
> > > > of
> > > > > > truth, the broker has some breathing room. Obviously our
> > microservice
> > > > > does
> > > > > > not go directly to the cloud storage, as that would be too slow. It
> > > > > stores
> > > > > > the payload in high speed memory cache and returns asap. That
> > memory
> > > is
> > > > > > eventually persisted into S3. The retrieval goest against the cache
> > > > > first,
> > > > > > then against the S3. Overall a rather cheappy and zippy solution. I
> > > > tried
> > > > > > proposing the KIP for this, but there was no excitement. Check this
> > > > out:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > >
> > > > > >
> > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID>
> > > > wrote:
> > > > > > >
> > > > > > > Hi, Andrew,
> > > > > > >
> > > > > > > If we want to focus only on reducing cross-zone replication
> > costs,
> > > > > there
> > > > > > is
> > > > > > > an alternative design in the KIP-1176 discussion thread that
> > seems
> > > > > > simpler
> > > > > > > than the proposal here. I am copying the outline of that approach
> > > > > below.
> > > > > > >
> > > > > > > 1. A new leader is elected.
> > > > > > > 2. Leader maintains a first tiered offset, which is initialized
> > to
> > > > log
> > > > > > end
> > > > > > > offset.
> > > > > > > 3. Leader writes produced data from the client to local log.
> > > > > > > 4. Leader uploads produced data from all local logs as a combined
> > > > > object
> > > > > > > 5. Leader stores the metadata for the combined object in memory.
> > > > > > > 6. If a follower fetch request has an offset >= first tiered
> > > offset,
> > > > > the
> > > > > > > metadata for the corresponding combined object is returned.
> > > > Otherwise,
> > > > > > the
> > > > > > > local data is returned.
> > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > >
> > > > > > > It's still a bit unnatural, but it could work.
> > > > > > >
> > > > > > > Hi, Ivan,
> > > > > > >
> > > > > > > Are you still committed to proceeding with the original design of
> > > > > > KIP-1150?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > andrew_schofield@live.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >> I’ve been following KIP-1150 and friends for a while. I’m going
> > to
> > > > > jump
> > > > > > >> into the discussions too.
> > > > > > >>
> > > > > > >> Looking back at Jack Vanlightly’s message, I am not quite so
> > > > convinced
> > > > > > >> that it’s a kind of fork in the road. The primary aim of the
> > > effort
> > > > is
> > > > > > to
> > > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > > prohibitively
> > > > > > >> expensive to use on cloud storage. I think it would be entirely
> > > > viable
> > > > > > to
> > > > > > >> prioritise code reuse for an initial implementation of diskless
> > > > > topics,
> > > > > > and
> > > > > > >> we could still have a more cloud-native design in the future.
> > It’s
> > > > > hard
> > > > > > to
> > > > > > >> predict what the community will prioritise in the future.
> > > > > > >>
> > > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> > support
> > > > > > >> leaderless produce requests, first writing WAL segments into
> > > object
> > > > > > >> storage, and then using the regular partition leaders to
> > sequence
> > > > the
> > > > > > >> records. The active log segment for a diskless topic will
> > > initially
> > > > > > contain
> > > > > > >> batch coordinates rather than record batches. The batch
> > > coordinates
> > > > > can
> > > > > > be
> > > > > > >> resolved from WAL segments for consumers, and also in order to
> > > > prepare
> > > > > > log
> > > > > > >> segments for uploading to tiered storage. Jun is probably
> > correct
> > > > that
> > > > > > we
> > > > > > >> need a more frequent object merging process than tiered storage
> > > > > > provides.
> > > > > > >> This is just the transition from write-optimised WAL segments to
> > > > > > >> read-optimised tiered segments, and all of the object
> > > storage-based
> > > > > > >> implementations of Kafka that I’m aware of do this
> > rearrangement.
> > > > But
> > > > > > >> perhaps this more frequent object merging is a pre-GA
> > improvement,
> > > > > > rather
> > > > > > >> than a strict requirement for an initial implementation for
> > early
> > > > > access
> > > > > > >> use.
> > > > > > >>
> > > > > > >> For zone-aligned share consumers, the share group assignor is
> > > > intended
> > > > > > to
> > > > > > >> be rack-aware. Consumers should be assigned to partitions with
> > > > leaders
> > > > > > in
> > > > > > >> their zone. The simple assignor is not rack-aware, but it easily
> > > > could
> > > > > > be
> > > > > > >> or we could have a rack-aware assignor.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Andrew
> > > > > > >>
> > > > > > >>
> > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID>
> > > > wrote:
> > > > > > >>>
> > > > > > >>> Hi, Ivan,
> > > > > > >>>
> > > > > > >>> Thanks for the reply.
> > > > > > >>>
> > > > > > >>> "As I understand, you’re speaking about locally materialized
> > > > > segments.
> > > > > > >> They
> > > > > > >>> will indeed consume some IOPS. See them as a cache that could
> > > > always
> > > > > be
> > > > > > >>> restored from the remote storage. While it’s not ideal, it's
> > > still
> > > > OK
> > > > > > to
> > > > > > >>> lose data in them due to a machine crash, for example. Because
> > of
> > > > > this,
> > > > > > >> we
> > > > > > >>> can avoid explicit flushing on local materialized segments at
> > all
> > > > and
> > > > > > let
> > > > > > >>> the file system and page cache figure out when to flush
> > > optimally.
> > > > > This
> > > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > > dramatically,
> > > > > > >>> depending on throughput for each partition. We, of course,
> > > continue
> > > > > > >>> flushing the metadata segments as before."
> > > > > > >>>
> > > > > > >>> If we have a mix of classic and diskless topics on the same
> > > broker,
> > > > > > it's
> > > > > > >>> important that the classic topics' data is flushed to disk as
> > > > quickly
> > > > > > as
> > > > > > >>> possible. To achieve this, users typically set
> > > > dirty_expire_centisecs
> > > > > > in
> > > > > > >>> the kernel based on the number of available disk IOPS. Once you
> > > set
> > > > > > this
> > > > > > >>> number, it applies to all dirty files, including the cached
> > data
> > > in
> > > > > > >>> diskless topics. So, if there are more files actively
> > > accumulating
> > > > > > data,
> > > > > > >>> the flush frequency and therefore RPO is reduced for classic
> > > > topics.
> > > > > > >>>
> > > > > > >>> "We should have mentioned this explicitly, but this step, in
> > > fact,
> > > > > > >> remains
> > > > > > >>> in the form of segments offloading to tiered storage. When we
> > > > > assemble
> > > > > > a
> > > > > > >>> segment and hand it over to RemoteLogManager, we’re effectively
> > > > doing
> > > > > > >>> metadata compaction: replacing a big number of pieces of
> > metadata
> > > > > about
> > > > > > >>> individual batches with a single record in
> > > __remote_log_metadata."
> > > > > > >>>
> > > > > > >>> The object merging in tier storage typically only kicks in
> > after
> > > a
> > > > > few
> > > > > > >>> hours. The impact is (1) the amount of accumulated metadata is
> > > > still
> > > > > > >> quite
> > > > > > >>> large; (2) there are many small objects, leading to poor read
> > > > > > >> performance.
> > > > > > >>> I think we need a more frequent object merging process than
> > tier
> > > > > > storage
> > > > > > >>> provides.
> > > > > > >>>
> > > > > > >>> Jun
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > ivan@ivanyu.me>
> > > > > > wrote:
> > > > > > >>>
> > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > >>>>
> > > > > > >>>> Thank you for your messages.
> > > > > > >>>>
> > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > >>>>
> > > > > > >>>>> First, it degrades the durability.
> > > > > > >>>>> For each partition, now there are two files being actively
> > > > written
> > > > > > at a
> > > > > > >>>>> given point of time, one for the data and another for the
> > > > metadata.
> > > > > > >>>>> Flushing each file requires a separate IO. If the disk has 1K
> > > > IOPS
> > > > > > and
> > > > > > >> we
> > > > > > >>>>> have 5K partitions in a broker, currently we can afford to
> > > flush
> > > > > each
> > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If
> > we
> > > > > > double
> > > > > > >>>> the
> > > > > > >>>>> number of files per partition, we can only flush each
> > partition
> > > > > every
> > > > > > >> 10
> > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > >>>>
> > > > > > >>>> As I understand, you’re speaking about locally materialized
> > > > > segments.
> > > > > > >> They
> > > > > > >>>> will indeed consume some IOPS. See them as a cache that could
> > > > always
> > > > > > be
> > > > > > >>>> restored from the remote storage. While it’s not ideal, it's
> > > still
> > > > > OK
> > > > > > to
> > > > > > >>>> lose data in them due to a machine crash, for example. Because
> > > of
> > > > > > this,
> > > > > > >> we
> > > > > > >>>> can avoid explicit flushing on local materialized segments at
> > > all
> > > > > and
> > > > > > >> let
> > > > > > >>>> the file system and page cache figure out when to flush
> > > optimally.
> > > > > > This
> > > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > > dramatically,
> > > > > > >>>> depending on throughput for each partition. We, of course,
> > > > continue
> > > > > > >>>> flushing the metadata segments as before.
> > > > > > >>>>
> > > > > > >>>> It’s worth making a note on caching. I think nobody will
> > > disagree
> > > > > that
> > > > > > >>>> doing direct reads from remote storage every time a batch is
> > > > > requested
> > > > > > >> by a
> > > > > > >>>> consumer will not be practical neither from the performance
> > nor
> > > > from
> > > > > > the
> > > > > > >>>> economy point of view. We need a way to keep the number of GET
> > > > > > requests
> > > > > > >>>> down. There are multiple options, for example:
> > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > >>>> 2. Local in-memory caching. Comes with less network chattiness
> > > and
> > > > > > >> works
> > > > > > >>>> well if we have more or less stable brokers to consume from.
> > > > > > >>>> 3. Materialization of diskless logs on local disk. Way lower
> > > > impact
> > > > > on
> > > > > > >>>> RAM and also requires stable brokers for consumption (using
> > just
> > > > > > >> assigned
> > > > > > >>>> replicas will probably work well).
> > > > > > >>>>
> > > > > > >>>> Materialization is one of possible options, but we can choose
> > > > > another
> > > > > > >> one.
> > > > > > >>>> However, we will have this dilemma regardless of whether we
> > have
> > > > an
> > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > >>>>
> > > > > > >>>>> Second, if we ever need this
> > > > > > >>>>> metadata somewhere else, say in the WAL file manager, the
> > > > consumer
> > > > > > >> needs
> > > > > > >>>> to
> > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > inefficient.
> > > > > > The
> > > > > > >>>>> actual benefit of this approach is also questionable. On the
> > > > > surface,
> > > > > > >> it
> > > > > > >>>>> might seem that we could reduce the number of lines that need
> > > to
> > > > be
> > > > > > >>>> changed
> > > > > > >>>>> for this KIP. However, the changes are quite intrusive to the
> > > > > classic
> > > > > > >>>>> partition's code path and will probably make the code base
> > > harder
> > > > > to
> > > > > > >>>>> maintain in the long run. I like the original approach based
> > on
> > > > the
> > > > > > >> batch
> > > > > > >>>>> coordinator much better than this one. We could probably
> > > refactor
> > > > > the
> > > > > > >>>>> producer state code so that it could be reused in the batch
> > > > > > >> coordinator.
> > > > > > >>>>
> > > > > > >>>> It’s hard to disagree with this. The explicit coordinator is
> > > more
> > > > a
> > > > > > side
> > > > > > >>>> thing, while coordinator-less approach is more about extending
> > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > >>>>
> > > > > > >>>>> Thanks for addressing the concerns on the number of RPCs in
> > the
> > > > > > produce
> > > > > > >>>>> path. I agree that with the metadata crafting mechanism, we
> > > could
> > > > > > >>>> mitigate
> > > > > > >>>>> the PRC problem. However, since we now require the metadata
> > to
> > > be
> > > > > > >>>>> collocated with the data on the same set of brokers, it's
> > weird
> > > > > that
> > > > > > >> they
> > > > > > >>>>> are now managed by different mechanisms. The data assignment
> > > now
> > > > > uses
> > > > > > >> the
> > > > > > >>>>> metadata crafting mechanism, but the metadata is stored in
> > the
> > > > > > classic
> > > > > > >>>>> partition using its own assignment strategy. It will be
> > > > complicated
> > > > > > to
> > > > > > >>>> keep
> > > > > > >>>>> them collocated.
> > > > > > >>>>
> > > > > > >>>> I would like to note that the metadata crafting is needed only
> > > to
> > > > > tell
> > > > > > >>>> producers which brokers they should send Produce requests to,
> > > but
> > > > > data
> > > > > > >> (as
> > > > > > >>>> in “locally materialized log”) is located on partition
> > replicas,
> > > > > i.e.
> > > > > > >>>> automatically co-located with metadata.
> > > > > > >>>>
> > > > > > >>>> As a side note, it would probably be better that instead of
> > > > > implicitly
> > > > > > >>>> crafting partition metadata, we extend the metadata protocol
> > so
> > > > that
> > > > > > for
> > > > > > >>>> diskless partitions we return not only the leader and
> > replicas,
> > > > but
> > > > > > also
> > > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > > performance
> > > > > > and
> > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > >>>>
> > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > compaction/merging
> > > > > > >>>>> step.
> > > > > > >>>>
> > > > > > >>>> We should have mentioned this explicitly, but this step, in
> > > fact,
> > > > > > >> remains
> > > > > > >>>> in the form of segments offloading to tiered storage. When we
> > > > > > assemble a
> > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > effectively
> > > > > doing
> > > > > > >>>> metadata compaction: replacing a big number of pieces of
> > > metadata
> > > > > > about
> > > > > > >>>> individual batches with a single record in
> > > __remote_log_metadata.
> > > > > > >>>>
> > > > > > >>>> We could create a Diskless-specific merging mechanism instead
> > if
> > > > > > needed.
> > > > > > >>>> It’s rather easy with the explicit coordinator approach. With
> > > the
> > > > > > >>>> coordinator-less approach, this would probably be a bit more
> > > > tricky
> > > > > > >>>> (rewriting the tail of the log by the leader + replicating
> > this
> > > > > change
> > > > > > >>>> reliably).
> > > > > > >>>>
> > > > > > >>>>> I see a tendency toward primarily optimizing for the fewest
> > > code
> > > > > > >> changes
> > > > > > >>>> in
> > > > > > >>>>> the KIP. Instead, our primary goal should be a clean design
> > > that
> > > > > can
> > > > > > >> last
> > > > > > >>>>> for the long term.
> > > > > > >>>>
> > > > > > >>>> Yes, totally agree.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Luke,
> > > > > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > > > > because
> > > > > > >> of
> > > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> > only
> > > > > focus
> > > > > > on
> > > > > > >>>> the
> > > > > > >>>>> "diskless" design to handle object compaction/merging to/from
> > > the
> > > > > > >> remote
> > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > >>>>
> > > > > > >>>> After some evolution of the original proposal, leaderless is
> > now
> > > > > > >> limited.
> > > > > > >>>> We only need to be able to accept Produce requests on more
> > than
> > > > one
> > > > > > >> broker
> > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get it
> > right
> > > > > that
> > > > > > >> you
> > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Let’s now look at this problem from a higher level, as Jack
> > > > > proposed.
> > > > > > As
> > > > > > >>>> it was said, the big choice we need to make is whether we 1)
> > > > create
> > > > > an
> > > > > > >>>> explicit batch coordinator; or 2) go for the coordinator-less
> > > > > > approach,
> > > > > > >>>> where each diskless partition is managed by its leader as in
> > > > classic
> > > > > > >> topics.
> > > > > > >>>>
> > > > > > >>>> If we try to compare the two approaches:
> > > > > > >>>>
> > > > > > >>>> Pluggability:
> > > > > > >>>> - Explicit coordinator: Possible. For example, some setups may
> > > > > benefit
> > > > > > >>>> from batch metadata being stored in a cloud database (such as
> > > AWS
> > > > > > >> DynamoDB
> > > > > > >>>> or GCP Spanner).
> > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > >>>>
> > > > > > >>>> Scalability and fault tolerance:
> > > > > > >>>> - Explicit coordinator: Depends on the implementation and it’s
> > > > also
> > > > > > >>>> necessary to actively work for it.
> > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics. Scaling is
> > > > done
> > > > > by
> > > > > > >>>> partition placement, partitions could fail independently.
> > > > > > >>>>
> > > > > > >>>> Separation of concerns:
> > > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > > independent
> > > > > > >> from
> > > > > > >>>> classic topics in terms of code and workflows. For example,
> > the
> > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > mechanism
> > > > > could
> > > > > > >> be
> > > > > > >>>> relatively simply implemented with it. As a flip side of this,
> > > > some
> > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > bringing
> > > > > > diskless
> > > > > > >>>> closer to classic topics. Some code paths and workflows could
> > be
> > > > > more
> > > > > > >>>> straightforwardly reused, but they will inevitably have to be
> > > > > adapted
> > > > > > to
> > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > >>>>
> > > > > > >>>> Cloud-nativeness. This is a vague concept, also related to the
> > > > > > previous,
> > > > > > >>>> but let’s try:
> > > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > > separately
> > > > > > makes
> > > > > > >>>> it easier for brokers to take different roles, be purely
> > > stateless
> > > > > if
> > > > > > >>>> needed, etc.
> > > > > > >>>> - Coordinator-less: Less so. Something could be achieved with
> > > > > creative
> > > > > > >>>> partition placement, but not much.
> > > > > > >>>>
> > > > > > >>>> Both seem to have their pros and cons. However, answering
> > Jack’s
> > > > > > >> question,
> > > > > > >>>> the explicit coordinator approach may indeed lead to a more
> > > > flexible
> > > > > > >> design.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> The purpose of this deviation in the discussion was to
> > receive a
> > > > > > >>>> preliminary community evaluation of the coordinator-less
> > > approach
> > > > > > >> without
> > > > > > >>>> taking on the task of writing a separate KIP and fitting it in
> > > the
> > > > > > >> system
> > > > > > >>>> of KIP-1150 and its children. We’re open to stopping it and
> > > > getting
> > > > > > >> back to
> > > > > > >>>> working out the coordinator design if the community doesn’t
> > > favor
> > > > > the
> > > > > > >>>> proposed approach.
> > > > > > >>>>
> > > > > > >>>> Best,
> > > > > > >>>> Ivan and Diskless team
> > > > > > >>>>
> > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > >>>>> Hi Ivan,
> > > > > > >>>>>
> > > > > > >>>>> As Jun pointed out, the updated design seems to have some
> > > > > > shortcomings
> > > > > > >>>>> although it simplifies the implementation.
> > > > > > >>>>>
> > > > > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > > > > because
> > > > > > >> of
> > > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> > only
> > > > > focus
> > > > > > on
> > > > > > >>>> the
> > > > > > >>>>> "diskless" design to handle object compaction/merging to/from
> > > the
> > > > > > >> remote
> > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> Thank you,
> > > > > > >>>>> Luke
> > > > > > >>>>>
> > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > <jun@confluent.io.invalid
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Hi, Ivan,
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks for the explanation.
> > > > > > >>>>>>
> > > > > > >>>>>> "we write the reference to the WAL file with the batch data"
> > > > > > >>>>>>
> > > > > > >>>>>> I understand the approach now, but I think it is a hacky
> > one.
> > > > > There
> > > > > > >> are
> > > > > > >>>>>> multiple short comings with this design. First, it degrades
> > > the
> > > > > > >>>> durability.
> > > > > > >>>>>> For each partition, now there are two files being actively
> > > > written
> > > > > > at
> > > > > > >> a
> > > > > > >>>>>> given point of time, one for the data and another for the
> > > > > metadata.
> > > > > > >>>>>> Flushing each file requires a separate IO. If the disk has
> > 1K
> > > > IOPS
> > > > > > and
> > > > > > >>>> we
> > > > > > >>>>>> have 5K partitions in a broker, currently we can afford to
> > > flush
> > > > > > each
> > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If
> > > we
> > > > > > double
> > > > > > >>>> the
> > > > > > >>>>>> number of files per partition, we can only flush each
> > > partition
> > > > > > every
> > > > > > >>>> 10
> > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we ever
> > need
> > > > > this
> > > > > > >>>>>> metadata somewhere else, say in the WAL file manager, the
> > > > consumer
> > > > > > >>>> needs to
> > > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > > inefficient.
> > > > > > The
> > > > > > >>>>>> actual benefit of this approach is also questionable. On the
> > > > > > surface,
> > > > > > >>>> it
> > > > > > >>>>>> might seem that we could reduce the number of lines that
> > need
> > > to
> > > > > be
> > > > > > >>>> changed
> > > > > > >>>>>> for this KIP. However, the changes are quite intrusive to
> > the
> > > > > > classic
> > > > > > >>>>>> partition's code path and will probably make the code base
> > > > harder
> > > > > to
> > > > > > >>>>>> maintain in the long run. I like the original approach based
> > > on
> > > > > the
> > > > > > >>>> batch
> > > > > > >>>>>> coordinator much better than this one. We could probably
> > > > refactor
> > > > > > the
> > > > > > >>>>>> producer state code so that it could be reused in the batch
> > > > > > >>>> coordinator.
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks for addressing the concerns on the number of RPCs in
> > > the
> > > > > > >> produce
> > > > > > >>>>>> path. I agree that with the metadata crafting mechanism, we
> > > > could
> > > > > > >>>> mitigate
> > > > > > >>>>>> the PRC problem. However, since we now require the metadata
> > to
> > > > be
> > > > > > >>>>>> collocated with the data on the same set of brokers, it's
> > > weird
> > > > > that
> > > > > > >>>> they
> > > > > > >>>>>> are now managed by different mechanisms. The data assignment
> > > now
> > > > > > uses
> > > > > > >>>> the
> > > > > > >>>>>> metadata crafting mechanism, but the metadata is stored in
> > the
> > > > > > classic
> > > > > > >>>>>> partition using its own assignment strategy. It will be
> > > > > complicated
> > > > > > to
> > > > > > >>>> keep
> > > > > > >>>>>> them collocated.
> > > > > > >>>>>>
> > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > compaction/merging
> > > > > > >>>>>> step. My first concern is on the amount of metadata that
> > need
> > > to
> > > > > be
> > > > > > >>>> kept.
> > > > > > >>>>>> Without object compcation, the metadata generated in the
> > > produce
> > > > > > path
> > > > > > >>>> can
> > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's say for
> > > > every
> > > > > > >>>> 250ms we
> > > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> > remoting
> > > > > > tiering
> > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K partitions,
> > we
> > > > need
> > > > > > to
> > > > > > >>>> keep
> > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata,
> > > > quite
> > > > > > >>>>>> signficant. A second concern is on performance. Every time
> > we
> > > > need
> > > > > > to
> > > > > > >>>>>> rebuild the caching data, we need to read a bunch of small
> > > > objects
> > > > > > >>>> from S3,
> > > > > > >>>>>> slowing down the building process. If a consumer happens to
> > > need
> > > > > > such
> > > > > > >>>> data,
> > > > > > >>>>>> it could slow down the application.
> > > > > > >>>>>>
> > > > > > >>>>>> I see a tendency toward primarily optimizing for the fewest
> > > code
> > > > > > >>>> changes in
> > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean design
> > > that
> > > > > can
> > > > > > >>>> last
> > > > > > >>>>>> for the long term.
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks,
> > > > > > >>>>>>
> > > > > > >>>>>> Jun
> > > > > > >>>>>>
> > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > ivan@ivanyu.me
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hi Jun,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed to
> > > clearly
> > > > > > >>>> explain
> > > > > > >>>>>> the
> > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Does each partition now have a metadata partition and a
> > > > separate
> > > > > > >>>> data
> > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > doubles
> > > > the
> > > > > > >>>> number
> > > > > > >>>>>>> of
> > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > descriptors
> > > > > and
> > > > > > >>>> the
> > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > > > > separate
> > > > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > > > creating
> > > > > > an
> > > > > > >>>>>>>> internal topic with an unbounded number of partitions.
> > > > > > >>>>>>>
> > > > > > >>>>>>> No. There will be only one physical partition per diskless
> > > > > > >>>> partition. Let
> > > > > > >>>>>>> me explain this with an example. Let’s say we have a
> > diskless
> > > > > > >>>> partition
> > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We
> > > > > produce
> > > > > > >>>> some
> > > > > > >>>>>>> batches to this partition. The content of the segment file
> > > will
> > > > > be
> > > > > > >>>>>>> something like this (for each batch):
> > > > > > >>>>>>>
> > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > >>>>>>> Magic: like in classic
> > > > > > >>>>>>> CRC: like in classic
> > > > > > >>>>>>> Attributes: like in classic
> > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > >>>>>>> ProducerId: like in classic
> > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > >>>>>>> Records:
> > > > > > >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > byte
> > > > > offset
> > > > > > >>>>>>> 123456
> > > > > > >>>>>>>
> > > > > > >>>>>>> It looks very much like classic log entries. The only
> > > > difference
> > > > > is
> > > > > > >>>> that
> > > > > > >>>>>>> instead of writing real Records, we write the reference to
> > > the
> > > > > WAL
> > > > > > >>>> file
> > > > > > >>>>>>> with the batch data (I guess we need only the name and the
> > > byte
> > > > > > >>>> offset,
> > > > > > >>>>>>> because the byte length is the standard field above).
> > > > Otherwise,
> > > > > > >>>> it’s a
> > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > >>>>>>>
> > > > > > >>>>>>> So we have as many partitions for diskless as for classic.
> > As
> > > > of
> > > > > > open
> > > > > > >>>>>> file
> > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Are the metadata and
> > > > > > >>>>>>>> the data for the same partition always collocated on the
> > > same
> > > > > > >>>> broker?
> > > > > > >>>>>> If
> > > > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > > > >>>>>>>
> > > > > > >>>>>>> The source of truth for the data is still in WAL files on
> > > > object
> > > > > > >>>> storage.
> > > > > > >>>>>>> The source of truth for the metadata is in segment files on
> > > the
> > > > > > >>>> brokers
> > > > > > >>>>>> in
> > > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > > independent
> > > > > > of
> > > > > > >>>> this
> > > > > > >>>>>>> new proposal, but I want to present them to give the idea
> > > that
> > > > > > only a
> > > > > > >>>>>>> limited amount of data files will be operated locally:
> > > > > > >>>>>>> - We want to assemble batches into segment files and
> > offload
> > > > them
> > > > > > to
> > > > > > >>>>>>> tiered storage in order to prevent the unbounded growth of
> > > > batch
> > > > > > >>>>>> metadata.
> > > > > > >>>>>>> For this, we need to open only  a few file descriptors (for
> > > the
> > > > > > >>>> segment
> > > > > > >>>>>>> file itself + the necessary indexes) before the segment is
> > > > fully
> > > > > > >>>> written
> > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > >>>>>>> - We want to assemble local segment files for caching
> > > purposes
> > > > as
> > > > > > >>>> well,
> > > > > > >>>>>>> i.e. to speed up fetching. This will not materialize the
> > full
> > > > > > >>>> content of
> > > > > > >>>>>>> the log, but only the hot set according to some policy (or
> > > > > > >>>> configurable
> > > > > > >>>>>>> policies), i.e. the number of segments and file descriptors
> > > > will
> > > > > > >>>> also be
> > > > > > >>>>>>> limited.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > > > higher.
> > > > > > For
> > > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > > cluster
> > > > > > >>>> with 100
> > > > > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > > > > requests.
> > > > > > >>>>>> This
> > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > >>>>>>>
> > > > > > >>>>>>> This is a valid concern that we considered, but this issue
> > > can
> > > > be
> > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > >>>>>>> The situation with a single broker is trivial: all the
> > commit
> > > > > > >>>> requests go
> > > > > > >>>>>>> from the broker to itself.
> > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but located in
> > > the
> > > > > > single
> > > > > > >>>>>> rack
> > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for diskless
> > > > > > >>>> partitions, but
> > > > > > >>>>>>> we can tell producers (through metadata) to always send
> > > Produce
> > > > > > >>>> requests
> > > > > > >>>>>> to
> > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader replicas
> > for
> > > > > > diskless
> > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> > Produce
> > > > > > >>>> requests
> > > > > > >>>>>> for
> > > > > > >>>>>>> these partitions in various combinations, but only for
> > them.
> > > > > > >>>>>>>
> > > > > > >>>>>>>                   Broker 0
> > > > > > >>>>>>>             +-----------------+
> > > > > > >>>>>>>             |    t1-0         |
> > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > >>>>>>> partitions   | +-------------+ |    for these partitions
> > > > > > >>>>>>>             |                 |
> > > > > > >>>>>>>             +-----------------+
> > > > > > >>>>>>>
> > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > Effectively,
> > > > > > each
> > > > > > >>>>>>> broker will commit only to itself, which effectively means
> > 1
> > > > > commit
> > > > > > >>>>>> request
> > > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls, if we
> > > > wish,
> > > > > > >>>> just a
> > > > > > >>>>>>> local function call).
> > > > > > >>>>>>>
> > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we
> > > > > cannot
> > > > > > >>>> always
> > > > > > >>>>>>> send Produce requests to the designated leaders of diskless
> > > > > > >>>> partitions:
> > > > > > >>>>>>> this would mean inter-AZ network traffic, which we would
> > like
> > > > to
> > > > > > >>>> avoid.
> > > > > > >>>>>> To
> > > > > > >>>>>>> avoid it, we say that every broker has a “diskless produce
> > > > > > >>>>>> representative”
> > > > > > >>>>>>> in every AZ. If we continue our example: when a Produce
> > > request
> > > > > for
> > > > > > >>>> t1-0,
> > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on
> > > > broker 0
> > > > > > >>>> (in the
> > > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > > However,
> > > > if
> > > > > > it
> > > > > > >>>>>> comes
> > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > > > > > >>>>>>>
> > > > > > >>>>>>> |produce requests         |produce requests        |produce
> > > > > > >>>> requests
> > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for
> > t1-0,
> > > > > t2-1,
> > > > > > >>>>>> t3-0
> > > > > > >>>>>>> |from AZ 0                |from AZ 1               |from
> > AZ 2
> > > > > > >>>>>>> v                         v                        v
> > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ
> > 2)
> > > > > > >>>>>>> +---------------+      +---------------+
> > > +---------------+
> > > > > > >>>>>>> |     t1-0      |      |               |      |
> > >  |
> > > > > > >>>>>>> |     t2-1      |      |               |      |
> > >  |
> > > > > > >>>>>>> |     t3-0      |      |               |      |
> > >  |
> > > > > > >>>>>>> +---------------+      +--------+------+
> > > +--------+------+
> > > > > > >>>>>>>    ^     ^                    |                      |
> > > > > > >>>>>>>    |     +--------------------+                      |
> > > > > > >>>>>>>    |     commit requests for these partitions        |
> > > > > > >>>>>>>    |                                                 |
> > > > > > >>>>>>>    +-------------------------------------------------+
> > > > > > >>>>>>>          commit requests for these partitions
> > > > > > >>>>>>>
> > > > > > >>>>>>> All the partitions that broker 0 is the leader of will be
> > > > > > >>>> “represented”
> > > > > > >>>>>> by
> > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Of course, this relationship goes both ways between AZs
> > (not
> > > > > > >>>> necessarily
> > > > > > >>>>>>> between the same brokers). It means that provided the
> > cluster
> > > > is
> > > > > > >>>> balanced
> > > > > > >>>>>>> by the number of brokers per AZ, each broker will represent
> > > > > > >>>>>> (number_of_azs
> > > > > > >>>>>>> - 1) other brokers. This will result in the situation that
> > > for
> > > > > the
> > > > > > >>>>>> majority
> > > > > > >>>>>>> of commits, each broker will do up to (number_of_azs - 1)
> > > > network
> > > > > > >>>> commit
> > > > > > >>>>>>> requests (plus one local). Cloud regions tend to have 3
> > AZs,
> > > > very
> > > > > > >>>> rarely
> > > > > > >>>>>>> more. That means, brokers will be doing up to 2 network
> > > commit
> > > > > > >>>> requests
> > > > > > >>>>>> per
> > > > > > >>>>>>> WAL file.
> > > > > > >>>>>>>
> > > > > > >>>>>>> There are the following exceptions:
> > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example, when we
> > > > have
> > > > > 2
> > > > > > >>>> AZs
> > > > > > >>>>>> and
> > > > > > >>>>>>> one has three brokers and another AZ has one. This one
> > broker
> > > > > will
> > > > > > do
> > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is not
> > an
> > > > > > extreme
> > > > > > >>>>>>> amplification. Such an imbalance is not healthy in most
> > > > practical
> > > > > > >>>> setups
> > > > > > >>>>>>> and should be avoided anyway.
> > > > > > >>>>>>> 2. Leadership changes and metadata propagation period. When
> > > the
> > > > > > >>>> partition
> > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > > producers
> > > > > > will
> > > > > > >>>> not
> > > > > > >>>>>>> know this immediately (unless we want to be strict and
> > > respond
> > > > > with
> > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will
> > come
> > > > > > >>>> together
> > > > > > >>>>>> in a
> > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two commit
> > > > requests:
> > > > > > to
> > > > > > >>>>>> broker
> > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0.
> > > This
> > > > > > >>>> situation
> > > > > > >>>>>>> is not permanent and as producers update the cluster
> > > metadata,
> > > > it
> > > > > > >>>> will be
> > > > > > >>>>>>> resolved.
> > > > > > >>>>>>>
> > > > > > >>>>>>> This all could be built with the metadata crafting
> > mechanism
> > > > only
> > > > > > >>>> (which
> > > > > > >>>>>>> is anyway needed for Diskless in one way or another to
> > direct
> > > > > > >>>> producers
> > > > > > >>>>>> and
> > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic), just
> > with
> > > > the
> > > > > > >>>> right
> > > > > > >>>>>>> policy for it (for example, some deterministic hash-based
> > > > > formula).
> > > > > > >>>> I.e.
> > > > > > >>>>>> no
> > > > > > >>>>>>> explicit support for “produce representative” or anything
> > > like
> > > > > this
> > > > > > >>>> is
> > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > > places,
> > > > > > >>>> partition
> > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> > > truth,
> > > > > and
> > > > > > >>>> how
> > > > > > >>>>>> do
> > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > >>>>>>>
> > > > > > >>>>>>> We do only two operations on WAL files that span multiple
> > > > > diskless
> > > > > > >>>>>>> partitions: committing and deleting. Commits can be done
> > > > > > >>>> independently as
> > > > > > >>>>>>> described above. But deletes are different, because when a
> > > file
> > > > > is
> > > > > > >>>>>> deleted,
> > > > > > >>>>>>> this affects all the partitions that still have alive
> > batches
> > > > in
> > > > > > this
> > > > > > >>>>>> file
> > > > > > >>>>>>> (if any).
> > > > > > >>>>>>>
> > > > > > >>>>>>> The WAL file manager is a necessary point of coordination
> > to
> > > > > delete
> > > > > > >>>> WAL
> > > > > > >>>>>>> files safely. We can say it is the source of truth about
> > > files
> > > > > > >>>>>> themselves,
> > > > > > >>>>>>> while the partition leaders and their logs hold the truth
> > > about
> > > > > > >>>> whether a
> > > > > > >>>>>>> particular file contains live batches of this particular
> > > > > partition.
> > > > > > >>>>>>>
> > > > > > >>>>>>> The file manager will do this important task: be able to
> > say
> > > > for
> > > > > > sure
> > > > > > >>>>>> that
> > > > > > >>>>>>> a file does not contain any live batch of any existing
> > > > partition.
> > > > > > For
> > > > > > >>>>>> this,
> > > > > > >>>>>>> it will have to periodically check against the partition
> > > > leaders.
> > > > > > >>>>>>> Considering that batch deletion is irreversible, when we
> > > > declare
> > > > > a
> > > > > > >>>> file
> > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > >>>>>>>
> > > > > > >>>>>>> The file manager has to know about files being committed to
> > > > start
> > > > > > >>>> track
> > > > > > >>>>>>> them and periodically check if they are empty. We can
> > > consider
> > > > > > >>>> various
> > > > > > >>>>>> ways
> > > > > > >>>>>>> to achieve this:
> > > > > > >>>>>>> 1. As was proposed in my previous message: best effort
> > commit
> > > > by
> > > > > > >>>> brokers
> > > > > > >>>>>> +
> > > > > > >>>>>>> periodic prefix scans of object storage to detect files
> > that
> > > > went
> > > > > > >>>> below
> > > > > > >>>>>> the
> > > > > > >>>>>>> radar due to network issue or the file manager temporary
> > > > > > >>>> unavailability.
> > > > > > >>>>>>> We’re speaking about listing the file names only and
> > opening
> > > > only
> > > > > > >>>>>>> previously unknown files in order to find the partitions
> > > > involved
> > > > > > >>>> with
> > > > > > >>>>>> them.
> > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill the
> > list
> > > of
> > > > > > files
> > > > > > >>>>>>> fully asynchronously and in the background. This may be not
> > > > ideal
> > > > > > >>>> due to
> > > > > > >>>>>>> costs and performance of scanning tons of files. However,
> > the
> > > > > > number
> > > > > > >>>> of
> > > > > > >>>>>>> live WAL files should be limited due to tiered storage
> > > > > offloading +
> > > > > > >>>> we
> > > > > > >>>>>> can
> > > > > > >>>>>>> optimize this if we give files some global soft order in
> > > their
> > > > > > names.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> I am not sure how this design simplifies the
> > implementation.
> > > > The
> > > > > > >>>>>> existing
> > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > Adjusting
> > > > both
> > > > > > >>>> the
> > > > > > >>>>>>> write
> > > > > > >>>>>>>> path in the leader and the replication path in the
> > follower
> > > to
> > > > > > >>>>>> understand
> > > > > > >>>>>>>> batch-header only data is quite intrusive to the existing
> > > > logic.
> > > > > > >>>>>>>
> > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > UnifiedLog
> > > in
> > > > > > >>>> order to
> > > > > > >>>>>>> support these changes. However, it seems that idempotence,
> > > > > > >>>> transactions,
> > > > > > >>>>>>> queues, tiered storage will have to be changed less than
> > with
> > > > the
> > > > > > >>>>>> original
> > > > > > >>>>>>> design. This is because the partition leader state would
> > > remain
> > > > > in
> > > > > > >>>> the
> > > > > > >>>>>> same
> > > > > > >>>>>>> place (on brokers) and existing workflows that involve it
> > > would
> > > > > > have
> > > > > > >>>> to
> > > > > > >>>>>> be
> > > > > > >>>>>>> changed less compared to the situation where we globalize
> > the
> > > > > > >>>> partition
> > > > > > >>>>>>> leader state in the batch coordinator. I admit this is hard
> > > to
> > > > > make
> > > > > > >>>>>>> convincing without both real implementations to hand :)
> > > > > > >>>>>>>
> > > > > > >>>>>>>> I am also
> > > > > > >>>>>>>> not sure how this enables seamless switching the topic
> > modes
> > > > > > >>>> between
> > > > > > >>>>>>>> diskless and classic. Could you provide more details on
> > > those?
> > > > > > >>>>>>>
> > > > > > >>>>>>> Let’s consider the scenario of turning a classic topic into
> > > > > > >>>> diskless. The
> > > > > > >>>>>>> user sets diskless.enabled=true, the leader receives this
> > > > > metadata
> > > > > > >>>> update
> > > > > > >>>>>>> and does the following:
> > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > >>>>>>> 3. Start a new segment that will be written in the diskless
> > > > > format
> > > > > > >>>> (i.e.
> > > > > > >>>>>>> without data).
> > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Since it’s the same log, the followers will know about that
> > > > > switch
> > > > > > >>>>>>> consistently. They will finish replicating the classic
> > > segments
> > > > > and
> > > > > > >>>> start
> > > > > > >>>>>>> replicating the diskless ones. They will always know where
> > > each
> > > > > > >>>> batch is
> > > > > > >>>>>>> located (either inside a classic segment or referenced by a
> > > > > > diskless
> > > > > > >>>>>> one).
> > > > > > >>>>>>> Switching back should be similar.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Doing this with the coordinator is possible, but has some
> > > > > caveats.
> > > > > > >>>> The
> > > > > > >>>>>>> leader must do the following:
> > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > >>>>>>> 3. Write a special control segment to persist and replicate
> > > the
> > > > > > fact
> > > > > > >>>> that
> > > > > > >>>>>>> from offset N the partition is now in the diskless mode.
> > > > > > >>>>>>> 4. Inform the coordinator about the first offset N of the
> > > > > “diskless
> > > > > > >>>> era”.
> > > > > > >>>>>>> 5. Inform the controller quorum that the transition has
> > > > finished
> > > > > > and
> > > > > > >>>> that
> > > > > > >>>>>>> brokers now can process diskless writes for this partition.
> > > > > > >>>>>>> This could fail at some points, so this will probably
> > require
> > > > > some
> > > > > > >>>>>>> explicit state machine with replication either in the
> > > partition
> > > > > log
> > > > > > >>>> or in
> > > > > > >>>>>>> KRaft.
> > > > > > >>>>>>>
> > > > > > >>>>>>> It seems that the coordinator-less approach makes this
> > > simpler
> > > > > > >>>> because
> > > > > > >>>>>> the
> > > > > > >>>>>>> “coordinator” for the partition and the partition leader
> > are
> > > > the
> > > > > > >>>> same and
> > > > > > >>>>>>> they store the partition metadata in the same log, too.
> > While
> > > > in
> > > > > > the
> > > > > > >>>>>>> coordinator approach we have to perform some kind of a
> > > > > distributed
> > > > > > >>>> commit
> > > > > > >>>>>>> to handover metadata management from the classic partition
> > > > leader
> > > > > > to
> > > > > > >>>> the
> > > > > > >>>>>>> batch coordinator.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I hope these explanations help to clarify the idea. Please
> > > let
> > > > me
> > > > > > >>>> know if
> > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Best,
> > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > >>>>>>>> Hi, Ivan,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks for the update.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I am not sure that I fully understand the new design, but
> > it
> > > > > seems
> > > > > > >>>> less
> > > > > > >>>>>>>> clean than before.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Does each partition now have a metadata partition and a
> > > > separate
> > > > > > >>>> data
> > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > doubles
> > > > the
> > > > > > >>>> number
> > > > > > >>>>>>> of
> > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > descriptors
> > > > > and
> > > > > > >>>> the
> > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > > > > separate
> > > > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > > > creating
> > > > > > an
> > > > > > >>>>>>>> internal topic with an unbounded number of partitions. Are
> > > the
> > > > > > >>>> metadata
> > > > > > >>>>>>> and
> > > > > > >>>>>>>> the data for the same partition always collocated on the
> > > same
> > > > > > >>>> broker?
> > > > > > >>>>>> If
> > > > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > > > higher.
> > > > > > For
> > > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > > cluster
> > > > > > >>>> with 100
> > > > > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > > > > requests.
> > > > > > >>>>>> This
> > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > > places,
> > > > > > >>>> partition
> > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> > > truth,
> > > > > and
> > > > > > >>>> how
> > > > > > >>>>>> do
> > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I am not sure how this design simplifies the
> > implementation.
> > > > The
> > > > > > >>>>>> existing
> > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > Adjusting
> > > > both
> > > > > > >>>> the
> > > > > > >>>>>>> write
> > > > > > >>>>>>>> path in the leader and the replication path in the
> > follower
> > > to
> > > > > > >>>>>> understand
> > > > > > >>>>>>>> batch-header only data is quite intrusive to the existing
> > > > > logic. I
> > > > > > >>>> am
> > > > > > >>>>>>> also
> > > > > > >>>>>>>> not sure how this enables seamless switching the topic
> > modes
> > > > > > >>>> between
> > > > > > >>>>>>>> diskless and classic. Could you provide more details on
> > > those?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Jun
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > ivan@ivanyu.me
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to have a
> > > > > separate
> > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role would
> > be
> > > to
> > > > > > >>>>>> centrally
> > > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> > topics.
> > > > > This
> > > > > > >>>>>>> component
> > > > > > >>>>>>>>> drew many reasonable comments from the community about
> > how
> > > it
> > > > > > >>>> would
> > > > > > >>>>>>> support
> > > > > > >>>>>>>>> various Kafka features (transactions, queues) and its
> > > > > > >>>> scalability.
> > > > > > >>>>>>> While we
> > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > concerns,
> > > > we
> > > > > > >>>> took a
> > > > > > >>>>>>> step
> > > > > > >>>>>>>>> back and looked at the problem from a different
> > > perspective.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> We would like to propose an alternative Diskless design
> > > > > *without
> > > > > > >>>> a
> > > > > > >>>>>>>>> centralized coordinator*. We believe this approach has
> > > > > potential
> > > > > > >>>> and
> > > > > > >>>>>>>>> propose to discuss it as it may be more appealing to the
> > > > > > >>>> community.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Let us explain the idea. Most of the complications with
> > the
> > > > > > >>>> original
> > > > > > >>>>>>>>> Diskless approach come from one necessary architecture
> > > > change:
> > > > > > >>>>>>> globalizing
> > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > coordinator.
> > > > > > >>>> This
> > > > > > >>>>>>> causes
> > > > > > >>>>>>>>> deviations to the established workflows in various
> > features
> > > > > like
> > > > > > >>>>>>> produce
> > > > > > >>>>>>>>> idempotence and transactions, queues, retention, etc.
> > These
> > > > > > >>>>>> deviations
> > > > > > >>>>>>> need
> > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > implemented
> > > > and
> > > > > > >>>>>>> tested. In
> > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > partition
> > > > > > >>>> leaders
> > > > > > >>>>>>> again
> > > > > > >>>>>>>>> responsible for managing their partitions, even in
> > diskless
> > > > > > >>>> topics.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata are
> > > blended
> > > > > > >>>> together
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>> the one partition log. The crux of the Diskless idea is
> > to
> > > > > > >>>> decouple
> > > > > > >>>>>>> them
> > > > > > >>>>>>>>> and move data to the remote storage, while keeping
> > metadata
> > > > > > >>>> somewhere
> > > > > > >>>>>>> else.
> > > > > > >>>>>>>>> Using the central batch coordinator for managing batch
> > > > metadata
> > > > > > >>>> is
> > > > > > >>>>>> one
> > > > > > >>>>>>> way,
> > > > > > >>>>>>>>> but not the only.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Let’s now think about managing metadata for each user
> > > > partition
> > > > > > >>>>>>>>> independently. Generally partitions are independent and
> > > don’t
> > > > > > >>>> share
> > > > > > >>>>>>>>> anything apart from that their data are mixed in WAL
> > files.
> > > > If
> > > > > we
> > > > > > >>>>>>> figure
> > > > > > >>>>>>>>> out how to commit and later delete WAL files safely, we
> > > will
> > > > > > >>>> achieve
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> > central
> > > > > batch
> > > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition will
> > be
> > > > > > >>>> managed
> > > > > > >>>>>> by
> > > > > > >>>>>>> its
> > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in classic
> > > > > > >>>> topics, the
> > > > > > >>>>>>>>> leader uses the partition log as the way to persist batch
> > > > > > >>>> metadata,
> > > > > > >>>>>>> i.e.
> > > > > > >>>>>>>>> the regular batch header + the information about how to
> > > find
> > > > > this
> > > > > > >>>>>>> batch on
> > > > > > >>>>>>>>> remote storage. In contrast to classic topics, batch data
> > > is
> > > > in
> > > > > > >>>>>> remote
> > > > > > >>>>>>>>> storage.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > >>>>>>>>> • Classic topics:
> > > > > > >>>>>>>>>  • Data and metadata are co-located in the partition log.
> > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > (metadata)|Batch
> > > > > > >>>> data].
> > > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > > > >>>> metadata.
> > > > > > >>>>>>>>> • Original Diskless:
> > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> > remote
> > > > > > >>>> storage.
> > > > > > >>>>>>>>>  • The partition state is global in the batch
> > coordinator.
> > > > > > >>>>>>>>> • New Diskless:
> > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on remote
> > > > storage.
> > > > > > >>>>>>>>>  • Partition log content: [Batch header (metadata)|Batch
> > > > > > >>>>>> coordinates
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>> remote storage].
> > > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > > > >>>> metadata.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the reminder of
> > the
> > > > > > >>>> original
> > > > > > >>>>>>>>> Diskless design:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The new approach could be depicted as the following:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> As you can see, the main difference is that now instead
> > of
> > > a
> > > > > > >>>> single
> > > > > > >>>>>>> commit
> > > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> > parallel
> > > > > > >>>> commit
> > > > > > >>>>>>> requests
> > > > > > >>>>>>>>> to all the leaders of each partition involved in the WAL
> > > > file.
> > > > > > >>>> Each
> > > > > > >>>>>> of
> > > > > > >>>>>>> them
> > > > > > >>>>>>>>> will commit its batches independently, without
> > coordinating
> > > > > with
> > > > > > >>>>>> other
> > > > > > >>>>>>>>> leaders and any other components. Batch data is addressed
> > > by
> > > > > the
> > > > > > >>>> WAL
> > > > > > >>>>>>> file
> > > > > > >>>>>>>>> name, the byte offset and size, which allows partitions
> > to
> > > > know
> > > > > > >>>>>> nothing
> > > > > > >>>>>>>>> about other partitions to access their data in shared WAL
> > > > > files.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The number of partitions involved in a single WAL file
> > may
> > > be
> > > > > > >>>> quite
> > > > > > >>>>>>> large,
> > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to commit one
> > > WAL
> > > > > > >>>> file is
> > > > > > >>>>>>> very
> > > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> > number:
> > > > > > >>>>>>>>> 1. Partition leaders are located on brokers. Requests to
> > > > > > >>>> leaders on
> > > > > > >>>>>>> one
> > > > > > >>>>>>>>> broker could be grouped together into a single physical
> > > > network
> > > > > > >>>>>> request
> > > > > > >>>>>>>>> (resembling the normal Produce request that may carry
> > > batches
> > > > > for
> > > > > > >>>>>> many
> > > > > > >>>>>>>>> partitions inside). This will cap the number of network
> > > > > requests
> > > > > > >>>> to
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make producers
> > send
> > > > > their
> > > > > > >>>>>>> requests
> > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> > achieve
> > > > the
> > > > > > >>>> higher
> > > > > > >>>>>>>>> concentration of logical commit requests in physical
> > > network
> > > > > > >>>> requests
> > > > > > >>>>>>>>> reducing the number of the latter ones even further,
> > > ideally
> > > > to
> > > > > > >>>> one.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Obviously, out of multiple commit requests some may fail
> > or
> > > > > time
> > > > > > >>>> out
> > > > > > >>>>>>> for a
> > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers will
> > > receive
> > > > > > >>>> totally
> > > > > > >>>>>>> or
> > > > > > >>>>>>>>> partially failed responses to their Produce requests,
> > > similar
> > > > > to
> > > > > > >>>> what
> > > > > > >>>>>>> they
> > > > > > >>>>>>>>> would have received when appending to a classic topic
> > fails
> > > > or
> > > > > > >>>> times
> > > > > > >>>>>>> out.
> > > > > > >>>>>>>>> If a partition experiences problems, other partitions
> > will
> > > > not
> > > > > be
> > > > > > >>>>>>> affected
> > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > uncommitted
> > > > > data
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > > (batches
> > > > > are
> > > > > > >>>>>>> constantly
> > > > > > >>>>>>>>> assembled into segments and offloaded to tiered storage),
> > > so
> > > > > this
> > > > > > >>>>>>> garbage
> > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> For safely deleting WAL files we now need to centrally
> > > manage
> > > > > > >>>> them,
> > > > > > >>>>>> as
> > > > > > >>>>>>>>> this is the only state and logic that spans multiple
> > > > > partitions.
> > > > > > >>>> On
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> diagram, you can see another commit request called
> > “Commit
> > > > file
> > > > > > >>>> (best
> > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This manager will
> > > be
> > > > > > >>>>>>> responsible
> > > > > > >>>>>>>>> for the following:
> > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and persisting
> > > > > > >>>> information
> > > > > > >>>>>>> about
> > > > > > >>>>>>>>> committed WAL files.
> > > > > > >>>>>>>>> 2. To handle potential failures in file information
> > > delivery,
> > > > > it
> > > > > > >>>>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> doing prefix scan on the remote storage periodically to
> > > find
> > > > > and
> > > > > > >>>>>>> register
> > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > configurable
> > > > and
> > > > > > >>>>>> ideally
> > > > > > >>>>>>>>> should be quite long.
> > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders (after a
> > > > grace
> > > > > > >>>>>>> period) if
> > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t anymore
> > > > referred
> > > > > > >>>> to by
> > > > > > >>>>>>> any
> > > > > > >>>>>>>>> partition.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > features
> > > > such
> > > > > > >>>> as
> > > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > > retention.
> > > > > > >>>> Now we
> > > > > > >>>>>>> don’t
> > > > > > >>>>>>>>> need to abstract away and reuse the code from partition
> > > > leaders
> > > > > > >>>> in
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use the
> > same
> > > > code
> > > > > > >>>> paths
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from classic
> > > > topics
> > > > > > >>>> mostly
> > > > > > >>>>>>>>> remain unchanged.
> > > > > > >>>>>>>>> For example, it seems that
> > > > > > >>>>>>>>>
> > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > and
> > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > transaction
> > > > > > >>>> support
> > > > > > >>>>>> on
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> partition leader side could be used for diskless topics
> > > with
> > > > > > >>>> little
> > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > idempotent
> > > > > > >>>> produce
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > >>>>>>>>> Another example is share groups support, where the share
> > > > > > >>>> partition
> > > > > > >>>>>>> leader,
> > > > > > >>>>>>>>> being co-located with the partition leader, would execute
> > > the
> > > > > > >>>> same
> > > > > > >>>>>>> logic
> > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > >>>>>>>>> 2. It returns to the familiar partition-based scaling
> > > model,
> > > > > > >>>> where
> > > > > > >>>>>>>>> partitions are independent.
> > > > > > >>>>>>>>> 3. It makes the operation and failure patterns closer to
> > > the
> > > > > > >>>>>> familiar
> > > > > > >>>>>>>>> ones from classic topics.
> > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless switching
> > > the
> > > > > > >>>> topics
> > > > > > >>>>>>> modes
> > > > > > >>>>>>>>> between diskless and classic.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The rest of the things remain unchanged compared to the
> > > > > previous
> > > > > > >>>>>>> Diskless
> > > > > > >>>>>>>>> design (after all previous discussions). Such things as
> > > local
> > > > > > >>>> segment
> > > > > > >>>>>>>>> materialization by replicas, the consume path, tiered
> > > storage
> > > > > > >>>>>>> integration,
> > > > > > >>>>>>>>> etc.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> If the community finds this design more suitable, we will
> > > > > update
> > > > > > >>>> the
> > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it. Please let
> > > us
> > > > > know
> > > > > > >>>>>> what
> > > > > > >>>>>>> you
> > > > > > >>>>>>>>> think.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Best regards,
> > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > > >>>>>>>>>> Hi Justine,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > transactions
> > > > > > >>>> for in
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> diskless coordinator for as long as the corresponding
> > > offsets
> > > > > are
> > > > > > >>>>>>> there.
> > > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> > earlier,
> > > > > this
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best,
> > > > > > >>>>>>>>>> Ivan
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks for the response. I think most of what you said
> > > made
> > > > > > >>>>>> sense,
> > > > > > >>>>>>> but
> > > > > > >>>>>>>>> I
> > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > > > > >>>> topics
> > > > > > >>>>>>> forgets
> > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > overpasses
> > > > > > >>>> it). The
> > > > > > >>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > > allowing
> > > > > > >>>>>>> partition
> > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > > > > > >>>> wrong. We
> > > > > > >>>>>>> think
> > > > > > >>>>>>>>>>> about relying on this with the batch coordinator and
> > > delete
> > > > > > >>>> the
> > > > > > >>>>>>>>> information
> > > > > > >>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > > > > >>>> replication
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> HWM
> > > > > > >>>>>>>>>>> advances immediately).
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I didn't quite understand this. In classic topics, we
> > > have
> > > > > > >>>> maps
> > > > > > >>>>>> for
> > > > > > >>>>>>>>> ongoing
> > > > > > >>>>>>>>>>> transactions which remove state when the transaction is
> > > > > > >>>> completed
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> an
> > > > > > >>>>>>>>>>> aborted transactions index which is retained for much
> > > > longer.
> > > > > > >>>>>> Once
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no longer
> > > > > > >>>> involved
> > > > > > >>>>>> in
> > > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> > subject
> > > to
> > > > > > >>>>>>> compaction
> > > > > > >>>>>>>>> etc.
> > > > > > >>>>>>>>>>> Looking back at the outline provided above, I didn't
> > see
> > > > much
> > > > > > >>>>>>> about the
> > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > > further.
> > > > I
> > > > > > >>>> saw
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> following in a response:
> > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > segment,
> > > > > > >>>> all the
> > > > > > >>>>>>>>> necessary
> > > > > > >>>>>>>>>>> control batches will be inserted and indices, including
> > > the
> > > > > > >>>>>>> transaction
> > > > > > >>>>>>>>>>> index will be built to serve FetchRequests exactly as
> > > they
> > > > > > >>>> are
> > > > > > >>>>>>> today.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain the
> > > > > > >>>> information
> > > > > > >>>>>>> about
> > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>> Justine
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > > >>>> ivan@ivanyu.me>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > identified
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer ID
> > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > > > > >>>> cached
> > > > > > >>>>>>>>> locally
> > > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > > >>>>>> transactions
> > > > > > >>>>>>> can
> > > > > > >>>>>>>>> be
> > > > > > >>>>>>>>>>>> used
> > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer id +
> > > > > > >>>>>>>>>>>>> epoch
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> > focused
> > > > > > >>>> only on
> > > > > > >>>>>>>>> version 2.
> > > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> > consider
> > > > > > >>>> using
> > > > > > >>>>>>> some
> > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > transactional
> > > > > > >>>>>>> checks
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > > > > >>>> like the
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > > > > >>>>>> checking
> > > > > > >>>>>>> if
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > > ongoing,
> > > > > > >>>> and
> > > > > > >>>>>> of
> > > > > > >>>>>>>>> course
> > > > > > >>>>>>>>>>>> the normal idempotence checks. What the partition
> > leader
> > > > > > >>>> in the
> > > > > > >>>>>>>>> classic
> > > > > > >>>>>>>>>>>> topics does before appending a batch to the local log
> > > > > > >>>> (e.g. in
> > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > > Diskless,
> > > > > > >>>> we
> > > > > > >>>>>>>>> unfortunately
> > > > > > >>>>>>>>>>>> cannot do these checks before appending the data to
> > the
> > > > WAL
> > > > > > >>>>>>> segment
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these batches in
> > > the
> > > > > > >>>> batch
> > > > > > >>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> > other
> > > > > > >>>> state
> > > > > > >>>>>>>>> mentioned
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > > > > >>>> what
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>> stored
> > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As the
> > > > > > >>>> partition
> > > > > > >>>>>>> leader
> > > > > > >>>>>>>>> tracks
> > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> > > > > > >>>> coordinator
> > > > > > >>>>>>> has
> > > > > > >>>>>>>>> to as
> > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > > >>>> transaction
> > > > > > >>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > > > > >>>> perhaps
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>> would
> > > > > > >>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed batches
> > > and
> > > > > > >>>> other
> > > > > > >>>>>>>>>>>> information, but for the sake of performance we’d
> > better
> > > > > > >>>> store
> > > > > > >>>>>> it
> > > > > > >>>>>>>>>>>> explicitly.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > transactional
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will
> > be
> > > > > > >>>>>> cleaned
> > > > > > >>>>>>> up?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > > > > >>>> topics
> > > > > > >>>>>>> forgets
> > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > overpasses
> > > > > > >>>> it).
> > > > > > >>>>>> The
> > > > > > >>>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > > > > >>>> allowing
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if this
> > is
> > > > > > >>>> wrong.
> > > > > > >>>>>> We
> > > > > > >>>>>>>>> think
> > > > > > >>>>>>>>>>>> about relying on this with the batch coordinator and
> > > > > > >>>> delete the
> > > > > > >>>>>>>>> information
> > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > > > > >>>>>> replication
> > > > > > >>>>>>>>> and HWM
> > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>> Ivan
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Excited to see some updates related to transactions!
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > identified
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer ID
> > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > > > > >>>> cached
> > > > > > >>>>>>>>> locally
> > > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > > >>>>>> transactions
> > > > > > >>>>>>> can
> > > > > > >>>>>>>>> be
> > > > > > >>>>>>>>>>>> used
> > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer id +
> > > > > > >>>>>>>>>>>>> epoch
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > transactional
> > > > > > >>>>>>> checks
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > > > > >>>> like the
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > > > > >>>>>> checking
> > > > > > >>>>>>> if
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is there
> > > state
> > > > > > >>>>>> about
> > > > > > >>>>>>>>> ongoing
> > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> > other
> > > > > > >>>> state
> > > > > > >>>>>>>>> mentioned
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > > > > >>>> what
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>> stored
> > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > > > > >>>> perhaps
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>> would
> > > > > > >>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > transactional
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will
> > be
> > > > > > >>>>>> cleaned
> > > > > > >>>>>>> up?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> > > > > > >>>>>> segments,
> > > > > > >>>>>>>>> making
> > > > > > >>>>>>>>>>>> use of
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > zero-copy
> > > > > > >>>>>>>>> functionality
> > > > > > >>>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
> > > > > > >>>> each
> > > > > > >>>>>>>>> partition in
> > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
> > > > > > >>>> need
> > > > > > >>>>>> to
> > > > > > >>>>>>> know
> > > > > > >>>>>>>>>>>> aborted
> > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > >>>> transaction is
> > > > > > >>>>>>>>> finished
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > corresponding
> > > > > > >>>> logs
> > > > > > >>>>>>> of the
> > > > > > >>>>>>>>>>>> involved
> > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
> > > > > > >>>> level,
> > > > > > >>>>>> no
> > > > > > >>>>>>>>> actual
> > > > > > >>>>>>>>>>>> control
> > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > transactional
> > > > > > >>>>>>> batches.
> > > > > > >>>>>>>>> How
> > > > > > >>>>>>>>>>>> does the
> > > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> > control
> > > > > > >>>>>> batch
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> each
> > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response needs to
> > > be
> > > > > > >>>>>>> built by
> > > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> > batches
> > > > > > >>>>>>> together?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition
> > leader
> > > > > > >>>> that
> > > > > > >>>>>>> all
> > > > > > >>>>>>>>>>>> consumers
> > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this
> > or
> > > > > > >>>> it's
> > > > > > >>>>>>> too
> > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> > include
> > > > > > >>>>>> enough
> > > > > > >>>>>>>>> details
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
> > > > > > >>>>>> producer
> > > > > > >>>>>>>>> contain a
> > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a producer
> > sends
> > > > > > >>>>>> every
> > > > > > >>>>>>>>> Produce
> > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests could
> > > > > > >>>> reach
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> batch
> > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to rejection
> > > > > > >>>> of the
> > > > > > >>>>>>>>> produce
> > > > > > >>>>>>>>>>>>>> requests.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > >>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details about
> > > > > > >>>>>>> transactions
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>>>> queues,
> > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic topics* is
> > > > > > >>>> based
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>> precise
> > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients (mostly
> > > > > > >>>>>>> producers,
> > > > > > >>>>>>>>> but
> > > > > > >>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > > >>>> classes),
> > > > > > >>>>>> and
> > > > > > >>>>>>>>>>>> transaction
> > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders
> > with
> > > > > > >>>>>> their
> > > > > > >>>>>>>>> local
> > > > > > >>>>>>>>>>>> state
> > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The high level (some details skipped) workflow is
> > the
> > > > > > >>>>>>>>> following.
> > > > > > >>>>>>>>>>>> When a
> > > > > > >>>>>>>>>>>>>>> transactional Produce request is received by the
> > > > > > >>>> broker:
> > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader checks
> > > > > > >>>> if a
> > > > > > >>>>>>>>> non-empty
> > > > > > >>>>>>>>>>>>>>> transaction is running for this partition. This is
> > > > > > >>>> done
> > > > > > >>>>>>> using
> > > > > > >>>>>>>>> its
> > > > > > >>>>>>>>>>>> local
> > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > >>>>>> (ProducerStateManager,
> > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed about
> > all
> > > > > > >>>> the
> > > > > > >>>>>>>>> partitions
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > transactional
> > > > > > >>>>>>> checks.
> > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the transactional
> > > > > > >>>> data to
> > > > > > >>>>>>>>> their logs
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>> update some of their state (for example, log the
> > fact
> > > > > > >>>>>> that
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>>>> transaction
> > > > > > >>>>>>>>>>>>>>> is running for the partition and its first offset).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > coordinator
> > > > > > >>>>>>> directly
> > > > > > >>>>>>>>> with
> > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > PREPARE_COMMIT
> > > > > > >>>> or
> > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> leaders
> > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> > > > > > >>>> markers to
> > > > > > >>>>>>>>> their logs
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction
> > state
> > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
> > > > > > >>>> of
> > > > > > >>>>>>>>> important
> > > > > > >>>>>>>>>>>> state
> > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is local.
> > The
> > > > > > >>>> main
> > > > > > >>>>>>>>> challenge
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
> > > > > > >>>> are no
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
> > > > > > >>>> to be
> > > > > > >>>>>>>>> globalized
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> > > > > > >>>> support
> > > > > > >>>>>>>>> idempotent
> > > > > > >>>>>>>>>>>>>> produce.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics* would
> > > > > > >>>> look
> > > > > > >>>>>>> very
> > > > > > >>>>>>>>>>>> similar:
> > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > > >>>> non-empty
> > > > > > >>>>>>>>> transaction
> > > > > > >>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to classic
> > > > > > >>>>>> topics,
> > > > > > >>>>>>>>> this is
> > > > > > >>>>>>>>>>>>>> checked
> > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
> > > > > > >>>> Since a
> > > > > > >>>>>>>>> transaction
> > > > > > >>>>>>>>>>>>>> could
> > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
> > > > > > >>>> the
> > > > > > >>>>>>> positive
> > > > > > >>>>>>>>>>>> result of
> > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the double
> > > > > > >>>>>>> configured
> > > > > > >>>>>>>>>>>> duration
> > > > > > >>>>>>>>>>>>>> of a
> > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > informed
> > > > > > >>>>>> about
> > > > > > >>>>>>> all
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
> > > > > > >>>> include
> > > > > > >>>>>>> them.
> > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
> > > > > > >>>> side.
> > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data to the
> > > > > > >>>>>> current
> > > > > > >>>>>>>>> shared
> > > > > > >>>>>>>>>>>> WAL
> > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
> > > > > > >>>> state
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> Diskless
> > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > > >>>> coordinator
> > > > > > >>>>>>> like
> > > > > > >>>>>>>>> in the
> > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > transactional
> > > > > > >>>>>>> checks
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> > errors
> > > > > > >>>> like
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> > batches
> > > > > > >>>>>> could
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>> rejected.
> > > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage in
> > the
> > > > > > >>>> WAL
> > > > > > >>>>>>>>> segment
> > > > > > >>>>>>>>>>>> file in
> > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is preferable to
> > > > > > >>>> doing
> > > > > > >>>>>>> more
> > > > > > >>>>>>>>>>>> network
> > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > segments
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>> relatively
> > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > coordinator
> > > > > > >>>>>>> directly
> > > > > > >>>>>>>>> with
> > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > PREPARE_COMMIT
> > > > > > >>>> or
> > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
> > > > > > >>>> batch
> > > > > > >>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> > > > > > >>>>>>> transaction is
> > > > > > >>>>>>>>>>>> finished
> > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > >>>> corresponding
> > > > > > >>>>>>> logs
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only on the
> > > > > > >>>>>> metadata
> > > > > > >>>>>>>>> level, no
> > > > > > >>>>>>>>>>>>>>> actual control batches are written to any file.
> > They
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>> dynamically
> > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations. We
> > could
> > > > > > >>>>>>>>> technically
> > > > > > >>>>>>>>>>>> write
> > > > > > >>>>>>>>>>>>>>> these control batches for real, but this would mean
> > > > > > >>>> extra
> > > > > > >>>>>>>>> produce
> > > > > > >>>>>>>>>>>>>> latency,
> > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > > >>>> coordinator
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> save
> > > > > > >>>>>>>>>>>> these
> > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> leaders
> > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
> > > > > > >>>> topics
> > > > > > >>>>>>> now.
> > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics write
> > the
> > > > > > >>>>>>>>> transaction
> > > > > > >>>>>>>>>>>> markers
> > > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction
> > state
> > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
> > > > > > >>>> get:
> > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between brokers and
> > > > > > >>>> the
> > > > > > >>>>>>> batch
> > > > > > >>>>>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>>>>> when a new partition appear in the transaction. To
> > > > > > >>>>>>> mitigate the
> > > > > > >>>>>>>>>>>> impact of
> > > > > > >>>>>>>>>>>>>>> them:
> > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one Produce
> > > > > > >>>>>>> request
> > > > > > >>>>>>>>> will be
> > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> > > > > > >>>> fast
> > > > > > >>>>>>>>> response to
> > > > > > >>>>>>>>>>>> these
> > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally will
> > > > > > >>>>>>> communicate
> > > > > > >>>>>>>>> with a
> > > > > > >>>>>>>>>>>>>>> single broker for the duration of the transaction
> > > > > > >>>> further
> > > > > > >>>>>>>>> reduces the
> > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> > > > > > >>>>>> coordinator
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> batch
> > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be able
> > to
> > > > > > >>>>>> span
> > > > > > >>>>>>> both
> > > > > > >>>>>>>>>>>> classic
> > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The share group coordination and management is a
> > > > > > >>>> side job
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>>>>> doesn't
> > > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > > >>>> replicas,
> > > > > > >>>>>>> physical
> > > > > > >>>>>>>>>>>> storage
> > > > > > >>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > consumers
> > > > > > >>>>>>> (Fetch and
> > > > > > >>>>>>>>>>>> Produce
> > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > affected.)
> > > > > > >>>> We
> > > > > > >>>>>>> don't
> > > > > > >>>>>>>>> see any
> > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
> > > > > > >>>> with
> > > > > > >>>>>>> share
> > > > > > >>>>>>>>>>>> groups the
> > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the code
> > > > > > >>>> level,
> > > > > > >>>>>> we
> > > > > > >>>>>>>>> don't
> > > > > > >>>>>>>>>>>> expect
> > > > > > >>>>>>>>>>>>>> any
> > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading routines are
> > > > > > >>>> used
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>> are
> > > > > > >>>>>>>>>>>> used for
> > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
> > > > > > >>>> too
> > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design input
> > > > > > >>>> on
> > > > > > >>>>>>>>> KIP-1150.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
> > > > > > >>>> new
> > > > > > >>>>>>>>> design. To
> > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> > > > > > >>>> existing
> > > > > > >>>>>>>>> KIP-405
> > > > > > >>>>>>>>>>>> Tiered
> > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
> > > > > > >>>>>>> Diskless
> > > > > > >>>>>>>>> topic
> > > > > > >>>>>>>>>>>> to be
> > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for the
> > > > > > >>>> Batch
> > > > > > >>>>>>>>> Coordinator
> > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> > > > > > >>>> Tiered
> > > > > > >>>>>>> Storage
> > > > > > >>>>>>>>>>>> plugin
> > > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative data
> > > > > > >>>>>> formats.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> > > > > > >>>> segments,
> > > > > > >>>>>>>>> making use
> > > > > > >>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > > >>>> zero-copy
> > > > > > >>>>>>>>> functionality
> > > > > > >>>>>>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered cache
> > > > > > >>>>>>> elements,
> > > > > > >>>>>>>>> do not
> > > > > > >>>>>>>>>>>> need
> > > > > > >>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > > >>>> contacting
> > > > > > >>>>>>> any
> > > > > > >>>>>>>>> other
> > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified substantially,
> > by
> > > > > > >>>>>>> removing
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> previous
> > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > >>>> component, and
> > > > > > >>>>>>>>> "object
> > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > > >>>> enabled by
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> Batch
> > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
> > > > > > >>>>>> earlier
> > > > > > >>>>>>>>> design,
> > > > > > >>>>>>>>>>>> while
> > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into the
> > > > > > >>>> existing
> > > > > > >>>>>>>>> ecosystem.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
> > > > > > >>>> the
> > > > > > >>>>>> new
> > > > > > >>>>>>>>> design.
> > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> > > > > > >>>> of
> > > > > > >>>>>> all
> > > > > > >>>>>>>>> existing
> > > > > > >>>>>>>>>>>>>> client
> > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > >>>> supporting
> > > > > > >>>>>> APIs
> > > > > > >>>>>>>>> like
> > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> > > > > > >>>>>>> status of
> > > > > > >>>>>>>>> this
> > > > > > >>>>>>>>>>>> KIP?
> > > > > > >>>>>>>>>>>>>> I
> > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> > > > > > >>>>>>> months,
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>> vote got
> > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> > > > > > >>>>>> answered
> > > > > > >>>>>>>>> since
> > > > > > >>>>>>>>>>>> then.
> > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > >>>> activity
> > > > > > >>>>>>> since
> > > > > > >>>>>>>>> May.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> > > > > > >>>>>> choices
> > > > > > >>>>>>>>> that can
> > > > > > >>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> > > > > > >>>> by
> > > > > > >>>>>> far
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>> dominant
> > > > > > >>>>>>>>>>>>>> one
> > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
> > > > > > >>>>>>>>> standalone
> > > > > > >>>>>>>>>>>>>> products.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> > > > > > >>>> hate to
> > > > > > >>>>>>> see a
> > > > > > >>>>>>>>>>>> stall. So
> > > > > > >>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> > > > > > >>>>>> McCabe -
> > > > > > >>>>>>>>> have your
> > > > > > >>>>>>>>>>>>>>> questions
> > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> > > > > > >>>> do
> > > > > > >>>>>> to
> > > > > > >>>>>>>>> help? I'm
> > > > > > >>>>>>>>>>>>>> deeply
> > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > > >>>> written
> > > > > > >>>>>>> about
> > > > > > >>>>>>>>> both?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > > >>>>>> Kozlovski
> > > > > > >>>>>>> <
> > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > > >>>> main
> > > > > > >>>>>> one
> > > > > > >>>>>>>>> (just a
> > > > > > >>>>>>>>>>>> bunch
> > > > > > >>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > >>>> example, I
> > > > > > >>>>>>> was
> > > > > > >>>>>>>>>>>> searching
> > > > > > >>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> > > > > > >>>> far
> > > > > > >>>>>> as
> > > > > > >>>>>>> I
> > > > > > >>>>>>>>> can
> > > > > > >>>>>>>>>>>> tell, it
> > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> > > > > > >>>> having
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> root
> > > > > > >>>>>>>>>>>> KIP be
> > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> > > > > > >>>> to
> > > > > > >>>>>>>>> sub-KIPs
> > > > > > >>>>>>>>>>>> where
> > > > > > >>>>>>>>>>>>>> were
> > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> > > > > > >>>> would
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> good to
> > > > > > >>>>>>>>>>>>>> create
> > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> > > > > > >>>>>>>>> introducing a
> > > > > > >>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>> type of
> > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > >>>> Storage as
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> primary
> > > > > > >>>>>>>>>>>>>>> source of
> > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > > >>>> decided
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>> split it
> > > > > > >>>>>>>>>>>>>> into
> > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > >>>> should aim
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>> have
> > > > > > >>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>> type of
> > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > > >>>>>> details
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>> how to
> > > > > > >>>>>>>>>>>>>>> implement
> > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
> > > > > > >>>>>>> discussed
> > > > > > >>>>>>>>>>>> KRaft.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> > > > > > >>>> hard
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>> discuss
> > > > > > >>>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > > >>>> (linked
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > >>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> > > > > > >>>>>>> threads
> > > > > > >>>>>>>>> for
> > > > > > >>>>>>>>>>>> each
> > > > > > >>>>>>>>>>>>>>> type of
> > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > >>>> whether
> > > > > > >>>>>>> Apache
> > > > > > >>>>>>>>> Kafka
> > > > > > >>>>>>>>>>>> wants
> > > > > > >>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > >>>> authors
> > > > > > >>>>>> of
> > > > > > >>>>>>>>> this KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> > > > > > >>>>>>> aiven.io
> > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > > >>>>>>> Valtonen,
> > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Anatolii Popov
> > Senior Software Developer, *Aiven OY*
> > m: +358505126242
> > w: aiven.io  e: anatolii.popov@aiven.io
> > <https://www.facebook.com/aivencloud>
> > <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Robin Han <ha...@gmail.com>]

I'm glad to see that the Diskless Topic has entered the voting stage in
other threads.
KIP-1163: The design of Diskless Core retains LogSegment and Tiered
Storage.
My understanding is that the main problem solved by the current Diskless
Topic is
the cross-AZ traffic cost. If this is the main purpose, there might be a
simpler solution
that does not require modifications to Coordinator, Fetch, Transaction,
etc.
(I apologize for raising this question at this time. It's because there
were quite a lot of discussions about Diskless Topic before, and I didn't
keep up with them.) The new design is inspired by KIP-1163, KIP-1176 and
AutoMQ *Produce Path*:

   1. Producers send Produce requests to any broker (BrokerN).
   2. The BrokerN accumulates Produce requests in a buffer until exceeding
   some size or time limit.
   3. When enough data accumulates or the timeout elapses, the BrokerN
   upload a object for all of the buffered batches to object storage.
   4. BrokerN sends a ProxyRequest to BrokerM which is the real leader of
   the partition. The ProxyRequest contains the object key and the location
   information of the batches.
   5. BrokerM reads Batches from the object and writes them into Partition
   (Leader) as the classic topic. And caches the metadata <partition, offset,
   timestamp, object, positionInObject> in memory for subsequent ISR
   replication.
   6. [ISR Replication] The Follower of the Partition sends a Fetch
   replication request to BrokerM.
   1. If BrokerM has a metadata cache of <partition, offset, timestamp,
      object, positionInObject>, return that metadata. The Follower reads the
      object, combines it with the offset and timestamp to generate the final
      RecordBatch, and writes it into the LocalLog.
      2. If BrokerM does not have a metadata cache, then BrokerM returns
      the data of LocalLog (Leader) and sends it back.
      7. BrokerM moves the highwatermark.
   8. BrokerM responds ProxyResponse to BrokerN and BrokerN responds to
   Producer.

The object is only used to solve the cross-az traffic issued by Produce
forwarding and ISR replication.
There is no need for persistent storage of metadata, and it can be designed
to be cleaned up 1 minute after writing.
Even if it is accidentally deleted, it can still continue to work：

   - Accidentally deleted during the Produce forwarding phase: return a
   sending failure to the client, and the client will retry sending.
   - Accidentally deleted during the ISR replication phase: ISR replication
   can fall back to the replication mode of directly returning LocalLog data
   for the classic topic.

*Consume Path*:
It's the same as the classic topic *Transaction*:
It's the same as the classic topic *Tiered Storage*:
It's the same as the classic topic

There is no additional Coordinator, no need to rewrite a large amount of
Kafka logic for Diskless Topic, just use Object to eliminate cross-AZ
traffic.


Jun Rao via dev <de...@kafka.apache.org> 于2026年1月13日周二 02:38写道：

> Hi, Anatolii,
>
> Thanks for the updated KIP. Looks good to me overall. A few more comments.
>
> JR10. "Even in the case where network traffic is not accounted; we believe
> that the operational benefits of Diskless topics are still appealing to
> Kafka users."
> Could you list other benefits such as better scalability and durability?
>
> JR11. "AWS charges 0.02$ per GiB"
> The object store is not free and has a different cost model based on
> requests. It would be useful summarize the tradeoff on cost and latency,
> and when diskless topics are cost effective.
>
> JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't been
> updated in the last 6 months. The metadata change in KIP-1181 is now
> included in KIP-1163.
>
> Jun
>
> On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <
> dev@kafka.apache.org>
> wrote:
>
> > Hi all,
> >
> > Thank you for the extensive feedback.
> >
> > We have substantially updated the KIP to address the points raised. Given
> > the scope of these changes, we have started a new VOTE thread to restart
> > the voting process cleanly.
> >
> > You can find the new thread here:
> > https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
> >
> > Best,
> > Anatolii
> >
> > On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> > dev@kafka.apache.org> wrote:
> >
> > > Hi all,
> > >
> > > We (the team at Slack) have been following the recent discussion
> > regarding
> > > the scope and timeline of KIP-1150. We agree with the community
> sentiment
> > > that Diskless Topics represents the right long-term architecture for
> > Kafka
> > > in the cloud, but we also recognize the valid concerns raised regarding
> > the
> > > engineering resources required to deliver such an ambitious change.
> > >
> > > To help address these concerns and accelerate the timeline, we are
> happy
> > to
> > > announce that we are partnering with the KIP-1150 authors to co-develop
> > > this feature.
> > >
> > > We previously proposed KIP-1176: Tiered Storage for Active Log Segments
> > to
> > > solve similar problems. However, rather than fragmenting the
> community's
> > > efforts across competing designs, we have decided to withdraw KIP-1176
> > and
> > > consolidate our engineering resources behind KIP-1150.
> > >
> > > To start, we plan to take ownership of Compaction for Tiered Storage.
> > This
> > > has long been a missing feature in KIP-405, and it becomes critical in
> a
> > > Diskless architecture where long-lived data is obligated to be tiered.
> By
> > > driving this related prerequisite feature, we hope to allow for faster
> > > delivery of KIP-1150.
> > >
> > > We are excited to collaborate on this to ensure a robust and timely
> > > delivery for the community.
> > >
> > > Best,
> > > Tom & Henry
> > >
> > > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi Greg,
> > > >
> > > > Thanks for sharing the meeting notes.
> > > > I agree we should keep polishing the contents of 1150 & high level
> > design
> > > > in 1163 to prepare for a vote.
> > > >
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > There was a video call between myself, Ivan Yurchenko, Jun Rao, and
> > > > Andrew
> > > > > Schofield pertaining to KIP-1150. Here are the notes from that
> > meeting:
> > > > >
> > > > > Ivan: What is the future state of Kafka in this area, in 5 years?
> > > > > Jun: Do we want something more cloud native? Yes, started with
> Tiered
> > > > > Storage. If there’s a better way, we should explore it. In the long
> > > term
> > > > > this will be useful
> > > > > Because Kafka is used so widely, we need to make sure everything we
> > add
> > > > is
> > > > > for the long term and for everyone, not just for a single company.
> > > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > > something
> > > > > that’s high quality/lasts/maintainable, and can work with all
> > existing
> > > > > capabilities.
> > > > > If both 1150 and 1176 proceed at the same time, it’s confusing.
> They
> > > > > overlap, but Diskless is more ambitious.
> > > > > If both KIPs are being seriously worked on, then we don’t really
> need
> > > > both,
> > > > > because Diskless clearly is better. Having multiple will confuse
> > > people.
> > > > It
> > > > > will duplicate some of the effort.
> > > > > If we want diskless ultimately, what is the short term strategy, to
> > get
> > > > > some early wins first?
> > > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > > Andrew: Eventually the architecture will change substantially, it
> may
> > > not
> > > > > be necessary to put all of that bill onto Diskless at once.
> > > > > Greg: We all agree on having a high quality feature merged
> upstream,
> > > and
> > > > > supporting all APIs
> > > > > Jun: We should try and keep things simple, but there is some
> minimum
> > > > > complexity needed.
> > > > > When doing the short term changes (1176), it doesn’t really
> progress
> > in
> > > > > changing to a more modern architecture.
> > > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > > Jun: The danger of only applying changes to some part of the API,
> you
> > > set
> > > > > the precedence that you only have to implement part of the API.
> > > > Supporting
> > > > > the full API set should be a minimum requirement.
> > > > > Andrew: When we started Kraft, how much did we know the design?
> > > > > Jun: For Kraft we didn’t really know much about the migration, but
> > the
> > > > > high-level was clear.
> > > > > Greg: Is 1150 votable in its current state?
> > > > > Jun: 1150 should promise to support all APIs. It doesn’t have to
> have
> > > all
> > > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > > We do need some high-level design enough to give confidence that
> the
> > > > > promise is able to be fulfilled.
> > > > > Greg: Is the draft version in 1163 enough detail or is more needed?
> > > > > Jun: We need to agree on the core design, such as leaderless etc.
> And
> > > how
> > > > > the APIs will be supported.
> > > > > Greg: Okay we can include these things, and provide a sketch of how
> > the
> > > > > other leader-based features operate.
> > > > > Jun: Yeah if at a high level the sketch appears to work, we can
> > approve
> > > > > that functionality.
> > > > > Are you committed to doing the more involved and big project?
> > > > > Greg: Yes, we’re committed to the 1163 design and can’t really
> accept
> > > > 1176.
> > > > > Jun: TS was slow because of Uber resourcing problems
> > > > > Greg: We’ll push internally for resources, and use the community
> > > > sentiment
> > > > > to motivate Aiven.
> > > > > How far into the future should we look? What sort of scale?
> > > > > Jun: As long as there’s a path forward, and we’re not closing off
> > > future
> > > > > improvements, we can figure out how to handle a larger scale when
> it
> > > > > arises.
> > > > > Greg: Random replica placement is very harmful, can we recommend
> > users
> > > to
> > > > > use an external tool like CruiseControl?
> > > > > Jun: Not everyone uses CruiseControl, we would probably need some
> > > > solution
> > > > > for this out of the box
> > > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > > Jun: Out-of-box experience should be good, good to allow other
> > > > > implementations
> > > > > Greg: But it could hurt Kafka feature/upgrade velocity when we wait
> > for
> > > > > plugin providers to implement it
> > > > > Ivan: We imagined that maybe cloud hyperscalers could implement it
> > with
> > > > > e.g. dynamodb
> > > > > Greg: Could we bake more details of the different providers into
> > Kafka,
> > > > or
> > > > > does it still make sense for it to be pluggable?
> > > > > Jun: Make it whatever is easiest to roll out and add new clients
> > > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > > Ivan: The vote is already open, we’re not too pressed for time.
> We’ll
> > > go
> > > > > improve the 1163 design and communication.
> > > > > Is 1176 a competing design? Someone will ask.
> > > > > Jun: If we are seriously working on something more ambitious, yeah
> we
> > > > > shouldn’t do the stop-gap solution.
> > > > > It’s diverting review resources. If we can get the short term thing
> > in
> > > > 1yr
> > > > > but Diskless solution is 2y it makes sense to go for Diskless. If
> > it’s
> > > > 5yr,
> > > > > that’s different and maybe the stop-gap solution is needed.
> > > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should we
> > > > > explicitly exclude 1176?
> > > > > Andrew: Put your arms around the feature set you actually want, and
> > use
> > > > > that to rule out 1176.
> > > > > Probably don’t need -1 votes, most likely KIPs just don’t receive
> > > votes.
> > > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > > Jun: Satish posted meeting notes regularly, we should do the same.
> > > > >
> > > > > To summarize, we will be polishing the contents of 1150 & high
> level
> > > > design
> > > > > in 1163 to prepare for a vote.
> > > > > We believe that the community should select the feature set of 1150
> > to
> > > > > fully eliminate producer cross-zone costs, and make the investment
> > in a
> > > > > high quality Diskless Topics implementation rather than in stop-gap
> > > > > solutions.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:
> > > > >
> > > > > > This may be a tangent, but we needed to offload storage off of
> > Kafka
> > > > into
> > > > > > S3. We are keeping Kafka not as a source of truth, but as a
> mostly
> > > > > > ephemeral broker that can come and go as it pleases. Be that
> > scaling
> > > or
> > > > > > outage. Disks can be destroyed and recreated at will, we still
> > retain
> > > > > data
> > > > > > and use broker for just that, brokering messages. Not only that,
> we
> > > > > reduced
> > > > > > the requirement on the actual Kafka resources by reducing the
> size
> > > of a
> > > > > > payload via a claim check pattern. Maybe this is an anti–pattern,
> > but
> > > > it
> > > > > is
> > > > > > super fast and highly cost efficient. We reworked ProducerRequest
> > to
> > > > > allow
> > > > > > plugins. We added a custom http plugin that submits every request
> > > via a
> > > > > > persisted connection to a microservice. Microservice stores the
> > > payload
> > > > > and
> > > > > > returns a tiny json metadata object,a claim check, that can be
> used
> > > to
> > > > > find
> > > > > > the actual data. Think of it as zipping the payload. This claim
> > check
> > > > > > metadata traverses the pipelines with consumers using the urls in
> > > > > metadata
> > > > > > to pull what they need. Think unzipping. This allowed us to also
> > pull
> > > > > ONLY
> > > > > > the data that we need in graphql like manner. So if you have a
> 100K
> > > > json
> > > > > > payload and you need only a subsection, you can pull that by
> > > jmespath.
> > > > > When
> > > > > > you have multiple consumer groups yanking down huge payloads it
> is
> > > > > > cumbersome on the broker. When you have the same consumer groups
> > > > yanking
> > > > > > down a claim check, and then going out of band directly to the
> > source
> > > > of
> > > > > > truth, the broker has some breathing room. Obviously our
> > microservice
> > > > > does
> > > > > > not go directly to the cloud storage, as that would be too slow.
> It
> > > > > stores
> > > > > > the payload in high speed memory cache and returns asap. That
> > memory
> > > is
> > > > > > eventually persisted into S3. The retrieval goest against the
> cache
> > > > > first,
> > > > > > then against the S3. Overall a rather cheappy and zippy
> solution. I
> > > > tried
> > > > > > proposing the KIP for this, but there was no excitement. Check
> this
> > > > out:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > > >
> > > > > >
> > > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID>
> > > > wrote:
> > > > > > >
> > > > > > > Hi, Andrew,
> > > > > > >
> > > > > > > If we want to focus only on reducing cross-zone replication
> > costs,
> > > > > there
> > > > > > is
> > > > > > > an alternative design in the KIP-1176 discussion thread that
> > seems
> > > > > > simpler
> > > > > > > than the proposal here. I am copying the outline of that
> approach
> > > > > below.
> > > > > > >
> > > > > > > 1. A new leader is elected.
> > > > > > > 2. Leader maintains a first tiered offset, which is initialized
> > to
> > > > log
> > > > > > end
> > > > > > > offset.
> > > > > > > 3. Leader writes produced data from the client to local log.
> > > > > > > 4. Leader uploads produced data from all local logs as a
> combined
> > > > > object
> > > > > > > 5. Leader stores the metadata for the combined object in
> memory.
> > > > > > > 6. If a follower fetch request has an offset >= first tiered
> > > offset,
> > > > > the
> > > > > > > metadata for the corresponding combined object is returned.
> > > > Otherwise,
> > > > > > the
> > > > > > > local data is returned.
> > > > > > > 7. Leader periodically advances first tiered offset.
> > > > > > >
> > > > > > > It's still a bit unnatural, but it could work.
> > > > > > >
> > > > > > > Hi, Ivan,
> > > > > > >
> > > > > > > Are you still committed to proceeding with the original design
> of
> > > > > > KIP-1150?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > > andrew_schofield@live.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >> I’ve been following KIP-1150 and friends for a while. I’m
> going
> > to
> > > > > jump
> > > > > > >> into the discussions too.
> > > > > > >>
> > > > > > >> Looking back at Jack Vanlightly’s message, I am not quite so
> > > > convinced
> > > > > > >> that it’s a kind of fork in the road. The primary aim of the
> > > effort
> > > > is
> > > > > > to
> > > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > > prohibitively
> > > > > > >> expensive to use on cloud storage. I think it would be
> entirely
> > > > viable
> > > > > > to
> > > > > > >> prioritise code reuse for an initial implementation of
> diskless
> > > > > topics,
> > > > > > and
> > > > > > >> we could still have a more cloud-native design in the future.
> > It’s
> > > > > hard
> > > > > > to
> > > > > > >> predict what the community will prioritise in the future.
> > > > > > >>
> > > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> > support
> > > > > > >> leaderless produce requests, first writing WAL segments into
> > > object
> > > > > > >> storage, and then using the regular partition leaders to
> > sequence
> > > > the
> > > > > > >> records. The active log segment for a diskless topic will
> > > initially
> > > > > > contain
> > > > > > >> batch coordinates rather than record batches. The batch
> > > coordinates
> > > > > can
> > > > > > be
> > > > > > >> resolved from WAL segments for consumers, and also in order to
> > > > prepare
> > > > > > log
> > > > > > >> segments for uploading to tiered storage. Jun is probably
> > correct
> > > > that
> > > > > > we
> > > > > > >> need a more frequent object merging process than tiered
> storage
> > > > > > provides.
> > > > > > >> This is just the transition from write-optimised WAL segments
> to
> > > > > > >> read-optimised tiered segments, and all of the object
> > > storage-based
> > > > > > >> implementations of Kafka that I’m aware of do this
> > rearrangement.
> > > > But
> > > > > > >> perhaps this more frequent object merging is a pre-GA
> > improvement,
> > > > > > rather
> > > > > > >> than a strict requirement for an initial implementation for
> > early
> > > > > access
> > > > > > >> use.
> > > > > > >>
> > > > > > >> For zone-aligned share consumers, the share group assignor is
> > > > intended
> > > > > > to
> > > > > > >> be rack-aware. Consumers should be assigned to partitions with
> > > > leaders
> > > > > > in
> > > > > > >> their zone. The simple assignor is not rack-aware, but it
> easily
> > > > could
> > > > > > be
> > > > > > >> or we could have a rack-aware assignor.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Andrew
> > > > > > >>
> > > > > > >>
> > > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID>
> > > > wrote:
> > > > > > >>>
> > > > > > >>> Hi, Ivan,
> > > > > > >>>
> > > > > > >>> Thanks for the reply.
> > > > > > >>>
> > > > > > >>> "As I understand, you’re speaking about locally materialized
> > > > > segments.
> > > > > > >> They
> > > > > > >>> will indeed consume some IOPS. See them as a cache that could
> > > > always
> > > > > be
> > > > > > >>> restored from the remote storage. While it’s not ideal, it's
> > > still
> > > > OK
> > > > > > to
> > > > > > >>> lose data in them due to a machine crash, for example.
> Because
> > of
> > > > > this,
> > > > > > >> we
> > > > > > >>> can avoid explicit flushing on local materialized segments at
> > all
> > > > and
> > > > > > let
> > > > > > >>> the file system and page cache figure out when to flush
> > > optimally.
> > > > > This
> > > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > > dramatically,
> > > > > > >>> depending on throughput for each partition. We, of course,
> > > continue
> > > > > > >>> flushing the metadata segments as before."
> > > > > > >>>
> > > > > > >>> If we have a mix of classic and diskless topics on the same
> > > broker,
> > > > > > it's
> > > > > > >>> important that the classic topics' data is flushed to disk as
> > > > quickly
> > > > > > as
> > > > > > >>> possible. To achieve this, users typically set
> > > > dirty_expire_centisecs
> > > > > > in
> > > > > > >>> the kernel based on the number of available disk IOPS. Once
> you
> > > set
> > > > > > this
> > > > > > >>> number, it applies to all dirty files, including the cached
> > data
> > > in
> > > > > > >>> diskless topics. So, if there are more files actively
> > > accumulating
> > > > > > data,
> > > > > > >>> the flush frequency and therefore RPO is reduced for classic
> > > > topics.
> > > > > > >>>
> > > > > > >>> "We should have mentioned this explicitly, but this step, in
> > > fact,
> > > > > > >> remains
> > > > > > >>> in the form of segments offloading to tiered storage. When we
> > > > > assemble
> > > > > > a
> > > > > > >>> segment and hand it over to RemoteLogManager, we’re
> effectively
> > > > doing
> > > > > > >>> metadata compaction: replacing a big number of pieces of
> > metadata
> > > > > about
> > > > > > >>> individual batches with a single record in
> > > __remote_log_metadata."
> > > > > > >>>
> > > > > > >>> The object merging in tier storage typically only kicks in
> > after
> > > a
> > > > > few
> > > > > > >>> hours. The impact is (1) the amount of accumulated metadata
> is
> > > > still
> > > > > > >> quite
> > > > > > >>> large; (2) there are many small objects, leading to poor read
> > > > > > >> performance.
> > > > > > >>> I think we need a more frequent object merging process than
> > tier
> > > > > > storage
> > > > > > >>> provides.
> > > > > > >>>
> > > > > > >>> Jun
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> > ivan@ivanyu.me>
> > > > > > wrote:
> > > > > > >>>
> > > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > > >>>>
> > > > > > >>>> Thank you for your messages.
> > > > > > >>>>
> > > > > > >>>> Let me first address some of Jun’s comments.
> > > > > > >>>>
> > > > > > >>>>> First, it degrades the durability.
> > > > > > >>>>> For each partition, now there are two files being actively
> > > > written
> > > > > > at a
> > > > > > >>>>> given point of time, one for the data and another for the
> > > > metadata.
> > > > > > >>>>> Flushing each file requires a separate IO. If the disk has
> 1K
> > > > IOPS
> > > > > > and
> > > > > > >> we
> > > > > > >>>>> have 5K partitions in a broker, currently we can afford to
> > > flush
> > > > > each
> > > > > > >>>>> partition every 5 seconds, achieving an RPO of 5 seconds.
> If
> > we
> > > > > > double
> > > > > > >>>> the
> > > > > > >>>>> number of files per partition, we can only flush each
> > partition
> > > > > every
> > > > > > >> 10
> > > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > > >>>>
> > > > > > >>>> As I understand, you’re speaking about locally materialized
> > > > > segments.
> > > > > > >> They
> > > > > > >>>> will indeed consume some IOPS. See them as a cache that
> could
> > > > always
> > > > > > be
> > > > > > >>>> restored from the remote storage. While it’s not ideal, it's
> > > still
> > > > > OK
> > > > > > to
> > > > > > >>>> lose data in them due to a machine crash, for example.
> Because
> > > of
> > > > > > this,
> > > > > > >> we
> > > > > > >>>> can avoid explicit flushing on local materialized segments
> at
> > > all
> > > > > and
> > > > > > >> let
> > > > > > >>>> the file system and page cache figure out when to flush
> > > optimally.
> > > > > > This
> > > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > > dramatically,
> > > > > > >>>> depending on throughput for each partition. We, of course,
> > > > continue
> > > > > > >>>> flushing the metadata segments as before.
> > > > > > >>>>
> > > > > > >>>> It’s worth making a note on caching. I think nobody will
> > > disagree
> > > > > that
> > > > > > >>>> doing direct reads from remote storage every time a batch is
> > > > > requested
> > > > > > >> by a
> > > > > > >>>> consumer will not be practical neither from the performance
> > nor
> > > > from
> > > > > > the
> > > > > > >>>> economy point of view. We need a way to keep the number of
> GET
> > > > > > requests
> > > > > > >>>> down. There are multiple options, for example:
> > > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > > >>>> 2. Local in-memory caching. Comes with less network
> chattiness
> > > and
> > > > > > >> works
> > > > > > >>>> well if we have more or less stable brokers to consume from.
> > > > > > >>>> 3. Materialization of diskless logs on local disk. Way lower
> > > > impact
> > > > > on
> > > > > > >>>> RAM and also requires stable brokers for consumption (using
> > just
> > > > > > >> assigned
> > > > > > >>>> replicas will probably work well).
> > > > > > >>>>
> > > > > > >>>> Materialization is one of possible options, but we can
> choose
> > > > > another
> > > > > > >> one.
> > > > > > >>>> However, we will have this dilemma regardless of whether we
> > have
> > > > an
> > > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > > >>>>
> > > > > > >>>>> Second, if we ever need this
> > > > > > >>>>> metadata somewhere else, say in the WAL file manager, the
> > > > consumer
> > > > > > >> needs
> > > > > > >>>> to
> > > > > > >>>>> subscribe to every partition in the cluster, which is
> > > > inefficient.
> > > > > > The
> > > > > > >>>>> actual benefit of this approach is also questionable. On
> the
> > > > > surface,
> > > > > > >> it
> > > > > > >>>>> might seem that we could reduce the number of lines that
> need
> > > to
> > > > be
> > > > > > >>>> changed
> > > > > > >>>>> for this KIP. However, the changes are quite intrusive to
> the
> > > > > classic
> > > > > > >>>>> partition's code path and will probably make the code base
> > > harder
> > > > > to
> > > > > > >>>>> maintain in the long run. I like the original approach
> based
> > on
> > > > the
> > > > > > >> batch
> > > > > > >>>>> coordinator much better than this one. We could probably
> > > refactor
> > > > > the
> > > > > > >>>>> producer state code so that it could be reused in the batch
> > > > > > >> coordinator.
> > > > > > >>>>
> > > > > > >>>> It’s hard to disagree with this. The explicit coordinator is
> > > more
> > > > a
> > > > > > side
> > > > > > >>>> thing, while coordinator-less approach is more about
> extending
> > > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > > >>>>
> > > > > > >>>>> Thanks for addressing the concerns on the number of RPCs in
> > the
> > > > > > produce
> > > > > > >>>>> path. I agree that with the metadata crafting mechanism, we
> > > could
> > > > > > >>>> mitigate
> > > > > > >>>>> the PRC problem. However, since we now require the metadata
> > to
> > > be
> > > > > > >>>>> collocated with the data on the same set of brokers, it's
> > weird
> > > > > that
> > > > > > >> they
> > > > > > >>>>> are now managed by different mechanisms. The data
> assignment
> > > now
> > > > > uses
> > > > > > >> the
> > > > > > >>>>> metadata crafting mechanism, but the metadata is stored in
> > the
> > > > > > classic
> > > > > > >>>>> partition using its own assignment strategy. It will be
> > > > complicated
> > > > > > to
> > > > > > >>>> keep
> > > > > > >>>>> them collocated.
> > > > > > >>>>
> > > > > > >>>> I would like to note that the metadata crafting is needed
> only
> > > to
> > > > > tell
> > > > > > >>>> producers which brokers they should send Produce requests
> to,
> > > but
> > > > > data
> > > > > > >> (as
> > > > > > >>>> in “locally materialized log”) is located on partition
> > replicas,
> > > > > i.e.
> > > > > > >>>> automatically co-located with metadata.
> > > > > > >>>>
> > > > > > >>>> As a side note, it would probably be better that instead of
> > > > > implicitly
> > > > > > >>>> crafting partition metadata, we extend the metadata protocol
> > so
> > > > that
> > > > > > for
> > > > > > >>>> diskless partitions we return not only the leader and
> > replicas,
> > > > but
> > > > > > also
> > > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > > performance
> > > > > > and
> > > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > > >>>>
> > > > > > >>>>> I am also concerned about the removal of the object
> > > > > > compaction/merging
> > > > > > >>>>> step.
> > > > > > >>>>
> > > > > > >>>> We should have mentioned this explicitly, but this step, in
> > > fact,
> > > > > > >> remains
> > > > > > >>>> in the form of segments offloading to tiered storage. When
> we
> > > > > > assemble a
> > > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> > effectively
> > > > > doing
> > > > > > >>>> metadata compaction: replacing a big number of pieces of
> > > metadata
> > > > > > about
> > > > > > >>>> individual batches with a single record in
> > > __remote_log_metadata.
> > > > > > >>>>
> > > > > > >>>> We could create a Diskless-specific merging mechanism
> instead
> > if
> > > > > > needed.
> > > > > > >>>> It’s rather easy with the explicit coordinator approach.
> With
> > > the
> > > > > > >>>> coordinator-less approach, this would probably be a bit more
> > > > tricky
> > > > > > >>>> (rewriting the tail of the log by the leader + replicating
> > this
> > > > > change
> > > > > > >>>> reliably).
> > > > > > >>>>
> > > > > > >>>>> I see a tendency toward primarily optimizing for the fewest
> > > code
> > > > > > >> changes
> > > > > > >>>> in
> > > > > > >>>>> the KIP. Instead, our primary goal should be a clean design
> > > that
> > > > > can
> > > > > > >> last
> > > > > > >>>>> for the long term.
> > > > > > >>>>
> > > > > > >>>> Yes, totally agree.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Luke,
> > > > > > >>>>> I'm wondering if the complexity of designing txn and queue
> is
> > > > > because
> > > > > > >> of
> > > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> > only
> > > > > focus
> > > > > > on
> > > > > > >>>> the
> > > > > > >>>>> "diskless" design to handle object compaction/merging
> to/from
> > > the
> > > > > > >> remote
> > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > >>>>
> > > > > > >>>> After some evolution of the original proposal, leaderless is
> > now
> > > > > > >> limited.
> > > > > > >>>> We only need to be able to accept Produce requests on more
> > than
> > > > one
> > > > > > >> broker
> > > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get it
> > right
> > > > > that
> > > > > > >> you
> > > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Let’s now look at this problem from a higher level, as Jack
> > > > > proposed.
> > > > > > As
> > > > > > >>>> it was said, the big choice we need to make is whether we 1)
> > > > create
> > > > > an
> > > > > > >>>> explicit batch coordinator; or 2) go for the
> coordinator-less
> > > > > > approach,
> > > > > > >>>> where each diskless partition is managed by its leader as in
> > > > classic
> > > > > > >> topics.
> > > > > > >>>>
> > > > > > >>>> If we try to compare the two approaches:
> > > > > > >>>>
> > > > > > >>>> Pluggability:
> > > > > > >>>> - Explicit coordinator: Possible. For example, some setups
> may
> > > > > benefit
> > > > > > >>>> from batch metadata being stored in a cloud database (such
> as
> > > AWS
> > > > > > >> DynamoDB
> > > > > > >>>> or GCP Spanner).
> > > > > > >>>> - Coordinator-less: Impossible.
> > > > > > >>>>
> > > > > > >>>> Scalability and fault tolerance:
> > > > > > >>>> - Explicit coordinator: Depends on the implementation and
> it’s
> > > > also
> > > > > > >>>> necessary to actively work for it.
> > > > > > >>>> - Coordinator-less: Closer to classic Kafka topics. Scaling
> is
> > > > done
> > > > > by
> > > > > > >>>> partition placement, partitions could fail independently.
> > > > > > >>>>
> > > > > > >>>> Separation of concerns:
> > > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > > independent
> > > > > > >> from
> > > > > > >>>> classic topics in terms of code and workflows. For example,
> > the
> > > > > > >>>> above-mentioned non-tiered storage metadata compaction
> > mechanism
> > > > > could
> > > > > > >> be
> > > > > > >>>> relatively simply implemented with it. As a flip side of
> this,
> > > > some
> > > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> > bringing
> > > > > > diskless
> > > > > > >>>> closer to classic topics. Some code paths and workflows
> could
> > be
> > > > > more
> > > > > > >>>> straightforwardly reused, but they will inevitably have to
> be
> > > > > adapted
> > > > > > to
> > > > > > >>>> accommodate both topic types as also discussed.
> > > > > > >>>>
> > > > > > >>>> Cloud-nativeness. This is a vague concept, also related to
> the
> > > > > > previous,
> > > > > > >>>> but let’s try:
> > > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > > separately
> > > > > > makes
> > > > > > >>>> it easier for brokers to take different roles, be purely
> > > stateless
> > > > > if
> > > > > > >>>> needed, etc.
> > > > > > >>>> - Coordinator-less: Less so. Something could be achieved
> with
> > > > > creative
> > > > > > >>>> partition placement, but not much.
> > > > > > >>>>
> > > > > > >>>> Both seem to have their pros and cons. However, answering
> > Jack’s
> > > > > > >> question,
> > > > > > >>>> the explicit coordinator approach may indeed lead to a more
> > > > flexible
> > > > > > >> design.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> The purpose of this deviation in the discussion was to
> > receive a
> > > > > > >>>> preliminary community evaluation of the coordinator-less
> > > approach
> > > > > > >> without
> > > > > > >>>> taking on the task of writing a separate KIP and fitting it
> in
> > > the
> > > > > > >> system
> > > > > > >>>> of KIP-1150 and its children. We’re open to stopping it and
> > > > getting
> > > > > > >> back to
> > > > > > >>>> working out the coordinator design if the community doesn’t
> > > favor
> > > > > the
> > > > > > >>>> proposed approach.
> > > > > > >>>>
> > > > > > >>>> Best,
> > > > > > >>>> Ivan and Diskless team
> > > > > > >>>>
> > > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > > >>>>> Hi Ivan,
> > > > > > >>>>>
> > > > > > >>>>> As Jun pointed out, the updated design seems to have some
> > > > > > shortcomings
> > > > > > >>>>> although it simplifies the implementation.
> > > > > > >>>>>
> > > > > > >>>>> I'm wondering if the complexity of designing txn and queue
> is
> > > > > because
> > > > > > >> of
> > > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> > only
> > > > > focus
> > > > > > on
> > > > > > >>>> the
> > > > > > >>>>> "diskless" design to handle object compaction/merging
> to/from
> > > the
> > > > > > >> remote
> > > > > > >>>>> storage to save the cross-AZ cost?
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> Thank you,
> > > > > > >>>>> Luke
> > > > > > >>>>>
> > > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > > <jun@confluent.io.invalid
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Hi, Ivan,
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks for the explanation.
> > > > > > >>>>>>
> > > > > > >>>>>> "we write the reference to the WAL file with the batch
> data"
> > > > > > >>>>>>
> > > > > > >>>>>> I understand the approach now, but I think it is a hacky
> > one.
> > > > > There
> > > > > > >> are
> > > > > > >>>>>> multiple short comings with this design. First, it
> degrades
> > > the
> > > > > > >>>> durability.
> > > > > > >>>>>> For each partition, now there are two files being actively
> > > > written
> > > > > > at
> > > > > > >> a
> > > > > > >>>>>> given point of time, one for the data and another for the
> > > > > metadata.
> > > > > > >>>>>> Flushing each file requires a separate IO. If the disk has
> > 1K
> > > > IOPS
> > > > > > and
> > > > > > >>>> we
> > > > > > >>>>>> have 5K partitions in a broker, currently we can afford to
> > > flush
> > > > > > each
> > > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5 seconds.
> If
> > > we
> > > > > > double
> > > > > > >>>> the
> > > > > > >>>>>> number of files per partition, we can only flush each
> > > partition
> > > > > > every
> > > > > > >>>> 10
> > > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we ever
> > need
> > > > > this
> > > > > > >>>>>> metadata somewhere else, say in the WAL file manager, the
> > > > consumer
> > > > > > >>>> needs to
> > > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > > inefficient.
> > > > > > The
> > > > > > >>>>>> actual benefit of this approach is also questionable. On
> the
> > > > > > surface,
> > > > > > >>>> it
> > > > > > >>>>>> might seem that we could reduce the number of lines that
> > need
> > > to
> > > > > be
> > > > > > >>>> changed
> > > > > > >>>>>> for this KIP. However, the changes are quite intrusive to
> > the
> > > > > > classic
> > > > > > >>>>>> partition's code path and will probably make the code base
> > > > harder
> > > > > to
> > > > > > >>>>>> maintain in the long run. I like the original approach
> based
> > > on
> > > > > the
> > > > > > >>>> batch
> > > > > > >>>>>> coordinator much better than this one. We could probably
> > > > refactor
> > > > > > the
> > > > > > >>>>>> producer state code so that it could be reused in the
> batch
> > > > > > >>>> coordinator.
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks for addressing the concerns on the number of RPCs
> in
> > > the
> > > > > > >> produce
> > > > > > >>>>>> path. I agree that with the metadata crafting mechanism,
> we
> > > > could
> > > > > > >>>> mitigate
> > > > > > >>>>>> the PRC problem. However, since we now require the
> metadata
> > to
> > > > be
> > > > > > >>>>>> collocated with the data on the same set of brokers, it's
> > > weird
> > > > > that
> > > > > > >>>> they
> > > > > > >>>>>> are now managed by different mechanisms. The data
> assignment
> > > now
> > > > > > uses
> > > > > > >>>> the
> > > > > > >>>>>> metadata crafting mechanism, but the metadata is stored in
> > the
> > > > > > classic
> > > > > > >>>>>> partition using its own assignment strategy. It will be
> > > > > complicated
> > > > > > to
> > > > > > >>>> keep
> > > > > > >>>>>> them collocated.
> > > > > > >>>>>>
> > > > > > >>>>>> I am also concerned about the removal of the object
> > > > > > compaction/merging
> > > > > > >>>>>> step. My first concern is on the amount of metadata that
> > need
> > > to
> > > > > be
> > > > > > >>>> kept.
> > > > > > >>>>>> Without object compcation, the metadata generated in the
> > > produce
> > > > > > path
> > > > > > >>>> can
> > > > > > >>>>>> only be deleted after remote tiering kicks in. Let's say
> for
> > > > every
> > > > > > >>>> 250ms we
> > > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> > remoting
> > > > > > tiering
> > > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K partitions,
> > we
> > > > need
> > > > > > to
> > > > > > >>>> keep
> > > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB
> metadata,
> > > > quite
> > > > > > >>>>>> signficant. A second concern is on performance. Every time
> > we
> > > > need
> > > > > > to
> > > > > > >>>>>> rebuild the caching data, we need to read a bunch of small
> > > > objects
> > > > > > >>>> from S3,
> > > > > > >>>>>> slowing down the building process. If a consumer happens
> to
> > > need
> > > > > > such
> > > > > > >>>> data,
> > > > > > >>>>>> it could slow down the application.
> > > > > > >>>>>>
> > > > > > >>>>>> I see a tendency toward primarily optimizing for the
> fewest
> > > code
> > > > > > >>>> changes in
> > > > > > >>>>>> the KIP. Instead, our primary goal should be a clean
> design
> > > that
> > > > > can
> > > > > > >>>> last
> > > > > > >>>>>> for the long term.
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks,
> > > > > > >>>>>>
> > > > > > >>>>>> Jun
> > > > > > >>>>>>
> > > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > > ivan@ivanyu.me
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hi Jun,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thank you for your message. I’m sorry that I failed to
> > > clearly
> > > > > > >>>> explain
> > > > > > >>>>>> the
> > > > > > >>>>>>> idea. Let me try to fix this.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Does each partition now have a metadata partition and a
> > > > separate
> > > > > > >>>> data
> > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > doubles
> > > > the
> > > > > > >>>> number
> > > > > > >>>>>>> of
> > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > descriptors
> > > > > and
> > > > > > >>>> the
> > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> have a
> > > > > > separate
> > > > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > > > creating
> > > > > > an
> > > > > > >>>>>>>> internal topic with an unbounded number of partitions.
> > > > > > >>>>>>>
> > > > > > >>>>>>> No. There will be only one physical partition per
> diskless
> > > > > > >>>> partition. Let
> > > > > > >>>>>>> me explain this with an example. Let’s say we have a
> > diskless
> > > > > > >>>> partition
> > > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader.
> We
> > > > > produce
> > > > > > >>>> some
> > > > > > >>>>>>> batches to this partition. The content of the segment
> file
> > > will
> > > > > be
> > > > > > >>>>>>> something like this (for each batch):
> > > > > > >>>>>>>
> > > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > > >>>>>>> Magic: like in classic
> > > > > > >>>>>>> CRC: like in classic
> > > > > > >>>>>>> Attributes: like in classic
> > > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > > >>>>>>> ProducerId: like in classic
> > > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > > >>>>>>> BaseSequence: like in classic
> > > > > > >>>>>>> RecordsCount: like in classic
> > > > > > >>>>>>> Records:
> > > > > > >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> > byte
> > > > > offset
> > > > > > >>>>>>> 123456
> > > > > > >>>>>>>
> > > > > > >>>>>>> It looks very much like classic log entries. The only
> > > > difference
> > > > > is
> > > > > > >>>> that
> > > > > > >>>>>>> instead of writing real Records, we write the reference
> to
> > > the
> > > > > WAL
> > > > > > >>>> file
> > > > > > >>>>>>> with the batch data (I guess we need only the name and
> the
> > > byte
> > > > > > >>>> offset,
> > > > > > >>>>>>> because the byte length is the standard field above).
> > > > Otherwise,
> > > > > > >>>> it’s a
> > > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > > >>>>>>>
> > > > > > >>>>>>> So we have as many partitions for diskless as for
> classic.
> > As
> > > > of
> > > > > > open
> > > > > > >>>>>> file
> > > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Are the metadata and
> > > > > > >>>>>>>> the data for the same partition always collocated on the
> > > same
> > > > > > >>>> broker?
> > > > > > >>>>>> If
> > > > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > > > >>>>>>>
> > > > > > >>>>>>> The source of truth for the data is still in WAL files on
> > > > object
> > > > > > >>>> storage.
> > > > > > >>>>>>> The source of truth for the metadata is in segment files
> on
> > > the
> > > > > > >>>> brokers
> > > > > > >>>>>> in
> > > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > > independent
> > > > > > of
> > > > > > >>>> this
> > > > > > >>>>>>> new proposal, but I want to present them to give the idea
> > > that
> > > > > > only a
> > > > > > >>>>>>> limited amount of data files will be operated locally:
> > > > > > >>>>>>> - We want to assemble batches into segment files and
> > offload
> > > > them
> > > > > > to
> > > > > > >>>>>>> tiered storage in order to prevent the unbounded growth
> of
> > > > batch
> > > > > > >>>>>> metadata.
> > > > > > >>>>>>> For this, we need to open only  a few file descriptors
> (for
> > > the
> > > > > > >>>> segment
> > > > > > >>>>>>> file itself + the necessary indexes) before the segment
> is
> > > > fully
> > > > > > >>>> written
> > > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > > >>>>>>> - We want to assemble local segment files for caching
> > > purposes
> > > > as
> > > > > > >>>> well,
> > > > > > >>>>>>> i.e. to speed up fetching. This will not materialize the
> > full
> > > > > > >>>> content of
> > > > > > >>>>>>> the log, but only the hot set according to some policy
> (or
> > > > > > >>>> configurable
> > > > > > >>>>>>> policies), i.e. the number of segments and file
> descriptors
> > > > will
> > > > > > >>>> also be
> > > > > > >>>>>>> limited.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > > > higher.
> > > > > > For
> > > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > > cluster
> > > > > > >>>> with 100
> > > > > > >>>>>>>> brokers, each produce request could generate 100 more
> RPC
> > > > > > requests.
> > > > > > >>>>>> This
> > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > >>>>>>>
> > > > > > >>>>>>> This is a valid concern that we considered, but this
> issue
> > > can
> > > > be
> > > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > > >>>>>>> The situation with a single broker is trivial: all the
> > commit
> > > > > > >>>> requests go
> > > > > > >>>>>>> from the broker to itself.
> > > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but located
> in
> > > the
> > > > > > single
> > > > > > >>>>>> rack
> > > > > > >>>>>>> (AZ). Any broker can accept Produce requests for diskless
> > > > > > >>>> partitions, but
> > > > > > >>>>>>> we can tell producers (through metadata) to always send
> > > Produce
> > > > > > >>>> requests
> > > > > > >>>>>> to
> > > > > > >>>>>>> leaders. For example, broker 0 hosts the leader replicas
> > for
> > > > > > diskless
> > > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> > Produce
> > > > > > >>>> requests
> > > > > > >>>>>> for
> > > > > > >>>>>>> these partitions in various combinations, but only for
> > them.
> > > > > > >>>>>>>
> > > > > > >>>>>>>                   Broker 0
> > > > > > >>>>>>>             +-----------------+
> > > > > > >>>>>>>             |    t1-0         |
> > > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > > >>>>>>>             |    t3-0         |            |
> > > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > > >>>>>>> partitions   | +-------------+ |    for these partitions
> > > > > > >>>>>>>             |                 |
> > > > > > >>>>>>>             +-----------------+
> > > > > > >>>>>>>
> > > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > > Effectively,
> > > > > > each
> > > > > > >>>>>>> broker will commit only to itself, which effectively
> means
> > 1
> > > > > commit
> > > > > > >>>>>> request
> > > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls, if
> we
> > > > wish,
> > > > > > >>>> just a
> > > > > > >>>>>>> local function call).
> > > > > > >>>>>>>
> > > > > > >>>>>>> Now let’s scale this to multiple racks (AZs). Obviously,
> we
> > > > > cannot
> > > > > > >>>> always
> > > > > > >>>>>>> send Produce requests to the designated leaders of
> diskless
> > > > > > >>>> partitions:
> > > > > > >>>>>>> this would mean inter-AZ network traffic, which we would
> > like
> > > > to
> > > > > > >>>> avoid.
> > > > > > >>>>>> To
> > > > > > >>>>>>> avoid it, we say that every broker has a “diskless
> produce
> > > > > > >>>>>> representative”
> > > > > > >>>>>>> in every AZ. If we continue our example: when a Produce
> > > request
> > > > > for
> > > > > > >>>> t1-0,
> > > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on
> > > > broker 0
> > > > > > >>>> (in the
> > > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > > However,
> > > > if
> > > > > > it
> > > > > > >>>>>> comes
> > > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > > > > > >>>>>>>
> > > > > > >>>>>>> |produce requests         |produce requests
> |produce
> > > > > > >>>> requests
> > > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for
> > t1-0,
> > > > > t2-1,
> > > > > > >>>>>> t3-0
> > > > > > >>>>>>> |from AZ 0                |from AZ 1               |from
> > AZ 2
> > > > > > >>>>>>> v                         v                        v
> > > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2
> (AZ
> > 2)
> > > > > > >>>>>>> +---------------+      +---------------+
> > > +---------------+
> > > > > > >>>>>>> |     t1-0      |      |               |      |
> > >  |
> > > > > > >>>>>>> |     t2-1      |      |               |      |
> > >  |
> > > > > > >>>>>>> |     t3-0      |      |               |      |
> > >  |
> > > > > > >>>>>>> +---------------+      +--------+------+
> > > +--------+------+
> > > > > > >>>>>>>    ^     ^                    |                      |
> > > > > > >>>>>>>    |     +--------------------+                      |
> > > > > > >>>>>>>    |     commit requests for these partitions        |
> > > > > > >>>>>>>    |                                                 |
> > > > > > >>>>>>>    +-------------------------------------------------+
> > > > > > >>>>>>>          commit requests for these partitions
> > > > > > >>>>>>>
> > > > > > >>>>>>> All the partitions that broker 0 is the leader of will be
> > > > > > >>>> “represented”
> > > > > > >>>>>> by
> > > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Of course, this relationship goes both ways between AZs
> > (not
> > > > > > >>>> necessarily
> > > > > > >>>>>>> between the same brokers). It means that provided the
> > cluster
> > > > is
> > > > > > >>>> balanced
> > > > > > >>>>>>> by the number of brokers per AZ, each broker will
> represent
> > > > > > >>>>>> (number_of_azs
> > > > > > >>>>>>> - 1) other brokers. This will result in the situation
> that
> > > for
> > > > > the
> > > > > > >>>>>> majority
> > > > > > >>>>>>> of commits, each broker will do up to (number_of_azs - 1)
> > > > network
> > > > > > >>>> commit
> > > > > > >>>>>>> requests (plus one local). Cloud regions tend to have 3
> > AZs,
> > > > very
> > > > > > >>>> rarely
> > > > > > >>>>>>> more. That means, brokers will be doing up to 2 network
> > > commit
> > > > > > >>>> requests
> > > > > > >>>>>> per
> > > > > > >>>>>>> WAL file.
> > > > > > >>>>>>>
> > > > > > >>>>>>> There are the following exceptions:
> > > > > > >>>>>>> 1. Broker count imbalance between AZs. For example, when
> we
> > > > have
> > > > > 2
> > > > > > >>>> AZs
> > > > > > >>>>>> and
> > > > > > >>>>>>> one has three brokers and another AZ has one. This one
> > broker
> > > > > will
> > > > > > do
> > > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is not
> > an
> > > > > > extreme
> > > > > > >>>>>>> amplification. Such an imbalance is not healthy in most
> > > > practical
> > > > > > >>>> setups
> > > > > > >>>>>>> and should be avoided anyway.
> > > > > > >>>>>>> 2. Leadership changes and metadata propagation period.
> When
> > > the
> > > > > > >>>> partition
> > > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > > producers
> > > > > > will
> > > > > > >>>> not
> > > > > > >>>>>>> know this immediately (unless we want to be strict and
> > > respond
> > > > > with
> > > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will
> > come
> > > > > > >>>> together
> > > > > > >>>>>> in a
> > > > > > >>>>>>> WAL buffer on broker 2, it will have to send two commit
> > > > requests:
> > > > > > to
> > > > > > >>>>>> broker
> > > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit
> t3-0.
> > > This
> > > > > > >>>> situation
> > > > > > >>>>>>> is not permanent and as producers update the cluster
> > > metadata,
> > > > it
> > > > > > >>>> will be
> > > > > > >>>>>>> resolved.
> > > > > > >>>>>>>
> > > > > > >>>>>>> This all could be built with the metadata crafting
> > mechanism
> > > > only
> > > > > > >>>> (which
> > > > > > >>>>>>> is anyway needed for Diskless in one way or another to
> > direct
> > > > > > >>>> producers
> > > > > > >>>>>> and
> > > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic), just
> > with
> > > > the
> > > > > > >>>> right
> > > > > > >>>>>>> policy for it (for example, some deterministic hash-based
> > > > > formula).
> > > > > > >>>> I.e.
> > > > > > >>>>>> no
> > > > > > >>>>>>> explicit support for “produce representative” or anything
> > > like
> > > > > this
> > > > > > >>>> is
> > > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > > places,
> > > > > > >>>> partition
> > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> > > truth,
> > > > > and
> > > > > > >>>> how
> > > > > > >>>>>> do
> > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > >>>>>>>
> > > > > > >>>>>>> We do only two operations on WAL files that span multiple
> > > > > diskless
> > > > > > >>>>>>> partitions: committing and deleting. Commits can be done
> > > > > > >>>> independently as
> > > > > > >>>>>>> described above. But deletes are different, because when
> a
> > > file
> > > > > is
> > > > > > >>>>>> deleted,
> > > > > > >>>>>>> this affects all the partitions that still have alive
> > batches
> > > > in
> > > > > > this
> > > > > > >>>>>> file
> > > > > > >>>>>>> (if any).
> > > > > > >>>>>>>
> > > > > > >>>>>>> The WAL file manager is a necessary point of coordination
> > to
> > > > > delete
> > > > > > >>>> WAL
> > > > > > >>>>>>> files safely. We can say it is the source of truth about
> > > files
> > > > > > >>>>>> themselves,
> > > > > > >>>>>>> while the partition leaders and their logs hold the truth
> > > about
> > > > > > >>>> whether a
> > > > > > >>>>>>> particular file contains live batches of this particular
> > > > > partition.
> > > > > > >>>>>>>
> > > > > > >>>>>>> The file manager will do this important task: be able to
> > say
> > > > for
> > > > > > sure
> > > > > > >>>>>> that
> > > > > > >>>>>>> a file does not contain any live batch of any existing
> > > > partition.
> > > > > > For
> > > > > > >>>>>> this,
> > > > > > >>>>>>> it will have to periodically check against the partition
> > > > leaders.
> > > > > > >>>>>>> Considering that batch deletion is irreversible, when we
> > > > declare
> > > > > a
> > > > > > >>>> file
> > > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > > >>>>>>>
> > > > > > >>>>>>> The file manager has to know about files being committed
> to
> > > > start
> > > > > > >>>> track
> > > > > > >>>>>>> them and periodically check if they are empty. We can
> > > consider
> > > > > > >>>> various
> > > > > > >>>>>> ways
> > > > > > >>>>>>> to achieve this:
> > > > > > >>>>>>> 1. As was proposed in my previous message: best effort
> > commit
> > > > by
> > > > > > >>>> brokers
> > > > > > >>>>>> +
> > > > > > >>>>>>> periodic prefix scans of object storage to detect files
> > that
> > > > went
> > > > > > >>>> below
> > > > > > >>>>>> the
> > > > > > >>>>>>> radar due to network issue or the file manager temporary
> > > > > > >>>> unavailability.
> > > > > > >>>>>>> We’re speaking about listing the file names only and
> > opening
> > > > only
> > > > > > >>>>>>> previously unknown files in order to find the partitions
> > > > involved
> > > > > > >>>> with
> > > > > > >>>>>> them.
> > > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill the
> > list
> > > of
> > > > > > files
> > > > > > >>>>>>> fully asynchronously and in the background. This may be
> not
> > > > ideal
> > > > > > >>>> due to
> > > > > > >>>>>>> costs and performance of scanning tons of files. However,
> > the
> > > > > > number
> > > > > > >>>> of
> > > > > > >>>>>>> live WAL files should be limited due to tiered storage
> > > > > offloading +
> > > > > > >>>> we
> > > > > > >>>>>> can
> > > > > > >>>>>>> optimize this if we give files some global soft order in
> > > their
> > > > > > names.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> I am not sure how this design simplifies the
> > implementation.
> > > > The
> > > > > > >>>>>> existing
> > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > Adjusting
> > > > both
> > > > > > >>>> the
> > > > > > >>>>>>> write
> > > > > > >>>>>>>> path in the leader and the replication path in the
> > follower
> > > to
> > > > > > >>>>>> understand
> > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> existing
> > > > logic.
> > > > > > >>>>>>>
> > > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> > UnifiedLog
> > > in
> > > > > > >>>> order to
> > > > > > >>>>>>> support these changes. However, it seems that
> idempotence,
> > > > > > >>>> transactions,
> > > > > > >>>>>>> queues, tiered storage will have to be changed less than
> > with
> > > > the
> > > > > > >>>>>> original
> > > > > > >>>>>>> design. This is because the partition leader state would
> > > remain
> > > > > in
> > > > > > >>>> the
> > > > > > >>>>>> same
> > > > > > >>>>>>> place (on brokers) and existing workflows that involve it
> > > would
> > > > > > have
> > > > > > >>>> to
> > > > > > >>>>>> be
> > > > > > >>>>>>> changed less compared to the situation where we globalize
> > the
> > > > > > >>>> partition
> > > > > > >>>>>>> leader state in the batch coordinator. I admit this is
> hard
> > > to
> > > > > make
> > > > > > >>>>>>> convincing without both real implementations to hand :)
> > > > > > >>>>>>>
> > > > > > >>>>>>>> I am also
> > > > > > >>>>>>>> not sure how this enables seamless switching the topic
> > modes
> > > > > > >>>> between
> > > > > > >>>>>>>> diskless and classic. Could you provide more details on
> > > those?
> > > > > > >>>>>>>
> > > > > > >>>>>>> Let’s consider the scenario of turning a classic topic
> into
> > > > > > >>>> diskless. The
> > > > > > >>>>>>> user sets diskless.enabled=true, the leader receives this
> > > > > metadata
> > > > > > >>>> update
> > > > > > >>>>>>> and does the following:
> > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > >>>>>>> 3. Start a new segment that will be written in the
> diskless
> > > > > format
> > > > > > >>>> (i.e.
> > > > > > >>>>>>> without data).
> > > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Since it’s the same log, the followers will know about
> that
> > > > > switch
> > > > > > >>>>>>> consistently. They will finish replicating the classic
> > > segments
> > > > > and
> > > > > > >>>> start
> > > > > > >>>>>>> replicating the diskless ones. They will always know
> where
> > > each
> > > > > > >>>> batch is
> > > > > > >>>>>>> located (either inside a classic segment or referenced
> by a
> > > > > > diskless
> > > > > > >>>>>> one).
> > > > > > >>>>>>> Switching back should be similar.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Doing this with the coordinator is possible, but has some
> > > > > caveats.
> > > > > > >>>> The
> > > > > > >>>>>>> leader must do the following:
> > > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > > >>>>>>> 2. Close the current active segment.
> > > > > > >>>>>>> 3. Write a special control segment to persist and
> replicate
> > > the
> > > > > > fact
> > > > > > >>>> that
> > > > > > >>>>>>> from offset N the partition is now in the diskless mode.
> > > > > > >>>>>>> 4. Inform the coordinator about the first offset N of the
> > > > > “diskless
> > > > > > >>>> era”.
> > > > > > >>>>>>> 5. Inform the controller quorum that the transition has
> > > > finished
> > > > > > and
> > > > > > >>>> that
> > > > > > >>>>>>> brokers now can process diskless writes for this
> partition.
> > > > > > >>>>>>> This could fail at some points, so this will probably
> > require
> > > > > some
> > > > > > >>>>>>> explicit state machine with replication either in the
> > > partition
> > > > > log
> > > > > > >>>> or in
> > > > > > >>>>>>> KRaft.
> > > > > > >>>>>>>
> > > > > > >>>>>>> It seems that the coordinator-less approach makes this
> > > simpler
> > > > > > >>>> because
> > > > > > >>>>>> the
> > > > > > >>>>>>> “coordinator” for the partition and the partition leader
> > are
> > > > the
> > > > > > >>>> same and
> > > > > > >>>>>>> they store the partition metadata in the same log, too.
> > While
> > > > in
> > > > > > the
> > > > > > >>>>>>> coordinator approach we have to perform some kind of a
> > > > > distributed
> > > > > > >>>> commit
> > > > > > >>>>>>> to handover metadata management from the classic
> partition
> > > > leader
> > > > > > to
> > > > > > >>>> the
> > > > > > >>>>>>> batch coordinator.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I hope these explanations help to clarify the idea.
> Please
> > > let
> > > > me
> > > > > > >>>> know if
> > > > > > >>>>>>> I should go deeper anywhere.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Best,
> > > > > > >>>>>>> Ivan and the Diskless team
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > > >>>>>>>> Hi, Ivan,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks for the update.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I am not sure that I fully understand the new design,
> but
> > it
> > > > > seems
> > > > > > >>>> less
> > > > > > >>>>>>>> clean than before.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Does each partition now have a metadata partition and a
> > > > separate
> > > > > > >>>> data
> > > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> > doubles
> > > > the
> > > > > > >>>> number
> > > > > > >>>>>>> of
> > > > > > >>>>>>>> partitions, which impacts the number of open file
> > > descriptors
> > > > > and
> > > > > > >>>> the
> > > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to
> have a
> > > > > > separate
> > > > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > > > creating
> > > > > > an
> > > > > > >>>>>>>> internal topic with an unbounded number of partitions.
> Are
> > > the
> > > > > > >>>> metadata
> > > > > > >>>>>>> and
> > > > > > >>>>>>>> the data for the same partition always collocated on the
> > > same
> > > > > > >>>> broker?
> > > > > > >>>>>> If
> > > > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > > > higher.
> > > > > > For
> > > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > > cluster
> > > > > > >>>> with 100
> > > > > > >>>>>>>> brokers, each produce request could generate 100 more
> RPC
> > > > > > requests.
> > > > > > >>>>>> This
> > > > > > >>>>>>>> will significantly increase the request rate.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > > places,
> > > > > > >>>> partition
> > > > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> > > truth,
> > > > > and
> > > > > > >>>> how
> > > > > > >>>>>> do
> > > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I am not sure how this design simplifies the
> > implementation.
> > > > The
> > > > > > >>>>>> existing
> > > > > > >>>>>>>> producer/replication code can't be simply reused.
> > Adjusting
> > > > both
> > > > > > >>>> the
> > > > > > >>>>>>> write
> > > > > > >>>>>>>> path in the leader and the replication path in the
> > follower
> > > to
> > > > > > >>>>>> understand
> > > > > > >>>>>>>> batch-header only data is quite intrusive to the
> existing
> > > > > logic. I
> > > > > > >>>> am
> > > > > > >>>>>>> also
> > > > > > >>>>>>>> not sure how this enables seamless switching the topic
> > modes
> > > > > > >>>> between
> > > > > > >>>>>>>> diskless and classic. Could you provide more details on
> > > those?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Jun
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > > ivan@ivanyu.me
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to have a
> > > > > separate
> > > > > > >>>>>>>>> component, batch/diskless coordinator, whose role would
> > be
> > > to
> > > > > > >>>>>> centrally
> > > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> > topics.
> > > > > This
> > > > > > >>>>>>> component
> > > > > > >>>>>>>>> drew many reasonable comments from the community about
> > how
> > > it
> > > > > > >>>> would
> > > > > > >>>>>>> support
> > > > > > >>>>>>>>> various Kafka features (transactions, queues) and its
> > > > > > >>>> scalability.
> > > > > > >>>>>>> While we
> > > > > > >>>>>>>>> believe we have good answers to all the expressed
> > concerns,
> > > > we
> > > > > > >>>> took a
> > > > > > >>>>>>> step
> > > > > > >>>>>>>>> back and looked at the problem from a different
> > > perspective.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> We would like to propose an alternative Diskless design
> > > > > *without
> > > > > > >>>> a
> > > > > > >>>>>>>>> centralized coordinator*. We believe this approach has
> > > > > potential
> > > > > > >>>> and
> > > > > > >>>>>>>>> propose to discuss it as it may be more appealing to
> the
> > > > > > >>>> community.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Let us explain the idea. Most of the complications with
> > the
> > > > > > >>>> original
> > > > > > >>>>>>>>> Diskless approach come from one necessary architecture
> > > > change:
> > > > > > >>>>>>> globalizing
> > > > > > >>>>>>>>> the local state of partition leader in the batch
> > > coordinator.
> > > > > > >>>> This
> > > > > > >>>>>>> causes
> > > > > > >>>>>>>>> deviations to the established workflows in various
> > features
> > > > > like
> > > > > > >>>>>>> produce
> > > > > > >>>>>>>>> idempotence and transactions, queues, retention, etc.
> > These
> > > > > > >>>>>> deviations
> > > > > > >>>>>>> need
> > > > > > >>>>>>>>> to be carefully considered, designed, and later
> > implemented
> > > > and
> > > > > > >>>>>>> tested. In
> > > > > > >>>>>>>>> the new approach we want to avoid this by making
> > partition
> > > > > > >>>> leaders
> > > > > > >>>>>>> again
> > > > > > >>>>>>>>> responsible for managing their partitions, even in
> > diskless
> > > > > > >>>> topics.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata are
> > > blended
> > > > > > >>>> together
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>> the one partition log. The crux of the Diskless idea is
> > to
> > > > > > >>>> decouple
> > > > > > >>>>>>> them
> > > > > > >>>>>>>>> and move data to the remote storage, while keeping
> > metadata
> > > > > > >>>> somewhere
> > > > > > >>>>>>> else.
> > > > > > >>>>>>>>> Using the central batch coordinator for managing batch
> > > > metadata
> > > > > > >>>> is
> > > > > > >>>>>> one
> > > > > > >>>>>>> way,
> > > > > > >>>>>>>>> but not the only.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Let’s now think about managing metadata for each user
> > > > partition
> > > > > > >>>>>>>>> independently. Generally partitions are independent and
> > > don’t
> > > > > > >>>> share
> > > > > > >>>>>>>>> anything apart from that their data are mixed in WAL
> > files.
> > > > If
> > > > > we
> > > > > > >>>>>>> figure
> > > > > > >>>>>>>>> out how to commit and later delete WAL files safely, we
> > > will
> > > > > > >>>> achieve
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> > central
> > > > > batch
> > > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition
> will
> > be
> > > > > > >>>> managed
> > > > > > >>>>>> by
> > > > > > >>>>>>> its
> > > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in
> classic
> > > > > > >>>> topics, the
> > > > > > >>>>>>>>> leader uses the partition log as the way to persist
> batch
> > > > > > >>>> metadata,
> > > > > > >>>>>>> i.e.
> > > > > > >>>>>>>>> the regular batch header + the information about how to
> > > find
> > > > > this
> > > > > > >>>>>>> batch on
> > > > > > >>>>>>>>> remote storage. In contrast to classic topics, batch
> data
> > > is
> > > > in
> > > > > > >>>>>> remote
> > > > > > >>>>>>>>> storage.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > > >>>>>>>>> • Classic topics:
> > > > > > >>>>>>>>>  • Data and metadata are co-located in the partition
> log.
> > > > > > >>>>>>>>>  • The partition log content: [Batch header
> > > (metadata)|Batch
> > > > > > >>>> data].
> > > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > > > >>>> metadata.
> > > > > > >>>>>>>>> • Original Diskless:
> > > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> > remote
> > > > > > >>>> storage.
> > > > > > >>>>>>>>>  • The partition state is global in the batch
> > coordinator.
> > > > > > >>>>>>>>> • New Diskless:
> > > > > > >>>>>>>>>  • Metadata is in the partition log, data is on remote
> > > > storage.
> > > > > > >>>>>>>>>  • Partition log content: [Batch header
> (metadata)|Batch
> > > > > > >>>>>> coordinates
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>> remote storage].
> > > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > > > >>>> metadata.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Let’s consider the produce path. Here’s the reminder of
> > the
> > > > > > >>>> original
> > > > > > >>>>>>>>> Diskless design:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The new approach could be depicted as the following:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> As you can see, the main difference is that now instead
> > of
> > > a
> > > > > > >>>> single
> > > > > > >>>>>>> commit
> > > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> > parallel
> > > > > > >>>> commit
> > > > > > >>>>>>> requests
> > > > > > >>>>>>>>> to all the leaders of each partition involved in the
> WAL
> > > > file.
> > > > > > >>>> Each
> > > > > > >>>>>> of
> > > > > > >>>>>>> them
> > > > > > >>>>>>>>> will commit its batches independently, without
> > coordinating
> > > > > with
> > > > > > >>>>>> other
> > > > > > >>>>>>>>> leaders and any other components. Batch data is
> addressed
> > > by
> > > > > the
> > > > > > >>>> WAL
> > > > > > >>>>>>> file
> > > > > > >>>>>>>>> name, the byte offset and size, which allows partitions
> > to
> > > > know
> > > > > > >>>>>> nothing
> > > > > > >>>>>>>>> about other partitions to access their data in shared
> WAL
> > > > > files.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The number of partitions involved in a single WAL file
> > may
> > > be
> > > > > > >>>> quite
> > > > > > >>>>>>> large,
> > > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to commit
> one
> > > WAL
> > > > > > >>>> file is
> > > > > > >>>>>>> very
> > > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> > number:
> > > > > > >>>>>>>>> 1. Partition leaders are located on brokers. Requests
> to
> > > > > > >>>> leaders on
> > > > > > >>>>>>> one
> > > > > > >>>>>>>>> broker could be grouped together into a single physical
> > > > network
> > > > > > >>>>>> request
> > > > > > >>>>>>>>> (resembling the normal Produce request that may carry
> > > batches
> > > > > for
> > > > > > >>>>>> many
> > > > > > >>>>>>>>> partitions inside). This will cap the number of network
> > > > > requests
> > > > > > >>>> to
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > > >>>>>>>>> 2. If we craft the cluster metadata to make producers
> > send
> > > > > their
> > > > > > >>>>>>> requests
> > > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> > achieve
> > > > the
> > > > > > >>>> higher
> > > > > > >>>>>>>>> concentration of logical commit requests in physical
> > > network
> > > > > > >>>> requests
> > > > > > >>>>>>>>> reducing the number of the latter ones even further,
> > > ideally
> > > > to
> > > > > > >>>> one.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Obviously, out of multiple commit requests some may
> fail
> > or
> > > > > time
> > > > > > >>>> out
> > > > > > >>>>>>> for a
> > > > > > >>>>>>>>> variety of reasons. This is fine. Some producers will
> > > receive
> > > > > > >>>> totally
> > > > > > >>>>>>> or
> > > > > > >>>>>>>>> partially failed responses to their Produce requests,
> > > similar
> > > > > to
> > > > > > >>>> what
> > > > > > >>>>>>> they
> > > > > > >>>>>>>>> would have received when appending to a classic topic
> > fails
> > > > or
> > > > > > >>>> times
> > > > > > >>>>>>> out.
> > > > > > >>>>>>>>> If a partition experiences problems, other partitions
> > will
> > > > not
> > > > > be
> > > > > > >>>>>>> affected
> > > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> > uncommitted
> > > > > data
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > > (batches
> > > > > are
> > > > > > >>>>>>> constantly
> > > > > > >>>>>>>>> assembled into segments and offloaded to tiered
> storage),
> > > so
> > > > > this
> > > > > > >>>>>>> garbage
> > > > > > >>>>>>>>> will be eventually deleted.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> For safely deleting WAL files we now need to centrally
> > > manage
> > > > > > >>>> them,
> > > > > > >>>>>> as
> > > > > > >>>>>>>>> this is the only state and logic that spans multiple
> > > > > partitions.
> > > > > > >>>> On
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> diagram, you can see another commit request called
> > “Commit
> > > > file
> > > > > > >>>> (best
> > > > > > >>>>>>>>> effort)” going to the WAL File Manager. This manager
> will
> > > be
> > > > > > >>>>>>> responsible
> > > > > > >>>>>>>>> for the following:
> > > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and persisting
> > > > > > >>>> information
> > > > > > >>>>>>> about
> > > > > > >>>>>>>>> committed WAL files.
> > > > > > >>>>>>>>> 2. To handle potential failures in file information
> > > delivery,
> > > > > it
> > > > > > >>>>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> doing prefix scan on the remote storage periodically to
> > > find
> > > > > and
> > > > > > >>>>>>> register
> > > > > > >>>>>>>>> unknown files. The period of this scan will be
> > configurable
> > > > and
> > > > > > >>>>>> ideally
> > > > > > >>>>>>>>> should be quite long.
> > > > > > >>>>>>>>> 3. Checking with the relevant partition leaders (after
> a
> > > > grace
> > > > > > >>>>>>> period) if
> > > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > > >>>>>>>>> 4. Physically deleting files when they aren’t anymore
> > > > referred
> > > > > > >>>> to by
> > > > > > >>>>>>> any
> > > > > > >>>>>>>>> partition.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> > features
> > > > such
> > > > > > >>>> as
> > > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > > retention.
> > > > > > >>>> Now we
> > > > > > >>>>>>> don’t
> > > > > > >>>>>>>>> need to abstract away and reuse the code from partition
> > > > leaders
> > > > > > >>>> in
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> batch coordinator. Instead, we will literally use the
> > same
> > > > code
> > > > > > >>>> paths
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>> leaders, with little adaptation. Workflows from classic
> > > > topics
> > > > > > >>>> mostly
> > > > > > >>>>>>>>> remain unchanged.
> > > > > > >>>>>>>>> For example, it seems that
> > > > > > >>>>>>>>>
> > ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > > and
> > > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> > transaction
> > > > > > >>>> support
> > > > > > >>>>>> on
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> partition leader side could be used for diskless topics
> > > with
> > > > > > >>>> little
> > > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > > idempotent
> > > > > > >>>> produce
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> transactions, would be reused.
> > > > > > >>>>>>>>> Another example is share groups support, where the
> share
> > > > > > >>>> partition
> > > > > > >>>>>>> leader,
> > > > > > >>>>>>>>> being co-located with the partition leader, would
> execute
> > > the
> > > > > > >>>> same
> > > > > > >>>>>>> logic
> > > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > > >>>>>>>>> 2. It returns to the familiar partition-based scaling
> > > model,
> > > > > > >>>> where
> > > > > > >>>>>>>>> partitions are independent.
> > > > > > >>>>>>>>> 3. It makes the operation and failure patterns closer
> to
> > > the
> > > > > > >>>>>> familiar
> > > > > > >>>>>>>>> ones from classic topics.
> > > > > > >>>>>>>>> 4. It opens a straightforward path to seamless
> switching
> > > the
> > > > > > >>>> topics
> > > > > > >>>>>>> modes
> > > > > > >>>>>>>>> between diskless and classic.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The rest of the things remain unchanged compared to the
> > > > > previous
> > > > > > >>>>>>> Diskless
> > > > > > >>>>>>>>> design (after all previous discussions). Such things as
> > > local
> > > > > > >>>> segment
> > > > > > >>>>>>>>> materialization by replicas, the consume path, tiered
> > > storage
> > > > > > >>>>>>> integration,
> > > > > > >>>>>>>>> etc.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> If the community finds this design more suitable, we
> will
> > > > > update
> > > > > > >>>> the
> > > > > > >>>>>>>>> KIP(s) accordingly and continue working on it. Please
> let
> > > us
> > > > > know
> > > > > > >>>>>> what
> > > > > > >>>>>>> you
> > > > > > >>>>>>>>> think.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Best regards,
> > > > > > >>>>>>>>> Ivan and Diskless team
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > > >>>>>>>>>> Hi Justine,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > > transactions
> > > > > > >>>> for in
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> diskless coordinator for as long as the corresponding
> > > offsets
> > > > > are
> > > > > > >>>>>>> there.
> > > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> > earlier,
> > > > > this
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best,
> > > > > > >>>>>>>>>> Ivan
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > > >>>>>>>>>>> Hey Ivan,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks for the response. I think most of what you
> said
> > > made
> > > > > > >>>>>> sense,
> > > > > > >>>>>>> but
> > > > > > >>>>>>>>> I
> > > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> classic
> > > > > > >>>> topics
> > > > > > >>>>>>> forgets
> > > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> > overpasses
> > > > > > >>>> it). The
> > > > > > >>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > > allowing
> > > > > > >>>>>>> partition
> > > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if this
> is
> > > > > > >>>> wrong. We
> > > > > > >>>>>>> think
> > > > > > >>>>>>>>>>> about relying on this with the batch coordinator and
> > > delete
> > > > > > >>>> the
> > > > > > >>>>>>>>> information
> > > > > > >>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > > > > >>>> replication
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> HWM
> > > > > > >>>>>>>>>>> advances immediately).
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I didn't quite understand this. In classic topics, we
> > > have
> > > > > > >>>> maps
> > > > > > >>>>>> for
> > > > > > >>>>>>>>> ongoing
> > > > > > >>>>>>>>>>> transactions which remove state when the transaction
> is
> > > > > > >>>> completed
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> an
> > > > > > >>>>>>>>>>> aborted transactions index which is retained for much
> > > > longer.
> > > > > > >>>>>> Once
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> transaction is completed, the coordinator is no
> longer
> > > > > > >>>> involved
> > > > > > >>>>>> in
> > > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> > subject
> > > to
> > > > > > >>>>>>> compaction
> > > > > > >>>>>>>>> etc.
> > > > > > >>>>>>>>>>> Looking back at the outline provided above, I didn't
> > see
> > > > much
> > > > > > >>>>>>> about the
> > > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > > further.
> > > > I
> > > > > > >>>> saw
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> following in a response:
> > > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> > segment,
> > > > > > >>>> all the
> > > > > > >>>>>>>>> necessary
> > > > > > >>>>>>>>>>> control batches will be inserted and indices,
> including
> > > the
> > > > > > >>>>>>> transaction
> > > > > > >>>>>>>>>>> index will be built to serve FetchRequests exactly as
> > > they
> > > > > > >>>> are
> > > > > > >>>>>>> today.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Based on this, it seems like we need to retain the
> > > > > > >>>> information
> > > > > > >>>>>>> about
> > > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>> Justine
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > > >>>> ivan@ivanyu.me>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > identified
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer ID
> > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check could
> be
> > > > > > >>>> cached
> > > > > > >>>>>>>>> locally
> > > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > > >>>>>> transactions
> > > > > > >>>>>>> can
> > > > > > >>>>>>>>> be
> > > > > > >>>>>>>>>>>> used
> > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> transactions
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer id +
> > > > > > >>>>>>>>>>>>> epoch
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> > focused
> > > > > > >>>> only on
> > > > > > >>>>>>>>> version 2.
> > > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> > consider
> > > > > > >>>> using
> > > > > > >>>>>>> some
> > > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > transactional
> > > > > > >>>>>>> checks
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> errors
> > > > > > >>>> like the
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you
> be
> > > > > > >>>>>> checking
> > > > > > >>>>>>> if
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > > ongoing,
> > > > > > >>>> and
> > > > > > >>>>>> of
> > > > > > >>>>>>>>> course
> > > > > > >>>>>>>>>>>> the normal idempotence checks. What the partition
> > leader
> > > > > > >>>> in the
> > > > > > >>>>>>>>> classic
> > > > > > >>>>>>>>>>>> topics does before appending a batch to the local
> log
> > > > > > >>>> (e.g. in
> > > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > > Diskless,
> > > > > > >>>> we
> > > > > > >>>>>>>>> unfortunately
> > > > > > >>>>>>>>>>>> cannot do these checks before appending the data to
> > the
> > > > WAL
> > > > > > >>>>>>> segment
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these batches
> in
> > > the
> > > > > > >>>> batch
> > > > > > >>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>> during the final commit.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> > other
> > > > > > >>>> state
> > > > > > >>>>>>>>> mentioned
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> clear
> > > > > > >>>> what
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>> stored
> > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Right, this should have been more explicit. As the
> > > > > > >>>> partition
> > > > > > >>>>>>> leader
> > > > > > >>>>>>>>> tracks
> > > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> > > > > > >>>> coordinator
> > > > > > >>>>>>> has
> > > > > > >>>>>>>>> to as
> > > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > > >>>> transaction
> > > > > > >>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO
> --
> > > > > > >>>> perhaps
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>> would
> > > > > > >>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed
> batches
> > > and
> > > > > > >>>> other
> > > > > > >>>>>>>>>>>> information, but for the sake of performance we’d
> > better
> > > > > > >>>> store
> > > > > > >>>>>> it
> > > > > > >>>>>>>>>>>> explicitly.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > transactional
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will
> > be
> > > > > > >>>>>> cleaned
> > > > > > >>>>>>> up?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> As we understand this, the partition leader in
> classic
> > > > > > >>>> topics
> > > > > > >>>>>>> forgets
> > > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> > overpasses
> > > > > > >>>> it).
> > > > > > >>>>>> The
> > > > > > >>>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > > > > >>>> allowing
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if this
> > is
> > > > > > >>>> wrong.
> > > > > > >>>>>> We
> > > > > > >>>>>>>>> think
> > > > > > >>>>>>>>>>>> about relying on this with the batch coordinator and
> > > > > > >>>> delete the
> > > > > > >>>>>>>>> information
> > > > > > >>>>>>>>>>>> about a transaction once it’s finished (as there’s
> no
> > > > > > >>>>>> replication
> > > > > > >>>>>>>>> and HWM
> > > > > > >>>>>>>>>>>> advances immediately).
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>> Ivan
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > >>>>>>>>>>>>> Hey folks,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Excited to see some updates related to
> transactions!
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> > identified
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer ID
> > > > > > >>>>>>>>>>>>> and epoch, the positive result of this check could
> be
> > > > > > >>>> cached
> > > > > > >>>>>>>>> locally
> > > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > > >>>>>> transactions
> > > > > > >>>>>>> can
> > > > > > >>>>>>>>> be
> > > > > > >>>>>>>>>>>> used
> > > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify
> transactions
> > > > > > >>>> with
> > > > > > >>>>>>>>> producer id +
> > > > > > >>>>>>>>>>>>> epoch
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > > transactional
> > > > > > >>>>>>> checks
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>> batches. This procedure would output the same
> errors
> > > > > > >>>> like the
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you
> be
> > > > > > >>>>>> checking
> > > > > > >>>>>>> if
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is there
> > > state
> > > > > > >>>>>> about
> > > > > > >>>>>>>>> ongoing
> > > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> > other
> > > > > > >>>> state
> > > > > > >>>>>>>>> mentioned
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> the End transaction section, but it's not super
> clear
> > > > > > >>>> what
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>> stored
> > > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO
> --
> > > > > > >>>> perhaps
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>> would
> > > > > > >>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > > transactional
> > > > > > >>>>>>> state is
> > > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will
> > be
> > > > > > >>>>>> cleaned
> > > > > > >>>>>>> up?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> > > > > > >>>>>> segments,
> > > > > > >>>>>>>>> making
> > > > > > >>>>>>>>>>>> use of
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > zero-copy
> > > > > > >>>>>>>>> functionality
> > > > > > >>>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state
> for
> > > > > > >>>> each
> > > > > > >>>>>>>>> partition in
> > > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer
> fetches
> > > > > > >>>> need
> > > > > > >>>>>> to
> > > > > > >>>>>>> know
> > > > > > >>>>>>>>>>>> aborted
> > > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > > >>>> transaction is
> > > > > > >>>>>>>>> finished
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> > corresponding
> > > > > > >>>> logs
> > > > > > >>>>>>> of the
> > > > > > >>>>>>>>>>>> involved
> > > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
> > > > > > >>>> level,
> > > > > > >>>>>> no
> > > > > > >>>>>>>>> actual
> > > > > > >>>>>>>>>>>> control
> > > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> > transactional
> > > > > > >>>>>>> batches.
> > > > > > >>>>>>>>> How
> > > > > > >>>>>>>>>>>> does the
> > > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> > control
> > > > > > >>>>>> batch
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> each
> > > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response needs
> to
> > > be
> > > > > > >>>>>>> built by
> > > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> > batches
> > > > > > >>>>>>> together?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition
> > leader
> > > > > > >>>> that
> > > > > > >>>>>>> all
> > > > > > >>>>>>>>>>>> consumers
> > > > > > >>>>>>>>>>>>>> are routed to?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include
> this
> > or
> > > > > > >>>> it's
> > > > > > >>>>>>> too
> > > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> > include
> > > > > > >>>>>> enough
> > > > > > >>>>>>>>> details
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a
> given
> > > > > > >>>>>> producer
> > > > > > >>>>>>>>> contain a
> > > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a producer
> > sends
> > > > > > >>>>>> every
> > > > > > >>>>>>>>> Produce
> > > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests
> could
> > > > > > >>>> reach
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> batch
> > > > > > >>>>>>>>>>>>>> coordinator in different order and lead to
> rejection
> > > > > > >>>> of the
> > > > > > >>>>>>>>> produce
> > > > > > >>>>>>>>>>>>>> requests.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > > > > >>>>>>> ivan@ivanyu.me>
> > > > > > >>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details about
> > > > > > >>>>>>> transactions
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>>>> queues,
> > > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The support for transactions in *classic topics*
> is
> > > > > > >>>> based
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>> precise
> > > > > > >>>>>>>>>>>>>>> interactions between three actors: clients
> (mostly
> > > > > > >>>>>>> producers,
> > > > > > >>>>>>>>> but
> > > > > > >>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > > >>>> classes),
> > > > > > >>>>>> and
> > > > > > >>>>>>>>>>>> transaction
> > > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders
> > with
> > > > > > >>>>>> their
> > > > > > >>>>>>>>> local
> > > > > > >>>>>>>>>>>> state
> > > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The high level (some details skipped) workflow is
> > the
> > > > > > >>>>>>>>> following.
> > > > > > >>>>>>>>>>>> When a
> > > > > > >>>>>>>>>>>>>>> transactional Produce request is received by the
> > > > > > >>>> broker:
> > > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader
> checks
> > > > > > >>>> if a
> > > > > > >>>>>>>>> non-empty
> > > > > > >>>>>>>>>>>>>>> transaction is running for this partition. This
> is
> > > > > > >>>> done
> > > > > > >>>>>>> using
> > > > > > >>>>>>>>> its
> > > > > > >>>>>>>>>>>> local
> > > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > > >>>>>> (ProducerStateManager,
> > > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed about
> > all
> > > > > > >>>> the
> > > > > > >>>>>>>>> partitions
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> > > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> > transactional
> > > > > > >>>>>>> checks.
> > > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the transactional
> > > > > > >>>> data to
> > > > > > >>>>>>>>> their logs
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>> update some of their state (for example, log the
> > fact
> > > > > > >>>>>> that
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>>>> transaction
> > > > > > >>>>>>>>>>>>>>> is running for the partition and its first
> offset).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > coordinator
> > > > > > >>>>>>> directly
> > > > > > >>>>>>>>> with
> > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > PREPARE_COMMIT
> > > > > > >>>> or
> > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> leaders
> > > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> > > > > > >>>> markers to
> > > > > > >>>>>>>>> their logs
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction
> > state
> > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders and
> lots
> > > > > > >>>> of
> > > > > > >>>>>>>>> important
> > > > > > >>>>>>>>>>>> state
> > > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is local.
> > The
> > > > > > >>>> main
> > > > > > >>>>>>>>> challenge
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact
> there
> > > > > > >>>> are no
> > > > > > >>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state
> need
> > > > > > >>>> to be
> > > > > > >>>>>>>>> globalized
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> > > > > > >>>> support
> > > > > > >>>>>>>>> idempotent
> > > > > > >>>>>>>>>>>>>> produce.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics*
> would
> > > > > > >>>> look
> > > > > > >>>>>>> very
> > > > > > >>>>>>>>>>>> similar:
> > > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > > >>>> non-empty
> > > > > > >>>>>>>>> transaction
> > > > > > >>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to
> classic
> > > > > > >>>>>> topics,
> > > > > > >>>>>>>>> this is
> > > > > > >>>>>>>>>>>>>> checked
> > > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
> > > > > > >>>> Since a
> > > > > > >>>>>>>>> transaction
> > > > > > >>>>>>>>>>>>>> could
> > > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and
> epoch,
> > > > > > >>>> the
> > > > > > >>>>>>> positive
> > > > > > >>>>>>>>>>>> result of
> > > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the
> double
> > > > > > >>>>>>> configured
> > > > > > >>>>>>>>>>>> duration
> > > > > > >>>>>>>>>>>>>> of a
> > > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> > informed
> > > > > > >>>>>> about
> > > > > > >>>>>>> all
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
> > > > > > >>>> include
> > > > > > >>>>>>> them.
> > > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
> > > > > > >>>> side.
> > > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data to
> the
> > > > > > >>>>>> current
> > > > > > >>>>>>>>> shared
> > > > > > >>>>>>>>>>>> WAL
> > > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any
> transaction-related
> > > > > > >>>> state
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> Diskless
> > > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > > >>>> coordinator
> > > > > > >>>>>>> like
> > > > > > >>>>>>>>> in the
> > > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> > transactional
> > > > > > >>>>>>> checks
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> > errors
> > > > > > >>>> like
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> > batches
> > > > > > >>>>>> could
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>> rejected.
> > > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage in
> > the
> > > > > > >>>> WAL
> > > > > > >>>>>>>>> segment
> > > > > > >>>>>>>>>>>> file in
> > > > > > >>>>>>>>>>>>>>> case of transactional errors. This is preferable
> to
> > > > > > >>>> doing
> > > > > > >>>>>>> more
> > > > > > >>>>>>>>>>>> network
> > > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> > segments
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>> relatively
> > > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> > coordinator
> > > > > > >>>>>>> directly
> > > > > > >>>>>>>>> with
> > > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> > PREPARE_COMMIT
> > > > > > >>>> or
> > > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs
> the
> > > > > > >>>> batch
> > > > > > >>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> > > > > > >>>>>>> transaction is
> > > > > > >>>>>>>>>>>> finished
> > > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > > >>>> corresponding
> > > > > > >>>>>>> logs
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only on
> the
> > > > > > >>>>>> metadata
> > > > > > >>>>>>>>> level, no
> > > > > > >>>>>>>>>>>>>>> actual control batches are written to any file.
> > They
> > > > > > >>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>> dynamically
> > > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations. We
> > could
> > > > > > >>>>>>>>> technically
> > > > > > >>>>>>>>>>>> write
> > > > > > >>>>>>>>>>>>>>> these control batches for real, but this would
> mean
> > > > > > >>>> extra
> > > > > > >>>>>>>>> produce
> > > > > > >>>>>>>>>>>>>> latency,
> > > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > > >>>> coordinator
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> save
> > > > > > >>>>>>>>>>>> these
> > > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> leaders
> > > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
> > > > > > >>>> topics
> > > > > > >>>>>>> now.
> > > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics write
> > the
> > > > > > >>>>>>>>> transaction
> > > > > > >>>>>>>>>>>> markers
> > > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction
> > state
> > > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce flow,
> we
> > > > > > >>>> get:
> > > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between brokers
> and
> > > > > > >>>> the
> > > > > > >>>>>>> batch
> > > > > > >>>>>>>>>>>> coordinator
> > > > > > >>>>>>>>>>>>>>> when a new partition appear in the transaction.
> To
> > > > > > >>>>>>> mitigate the
> > > > > > >>>>>>>>>>>> impact of
> > > > > > >>>>>>>>>>>>>>> them:
> > > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one
> Produce
> > > > > > >>>>>>> request
> > > > > > >>>>>>>>> will be
> > > > > > >>>>>>>>>>>>>>> grouped.
> > > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> > > > > > >>>> fast
> > > > > > >>>>>>>>> response to
> > > > > > >>>>>>>>>>>> these
> > > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally will
> > > > > > >>>>>>> communicate
> > > > > > >>>>>>>>> with a
> > > > > > >>>>>>>>>>>>>>> single broker for the duration of the transaction
> > > > > > >>>> further
> > > > > > >>>>>>>>> reduces the
> > > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> > > > > > >>>>>> coordinator
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> batch
> > > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be
> able
> > to
> > > > > > >>>>>> span
> > > > > > >>>>>>> both
> > > > > > >>>>>>>>>>>> classic
> > > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> The share group coordination and management is a
> > > > > > >>>> side job
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>>>>> doesn't
> > > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > > >>>> replicas,
> > > > > > >>>>>>> physical
> > > > > > >>>>>>>>>>>> storage
> > > > > > >>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> > consumers
> > > > > > >>>>>>> (Fetch and
> > > > > > >>>>>>>>>>>> Produce
> > > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> > affected.)
> > > > > > >>>> We
> > > > > > >>>>>>> don't
> > > > > > >>>>>>>>> see any
> > > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics
> compatible
> > > > > > >>>> with
> > > > > > >>>>>>> share
> > > > > > >>>>>>>>>>>> groups the
> > > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the code
> > > > > > >>>> level,
> > > > > > >>>>>> we
> > > > > > >>>>>>>>> don't
> > > > > > >>>>>>>>>>>> expect
> > > > > > >>>>>>>>>>>>>> any
> > > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading routines
> are
> > > > > > >>>> used
> > > > > > >>>>>>> that
> > > > > > >>>>>>>>> are
> > > > > > >>>>>>>>>>>> used for
> > > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this or
> it's
> > > > > > >>>> too
> > > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > > >>>>>>>>>>>>>>> Ivan
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design
> input
> > > > > > >>>> on
> > > > > > >>>>>>>>> KIP-1150.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with
> a
> > > > > > >>>> new
> > > > > > >>>>>>>>> design. To
> > > > > > >>>>>>>>>>>>>>> summarize
> > > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> > > > > > >>>> existing
> > > > > > >>>>>>>>> KIP-405
> > > > > > >>>>>>>>>>>> Tiered
> > > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to
> a
> > > > > > >>>>>>> Diskless
> > > > > > >>>>>>>>> topic
> > > > > > >>>>>>>>>>>> to be
> > > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for the
> > > > > > >>>> Batch
> > > > > > >>>>>>>>> Coordinator
> > > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> > > > > > >>>> Tiered
> > > > > > >>>>>>> Storage
> > > > > > >>>>>>>>>>>> plugin
> > > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative data
> > > > > > >>>>>> formats.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> > > > > > >>>> segments,
> > > > > > >>>>>>>>> making use
> > > > > > >>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > > >>>> zero-copy
> > > > > > >>>>>>>>> functionality
> > > > > > >>>>>>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > > >>>>>>>>>>>>>>>> However, local segments are now considered cache
> > > > > > >>>>>>> elements,
> > > > > > >>>>>>>>> do not
> > > > > > >>>>>>>>>>>> need
> > > > > > >>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > > >>>> contacting
> > > > > > >>>>>>> any
> > > > > > >>>>>>>>> other
> > > > > > >>>>>>>>>>>>>>> replicas.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified substantially,
> > by
> > > > > > >>>>>>> removing
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> previous
> > > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > > >>>> component, and
> > > > > > >>>>>>>>> "object
> > > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > > >>>> enabled by
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> Batch
> > > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as
> the
> > > > > > >>>>>> earlier
> > > > > > >>>>>>>>> design,
> > > > > > >>>>>>>>>>>> while
> > > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into the
> > > > > > >>>> existing
> > > > > > >>>>>>>>> ecosystem.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback
> on
> > > > > > >>>> the
> > > > > > >>>>>> new
> > > > > > >>>>>>>>> design.
> > > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> > > > > > >>>> of
> > > > > > >>>>>> all
> > > > > > >>>>>>>>> existing
> > > > > > >>>>>>>>>>>>>> client
> > > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > > >>>> supporting
> > > > > > >>>>>> APIs
> > > > > > >>>>>>>>> like
> > > > > > >>>>>>>>>>>>>>> transactions
> > > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> > > > > > >>>>>>> status of
> > > > > > >>>>>>>>> this
> > > > > > >>>>>>>>>>>> KIP?
> > > > > > >>>>>>>>>>>>>> I
> > > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> > > > > > >>>>>>> months,
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>> vote got
> > > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> > > > > > >>>>>> answered
> > > > > > >>>>>>>>> since
> > > > > > >>>>>>>>>>>> then.
> > > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > > >>>> activity
> > > > > > >>>>>>> since
> > > > > > >>>>>>>>> May.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> > > > > > >>>>>> choices
> > > > > > >>>>>>>>> that can
> > > > > > >>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> > > > > > >>>> by
> > > > > > >>>>>> far
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>> dominant
> > > > > > >>>>>>>>>>>>>> one
> > > > > > >>>>>>>>>>>>>>>>> when
> > > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these
> are
> > > > > > >>>>>>>>> standalone
> > > > > > >>>>>>>>>>>>>> products.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> > > > > > >>>> hate to
> > > > > > >>>>>>> see a
> > > > > > >>>>>>>>>>>> stall. So
> > > > > > >>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> > > > > > >>>>>> McCabe -
> > > > > > >>>>>>>>> have your
> > > > > > >>>>>>>>>>>>>>> questions
> > > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> > > > > > >>>> do
> > > > > > >>>>>> to
> > > > > > >>>>>>>>> help? I'm
> > > > > > >>>>>>>>>>>>>> deeply
> > > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > > >>>> written
> > > > > > >>>>>>> about
> > > > > > >>>>>>>>> both?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > > >>>>>> Kozlovski
> > > > > > >>>>>>> <
> > > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > > >>>> main
> > > > > > >>>>>> one
> > > > > > >>>>>>>>> (just a
> > > > > > >>>>>>>>>>>> bunch
> > > > > > >>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>>>> links
> > > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > > >>>> example, I
> > > > > > >>>>>>> was
> > > > > > >>>>>>>>>>>> searching
> > > > > > >>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> > > > > > >>>> far
> > > > > > >>>>>> as
> > > > > > >>>>>>> I
> > > > > > >>>>>>>>> can
> > > > > > >>>>>>>>>>>> tell, it
> > > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> > > > > > >>>> having
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> root
> > > > > > >>>>>>>>>>>> KIP be
> > > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> > > > > > >>>> to
> > > > > > >>>>>>>>> sub-KIPs
> > > > > > >>>>>>>>>>>> where
> > > > > > >>>>>>>>>>>>>> were
> > > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> > > > > > >>>> would
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>> good to
> > > > > > >>>>>>>>>>>>>> create
> > > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> > > > > > >>>>>>>>> introducing a
> > > > > > >>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>> type of
> > > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > > >>>> Storage as
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> primary
> > > > > > >>>>>>>>>>>>>>> source of
> > > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > > >>>> decided
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>> split it
> > > > > > >>>>>>>>>>>>>> into
> > > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > > >>>> should aim
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>> have
> > > > > > >>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>> type of
> > > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > > >>>>>> details
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>> how to
> > > > > > >>>>>>>>>>>>>>> implement
> > > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
> > > > > > >>>>>>> discussed
> > > > > > >>>>>>>>>>>> KRaft.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> > > > > > >>>> hard
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>> discuss
> > > > > > >>>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>> meta
> > > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > > >>>> (linked
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>>>>> KIP-1150)
> > > > > > >>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>> offer
> > > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> > > > > > >>>>>>> threads
> > > > > > >>>>>>>>> for
> > > > > > >>>>>>>>>>>> each
> > > > > > >>>>>>>>>>>>>>> type of
> > > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > > >>>> whether
> > > > > > >>>>>>> Apache
> > > > > > >>>>>>>>> Kafka
> > > > > > >>>>>>>>>>>> wants
> > > > > > >>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> have
> > > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > > >>>> authors
> > > > > > >>>>>> of
> > > > > > >>>>>>>>> this KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> > > > > > >>>>>>> aiven.io
> > > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > > >>>>>>> Valtonen,
> > > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Anatolii Popov
> > Senior Software Developer, *Aiven OY*
> > m: +358505126242
> > w: aiven.io  e: anatolii.popov@aiven.io
> > <https://www.facebook.com/aivencloud>
> > <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao via dev <de...@kafka.apache.org>]

Hi, Anatolii,

Thanks for the updated KIP. Looks good to me overall. A few more comments.

JR10. "Even in the case where network traffic is not accounted; we believe
that the operational benefits of Diskless topics are still appealing to
Kafka users."
Could you list other benefits such as better scalability and durability?

JR11. "AWS charges 0.02$ per GiB"
The object store is not free and has a different cost model based on
requests. It would be useful summarize the tradeoff on cost and latency,
and when diskless topics are cost effective.

JR12. Are the followup KIPs up to date? KIP-1164 and KIP-1181 haven't been
updated in the last 6 months. The metadata change in KIP-1181 is now
included in KIP-1163.

Jun

On Fri, Jan 9, 2026 at 5:09 AM Anatolii Popov via dev <de...@kafka.apache.org>
wrote:

> Hi all,
>
> Thank you for the extensive feedback.
>
> We have substantially updated the KIP to address the points raised. Given
> the scope of these changes, we have started a new VOTE thread to restart
> the voting process cleanly.
>
> You can find the new thread here:
> https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q
>
> Best,
> Anatolii
>
> On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
> dev@kafka.apache.org> wrote:
>
> > Hi all,
> >
> > We (the team at Slack) have been following the recent discussion
> regarding
> > the scope and timeline of KIP-1150. We agree with the community sentiment
> > that Diskless Topics represents the right long-term architecture for
> Kafka
> > in the cloud, but we also recognize the valid concerns raised regarding
> the
> > engineering resources required to deliver such an ambitious change.
> >
> > To help address these concerns and accelerate the timeline, we are happy
> to
> > announce that we are partnering with the KIP-1150 authors to co-develop
> > this feature.
> >
> > We previously proposed KIP-1176: Tiered Storage for Active Log Segments
> to
> > solve similar problems. However, rather than fragmenting the community's
> > efforts across competing designs, we have decided to withdraw KIP-1176
> and
> > consolidate our engineering resources behind KIP-1150.
> >
> > To start, we plan to take ownership of Compaction for Tiered Storage.
> This
> > has long been a missing feature in KIP-405, and it becomes critical in a
> > Diskless architecture where long-lived data is obligated to be tiered. By
> > driving this related prerequisite feature, we hope to allow for faster
> > delivery of KIP-1150.
> >
> > We are excited to collaborate on this to ensure a robust and timely
> > delivery for the community.
> >
> > Best,
> > Tom & Henry
> >
> > On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi Greg,
> > >
> > > Thanks for sharing the meeting notes.
> > > I agree we should keep polishing the contents of 1150 & high level
> design
> > > in 1163 to prepare for a vote.
> > >
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris
> <greg.harris@aiven.io.invalid
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > There was a video call between myself, Ivan Yurchenko, Jun Rao, and
> > > Andrew
> > > > Schofield pertaining to KIP-1150. Here are the notes from that
> meeting:
> > > >
> > > > Ivan: What is the future state of Kafka in this area, in 5 years?
> > > > Jun: Do we want something more cloud native? Yes, started with Tiered
> > > > Storage. If there’s a better way, we should explore it. In the long
> > term
> > > > this will be useful
> > > > Because Kafka is used so widely, we need to make sure everything we
> add
> > > is
> > > > for the long term and for everyone, not just for a single company.
> > > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> > something
> > > > that’s high quality/lasts/maintainable, and can work with all
> existing
> > > > capabilities.
> > > > If both 1150 and 1176 proceed at the same time, it’s confusing. They
> > > > overlap, but Diskless is more ambitious.
> > > > If both KIPs are being seriously worked on, then we don’t really need
> > > both,
> > > > because Diskless clearly is better. Having multiple will confuse
> > people.
> > > It
> > > > will duplicate some of the effort.
> > > > If we want diskless ultimately, what is the short term strategy, to
> get
> > > > some early wins first?
> > > > Ivan: Andrew, do you want a more revolutionary approach?
> > > > Andrew: Eventually the architecture will change substantially, it may
> > not
> > > > be necessary to put all of that bill onto Diskless at once.
> > > > Greg: We all agree on having a high quality feature merged upstream,
> > and
> > > > supporting all APIs
> > > > Jun: We should try and keep things simple, but there is some minimum
> > > > complexity needed.
> > > > When doing the short term changes (1176), it doesn’t really progress
> in
> > > > changing to a more modern architecture.
> > > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > > Jun: The danger of only applying changes to some part of the API, you
> > set
> > > > the precedence that you only have to implement part of the API.
> > > Supporting
> > > > the full API set should be a minimum requirement.
> > > > Andrew: When we started Kraft, how much did we know the design?
> > > > Jun: For Kraft we didn’t really know much about the migration, but
> the
> > > > high-level was clear.
> > > > Greg: Is 1150 votable in its current state?
> > > > Jun: 1150 should promise to support all APIs. It doesn’t have to have
> > all
> > > > the details/apis/etc. KIP-500 didn’t have it.
> > > > We do need some high-level design enough to give confidence that the
> > > > promise is able to be fulfilled.
> > > > Greg: Is the draft version in 1163 enough detail or is more needed?
> > > > Jun: We need to agree on the core design, such as leaderless etc. And
> > how
> > > > the APIs will be supported.
> > > > Greg: Okay we can include these things, and provide a sketch of how
> the
> > > > other leader-based features operate.
> > > > Jun: Yeah if at a high level the sketch appears to work, we can
> approve
> > > > that functionality.
> > > > Are you committed to doing the more involved and big project?
> > > > Greg: Yes, we’re committed to the 1163 design and can’t really accept
> > > 1176.
> > > > Jun: TS was slow because of Uber resourcing problems
> > > > Greg: We’ll push internally for resources, and use the community
> > > sentiment
> > > > to motivate Aiven.
> > > > How far into the future should we look? What sort of scale?
> > > > Jun: As long as there’s a path forward, and we’re not closing off
> > future
> > > > improvements, we can figure out how to handle a larger scale when it
> > > > arises.
> > > > Greg: Random replica placement is very harmful, can we recommend
> users
> > to
> > > > use an external tool like CruiseControl?
> > > > Jun: Not everyone uses CruiseControl, we would probably need some
> > > solution
> > > > for this out of the box
> > > > Ivan: Should the Batch Coordinator be pluggable?
> > > > Jun: Out-of-box experience should be good, good to allow other
> > > > implementations
> > > > Greg: But it could hurt Kafka feature/upgrade velocity when we wait
> for
> > > > plugin providers to implement it
> > > > Ivan: We imagined that maybe cloud hyperscalers could implement it
> with
> > > > e.g. dynamodb
> > > > Greg: Could we bake more details of the different providers into
> Kafka,
> > > or
> > > > does it still make sense for it to be pluggable?
> > > > Jun: Make it whatever is easiest to roll out and add new clients
> > > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > > Ivan: The vote is already open, we’re not too pressed for time. We’ll
> > go
> > > > improve the 1163 design and communication.
> > > > Is 1176 a competing design? Someone will ask.
> > > > Jun: If we are seriously working on something more ambitious, yeah we
> > > > shouldn’t do the stop-gap solution.
> > > > It’s diverting review resources. If we can get the short term thing
> in
> > > 1yr
> > > > but Diskless solution is 2y it makes sense to go for Diskless. If
> it’s
> > > 5yr,
> > > > that’s different and maybe the stop-gap solution is needed.
> > > > Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should we
> > > > explicitly exclude 1176?
> > > > Andrew: Put your arms around the feature set you actually want, and
> use
> > > > that to rule out 1176.
> > > > Probably don’t need -1 votes, most likely KIPs just don’t receive
> > votes.
> > > > Ivan: Should we have sync meetings like tiered storage did?
> > > > Jun: Satish posted meeting notes regularly, we should do the same.
> > > >
> > > > To summarize, we will be polishing the contents of 1150 & high level
> > > design
> > > > in 1163 to prepare for a vote.
> > > > We believe that the community should select the feature set of 1150
> to
> > > > fully eliminate producer cross-zone costs, and make the investment
> in a
> > > > high quality Diskless Topics implementation rather than in stop-gap
> > > > solutions.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:
> > > >
> > > > > This may be a tangent, but we needed to offload storage off of
> Kafka
> > > into
> > > > > S3. We are keeping Kafka not as a source of truth, but as a mostly
> > > > > ephemeral broker that can come and go as it pleases. Be that
> scaling
> > or
> > > > > outage. Disks can be destroyed and recreated at will, we still
> retain
> > > > data
> > > > > and use broker for just that, brokering messages. Not only that, we
> > > > reduced
> > > > > the requirement on the actual Kafka resources by reducing the size
> > of a
> > > > > payload via a claim check pattern. Maybe this is an anti–pattern,
> but
> > > it
> > > > is
> > > > > super fast and highly cost efficient. We reworked ProducerRequest
> to
> > > > allow
> > > > > plugins. We added a custom http plugin that submits every request
> > via a
> > > > > persisted connection to a microservice. Microservice stores the
> > payload
> > > > and
> > > > > returns a tiny json metadata object,a claim check, that can be used
> > to
> > > > find
> > > > > the actual data. Think of it as zipping the payload. This claim
> check
> > > > > metadata traverses the pipelines with consumers using the urls in
> > > > metadata
> > > > > to pull what they need. Think unzipping. This allowed us to also
> pull
> > > > ONLY
> > > > > the data that we need in graphql like manner. So if you have a 100K
> > > json
> > > > > payload and you need only a subsection, you can pull that by
> > jmespath.
> > > > When
> > > > > you have multiple consumer groups yanking down huge payloads it is
> > > > > cumbersome on the broker. When you have the same consumer groups
> > > yanking
> > > > > down a claim check, and then going out of band directly to the
> source
> > > of
> > > > > truth, the broker has some breathing room. Obviously our
> microservice
> > > > does
> > > > > not go directly to the cloud storage, as that would be too slow. It
> > > > stores
> > > > > the payload in high speed memory cache and returns asap. That
> memory
> > is
> > > > > eventually persisted into S3. The retrieval goest against the cache
> > > > first,
> > > > > then against the S3. Overall a rather cheappy and zippy solution. I
> > > tried
> > > > > proposing the KIP for this, but there was no excitement. Check this
> > > out:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > > >
> > > > >
> > > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID>
> > > wrote:
> > > > > >
> > > > > > Hi, Andrew,
> > > > > >
> > > > > > If we want to focus only on reducing cross-zone replication
> costs,
> > > > there
> > > > > is
> > > > > > an alternative design in the KIP-1176 discussion thread that
> seems
> > > > > simpler
> > > > > > than the proposal here. I am copying the outline of that approach
> > > > below.
> > > > > >
> > > > > > 1. A new leader is elected.
> > > > > > 2. Leader maintains a first tiered offset, which is initialized
> to
> > > log
> > > > > end
> > > > > > offset.
> > > > > > 3. Leader writes produced data from the client to local log.
> > > > > > 4. Leader uploads produced data from all local logs as a combined
> > > > object
> > > > > > 5. Leader stores the metadata for the combined object in memory.
> > > > > > 6. If a follower fetch request has an offset >= first tiered
> > offset,
> > > > the
> > > > > > metadata for the corresponding combined object is returned.
> > > Otherwise,
> > > > > the
> > > > > > local data is returned.
> > > > > > 7. Leader periodically advances first tiered offset.
> > > > > >
> > > > > > It's still a bit unnatural, but it could work.
> > > > > >
> > > > > > Hi, Ivan,
> > > > > >
> > > > > > Are you still committed to proceeding with the original design of
> > > > > KIP-1150?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > > andrew_schofield@live.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >> I’ve been following KIP-1150 and friends for a while. I’m going
> to
> > > > jump
> > > > > >> into the discussions too.
> > > > > >>
> > > > > >> Looking back at Jack Vanlightly’s message, I am not quite so
> > > convinced
> > > > > >> that it’s a kind of fork in the road. The primary aim of the
> > effort
> > > is
> > > > > to
> > > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > > prohibitively
> > > > > >> expensive to use on cloud storage. I think it would be entirely
> > > viable
> > > > > to
> > > > > >> prioritise code reuse for an initial implementation of diskless
> > > > topics,
> > > > > and
> > > > > >> we could still have a more cloud-native design in the future.
> It’s
> > > > hard
> > > > > to
> > > > > >> predict what the community will prioritise in the future.
> > > > > >>
> > > > > >> Of the three major revisions, I’m in the rev3 camp. We can
> support
> > > > > >> leaderless produce requests, first writing WAL segments into
> > object
> > > > > >> storage, and then using the regular partition leaders to
> sequence
> > > the
> > > > > >> records. The active log segment for a diskless topic will
> > initially
> > > > > contain
> > > > > >> batch coordinates rather than record batches. The batch
> > coordinates
> > > > can
> > > > > be
> > > > > >> resolved from WAL segments for consumers, and also in order to
> > > prepare
> > > > > log
> > > > > >> segments for uploading to tiered storage. Jun is probably
> correct
> > > that
> > > > > we
> > > > > >> need a more frequent object merging process than tiered storage
> > > > > provides.
> > > > > >> This is just the transition from write-optimised WAL segments to
> > > > > >> read-optimised tiered segments, and all of the object
> > storage-based
> > > > > >> implementations of Kafka that I’m aware of do this
> rearrangement.
> > > But
> > > > > >> perhaps this more frequent object merging is a pre-GA
> improvement,
> > > > > rather
> > > > > >> than a strict requirement for an initial implementation for
> early
> > > > access
> > > > > >> use.
> > > > > >>
> > > > > >> For zone-aligned share consumers, the share group assignor is
> > > intended
> > > > > to
> > > > > >> be rack-aware. Consumers should be assigned to partitions with
> > > leaders
> > > > > in
> > > > > >> their zone. The simple assignor is not rack-aware, but it easily
> > > could
> > > > > be
> > > > > >> or we could have a rack-aware assignor.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Andrew
> > > > > >>
> > > > > >>
> > > > > >>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID>
> > > wrote:
> > > > > >>>
> > > > > >>> Hi, Ivan,
> > > > > >>>
> > > > > >>> Thanks for the reply.
> > > > > >>>
> > > > > >>> "As I understand, you’re speaking about locally materialized
> > > > segments.
> > > > > >> They
> > > > > >>> will indeed consume some IOPS. See them as a cache that could
> > > always
> > > > be
> > > > > >>> restored from the remote storage. While it’s not ideal, it's
> > still
> > > OK
> > > > > to
> > > > > >>> lose data in them due to a machine crash, for example. Because
> of
> > > > this,
> > > > > >> we
> > > > > >>> can avoid explicit flushing on local materialized segments at
> all
> > > and
> > > > > let
> > > > > >>> the file system and page cache figure out when to flush
> > optimally.
> > > > This
> > > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > > dramatically,
> > > > > >>> depending on throughput for each partition. We, of course,
> > continue
> > > > > >>> flushing the metadata segments as before."
> > > > > >>>
> > > > > >>> If we have a mix of classic and diskless topics on the same
> > broker,
> > > > > it's
> > > > > >>> important that the classic topics' data is flushed to disk as
> > > quickly
> > > > > as
> > > > > >>> possible. To achieve this, users typically set
> > > dirty_expire_centisecs
> > > > > in
> > > > > >>> the kernel based on the number of available disk IOPS. Once you
> > set
> > > > > this
> > > > > >>> number, it applies to all dirty files, including the cached
> data
> > in
> > > > > >>> diskless topics. So, if there are more files actively
> > accumulating
> > > > > data,
> > > > > >>> the flush frequency and therefore RPO is reduced for classic
> > > topics.
> > > > > >>>
> > > > > >>> "We should have mentioned this explicitly, but this step, in
> > fact,
> > > > > >> remains
> > > > > >>> in the form of segments offloading to tiered storage. When we
> > > > assemble
> > > > > a
> > > > > >>> segment and hand it over to RemoteLogManager, we’re effectively
> > > doing
> > > > > >>> metadata compaction: replacing a big number of pieces of
> metadata
> > > > about
> > > > > >>> individual batches with a single record in
> > __remote_log_metadata."
> > > > > >>>
> > > > > >>> The object merging in tier storage typically only kicks in
> after
> > a
> > > > few
> > > > > >>> hours. The impact is (1) the amount of accumulated metadata is
> > > still
> > > > > >> quite
> > > > > >>> large; (2) there are many small objects, leading to poor read
> > > > > >> performance.
> > > > > >>> I think we need a more frequent object merging process than
> tier
> > > > > storage
> > > > > >>> provides.
> > > > > >>>
> > > > > >>> Jun
> > > > > >>>
> > > > > >>>
> > > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <
> ivan@ivanyu.me>
> > > > > wrote:
> > > > > >>>
> > > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > > >>>>
> > > > > >>>> Thank you for your messages.
> > > > > >>>>
> > > > > >>>> Let me first address some of Jun’s comments.
> > > > > >>>>
> > > > > >>>>> First, it degrades the durability.
> > > > > >>>>> For each partition, now there are two files being actively
> > > written
> > > > > at a
> > > > > >>>>> given point of time, one for the data and another for the
> > > metadata.
> > > > > >>>>> Flushing each file requires a separate IO. If the disk has 1K
> > > IOPS
> > > > > and
> > > > > >> we
> > > > > >>>>> have 5K partitions in a broker, currently we can afford to
> > flush
> > > > each
> > > > > >>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If
> we
> > > > > double
> > > > > >>>> the
> > > > > >>>>> number of files per partition, we can only flush each
> partition
> > > > every
> > > > > >> 10
> > > > > >>>>> seconds, which makes RPO twice as bad.
> > > > > >>>>
> > > > > >>>> As I understand, you’re speaking about locally materialized
> > > > segments.
> > > > > >> They
> > > > > >>>> will indeed consume some IOPS. See them as a cache that could
> > > always
> > > > > be
> > > > > >>>> restored from the remote storage. While it’s not ideal, it's
> > still
> > > > OK
> > > > > to
> > > > > >>>> lose data in them due to a machine crash, for example. Because
> > of
> > > > > this,
> > > > > >> we
> > > > > >>>> can avoid explicit flushing on local materialized segments at
> > all
> > > > and
> > > > > >> let
> > > > > >>>> the file system and page cache figure out when to flush
> > optimally.
> > > > > This
> > > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > > dramatically,
> > > > > >>>> depending on throughput for each partition. We, of course,
> > > continue
> > > > > >>>> flushing the metadata segments as before.
> > > > > >>>>
> > > > > >>>> It’s worth making a note on caching. I think nobody will
> > disagree
> > > > that
> > > > > >>>> doing direct reads from remote storage every time a batch is
> > > > requested
> > > > > >> by a
> > > > > >>>> consumer will not be practical neither from the performance
> nor
> > > from
> > > > > the
> > > > > >>>> economy point of view. We need a way to keep the number of GET
> > > > > requests
> > > > > >>>> down. There are multiple options, for example:
> > > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > > >>>> 2. Local in-memory caching. Comes with less network chattiness
> > and
> > > > > >> works
> > > > > >>>> well if we have more or less stable brokers to consume from.
> > > > > >>>> 3. Materialization of diskless logs on local disk. Way lower
> > > impact
> > > > on
> > > > > >>>> RAM and also requires stable brokers for consumption (using
> just
> > > > > >> assigned
> > > > > >>>> replicas will probably work well).
> > > > > >>>>
> > > > > >>>> Materialization is one of possible options, but we can choose
> > > > another
> > > > > >> one.
> > > > > >>>> However, we will have this dilemma regardless of whether we
> have
> > > an
> > > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > > >>>>
> > > > > >>>>> Second, if we ever need this
> > > > > >>>>> metadata somewhere else, say in the WAL file manager, the
> > > consumer
> > > > > >> needs
> > > > > >>>> to
> > > > > >>>>> subscribe to every partition in the cluster, which is
> > > inefficient.
> > > > > The
> > > > > >>>>> actual benefit of this approach is also questionable. On the
> > > > surface,
> > > > > >> it
> > > > > >>>>> might seem that we could reduce the number of lines that need
> > to
> > > be
> > > > > >>>> changed
> > > > > >>>>> for this KIP. However, the changes are quite intrusive to the
> > > > classic
> > > > > >>>>> partition's code path and will probably make the code base
> > harder
> > > > to
> > > > > >>>>> maintain in the long run. I like the original approach based
> on
> > > the
> > > > > >> batch
> > > > > >>>>> coordinator much better than this one. We could probably
> > refactor
> > > > the
> > > > > >>>>> producer state code so that it could be reused in the batch
> > > > > >> coordinator.
> > > > > >>>>
> > > > > >>>> It’s hard to disagree with this. The explicit coordinator is
> > more
> > > a
> > > > > side
> > > > > >>>> thing, while coordinator-less approach is more about extending
> > > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > > >>>>
> > > > > >>>>> Thanks for addressing the concerns on the number of RPCs in
> the
> > > > > produce
> > > > > >>>>> path. I agree that with the metadata crafting mechanism, we
> > could
> > > > > >>>> mitigate
> > > > > >>>>> the PRC problem. However, since we now require the metadata
> to
> > be
> > > > > >>>>> collocated with the data on the same set of brokers, it's
> weird
> > > > that
> > > > > >> they
> > > > > >>>>> are now managed by different mechanisms. The data assignment
> > now
> > > > uses
> > > > > >> the
> > > > > >>>>> metadata crafting mechanism, but the metadata is stored in
> the
> > > > > classic
> > > > > >>>>> partition using its own assignment strategy. It will be
> > > complicated
> > > > > to
> > > > > >>>> keep
> > > > > >>>>> them collocated.
> > > > > >>>>
> > > > > >>>> I would like to note that the metadata crafting is needed only
> > to
> > > > tell
> > > > > >>>> producers which brokers they should send Produce requests to,
> > but
> > > > data
> > > > > >> (as
> > > > > >>>> in “locally materialized log”) is located on partition
> replicas,
> > > > i.e.
> > > > > >>>> automatically co-located with metadata.
> > > > > >>>>
> > > > > >>>> As a side note, it would probably be better that instead of
> > > > implicitly
> > > > > >>>> crafting partition metadata, we extend the metadata protocol
> so
> > > that
> > > > > for
> > > > > >>>> diskless partitions we return not only the leader and
> replicas,
> > > but
> > > > > also
> > > > > >>>> some “recommended produce brokers”, selected for optimal
> > > performance
> > > > > and
> > > > > >>>> costs. Producers will pick ones in their racks.
> > > > > >>>>
> > > > > >>>>> I am also concerned about the removal of the object
> > > > > compaction/merging
> > > > > >>>>> step.
> > > > > >>>>
> > > > > >>>> We should have mentioned this explicitly, but this step, in
> > fact,
> > > > > >> remains
> > > > > >>>> in the form of segments offloading to tiered storage. When we
> > > > > assemble a
> > > > > >>>> segment and hand it over to RemoteLogManager, we’re
> effectively
> > > > doing
> > > > > >>>> metadata compaction: replacing a big number of pieces of
> > metadata
> > > > > about
> > > > > >>>> individual batches with a single record in
> > __remote_log_metadata.
> > > > > >>>>
> > > > > >>>> We could create a Diskless-specific merging mechanism instead
> if
> > > > > needed.
> > > > > >>>> It’s rather easy with the explicit coordinator approach. With
> > the
> > > > > >>>> coordinator-less approach, this would probably be a bit more
> > > tricky
> > > > > >>>> (rewriting the tail of the log by the leader + replicating
> this
> > > > change
> > > > > >>>> reliably).
> > > > > >>>>
> > > > > >>>>> I see a tendency toward primarily optimizing for the fewest
> > code
> > > > > >> changes
> > > > > >>>> in
> > > > > >>>>> the KIP. Instead, our primary goal should be a clean design
> > that
> > > > can
> > > > > >> last
> > > > > >>>>> for the long term.
> > > > > >>>>
> > > > > >>>> Yes, totally agree.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Luke,
> > > > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > > > because
> > > > > >> of
> > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> only
> > > > focus
> > > > > on
> > > > > >>>> the
> > > > > >>>>> "diskless" design to handle object compaction/merging to/from
> > the
> > > > > >> remote
> > > > > >>>>> storage to save the cross-AZ cost?
> > > > > >>>>
> > > > > >>>> After some evolution of the original proposal, leaderless is
> now
> > > > > >> limited.
> > > > > >>>> We only need to be able to accept Produce requests on more
> than
> > > one
> > > > > >> broker
> > > > > >>>> to eliminate the cross-AZ costs for producers. Do I get it
> right
> > > > that
> > > > > >> you
> > > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Let’s now look at this problem from a higher level, as Jack
> > > > proposed.
> > > > > As
> > > > > >>>> it was said, the big choice we need to make is whether we 1)
> > > create
> > > > an
> > > > > >>>> explicit batch coordinator; or 2) go for the coordinator-less
> > > > > approach,
> > > > > >>>> where each diskless partition is managed by its leader as in
> > > classic
> > > > > >> topics.
> > > > > >>>>
> > > > > >>>> If we try to compare the two approaches:
> > > > > >>>>
> > > > > >>>> Pluggability:
> > > > > >>>> - Explicit coordinator: Possible. For example, some setups may
> > > > benefit
> > > > > >>>> from batch metadata being stored in a cloud database (such as
> > AWS
> > > > > >> DynamoDB
> > > > > >>>> or GCP Spanner).
> > > > > >>>> - Coordinator-less: Impossible.
> > > > > >>>>
> > > > > >>>> Scalability and fault tolerance:
> > > > > >>>> - Explicit coordinator: Depends on the implementation and it’s
> > > also
> > > > > >>>> necessary to actively work for it.
> > > > > >>>> - Coordinator-less: Closer to classic Kafka topics. Scaling is
> > > done
> > > > by
> > > > > >>>> partition placement, partitions could fail independently.
> > > > > >>>>
> > > > > >>>> Separation of concerns:
> > > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > > independent
> > > > > >> from
> > > > > >>>> classic topics in terms of code and workflows. For example,
> the
> > > > > >>>> above-mentioned non-tiered storage metadata compaction
> mechanism
> > > > could
> > > > > >> be
> > > > > >>>> relatively simply implemented with it. As a flip side of this,
> > > some
> > > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > > >>>> - Coordinator-less: Less so. It leans to the opposite:
> bringing
> > > > > diskless
> > > > > >>>> closer to classic topics. Some code paths and workflows could
> be
> > > > more
> > > > > >>>> straightforwardly reused, but they will inevitably have to be
> > > > adapted
> > > > > to
> > > > > >>>> accommodate both topic types as also discussed.
> > > > > >>>>
> > > > > >>>> Cloud-nativeness. This is a vague concept, also related to the
> > > > > previous,
> > > > > >>>> but let’s try:
> > > > > >>>> - Explicit coordinator: Storing and processing metadata
> > separately
> > > > > makes
> > > > > >>>> it easier for brokers to take different roles, be purely
> > stateless
> > > > if
> > > > > >>>> needed, etc.
> > > > > >>>> - Coordinator-less: Less so. Something could be achieved with
> > > > creative
> > > > > >>>> partition placement, but not much.
> > > > > >>>>
> > > > > >>>> Both seem to have their pros and cons. However, answering
> Jack’s
> > > > > >> question,
> > > > > >>>> the explicit coordinator approach may indeed lead to a more
> > > flexible
> > > > > >> design.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> The purpose of this deviation in the discussion was to
> receive a
> > > > > >>>> preliminary community evaluation of the coordinator-less
> > approach
> > > > > >> without
> > > > > >>>> taking on the task of writing a separate KIP and fitting it in
> > the
> > > > > >> system
> > > > > >>>> of KIP-1150 and its children. We’re open to stopping it and
> > > getting
> > > > > >> back to
> > > > > >>>> working out the coordinator design if the community doesn’t
> > favor
> > > > the
> > > > > >>>> proposed approach.
> > > > > >>>>
> > > > > >>>> Best,
> > > > > >>>> Ivan and Diskless team
> > > > > >>>>
> > > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > > >>>>> Hi Ivan,
> > > > > >>>>>
> > > > > >>>>> As Jun pointed out, the updated design seems to have some
> > > > > shortcomings
> > > > > >>>>> although it simplifies the implementation.
> > > > > >>>>>
> > > > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > > > because
> > > > > >> of
> > > > > >>>>> leaderless cluster, do you think it will be simpler if we
> only
> > > > focus
> > > > > on
> > > > > >>>> the
> > > > > >>>>> "diskless" design to handle object compaction/merging to/from
> > the
> > > > > >> remote
> > > > > >>>>> storage to save the cross-AZ cost?
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> Thank you,
> > > > > >>>>> Luke
> > > > > >>>>>
> > > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > >>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Hi, Ivan,
> > > > > >>>>>>
> > > > > >>>>>> Thanks for the explanation.
> > > > > >>>>>>
> > > > > >>>>>> "we write the reference to the WAL file with the batch data"
> > > > > >>>>>>
> > > > > >>>>>> I understand the approach now, but I think it is a hacky
> one.
> > > > There
> > > > > >> are
> > > > > >>>>>> multiple short comings with this design. First, it degrades
> > the
> > > > > >>>> durability.
> > > > > >>>>>> For each partition, now there are two files being actively
> > > written
> > > > > at
> > > > > >> a
> > > > > >>>>>> given point of time, one for the data and another for the
> > > > metadata.
> > > > > >>>>>> Flushing each file requires a separate IO. If the disk has
> 1K
> > > IOPS
> > > > > and
> > > > > >>>> we
> > > > > >>>>>> have 5K partitions in a broker, currently we can afford to
> > flush
> > > > > each
> > > > > >>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If
> > we
> > > > > double
> > > > > >>>> the
> > > > > >>>>>> number of files per partition, we can only flush each
> > partition
> > > > > every
> > > > > >>>> 10
> > > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we ever
> need
> > > > this
> > > > > >>>>>> metadata somewhere else, say in the WAL file manager, the
> > > consumer
> > > > > >>>> needs to
> > > > > >>>>>> subscribe to every partition in the cluster, which is
> > > inefficient.
> > > > > The
> > > > > >>>>>> actual benefit of this approach is also questionable. On the
> > > > > surface,
> > > > > >>>> it
> > > > > >>>>>> might seem that we could reduce the number of lines that
> need
> > to
> > > > be
> > > > > >>>> changed
> > > > > >>>>>> for this KIP. However, the changes are quite intrusive to
> the
> > > > > classic
> > > > > >>>>>> partition's code path and will probably make the code base
> > > harder
> > > > to
> > > > > >>>>>> maintain in the long run. I like the original approach based
> > on
> > > > the
> > > > > >>>> batch
> > > > > >>>>>> coordinator much better than this one. We could probably
> > > refactor
> > > > > the
> > > > > >>>>>> producer state code so that it could be reused in the batch
> > > > > >>>> coordinator.
> > > > > >>>>>>
> > > > > >>>>>> Thanks for addressing the concerns on the number of RPCs in
> > the
> > > > > >> produce
> > > > > >>>>>> path. I agree that with the metadata crafting mechanism, we
> > > could
> > > > > >>>> mitigate
> > > > > >>>>>> the PRC problem. However, since we now require the metadata
> to
> > > be
> > > > > >>>>>> collocated with the data on the same set of brokers, it's
> > weird
> > > > that
> > > > > >>>> they
> > > > > >>>>>> are now managed by different mechanisms. The data assignment
> > now
> > > > > uses
> > > > > >>>> the
> > > > > >>>>>> metadata crafting mechanism, but the metadata is stored in
> the
> > > > > classic
> > > > > >>>>>> partition using its own assignment strategy. It will be
> > > > complicated
> > > > > to
> > > > > >>>> keep
> > > > > >>>>>> them collocated.
> > > > > >>>>>>
> > > > > >>>>>> I am also concerned about the removal of the object
> > > > > compaction/merging
> > > > > >>>>>> step. My first concern is on the amount of metadata that
> need
> > to
> > > > be
> > > > > >>>> kept.
> > > > > >>>>>> Without object compcation, the metadata generated in the
> > produce
> > > > > path
> > > > > >>>> can
> > > > > >>>>>> only be deleted after remote tiering kicks in. Let's say for
> > > every
> > > > > >>>> 250ms we
> > > > > >>>>>> produce 100 byte of metadata per partition. Let's say
> remoting
> > > > > tiering
> > > > > >>>>>> kicks in after 5 hours. In a cluster with 100K partitions,
> we
> > > need
> > > > > to
> > > > > >>>> keep
> > > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata,
> > > quite
> > > > > >>>>>> signficant. A second concern is on performance. Every time
> we
> > > need
> > > > > to
> > > > > >>>>>> rebuild the caching data, we need to read a bunch of small
> > > objects
> > > > > >>>> from S3,
> > > > > >>>>>> slowing down the building process. If a consumer happens to
> > need
> > > > > such
> > > > > >>>> data,
> > > > > >>>>>> it could slow down the application.
> > > > > >>>>>>
> > > > > >>>>>> I see a tendency toward primarily optimizing for the fewest
> > code
> > > > > >>>> changes in
> > > > > >>>>>> the KIP. Instead, our primary goal should be a clean design
> > that
> > > > can
> > > > > >>>> last
> > > > > >>>>>> for the long term.
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>>
> > > > > >>>>>> Jun
> > > > > >>>>>>
> > > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> > ivan@ivanyu.me
> > > >
> > > > > >>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hi Jun,
> > > > > >>>>>>>
> > > > > >>>>>>> Thank you for your message. I’m sorry that I failed to
> > clearly
> > > > > >>>> explain
> > > > > >>>>>> the
> > > > > >>>>>>> idea. Let me try to fix this.
> > > > > >>>>>>>
> > > > > >>>>>>>> Does each partition now have a metadata partition and a
> > > separate
> > > > > >>>> data
> > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> doubles
> > > the
> > > > > >>>> number
> > > > > >>>>>>> of
> > > > > >>>>>>>> partitions, which impacts the number of open file
> > descriptors
> > > > and
> > > > > >>>> the
> > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > > > separate
> > > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > > creating
> > > > > an
> > > > > >>>>>>>> internal topic with an unbounded number of partitions.
> > > > > >>>>>>>
> > > > > >>>>>>> No. There will be only one physical partition per diskless
> > > > > >>>> partition. Let
> > > > > >>>>>>> me explain this with an example. Let’s say we have a
> diskless
> > > > > >>>> partition
> > > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We
> > > > produce
> > > > > >>>> some
> > > > > >>>>>>> batches to this partition. The content of the segment file
> > will
> > > > be
> > > > > >>>>>>> something like this (for each batch):
> > > > > >>>>>>>
> > > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > > >>>>>>> Length: 123456 (like in classic)
> > > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > > >>>>>>> Magic: like in classic
> > > > > >>>>>>> CRC: like in classic
> > > > > >>>>>>> Attributes: like in classic
> > > > > >>>>>>> LastOffsetDelta: like in classic
> > > > > >>>>>>> BaseTimestamp: like in classic
> > > > > >>>>>>> MaxTimestamp: like in classic
> > > > > >>>>>>> ProducerId: like in classic
> > > > > >>>>>>> ProducerEpoch: like in classic
> > > > > >>>>>>> BaseSequence: like in classic
> > > > > >>>>>>> RecordsCount: like in classic
> > > > > >>>>>>> Records:
> > > > > >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a;
> byte
> > > > offset
> > > > > >>>>>>> 123456
> > > > > >>>>>>>
> > > > > >>>>>>> It looks very much like classic log entries. The only
> > > difference
> > > > is
> > > > > >>>> that
> > > > > >>>>>>> instead of writing real Records, we write the reference to
> > the
> > > > WAL
> > > > > >>>> file
> > > > > >>>>>>> with the batch data (I guess we need only the name and the
> > byte
> > > > > >>>> offset,
> > > > > >>>>>>> because the byte length is the standard field above).
> > > Otherwise,
> > > > > >>>> it’s a
> > > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > > >>>>>>>
> > > > > >>>>>>> So we have as many partitions for diskless as for classic.
> As
> > > of
> > > > > open
> > > > > >>>>>> file
> > > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > > >>>>>>>
> > > > > >>>>>>>> Are the metadata and
> > > > > >>>>>>>> the data for the same partition always collocated on the
> > same
> > > > > >>>> broker?
> > > > > >>>>>> If
> > > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > > >>>>>>>
> > > > > >>>>>>> The source of truth for the data is still in WAL files on
> > > object
> > > > > >>>> storage.
> > > > > >>>>>>> The source of truth for the metadata is in segment files on
> > the
> > > > > >>>> brokers
> > > > > >>>>>> in
> > > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > > independent
> > > > > of
> > > > > >>>> this
> > > > > >>>>>>> new proposal, but I want to present them to give the idea
> > that
> > > > > only a
> > > > > >>>>>>> limited amount of data files will be operated locally:
> > > > > >>>>>>> - We want to assemble batches into segment files and
> offload
> > > them
> > > > > to
> > > > > >>>>>>> tiered storage in order to prevent the unbounded growth of
> > > batch
> > > > > >>>>>> metadata.
> > > > > >>>>>>> For this, we need to open only  a few file descriptors (for
> > the
> > > > > >>>> segment
> > > > > >>>>>>> file itself + the necessary indexes) before the segment is
> > > fully
> > > > > >>>> written
> > > > > >>>>>>> and handed over to RemoteLogManager.
> > > > > >>>>>>> - We want to assemble local segment files for caching
> > purposes
> > > as
> > > > > >>>> well,
> > > > > >>>>>>> i.e. to speed up fetching. This will not materialize the
> full
> > > > > >>>> content of
> > > > > >>>>>>> the log, but only the hot set according to some policy (or
> > > > > >>>> configurable
> > > > > >>>>>>> policies), i.e. the number of segments and file descriptors
> > > will
> > > > > >>>> also be
> > > > > >>>>>>> limited.
> > > > > >>>>>>>
> > > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > > higher.
> > > > > For
> > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > cluster
> > > > > >>>> with 100
> > > > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > > > requests.
> > > > > >>>>>> This
> > > > > >>>>>>>> will significantly increase the request rate.
> > > > > >>>>>>>
> > > > > >>>>>>> This is a valid concern that we considered, but this issue
> > can
> > > be
> > > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > > >>>>>>> The situation with a single broker is trivial: all the
> commit
> > > > > >>>> requests go
> > > > > >>>>>>> from the broker to itself.
> > > > > >>>>>>> Let’s scale this to a multi-broker cluster, but located in
> > the
> > > > > single
> > > > > >>>>>> rack
> > > > > >>>>>>> (AZ). Any broker can accept Produce requests for diskless
> > > > > >>>> partitions, but
> > > > > >>>>>>> we can tell producers (through metadata) to always send
> > Produce
> > > > > >>>> requests
> > > > > >>>>>> to
> > > > > >>>>>>> leaders. For example, broker 0 hosts the leader replicas
> for
> > > > > diskless
> > > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless
> Produce
> > > > > >>>> requests
> > > > > >>>>>> for
> > > > > >>>>>>> these partitions in various combinations, but only for
> them.
> > > > > >>>>>>>
> > > > > >>>>>>>                   Broker 0
> > > > > >>>>>>>             +-----------------+
> > > > > >>>>>>>             |    t1-0         |
> > > > > >>>>>>>             |    t2-1 <--------------------+
> > > > > >>>>>>>             |    t3-0         |            |
> > > > > >>>>>>> produce      | +-------------+ |            |
> > > > > >>>>>>> requests     | |  diskless   | |            |
> > > > > >>>>>>> --------------->|   produce   +--------------+
> > > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > > >>>>>>> partitions   | +-------------+ |    for these partitions
> > > > > >>>>>>>             |                 |
> > > > > >>>>>>>             +-----------------+
> > > > > >>>>>>>
> > > > > >>>>>>> The same applies for other brokers in this cluster.
> > > Effectively,
> > > > > each
> > > > > >>>>>>> broker will commit only to itself, which effectively means
> 1
> > > > commit
> > > > > >>>>>> request
> > > > > >>>>>>> per WAL buffer (this may be 0 physical network calls, if we
> > > wish,
> > > > > >>>> just a
> > > > > >>>>>>> local function call).
> > > > > >>>>>>>
> > > > > >>>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we
> > > > cannot
> > > > > >>>> always
> > > > > >>>>>>> send Produce requests to the designated leaders of diskless
> > > > > >>>> partitions:
> > > > > >>>>>>> this would mean inter-AZ network traffic, which we would
> like
> > > to
> > > > > >>>> avoid.
> > > > > >>>>>> To
> > > > > >>>>>>> avoid it, we say that every broker has a “diskless produce
> > > > > >>>>>> representative”
> > > > > >>>>>>> in every AZ. If we continue our example: when a Produce
> > request
> > > > for
> > > > > >>>> t1-0,
> > > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on
> > > broker 0
> > > > > >>>> (in the
> > > > > >>>>>>> broker’s AZ the representative is the broker itself).
> > However,
> > > if
> > > > > it
> > > > > >>>>>> comes
> > > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > > > > >>>>>>>
> > > > > >>>>>>> |produce requests         |produce requests        |produce
> > > > > >>>> requests
> > > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for
> t1-0,
> > > > t2-1,
> > > > > >>>>>> t3-0
> > > > > >>>>>>> |from AZ 0                |from AZ 1               |from
> AZ 2
> > > > > >>>>>>> v                         v                        v
> > > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ
> 2)
> > > > > >>>>>>> +---------------+      +---------------+
> > +---------------+
> > > > > >>>>>>> |     t1-0      |      |               |      |
> >  |
> > > > > >>>>>>> |     t2-1      |      |               |      |
> >  |
> > > > > >>>>>>> |     t3-0      |      |               |      |
> >  |
> > > > > >>>>>>> +---------------+      +--------+------+
> > +--------+------+
> > > > > >>>>>>>    ^     ^                    |                      |
> > > > > >>>>>>>    |     +--------------------+                      |
> > > > > >>>>>>>    |     commit requests for these partitions        |
> > > > > >>>>>>>    |                                                 |
> > > > > >>>>>>>    +-------------------------------------------------+
> > > > > >>>>>>>          commit requests for these partitions
> > > > > >>>>>>>
> > > > > >>>>>>> All the partitions that broker 0 is the leader of will be
> > > > > >>>> “represented”
> > > > > >>>>>> by
> > > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > > >>>>>>>
> > > > > >>>>>>> Of course, this relationship goes both ways between AZs
> (not
> > > > > >>>> necessarily
> > > > > >>>>>>> between the same brokers). It means that provided the
> cluster
> > > is
> > > > > >>>> balanced
> > > > > >>>>>>> by the number of brokers per AZ, each broker will represent
> > > > > >>>>>> (number_of_azs
> > > > > >>>>>>> - 1) other brokers. This will result in the situation that
> > for
> > > > the
> > > > > >>>>>> majority
> > > > > >>>>>>> of commits, each broker will do up to (number_of_azs - 1)
> > > network
> > > > > >>>> commit
> > > > > >>>>>>> requests (plus one local). Cloud regions tend to have 3
> AZs,
> > > very
> > > > > >>>> rarely
> > > > > >>>>>>> more. That means, brokers will be doing up to 2 network
> > commit
> > > > > >>>> requests
> > > > > >>>>>> per
> > > > > >>>>>>> WAL file.
> > > > > >>>>>>>
> > > > > >>>>>>> There are the following exceptions:
> > > > > >>>>>>> 1. Broker count imbalance between AZs. For example, when we
> > > have
> > > > 2
> > > > > >>>> AZs
> > > > > >>>>>> and
> > > > > >>>>>>> one has three brokers and another AZ has one. This one
> broker
> > > > will
> > > > > do
> > > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is not
> an
> > > > > extreme
> > > > > >>>>>>> amplification. Such an imbalance is not healthy in most
> > > practical
> > > > > >>>> setups
> > > > > >>>>>>> and should be avoided anyway.
> > > > > >>>>>>> 2. Leadership changes and metadata propagation period. When
> > the
> > > > > >>>> partition
> > > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> > producers
> > > > > will
> > > > > >>>> not
> > > > > >>>>>>> know this immediately (unless we want to be strict and
> > respond
> > > > with
> > > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will
> come
> > > > > >>>> together
> > > > > >>>>>> in a
> > > > > >>>>>>> WAL buffer on broker 2, it will have to send two commit
> > > requests:
> > > > > to
> > > > > >>>>>> broker
> > > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0.
> > This
> > > > > >>>> situation
> > > > > >>>>>>> is not permanent and as producers update the cluster
> > metadata,
> > > it
> > > > > >>>> will be
> > > > > >>>>>>> resolved.
> > > > > >>>>>>>
> > > > > >>>>>>> This all could be built with the metadata crafting
> mechanism
> > > only
> > > > > >>>> (which
> > > > > >>>>>>> is anyway needed for Diskless in one way or another to
> direct
> > > > > >>>> producers
> > > > > >>>>>> and
> > > > > >>>>>>> consumers where we need to avoid inter-AZ traffic), just
> with
> > > the
> > > > > >>>> right
> > > > > >>>>>>> policy for it (for example, some deterministic hash-based
> > > > formula).
> > > > > >>>> I.e.
> > > > > >>>>>> no
> > > > > >>>>>>> explicit support for “produce representative” or anything
> > like
> > > > this
> > > > > >>>> is
> > > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > > >>>>>>>
> > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > places,
> > > > > >>>> partition
> > > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> > truth,
> > > > and
> > > > > >>>> how
> > > > > >>>>>> do
> > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > >>>>>>>
> > > > > >>>>>>> We do only two operations on WAL files that span multiple
> > > > diskless
> > > > > >>>>>>> partitions: committing and deleting. Commits can be done
> > > > > >>>> independently as
> > > > > >>>>>>> described above. But deletes are different, because when a
> > file
> > > > is
> > > > > >>>>>> deleted,
> > > > > >>>>>>> this affects all the partitions that still have alive
> batches
> > > in
> > > > > this
> > > > > >>>>>> file
> > > > > >>>>>>> (if any).
> > > > > >>>>>>>
> > > > > >>>>>>> The WAL file manager is a necessary point of coordination
> to
> > > > delete
> > > > > >>>> WAL
> > > > > >>>>>>> files safely. We can say it is the source of truth about
> > files
> > > > > >>>>>> themselves,
> > > > > >>>>>>> while the partition leaders and their logs hold the truth
> > about
> > > > > >>>> whether a
> > > > > >>>>>>> particular file contains live batches of this particular
> > > > partition.
> > > > > >>>>>>>
> > > > > >>>>>>> The file manager will do this important task: be able to
> say
> > > for
> > > > > sure
> > > > > >>>>>> that
> > > > > >>>>>>> a file does not contain any live batch of any existing
> > > partition.
> > > > > For
> > > > > >>>>>> this,
> > > > > >>>>>>> it will have to periodically check against the partition
> > > leaders.
> > > > > >>>>>>> Considering that batch deletion is irreversible, when we
> > > declare
> > > > a
> > > > > >>>> file
> > > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > > >>>>>>>
> > > > > >>>>>>> The file manager has to know about files being committed to
> > > start
> > > > > >>>> track
> > > > > >>>>>>> them and periodically check if they are empty. We can
> > consider
> > > > > >>>> various
> > > > > >>>>>> ways
> > > > > >>>>>>> to achieve this:
> > > > > >>>>>>> 1. As was proposed in my previous message: best effort
> commit
> > > by
> > > > > >>>> brokers
> > > > > >>>>>> +
> > > > > >>>>>>> periodic prefix scans of object storage to detect files
> that
> > > went
> > > > > >>>> below
> > > > > >>>>>> the
> > > > > >>>>>>> radar due to network issue or the file manager temporary
> > > > > >>>> unavailability.
> > > > > >>>>>>> We’re speaking about listing the file names only and
> opening
> > > only
> > > > > >>>>>>> previously unknown files in order to find the partitions
> > > involved
> > > > > >>>> with
> > > > > >>>>>> them.
> > > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill the
> list
> > of
> > > > > files
> > > > > >>>>>>> fully asynchronously and in the background. This may be not
> > > ideal
> > > > > >>>> due to
> > > > > >>>>>>> costs and performance of scanning tons of files. However,
> the
> > > > > number
> > > > > >>>> of
> > > > > >>>>>>> live WAL files should be limited due to tiered storage
> > > > offloading +
> > > > > >>>> we
> > > > > >>>>>> can
> > > > > >>>>>>> optimize this if we give files some global soft order in
> > their
> > > > > names.
> > > > > >>>>>>>
> > > > > >>>>>>>> I am not sure how this design simplifies the
> implementation.
> > > The
> > > > > >>>>>> existing
> > > > > >>>>>>>> producer/replication code can't be simply reused.
> Adjusting
> > > both
> > > > > >>>> the
> > > > > >>>>>>> write
> > > > > >>>>>>>> path in the leader and the replication path in the
> follower
> > to
> > > > > >>>>>> understand
> > > > > >>>>>>>> batch-header only data is quite intrusive to the existing
> > > logic.
> > > > > >>>>>>>
> > > > > >>>>>>> It is true that we’ll have to change LocalLog and
> UnifiedLog
> > in
> > > > > >>>> order to
> > > > > >>>>>>> support these changes. However, it seems that idempotence,
> > > > > >>>> transactions,
> > > > > >>>>>>> queues, tiered storage will have to be changed less than
> with
> > > the
> > > > > >>>>>> original
> > > > > >>>>>>> design. This is because the partition leader state would
> > remain
> > > > in
> > > > > >>>> the
> > > > > >>>>>> same
> > > > > >>>>>>> place (on brokers) and existing workflows that involve it
> > would
> > > > > have
> > > > > >>>> to
> > > > > >>>>>> be
> > > > > >>>>>>> changed less compared to the situation where we globalize
> the
> > > > > >>>> partition
> > > > > >>>>>>> leader state in the batch coordinator. I admit this is hard
> > to
> > > > make
> > > > > >>>>>>> convincing without both real implementations to hand :)
> > > > > >>>>>>>
> > > > > >>>>>>>> I am also
> > > > > >>>>>>>> not sure how this enables seamless switching the topic
> modes
> > > > > >>>> between
> > > > > >>>>>>>> diskless and classic. Could you provide more details on
> > those?
> > > > > >>>>>>>
> > > > > >>>>>>> Let’s consider the scenario of turning a classic topic into
> > > > > >>>> diskless. The
> > > > > >>>>>>> user sets diskless.enabled=true, the leader receives this
> > > > metadata
> > > > > >>>> update
> > > > > >>>>>>> and does the following:
> > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > >>>>>>> 2. Close the current active segment.
> > > > > >>>>>>> 3. Start a new segment that will be written in the diskless
> > > > format
> > > > > >>>> (i.e.
> > > > > >>>>>>> without data).
> > > > > >>>>>>> 4. Start accepting diskless commits.
> > > > > >>>>>>>
> > > > > >>>>>>> Since it’s the same log, the followers will know about that
> > > > switch
> > > > > >>>>>>> consistently. They will finish replicating the classic
> > segments
> > > > and
> > > > > >>>> start
> > > > > >>>>>>> replicating the diskless ones. They will always know where
> > each
> > > > > >>>> batch is
> > > > > >>>>>>> located (either inside a classic segment or referenced by a
> > > > > diskless
> > > > > >>>>>> one).
> > > > > >>>>>>> Switching back should be similar.
> > > > > >>>>>>>
> > > > > >>>>>>> Doing this with the coordinator is possible, but has some
> > > > caveats.
> > > > > >>>> The
> > > > > >>>>>>> leader must do the following:
> > > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > > >>>>>>> 2. Close the current active segment.
> > > > > >>>>>>> 3. Write a special control segment to persist and replicate
> > the
> > > > > fact
> > > > > >>>> that
> > > > > >>>>>>> from offset N the partition is now in the diskless mode.
> > > > > >>>>>>> 4. Inform the coordinator about the first offset N of the
> > > > “diskless
> > > > > >>>> era”.
> > > > > >>>>>>> 5. Inform the controller quorum that the transition has
> > > finished
> > > > > and
> > > > > >>>> that
> > > > > >>>>>>> brokers now can process diskless writes for this partition.
> > > > > >>>>>>> This could fail at some points, so this will probably
> require
> > > > some
> > > > > >>>>>>> explicit state machine with replication either in the
> > partition
> > > > log
> > > > > >>>> or in
> > > > > >>>>>>> KRaft.
> > > > > >>>>>>>
> > > > > >>>>>>> It seems that the coordinator-less approach makes this
> > simpler
> > > > > >>>> because
> > > > > >>>>>> the
> > > > > >>>>>>> “coordinator” for the partition and the partition leader
> are
> > > the
> > > > > >>>> same and
> > > > > >>>>>>> they store the partition metadata in the same log, too.
> While
> > > in
> > > > > the
> > > > > >>>>>>> coordinator approach we have to perform some kind of a
> > > > distributed
> > > > > >>>> commit
> > > > > >>>>>>> to handover metadata management from the classic partition
> > > leader
> > > > > to
> > > > > >>>> the
> > > > > >>>>>>> batch coordinator.
> > > > > >>>>>>>
> > > > > >>>>>>> I hope these explanations help to clarify the idea. Please
> > let
> > > me
> > > > > >>>> know if
> > > > > >>>>>>> I should go deeper anywhere.
> > > > > >>>>>>>
> > > > > >>>>>>> Best,
> > > > > >>>>>>> Ivan and the Diskless team
> > > > > >>>>>>>
> > > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > >>>>>>>> Hi, Ivan,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks for the update.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I am not sure that I fully understand the new design, but
> it
> > > > seems
> > > > > >>>> less
> > > > > >>>>>>>> clean than before.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Does each partition now have a metadata partition and a
> > > separate
> > > > > >>>> data
> > > > > >>>>>>>> partition? If so, I am concerned that it essentially
> doubles
> > > the
> > > > > >>>> number
> > > > > >>>>>>> of
> > > > > >>>>>>>> partitions, which impacts the number of open file
> > descriptors
> > > > and
> > > > > >>>> the
> > > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > > > separate
> > > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > > creating
> > > > > an
> > > > > >>>>>>>> internal topic with an unbounded number of partitions. Are
> > the
> > > > > >>>> metadata
> > > > > >>>>>>> and
> > > > > >>>>>>>> the data for the same partition always collocated on the
> > same
> > > > > >>>> broker?
> > > > > >>>>>> If
> > > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > > >>>>>>>>
> > > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > > higher.
> > > > > For
> > > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> > cluster
> > > > > >>>> with 100
> > > > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > > > requests.
> > > > > >>>>>> This
> > > > > >>>>>>>> will significantly increase the request rate.
> > > > > >>>>>>>>
> > > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> > places,
> > > > > >>>> partition
> > > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> > truth,
> > > > and
> > > > > >>>> how
> > > > > >>>>>> do
> > > > > >>>>>>>> we maintain consistency between the two places?
> > > > > >>>>>>>>
> > > > > >>>>>>>> I am not sure how this design simplifies the
> implementation.
> > > The
> > > > > >>>>>> existing
> > > > > >>>>>>>> producer/replication code can't be simply reused.
> Adjusting
> > > both
> > > > > >>>> the
> > > > > >>>>>>> write
> > > > > >>>>>>>> path in the leader and the replication path in the
> follower
> > to
> > > > > >>>>>> understand
> > > > > >>>>>>>> batch-header only data is quite intrusive to the existing
> > > > logic. I
> > > > > >>>> am
> > > > > >>>>>>> also
> > > > > >>>>>>>> not sure how this enables seamless switching the topic
> modes
> > > > > >>>> between
> > > > > >>>>>>>> diskless and classic. Could you provide more details on
> > those?
> > > > > >>>>>>>>
> > > > > >>>>>>>> Jun
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> > ivan@ivanyu.me
> > > >
> > > > > >>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi dear Kafka community,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> In the initial Diskless proposal, we proposed to have a
> > > > separate
> > > > > >>>>>>>>> component, batch/diskless coordinator, whose role would
> be
> > to
> > > > > >>>>>> centrally
> > > > > >>>>>>>>> manage the batch and WAL file metadata for diskless
> topics.
> > > > This
> > > > > >>>>>>> component
> > > > > >>>>>>>>> drew many reasonable comments from the community about
> how
> > it
> > > > > >>>> would
> > > > > >>>>>>> support
> > > > > >>>>>>>>> various Kafka features (transactions, queues) and its
> > > > > >>>> scalability.
> > > > > >>>>>>> While we
> > > > > >>>>>>>>> believe we have good answers to all the expressed
> concerns,
> > > we
> > > > > >>>> took a
> > > > > >>>>>>> step
> > > > > >>>>>>>>> back and looked at the problem from a different
> > perspective.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> We would like to propose an alternative Diskless design
> > > > *without
> > > > > >>>> a
> > > > > >>>>>>>>> centralized coordinator*. We believe this approach has
> > > > potential
> > > > > >>>> and
> > > > > >>>>>>>>> propose to discuss it as it may be more appealing to the
> > > > > >>>> community.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Let us explain the idea. Most of the complications with
> the
> > > > > >>>> original
> > > > > >>>>>>>>> Diskless approach come from one necessary architecture
> > > change:
> > > > > >>>>>>> globalizing
> > > > > >>>>>>>>> the local state of partition leader in the batch
> > coordinator.
> > > > > >>>> This
> > > > > >>>>>>> causes
> > > > > >>>>>>>>> deviations to the established workflows in various
> features
> > > > like
> > > > > >>>>>>> produce
> > > > > >>>>>>>>> idempotence and transactions, queues, retention, etc.
> These
> > > > > >>>>>> deviations
> > > > > >>>>>>> need
> > > > > >>>>>>>>> to be carefully considered, designed, and later
> implemented
> > > and
> > > > > >>>>>>> tested. In
> > > > > >>>>>>>>> the new approach we want to avoid this by making
> partition
> > > > > >>>> leaders
> > > > > >>>>>>> again
> > > > > >>>>>>>>> responsible for managing their partitions, even in
> diskless
> > > > > >>>> topics.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> In classic Kafka topics, batch data and metadata are
> > blended
> > > > > >>>> together
> > > > > >>>>>>> in
> > > > > >>>>>>>>> the one partition log. The crux of the Diskless idea is
> to
> > > > > >>>> decouple
> > > > > >>>>>>> them
> > > > > >>>>>>>>> and move data to the remote storage, while keeping
> metadata
> > > > > >>>> somewhere
> > > > > >>>>>>> else.
> > > > > >>>>>>>>> Using the central batch coordinator for managing batch
> > > metadata
> > > > > >>>> is
> > > > > >>>>>> one
> > > > > >>>>>>> way,
> > > > > >>>>>>>>> but not the only.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Let’s now think about managing metadata for each user
> > > partition
> > > > > >>>>>>>>> independently. Generally partitions are independent and
> > don’t
> > > > > >>>> share
> > > > > >>>>>>>>> anything apart from that their data are mixed in WAL
> files.
> > > If
> > > > we
> > > > > >>>>>>> figure
> > > > > >>>>>>>>> out how to commit and later delete WAL files safely, we
> > will
> > > > > >>>> achieve
> > > > > >>>>>>> the
> > > > > >>>>>>>>> necessary autonomy that allows us to get rid of the
> central
> > > > batch
> > > > > >>>>>>>>> coordinator. Instead, *each diskless user partition will
> be
> > > > > >>>> managed
> > > > > >>>>>> by
> > > > > >>>>>>> its
> > > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in classic
> > > > > >>>> topics, the
> > > > > >>>>>>>>> leader uses the partition log as the way to persist batch
> > > > > >>>> metadata,
> > > > > >>>>>>> i.e.
> > > > > >>>>>>>>> the regular batch header + the information about how to
> > find
> > > > this
> > > > > >>>>>>> batch on
> > > > > >>>>>>>>> remote storage. In contrast to classic topics, batch data
> > is
> > > in
> > > > > >>>>>> remote
> > > > > >>>>>>>>> storage.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > > >>>>>>>>> • Classic topics:
> > > > > >>>>>>>>>  • Data and metadata are co-located in the partition log.
> > > > > >>>>>>>>>  • The partition log content: [Batch header
> > (metadata)|Batch
> > > > > >>>> data].
> > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > > >>>> metadata.
> > > > > >>>>>>>>> • Original Diskless:
> > > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on
> remote
> > > > > >>>> storage.
> > > > > >>>>>>>>>  • The partition state is global in the batch
> coordinator.
> > > > > >>>>>>>>> • New Diskless:
> > > > > >>>>>>>>>  • Metadata is in the partition log, data is on remote
> > > storage.
> > > > > >>>>>>>>>  • Partition log content: [Batch header (metadata)|Batch
> > > > > >>>>>> coordinates
> > > > > >>>>>>> on
> > > > > >>>>>>>>> remote storage].
> > > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > > >>>> metadata.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Let’s consider the produce path. Here’s the reminder of
> the
> > > > > >>>> original
> > > > > >>>>>>>>> Diskless design:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> The new approach could be depicted as the following:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> As you can see, the main difference is that now instead
> of
> > a
> > > > > >>>> single
> > > > > >>>>>>> commit
> > > > > >>>>>>>>> request to the batch coordinator, we send multiple
> parallel
> > > > > >>>> commit
> > > > > >>>>>>> requests
> > > > > >>>>>>>>> to all the leaders of each partition involved in the WAL
> > > file.
> > > > > >>>> Each
> > > > > >>>>>> of
> > > > > >>>>>>> them
> > > > > >>>>>>>>> will commit its batches independently, without
> coordinating
> > > > with
> > > > > >>>>>> other
> > > > > >>>>>>>>> leaders and any other components. Batch data is addressed
> > by
> > > > the
> > > > > >>>> WAL
> > > > > >>>>>>> file
> > > > > >>>>>>>>> name, the byte offset and size, which allows partitions
> to
> > > know
> > > > > >>>>>> nothing
> > > > > >>>>>>>>> about other partitions to access their data in shared WAL
> > > > files.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> The number of partitions involved in a single WAL file
> may
> > be
> > > > > >>>> quite
> > > > > >>>>>>> large,
> > > > > >>>>>>>>> e.g. a hundred. A hundred network requests to commit one
> > WAL
> > > > > >>>> file is
> > > > > >>>>>>> very
> > > > > >>>>>>>>> impractical. However, there are ways to reduce this
> number:
> > > > > >>>>>>>>> 1. Partition leaders are located on brokers. Requests to
> > > > > >>>> leaders on
> > > > > >>>>>>> one
> > > > > >>>>>>>>> broker could be grouped together into a single physical
> > > network
> > > > > >>>>>> request
> > > > > >>>>>>>>> (resembling the normal Produce request that may carry
> > batches
> > > > for
> > > > > >>>>>> many
> > > > > >>>>>>>>> partitions inside). This will cap the number of network
> > > > requests
> > > > > >>>> to
> > > > > >>>>>> the
> > > > > >>>>>>>>> number of brokers in the cluster.
> > > > > >>>>>>>>> 2. If we craft the cluster metadata to make producers
> send
> > > > their
> > > > > >>>>>>> requests
> > > > > >>>>>>>>> to the right brokers (with respect to AZs), we may
> achieve
> > > the
> > > > > >>>> higher
> > > > > >>>>>>>>> concentration of logical commit requests in physical
> > network
> > > > > >>>> requests
> > > > > >>>>>>>>> reducing the number of the latter ones even further,
> > ideally
> > > to
> > > > > >>>> one.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Obviously, out of multiple commit requests some may fail
> or
> > > > time
> > > > > >>>> out
> > > > > >>>>>>> for a
> > > > > >>>>>>>>> variety of reasons. This is fine. Some producers will
> > receive
> > > > > >>>> totally
> > > > > >>>>>>> or
> > > > > >>>>>>>>> partially failed responses to their Produce requests,
> > similar
> > > > to
> > > > > >>>> what
> > > > > >>>>>>> they
> > > > > >>>>>>>>> would have received when appending to a classic topic
> fails
> > > or
> > > > > >>>> times
> > > > > >>>>>>> out.
> > > > > >>>>>>>>> If a partition experiences problems, other partitions
> will
> > > not
> > > > be
> > > > > >>>>>>> affected
> > > > > >>>>>>>>> (again, like in classic topics). Of course, the
> uncommitted
> > > > data
> > > > > >>>> will
> > > > > >>>>>>> be
> > > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> > (batches
> > > > are
> > > > > >>>>>>> constantly
> > > > > >>>>>>>>> assembled into segments and offloaded to tiered storage),
> > so
> > > > this
> > > > > >>>>>>> garbage
> > > > > >>>>>>>>> will be eventually deleted.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> For safely deleting WAL files we now need to centrally
> > manage
> > > > > >>>> them,
> > > > > >>>>>> as
> > > > > >>>>>>>>> this is the only state and logic that spans multiple
> > > > partitions.
> > > > > >>>> On
> > > > > >>>>>> the
> > > > > >>>>>>>>> diagram, you can see another commit request called
> “Commit
> > > file
> > > > > >>>> (best
> > > > > >>>>>>>>> effort)” going to the WAL File Manager. This manager will
> > be
> > > > > >>>>>>> responsible
> > > > > >>>>>>>>> for the following:
> > > > > >>>>>>>>> 1. Collecting (by requests from brokers) and persisting
> > > > > >>>> information
> > > > > >>>>>>> about
> > > > > >>>>>>>>> committed WAL files.
> > > > > >>>>>>>>> 2. To handle potential failures in file information
> > delivery,
> > > > it
> > > > > >>>>>> will
> > > > > >>>>>>> be
> > > > > >>>>>>>>> doing prefix scan on the remote storage periodically to
> > find
> > > > and
> > > > > >>>>>>> register
> > > > > >>>>>>>>> unknown files. The period of this scan will be
> configurable
> > > and
> > > > > >>>>>> ideally
> > > > > >>>>>>>>> should be quite long.
> > > > > >>>>>>>>> 3. Checking with the relevant partition leaders (after a
> > > grace
> > > > > >>>>>>> period) if
> > > > > >>>>>>>>> they still have batches in a particular file.
> > > > > >>>>>>>>> 4. Physically deleting files when they aren’t anymore
> > > referred
> > > > > >>>> to by
> > > > > >>>>>>> any
> > > > > >>>>>>>>> partition.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> This new design offers the following advantages:
> > > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka
> features
> > > such
> > > > > >>>> as
> > > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> > retention.
> > > > > >>>> Now we
> > > > > >>>>>>> don’t
> > > > > >>>>>>>>> need to abstract away and reuse the code from partition
> > > leaders
> > > > > >>>> in
> > > > > >>>>>> the
> > > > > >>>>>>>>> batch coordinator. Instead, we will literally use the
> same
> > > code
> > > > > >>>> paths
> > > > > >>>>>>> in
> > > > > >>>>>>>>> leaders, with little adaptation. Workflows from classic
> > > topics
> > > > > >>>> mostly
> > > > > >>>>>>>>> remain unchanged.
> > > > > >>>>>>>>> For example, it seems that
> > > > > >>>>>>>>>
> ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > > and
> > > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for
> transaction
> > > > > >>>> support
> > > > > >>>>>> on
> > > > > >>>>>>> the
> > > > > >>>>>>>>> partition leader side could be used for diskless topics
> > with
> > > > > >>>> little
> > > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> > idempotent
> > > > > >>>> produce
> > > > > >>>>>>> and
> > > > > >>>>>>>>> transactions, would be reused.
> > > > > >>>>>>>>> Another example is share groups support, where the share
> > > > > >>>> partition
> > > > > >>>>>>> leader,
> > > > > >>>>>>>>> being co-located with the partition leader, would execute
> > the
> > > > > >>>> same
> > > > > >>>>>>> logic
> > > > > >>>>>>>>> for both diskless and classic topics.
> > > > > >>>>>>>>> 2. It returns to the familiar partition-based scaling
> > model,
> > > > > >>>> where
> > > > > >>>>>>>>> partitions are independent.
> > > > > >>>>>>>>> 3. It makes the operation and failure patterns closer to
> > the
> > > > > >>>>>> familiar
> > > > > >>>>>>>>> ones from classic topics.
> > > > > >>>>>>>>> 4. It opens a straightforward path to seamless switching
> > the
> > > > > >>>> topics
> > > > > >>>>>>> modes
> > > > > >>>>>>>>> between diskless and classic.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> The rest of the things remain unchanged compared to the
> > > > previous
> > > > > >>>>>>> Diskless
> > > > > >>>>>>>>> design (after all previous discussions). Such things as
> > local
> > > > > >>>> segment
> > > > > >>>>>>>>> materialization by replicas, the consume path, tiered
> > storage
> > > > > >>>>>>> integration,
> > > > > >>>>>>>>> etc.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> If the community finds this design more suitable, we will
> > > > update
> > > > > >>>> the
> > > > > >>>>>>>>> KIP(s) accordingly and continue working on it. Please let
> > us
> > > > know
> > > > > >>>>>> what
> > > > > >>>>>>> you
> > > > > >>>>>>>>> think.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Best regards,
> > > > > >>>>>>>>> Ivan and Diskless team
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > >>>>>>>>>> Hi Justine,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> > transactions
> > > > > >>>> for in
> > > > > >>>>>>> the
> > > > > >>>>>>>>> diskless coordinator for as long as the corresponding
> > offsets
> > > > are
> > > > > >>>>>>> there.
> > > > > >>>>>>>>> With the tiered storage unification Greg mentioned
> earlier,
> > > > this
> > > > > >>>> will
> > > > > >>>>>>> be
> > > > > >>>>>>>>> finite time even for infinite data retention.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Best,
> > > > > >>>>>>>>>> Ivan
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > >>>>>>>>>>> Hey Ivan,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks for the response. I think most of what you said
> > made
> > > > > >>>>>> sense,
> > > > > >>>>>>> but
> > > > > >>>>>>>>> I
> > > > > >>>>>>>>>>> did have some questions about this part:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > > > >>>> topics
> > > > > >>>>>>> forgets
> > > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM
> overpasses
> > > > > >>>> it). The
> > > > > >>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > allowing
> > > > > >>>>>>> partition
> > > > > >>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > > > > >>>> wrong. We
> > > > > >>>>>>> think
> > > > > >>>>>>>>>>> about relying on this with the batch coordinator and
> > delete
> > > > > >>>> the
> > > > > >>>>>>>>> information
> > > > > >>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > > > >>>> replication
> > > > > >>>>>>> and
> > > > > >>>>>>>>> HWM
> > > > > >>>>>>>>>>> advances immediately).
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I didn't quite understand this. In classic topics, we
> > have
> > > > > >>>> maps
> > > > > >>>>>> for
> > > > > >>>>>>>>> ongoing
> > > > > >>>>>>>>>>> transactions which remove state when the transaction is
> > > > > >>>> completed
> > > > > >>>>>>> and
> > > > > >>>>>>>>> an
> > > > > >>>>>>>>>>> aborted transactions index which is retained for much
> > > longer.
> > > > > >>>>>> Once
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>> transaction is completed, the coordinator is no longer
> > > > > >>>> involved
> > > > > >>>>>> in
> > > > > >>>>>>>>>>> maintaining this partition side state, and it is
> subject
> > to
> > > > > >>>>>>> compaction
> > > > > >>>>>>>>> etc.
> > > > > >>>>>>>>>>> Looking back at the outline provided above, I didn't
> see
> > > much
> > > > > >>>>>>> about the
> > > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> > further.
> > > I
> > > > > >>>> saw
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>> following in a response:
> > > > > >>>>>>>>>>>> When the broker constructs a fully valid local
> segment,
> > > > > >>>> all the
> > > > > >>>>>>>>> necessary
> > > > > >>>>>>>>>>> control batches will be inserted and indices, including
> > the
> > > > > >>>>>>> transaction
> > > > > >>>>>>>>>>> index will be built to serve FetchRequests exactly as
> > they
> > > > > >>>> are
> > > > > >>>>>>> today.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Based on this, it seems like we need to retain the
> > > > > >>>> information
> > > > > >>>>>>> about
> > > > > >>>>>>>>>>> aborted txns for longer.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>> Justine
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > > >>>> ivan@ivanyu.me>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Hi Justine and all,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Thank you for your questions!
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> identified
> > > > > >>>> with
> > > > > >>>>>>>>> producer ID
> > > > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > > > >>>> cached
> > > > > >>>>>>>>> locally
> > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > >>>>>> transactions
> > > > > >>>>>>> can
> > > > > >>>>>>>>> be
> > > > > >>>>>>>>>>>> used
> > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > > > >>>> with
> > > > > >>>>>>>>> producer id +
> > > > > >>>>>>>>>>>>> epoch
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> You’re right that we (probably unintentionally)
> focused
> > > > > >>>> only on
> > > > > >>>>>>>>> version 2.
> > > > > >>>>>>>>>>>> We can either limit the support to version 2 or
> consider
> > > > > >>>> using
> > > > > >>>>>>> some
> > > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > transactional
> > > > > >>>>>>> checks
> > > > > >>>>>>>>> of the
> > > > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > > > >>>> like the
> > > > > >>>>>>>>> partition
> > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > > > >>>>>> checking
> > > > > >>>>>>> if
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> > ongoing,
> > > > > >>>> and
> > > > > >>>>>> of
> > > > > >>>>>>>>> course
> > > > > >>>>>>>>>>>> the normal idempotence checks. What the partition
> leader
> > > > > >>>> in the
> > > > > >>>>>>>>> classic
> > > > > >>>>>>>>>>>> topics does before appending a batch to the local log
> > > > > >>>> (e.g. in
> > > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> > Diskless,
> > > > > >>>> we
> > > > > >>>>>>>>> unfortunately
> > > > > >>>>>>>>>>>> cannot do these checks before appending the data to
> the
> > > WAL
> > > > > >>>>>>> segment
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these batches in
> > the
> > > > > >>>> batch
> > > > > >>>>>>>>> coordinator
> > > > > >>>>>>>>>>>> during the final commit.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> other
> > > > > >>>> state
> > > > > >>>>>>>>> mentioned
> > > > > >>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > > > >>>> what
> > > > > >>>>>>> state is
> > > > > >>>>>>>>>>>> stored
> > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Right, this should have been more explicit. As the
> > > > > >>>> partition
> > > > > >>>>>>> leader
> > > > > >>>>>>>>> tracks
> > > > > >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> > > > > >>>> coordinator
> > > > > >>>>>>> has
> > > > > >>>>>>>>> to as
> > > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > > >>>> transaction
> > > > > >>>>>>>>> coordinator
> > > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > > > >>>> perhaps
> > > > > >>>>>>> that
> > > > > >>>>>>>>> would
> > > > > >>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Yes. This could be deduced from the committed batches
> > and
> > > > > >>>> other
> > > > > >>>>>>>>>>>> information, but for the sake of performance we’d
> better
> > > > > >>>> store
> > > > > >>>>>> it
> > > > > >>>>>>>>>>>> explicitly.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > transactional
> > > > > >>>>>>> state is
> > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will
> be
> > > > > >>>>>> cleaned
> > > > > >>>>>>> up?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > > > >>>> topics
> > > > > >>>>>>> forgets
> > > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM
> overpasses
> > > > > >>>> it).
> > > > > >>>>>> The
> > > > > >>>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > > > >>>> allowing
> > > > > >>>>>>>>> partition
> > > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if this
> is
> > > > > >>>> wrong.
> > > > > >>>>>> We
> > > > > >>>>>>>>> think
> > > > > >>>>>>>>>>>> about relying on this with the batch coordinator and
> > > > > >>>> delete the
> > > > > >>>>>>>>> information
> > > > > >>>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > > > >>>>>> replication
> > > > > >>>>>>>>> and HWM
> > > > > >>>>>>>>>>>> advances immediately).
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>> Ivan
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > >>>>>>>>>>>>> Hey folks,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Excited to see some updates related to transactions!
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I had a few questions.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely
> identified
> > > > > >>>> with
> > > > > >>>>>>>>> producer ID
> > > > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > > > >>>> cached
> > > > > >>>>>>>>> locally
> > > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > > >>>>>> transactions
> > > > > >>>>>>> can
> > > > > >>>>>>>>> be
> > > > > >>>>>>>>>>>> used
> > > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > > > >>>> with
> > > > > >>>>>>>>> producer id +
> > > > > >>>>>>>>>>>>> epoch
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> > transactional
> > > > > >>>>>>> checks
> > > > > >>>>>>>>> of the
> > > > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > > > >>>> like the
> > > > > >>>>>>>>> partition
> > > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > > > >>>>>> checking
> > > > > >>>>>>> if
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is there
> > state
> > > > > >>>>>> about
> > > > > >>>>>>>>> ongoing
> > > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some
> other
> > > > > >>>> state
> > > > > >>>>>>>>> mentioned
> > > > > >>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > > > >>>> what
> > > > > >>>>>>> state is
> > > > > >>>>>>>>>>>> stored
> > > > > >>>>>>>>>>>>> and when it is stored.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > > > >>>> perhaps
> > > > > >>>>>>> that
> > > > > >>>>>>>>> would
> > > > > >>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> > transactional
> > > > > >>>>>>> state is
> > > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will
> be
> > > > > >>>>>> cleaned
> > > > > >>>>>>> up?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > > >>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> > > > > >>>>>> segments,
> > > > > >>>>>>>>> making
> > > > > >>>>>>>>>>>> use of
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> zero-copy
> > > > > >>>>>>>>> functionality
> > > > > >>>>>>>>>>>> already
> > > > > >>>>>>>>>>>>>> built into classic topics."
> > > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
> > > > > >>>> each
> > > > > >>>>>>>>> partition in
> > > > > >>>>>>>>>>>>>> diskless topics?
> > > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
> > > > > >>>> need
> > > > > >>>>>> to
> > > > > >>>>>>> know
> > > > > >>>>>>>>>>>> aborted
> > > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > > >>>> transaction is
> > > > > >>>>>>>>> finished
> > > > > >>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>> also inserts the control batches in the
> corresponding
> > > > > >>>> logs
> > > > > >>>>>>> of the
> > > > > >>>>>>>>>>>> involved
> > > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
> > > > > >>>> level,
> > > > > >>>>>> no
> > > > > >>>>>>>>> actual
> > > > > >>>>>>>>>>>> control
> > > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > > >>>>>>>>>>>>>> A fetch response could include multiple
> transactional
> > > > > >>>>>>> batches.
> > > > > >>>>>>>>> How
> > > > > >>>>>>>>>>>> does the
> > > > > >>>>>>>>>>>>>> broker obtain the information about the ending
> control
> > > > > >>>>>> batch
> > > > > >>>>>>> for
> > > > > >>>>>>>>> each
> > > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response needs to
> > be
> > > > > >>>>>>> built by
> > > > > >>>>>>>>>>>>>> stitching record batches and generated control
> batches
> > > > > >>>>>>> together?
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition
> leader
> > > > > >>>> that
> > > > > >>>>>>> all
> > > > > >>>>>>>>>>>> consumers
> > > > > >>>>>>>>>>>>>> are routed to?
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this
> or
> > > > > >>>> it's
> > > > > >>>>>>> too
> > > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to
> include
> > > > > >>>>>> enough
> > > > > >>>>>>>>> details
> > > > > >>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
> > > > > >>>>>> producer
> > > > > >>>>>>>>> contain a
> > > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a producer
> sends
> > > > > >>>>>> every
> > > > > >>>>>>>>> Produce
> > > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests could
> > > > > >>>> reach
> > > > > >>>>>>> the
> > > > > >>>>>>>>> batch
> > > > > >>>>>>>>>>>>>> coordinator in different order and lead to rejection
> > > > > >>>> of the
> > > > > >>>>>>>>> produce
> > > > > >>>>>>>>>>>>>> requests.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > > > >>>>>>> ivan@ivanyu.me>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Hi all,
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> We have also thought in a bit more details about
> > > > > >>>>>>> transactions
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>>>>> queues,
> > > > > >>>>>>>>>>>>>>> here's the plan.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> *Transactions*
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> The support for transactions in *classic topics* is
> > > > > >>>> based
> > > > > >>>>>>> on
> > > > > >>>>>>>>> precise
> > > > > >>>>>>>>>>>>>>> interactions between three actors: clients (mostly
> > > > > >>>>>>> producers,
> > > > > >>>>>>>>> but
> > > > > >>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > > >>>> classes),
> > > > > >>>>>> and
> > > > > >>>>>>>>>>>> transaction
> > > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders
> with
> > > > > >>>>>> their
> > > > > >>>>>>>>> local
> > > > > >>>>>>>>>>>> state
> > > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> The high level (some details skipped) workflow is
> the
> > > > > >>>>>>>>> following.
> > > > > >>>>>>>>>>>> When a
> > > > > >>>>>>>>>>>>>>> transactional Produce request is received by the
> > > > > >>>> broker:
> > > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader checks
> > > > > >>>> if a
> > > > > >>>>>>>>> non-empty
> > > > > >>>>>>>>>>>>>>> transaction is running for this partition. This is
> > > > > >>>> done
> > > > > >>>>>>> using
> > > > > >>>>>>>>> its
> > > > > >>>>>>>>>>>> local
> > > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > > >>>>>> (ProducerStateManager,
> > > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed about
> all
> > > > > >>>> the
> > > > > >>>>>>>>> partitions
> > > > > >>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> > > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional
> transactional
> > > > > >>>>>>> checks.
> > > > > >>>>>>>>>>>>>>> 4. The partition leaders append the transactional
> > > > > >>>> data to
> > > > > >>>>>>>>> their logs
> > > > > >>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>> update some of their state (for example, log the
> fact
> > > > > >>>>>> that
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>>>> transaction
> > > > > >>>>>>>>>>>>>>> is running for the partition and its first offset).
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> coordinator
> > > > > >>>>>>> directly
> > > > > >>>>>>>>> with
> > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> PREPARE_COMMIT
> > > > > >>>> or
> > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>>>> leaders
> > > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > > >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> > > > > >>>> markers to
> > > > > >>>>>>>>> their logs
> > > > > >>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction
> state
> > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
> > > > > >>>> of
> > > > > >>>>>>>>> important
> > > > > >>>>>>>>>>>> state
> > > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is local.
> The
> > > > > >>>> main
> > > > > >>>>>>>>> challenge
> > > > > >>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
> > > > > >>>> are no
> > > > > >>>>>>>>> partition
> > > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
> > > > > >>>> to be
> > > > > >>>>>>>>> globalized
> > > > > >>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> > > > > >>>> support
> > > > > >>>>>>>>> idempotent
> > > > > >>>>>>>>>>>>>> produce.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics* would
> > > > > >>>> look
> > > > > >>>>>>> very
> > > > > >>>>>>>>>>>> similar:
> > > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > > >>>> non-empty
> > > > > >>>>>>>>> transaction
> > > > > >>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>> running for this partition. In contrast to classic
> > > > > >>>>>> topics,
> > > > > >>>>>>>>> this is
> > > > > >>>>>>>>>>>>>> checked
> > > > > >>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
> > > > > >>>> Since a
> > > > > >>>>>>>>> transaction
> > > > > >>>>>>>>>>>>>> could
> > > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
> > > > > >>>> the
> > > > > >>>>>>> positive
> > > > > >>>>>>>>>>>> result of
> > > > > >>>>>>>>>>>>>>> this check could be cached locally (for the double
> > > > > >>>>>>> configured
> > > > > >>>>>>>>>>>> duration
> > > > > >>>>>>>>>>>>>> of a
> > > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is
> informed
> > > > > >>>>>> about
> > > > > >>>>>>> all
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
> > > > > >>>> include
> > > > > >>>>>>> them.
> > > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
> > > > > >>>> side.
> > > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data to the
> > > > > >>>>>> current
> > > > > >>>>>>>>> shared
> > > > > >>>>>>>>>>>> WAL
> > > > > >>>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
> > > > > >>>> state
> > > > > >>>>>>> for
> > > > > >>>>>>>>> Diskless
> > > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > > >>>> coordinator
> > > > > >>>>>>> like
> > > > > >>>>>>>>> in the
> > > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final
> transactional
> > > > > >>>>>>> checks
> > > > > >>>>>>>>> of the
> > > > > >>>>>>>>>>>>>>> batches. This procedure would output the same
> errors
> > > > > >>>> like
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>> partition
> > > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some
> batches
> > > > > >>>>>> could
> > > > > >>>>>>> be
> > > > > >>>>>>>>>>>> rejected.
> > > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage in
> the
> > > > > >>>> WAL
> > > > > >>>>>>>>> segment
> > > > > >>>>>>>>>>>> file in
> > > > > >>>>>>>>>>>>>>> case of transactional errors. This is preferable to
> > > > > >>>> doing
> > > > > >>>>>>> more
> > > > > >>>>>>>>>>>> network
> > > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL
> segments
> > > > > >>>> will
> > > > > >>>>>>> be
> > > > > >>>>>>>>>>>> relatively
> > > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction
> coordinator
> > > > > >>>>>>> directly
> > > > > >>>>>>>>> with
> > > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes
> PREPARE_COMMIT
> > > > > >>>> or
> > > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
> > > > > >>>> batch
> > > > > >>>>>>>>> coordinator
> > > > > >>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> > > > > >>>>>>> transaction is
> > > > > >>>>>>>>>>>> finished
> > > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > > >>>> corresponding
> > > > > >>>>>>> logs
> > > > > >>>>>>>>> of the
> > > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only on the
> > > > > >>>>>> metadata
> > > > > >>>>>>>>> level, no
> > > > > >>>>>>>>>>>>>>> actual control batches are written to any file.
> They
> > > > > >>>> will
> > > > > >>>>>>> be
> > > > > >>>>>>>>>>>> dynamically
> > > > > >>>>>>>>>>>>>>> created on Fetch and other read operations. We
> could
> > > > > >>>>>>>>> technically
> > > > > >>>>>>>>>>>> write
> > > > > >>>>>>>>>>>>>>> these control batches for real, but this would mean
> > > > > >>>> extra
> > > > > >>>>>>>>> produce
> > > > > >>>>>>>>>>>>>> latency,
> > > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > > >>>> coordinator
> > > > > >>>>>>> and
> > > > > >>>>>>>>> save
> > > > > >>>>>>>>>>>> these
> > > > > >>>>>>>>>>>>>>> milliseconds.
> > > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > > >>>>>>> WriteTxnMarkersRequest to
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>>>> leaders
> > > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
> > > > > >>>> topics
> > > > > >>>>>>> now.
> > > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics write
> the
> > > > > >>>>>>>>> transaction
> > > > > >>>>>>>>>>>> markers
> > > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction
> state
> > > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
> > > > > >>>> get:
> > > > > >>>>>>>>>>>>>>> 1. An extra network round trip between brokers and
> > > > > >>>> the
> > > > > >>>>>>> batch
> > > > > >>>>>>>>>>>> coordinator
> > > > > >>>>>>>>>>>>>>> when a new partition appear in the transaction. To
> > > > > >>>>>>> mitigate the
> > > > > >>>>>>>>>>>> impact of
> > > > > >>>>>>>>>>>>>>> them:
> > > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one Produce
> > > > > >>>>>>> request
> > > > > >>>>>>>>> will be
> > > > > >>>>>>>>>>>>>>> grouped.
> > > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> > > > > >>>> fast
> > > > > >>>>>>>>> response to
> > > > > >>>>>>>>>>>> these
> > > > > >>>>>>>>>>>>>>> RPCs.
> > > > > >>>>>>>>>>>>>>> - The fact that a single producer normally will
> > > > > >>>>>>> communicate
> > > > > >>>>>>>>> with a
> > > > > >>>>>>>>>>>>>>> single broker for the duration of the transaction
> > > > > >>>> further
> > > > > >>>>>>>>> reduces the
> > > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > > >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> > > > > >>>>>> coordinator
> > > > > >>>>>>> and
> > > > > >>>>>>>>> batch
> > > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be able
> to
> > > > > >>>>>> span
> > > > > >>>>>>> both
> > > > > >>>>>>>>>>>> classic
> > > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> *Queues*
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> The share group coordination and management is a
> > > > > >>>> side job
> > > > > >>>>>>> that
> > > > > >>>>>>>>>>>> doesn't
> > > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > > >>>> replicas,
> > > > > >>>>>>> physical
> > > > > >>>>>>>>>>>> storage
> > > > > >>>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and
> consumers
> > > > > >>>>>>> (Fetch and
> > > > > >>>>>>>>>>>> Produce
> > > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not
> affected.)
> > > > > >>>> We
> > > > > >>>>>>> don't
> > > > > >>>>>>>>> see any
> > > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
> > > > > >>>> with
> > > > > >>>>>>> share
> > > > > >>>>>>>>>>>> groups the
> > > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the code
> > > > > >>>> level,
> > > > > >>>>>> we
> > > > > >>>>>>>>> don't
> > > > > >>>>>>>>>>>> expect
> > > > > >>>>>>>>>>>>>> any
> > > > > >>>>>>>>>>>>>>> serious refactoring: the same reading routines are
> > > > > >>>> used
> > > > > >>>>>>> that
> > > > > >>>>>>>>> are
> > > > > >>>>>>>>>>>> used for
> > > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
> > > > > >>>> too
> > > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Best regards,
> > > > > >>>>>>>>>>>>>>> Ivan
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > >>>>>>>>>>>>>>>> Hi all,
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design input
> > > > > >>>> on
> > > > > >>>>>>>>> KIP-1150.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
> > > > > >>>> new
> > > > > >>>>>>>>> design. To
> > > > > >>>>>>>>>>>>>>> summarize
> > > > > >>>>>>>>>>>>>>>> the changes:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> > > > > >>>> existing
> > > > > >>>>>>>>> KIP-405
> > > > > >>>>>>>>>>>> Tiered
> > > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
> > > > > >>>>>>> Diskless
> > > > > >>>>>>>>> topic
> > > > > >>>>>>>>>>>> to be
> > > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for the
> > > > > >>>> Batch
> > > > > >>>>>>>>> Coordinator
> > > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> > > > > >>>> Tiered
> > > > > >>>>>>> Storage
> > > > > >>>>>>>>>>>> plugin
> > > > > >>>>>>>>>>>>>>>> features such as encryption and alternative data
> > > > > >>>>>> formats.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> > > > > >>>> segments,
> > > > > >>>>>>>>> making use
> > > > > >>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > > >>>> zero-copy
> > > > > >>>>>>>>> functionality
> > > > > >>>>>>>>>>>>>>> already
> > > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > > >>>>>>>>>>>>>>>> However, local segments are now considered cache
> > > > > >>>>>>> elements,
> > > > > >>>>>>>>> do not
> > > > > >>>>>>>>>>>> need
> > > > > >>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > > >>>> contacting
> > > > > >>>>>>> any
> > > > > >>>>>>>>> other
> > > > > >>>>>>>>>>>>>>> replicas.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> 3. The design has been simplified substantially,
> by
> > > > > >>>>>>> removing
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>>>> previous
> > > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > > >>>> component, and
> > > > > >>>>>>>>> "object
> > > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > > >>>> enabled by
> > > > > >>>>>>> the
> > > > > >>>>>>>>> Batch
> > > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
> > > > > >>>>>> earlier
> > > > > >>>>>>>>> design,
> > > > > >>>>>>>>>>>> while
> > > > > >>>>>>>>>>>>>>>> being simpler and integrating better into the
> > > > > >>>> existing
> > > > > >>>>>>>>> ecosystem.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
> > > > > >>>> the
> > > > > >>>>>> new
> > > > > >>>>>>>>> design.
> > > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > > >>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> > > > > >>>> of
> > > > > >>>>>> all
> > > > > >>>>>>>>> existing
> > > > > >>>>>>>>>>>>>> client
> > > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > > >>>> supporting
> > > > > >>>>>> APIs
> > > > > >>>>>>>>> like
> > > > > >>>>>>>>>>>>>>> transactions
> > > > > >>>>>>>>>>>>>>>>> and queues.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> > > > > >>>>>>> status of
> > > > > >>>>>>>>> this
> > > > > >>>>>>>>>>>> KIP?
> > > > > >>>>>>>>>>>>>> I
> > > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> > > > > >>>>>>> months,
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>> vote got
> > > > > >>>>>>>>>>>>>>>>> vetoed
> > > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> > > > > >>>>>> answered
> > > > > >>>>>>>>> since
> > > > > >>>>>>>>>>>> then.
> > > > > >>>>>>>>>>>>>>> KIP-1183
> > > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > > >>>> activity
> > > > > >>>>>>> since
> > > > > >>>>>>>>> May.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> > > > > >>>>>> choices
> > > > > >>>>>>>>> that can
> > > > > >>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> > > > > >>>> by
> > > > > >>>>>> far
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>> dominant
> > > > > >>>>>>>>>>>>>> one
> > > > > >>>>>>>>>>>>>>>>> when
> > > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
> > > > > >>>>>>>>> standalone
> > > > > >>>>>>>>>>>>>> products.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> > > > > >>>> hate to
> > > > > >>>>>>> see a
> > > > > >>>>>>>>>>>> stall. So
> > > > > >>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> > > > > >>>>>> McCabe -
> > > > > >>>>>>>>> have your
> > > > > >>>>>>>>>>>>>>> questions
> > > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> > > > > >>>> do
> > > > > >>>>>> to
> > > > > >>>>>>>>> help? I'm
> > > > > >>>>>>>>>>>>>> deeply
> > > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > > >>>> written
> > > > > >>>>>>> about
> > > > > >>>>>>>>> both?
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > > >>>>>>>>>>>>>>>>>> Jan
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > > >>>>>> Kozlovski
> > > > > >>>>>>> <
> > > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > > >>>> main
> > > > > >>>>>> one
> > > > > >>>>>>>>> (just a
> > > > > >>>>>>>>>>>> bunch
> > > > > >>>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>>>> links
> > > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > > >>>> example, I
> > > > > >>>>>>> was
> > > > > >>>>>>>>>>>> searching
> > > > > >>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> > > > > >>>> far
> > > > > >>>>>> as
> > > > > >>>>>>> I
> > > > > >>>>>>>>> can
> > > > > >>>>>>>>>>>> tell, it
> > > > > >>>>>>>>>>>>>>>>> doesn't
> > > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> > > > > >>>> having
> > > > > >>>>>>> the
> > > > > >>>>>>>>> root
> > > > > >>>>>>>>>>>> KIP be
> > > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> > > > > >>>> to
> > > > > >>>>>>>>> sub-KIPs
> > > > > >>>>>>>>>>>> where
> > > > > >>>>>>>>>>>>>> were
> > > > > >>>>>>>>>>>>>>>>> aimed
> > > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> > > > > >>>> would
> > > > > >>>>>>> be
> > > > > >>>>>>>>> good to
> > > > > >>>>>>>>>>>>>> create
> > > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> > > > > >>>>>>>>> introducing a
> > > > > >>>>>>>>>>>> new
> > > > > >>>>>>>>>>>>>>> type of
> > > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > > >>>> Storage as
> > > > > >>>>>>> the
> > > > > >>>>>>>>> primary
> > > > > >>>>>>>>>>>>>>> source of
> > > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > > >>>> decided
> > > > > >>>>>>> to
> > > > > >>>>>>>>> split it
> > > > > >>>>>>>>>>>>>> into
> > > > > >>>>>>>>>>>>>>>>>> multiple
> > > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > >>>>>>>>>>>>>>>>>>> )
> > > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > > >>>> should aim
> > > > > >>>>>>> to
> > > > > >>>>>>>>> have
> > > > > >>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>> type of
> > > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > > >>>>>> details
> > > > > >>>>>>> on
> > > > > >>>>>>>>> how to
> > > > > >>>>>>>>>>>>>>> implement
> > > > > >>>>>>>>>>>>>>>>>> it.
> > > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
> > > > > >>>>>>> discussed
> > > > > >>>>>>>>>>>> KRaft.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> > > > > >>>> hard
> > > > > >>>>>>> to
> > > > > >>>>>>>>> discuss
> > > > > >>>>>>>>>>>> on
> > > > > >>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>> meta
> > > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > > >>>> (linked
> > > > > >>>>>>> in
> > > > > >>>>>>>>>>>> KIP-1150)
> > > > > >>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>> offer
> > > > > >>>>>>>>>>>>>>>>>>> an
> > > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> > > > > >>>>>>> threads
> > > > > >>>>>>>>> for
> > > > > >>>>>>>>>>>> each
> > > > > >>>>>>>>>>>>>>> type of
> > > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > > >>>> whether
> > > > > >>>>>>> Apache
> > > > > >>>>>>>>> Kafka
> > > > > >>>>>>>>>>>> wants
> > > > > >>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> have
> > > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > > >>>> authors
> > > > > >>>>>> of
> > > > > >>>>>>>>> this KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> > > > > >>>>>>> aiven.io
> > > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > > >>>>>>> Valtonen,
> > > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Anatolii Popov
> Senior Software Developer, *Aiven OY*
> m: +358505126242
> w: aiven.io  e: anatolii.popov@aiven.io
> <https://www.facebook.com/aivencloud>
> <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Anatolii Popov via dev <de...@kafka.apache.org>]

Hi all,

Thank you for the extensive feedback.

We have substantially updated the KIP to address the points raised. Given
the scope of these changes, we have started a new VOTE thread to restart
the voting process cleanly.

You can find the new thread here:
https://lists.apache.org/thread/m42nj2qm5z4w2x8kt7x4kgghfzrdwl7q

Best,
Anatolii

On Mon, Dec 15, 2025 at 9:47 PM Thomas Thornton via dev <
dev@kafka.apache.org> wrote:

> Hi all,
>
> We (the team at Slack) have been following the recent discussion regarding
> the scope and timeline of KIP-1150. We agree with the community sentiment
> that Diskless Topics represents the right long-term architecture for Kafka
> in the cloud, but we also recognize the valid concerns raised regarding the
> engineering resources required to deliver such an ambitious change.
>
> To help address these concerns and accelerate the timeline, we are happy to
> announce that we are partnering with the KIP-1150 authors to co-develop
> this feature.
>
> We previously proposed KIP-1176: Tiered Storage for Active Log Segments to
> solve similar problems. However, rather than fragmenting the community's
> efforts across competing designs, we have decided to withdraw KIP-1176 and
> consolidate our engineering resources behind KIP-1150.
>
> To start, we plan to take ownership of Compaction for Tiered Storage. This
> has long been a missing feature in KIP-405, and it becomes critical in a
> Diskless architecture where long-lived data is obligated to be tiered. By
> driving this related prerequisite feature, we hope to allow for faster
> delivery of KIP-1150.
>
> We are excited to collaborate on this to ensure a robust and timely
> delivery for the community.
>
> Best,
> Tom & Henry
>
> On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Greg,
> >
> > Thanks for sharing the meeting notes.
> > I agree we should keep polishing the contents of 1150 & high level design
> > in 1163 to prepare for a vote.
> >
> >
> > Thanks.
> > Luke
> >
> > On Fri, Nov 14, 2025 at 3:54 AM Greg Harris <greg.harris@aiven.io.invalid
> >
> > wrote:
> >
> > > Hi all,
> > >
> > > There was a video call between myself, Ivan Yurchenko, Jun Rao, and
> > Andrew
> > > Schofield pertaining to KIP-1150. Here are the notes from that meeting:
> > >
> > > Ivan: What is the future state of Kafka in this area, in 5 years?
> > > Jun: Do we want something more cloud native? Yes, started with Tiered
> > > Storage. If there’s a better way, we should explore it. In the long
> term
> > > this will be useful
> > > Because Kafka is used so widely, we need to make sure everything we add
> > is
> > > for the long term and for everyone, not just for a single company.
> > > When we add TS, it doesn’t just solve Uber’s use-case. We want
> something
> > > that’s high quality/lasts/maintainable, and can work with all existing
> > > capabilities.
> > > If both 1150 and 1176 proceed at the same time, it’s confusing. They
> > > overlap, but Diskless is more ambitious.
> > > If both KIPs are being seriously worked on, then we don’t really need
> > both,
> > > because Diskless clearly is better. Having multiple will confuse
> people.
> > It
> > > will duplicate some of the effort.
> > > If we want diskless ultimately, what is the short term strategy, to get
> > > some early wins first?
> > > Ivan: Andrew, do you want a more revolutionary approach?
> > > Andrew: Eventually the architecture will change substantially, it may
> not
> > > be necessary to put all of that bill onto Diskless at once.
> > > Greg: We all agree on having a high quality feature merged upstream,
> and
> > > supporting all APIs
> > > Jun: We should try and keep things simple, but there is some minimum
> > > complexity needed.
> > > When doing the short term changes (1176), it doesn’t really progress in
> > > changing to a more modern architecture.
> > > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > > Jun: The danger of only applying changes to some part of the API, you
> set
> > > the precedence that you only have to implement part of the API.
> > Supporting
> > > the full API set should be a minimum requirement.
> > > Andrew: When we started Kraft, how much did we know the design?
> > > Jun: For Kraft we didn’t really know much about the migration, but the
> > > high-level was clear.
> > > Greg: Is 1150 votable in its current state?
> > > Jun: 1150 should promise to support all APIs. It doesn’t have to have
> all
> > > the details/apis/etc. KIP-500 didn’t have it.
> > > We do need some high-level design enough to give confidence that the
> > > promise is able to be fulfilled.
> > > Greg: Is the draft version in 1163 enough detail or is more needed?
> > > Jun: We need to agree on the core design, such as leaderless etc. And
> how
> > > the APIs will be supported.
> > > Greg: Okay we can include these things, and provide a sketch of how the
> > > other leader-based features operate.
> > > Jun: Yeah if at a high level the sketch appears to work, we can approve
> > > that functionality.
> > > Are you committed to doing the more involved and big project?
> > > Greg: Yes, we’re committed to the 1163 design and can’t really accept
> > 1176.
> > > Jun: TS was slow because of Uber resourcing problems
> > > Greg: We’ll push internally for resources, and use the community
> > sentiment
> > > to motivate Aiven.
> > > How far into the future should we look? What sort of scale?
> > > Jun: As long as there’s a path forward, and we’re not closing off
> future
> > > improvements, we can figure out how to handle a larger scale when it
> > > arises.
> > > Greg: Random replica placement is very harmful, can we recommend users
> to
> > > use an external tool like CruiseControl?
> > > Jun: Not everyone uses CruiseControl, we would probably need some
> > solution
> > > for this out of the box
> > > Ivan: Should the Batch Coordinator be pluggable?
> > > Jun: Out-of-box experience should be good, good to allow other
> > > implementations
> > > Greg: But it could hurt Kafka feature/upgrade velocity when we wait for
> > > plugin providers to implement it
> > > Ivan: We imagined that maybe cloud hyperscalers could implement it with
> > > e.g. dynamodb
> > > Greg: Could we bake more details of the different providers into Kafka,
> > or
> > > does it still make sense for it to be pluggable?
> > > Jun: Make it whatever is easiest to roll out and add new clients
> > > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > > Ivan: The vote is already open, we’re not too pressed for time. We’ll
> go
> > > improve the 1163 design and communication.
> > > Is 1176 a competing design? Someone will ask.
> > > Jun: If we are seriously working on something more ambitious, yeah we
> > > shouldn’t do the stop-gap solution.
> > > It’s diverting review resources. If we can get the short term thing in
> > 1yr
> > > but Diskless solution is 2y it makes sense to go for Diskless. If it’s
> > 5yr,
> > > that’s different and maybe the stop-gap solution is needed.
> > > Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should we
> > > explicitly exclude 1176?
> > > Andrew: Put your arms around the feature set you actually want, and use
> > > that to rule out 1176.
> > > Probably don’t need -1 votes, most likely KIPs just don’t receive
> votes.
> > > Ivan: Should we have sync meetings like tiered storage did?
> > > Jun: Satish posted meeting notes regularly, we should do the same.
> > >
> > > To summarize, we will be polishing the contents of 1150 & high level
> > design
> > > in 1163 to prepare for a vote.
> > > We believe that the community should select the feature set of 1150 to
> > > fully eliminate producer cross-zone costs, and make the investment in a
> > > high quality Diskless Topics implementation rather than in stop-gap
> > > solutions.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:
> > >
> > > > This may be a tangent, but we needed to offload storage off of Kafka
> > into
> > > > S3. We are keeping Kafka not as a source of truth, but as a mostly
> > > > ephemeral broker that can come and go as it pleases. Be that scaling
> or
> > > > outage. Disks can be destroyed and recreated at will, we still retain
> > > data
> > > > and use broker for just that, brokering messages. Not only that, we
> > > reduced
> > > > the requirement on the actual Kafka resources by reducing the size
> of a
> > > > payload via a claim check pattern. Maybe this is an anti–pattern, but
> > it
> > > is
> > > > super fast and highly cost efficient. We reworked ProducerRequest to
> > > allow
> > > > plugins. We added a custom http plugin that submits every request
> via a
> > > > persisted connection to a microservice. Microservice stores the
> payload
> > > and
> > > > returns a tiny json metadata object,a claim check, that can be used
> to
> > > find
> > > > the actual data. Think of it as zipping the payload. This claim check
> > > > metadata traverses the pipelines with consumers using the urls in
> > > metadata
> > > > to pull what they need. Think unzipping. This allowed us to also pull
> > > ONLY
> > > > the data that we need in graphql like manner. So if you have a 100K
> > json
> > > > payload and you need only a subsection, you can pull that by
> jmespath.
> > > When
> > > > you have multiple consumer groups yanking down huge payloads it is
> > > > cumbersome on the broker. When you have the same consumer groups
> > yanking
> > > > down a claim check, and then going out of band directly to the source
> > of
> > > > truth, the broker has some breathing room. Obviously our microservice
> > > does
> > > > not go directly to the cloud storage, as that would be too slow. It
> > > stores
> > > > the payload in high speed memory cache and returns asap. That memory
> is
> > > > eventually persisted into S3. The retrieval goest against the cache
> > > first,
> > > > then against the S3. Overall a rather cheappy and zippy solution. I
> > tried
> > > > proposing the KIP for this, but there was no excitement. Check this
> > out:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > > >
> > > >
> > > > > On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID>
> > wrote:
> > > > >
> > > > > Hi, Andrew,
> > > > >
> > > > > If we want to focus only on reducing cross-zone replication costs,
> > > there
> > > > is
> > > > > an alternative design in the KIP-1176 discussion thread that seems
> > > > simpler
> > > > > than the proposal here. I am copying the outline of that approach
> > > below.
> > > > >
> > > > > 1. A new leader is elected.
> > > > > 2. Leader maintains a first tiered offset, which is initialized to
> > log
> > > > end
> > > > > offset.
> > > > > 3. Leader writes produced data from the client to local log.
> > > > > 4. Leader uploads produced data from all local logs as a combined
> > > object
> > > > > 5. Leader stores the metadata for the combined object in memory.
> > > > > 6. If a follower fetch request has an offset >= first tiered
> offset,
> > > the
> > > > > metadata for the corresponding combined object is returned.
> > Otherwise,
> > > > the
> > > > > local data is returned.
> > > > > 7. Leader periodically advances first tiered offset.
> > > > >
> > > > > It's still a bit unnatural, but it could work.
> > > > >
> > > > > Hi, Ivan,
> > > > >
> > > > > Are you still committed to proceeding with the original design of
> > > > KIP-1150?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > > andrew_schofield@live.com>
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >> I’ve been following KIP-1150 and friends for a while. I’m going to
> > > jump
> > > > >> into the discussions too.
> > > > >>
> > > > >> Looking back at Jack Vanlightly’s message, I am not quite so
> > convinced
> > > > >> that it’s a kind of fork in the road. The primary aim of the
> effort
> > is
> > > > to
> > > > >> reduce cross-zone replication costs so Apache Kafka is not
> > > prohibitively
> > > > >> expensive to use on cloud storage. I think it would be entirely
> > viable
> > > > to
> > > > >> prioritise code reuse for an initial implementation of diskless
> > > topics,
> > > > and
> > > > >> we could still have a more cloud-native design in the future. It’s
> > > hard
> > > > to
> > > > >> predict what the community will prioritise in the future.
> > > > >>
> > > > >> Of the three major revisions, I’m in the rev3 camp. We can support
> > > > >> leaderless produce requests, first writing WAL segments into
> object
> > > > >> storage, and then using the regular partition leaders to sequence
> > the
> > > > >> records. The active log segment for a diskless topic will
> initially
> > > > contain
> > > > >> batch coordinates rather than record batches. The batch
> coordinates
> > > can
> > > > be
> > > > >> resolved from WAL segments for consumers, and also in order to
> > prepare
> > > > log
> > > > >> segments for uploading to tiered storage. Jun is probably correct
> > that
> > > > we
> > > > >> need a more frequent object merging process than tiered storage
> > > > provides.
> > > > >> This is just the transition from write-optimised WAL segments to
> > > > >> read-optimised tiered segments, and all of the object
> storage-based
> > > > >> implementations of Kafka that I’m aware of do this rearrangement.
> > But
> > > > >> perhaps this more frequent object merging is a pre-GA improvement,
> > > > rather
> > > > >> than a strict requirement for an initial implementation for early
> > > access
> > > > >> use.
> > > > >>
> > > > >> For zone-aligned share consumers, the share group assignor is
> > intended
> > > > to
> > > > >> be rack-aware. Consumers should be assigned to partitions with
> > leaders
> > > > in
> > > > >> their zone. The simple assignor is not rack-aware, but it easily
> > could
> > > > be
> > > > >> or we could have a rack-aware assignor.
> > > > >>
> > > > >> Thanks,
> > > > >> Andrew
> > > > >>
> > > > >>
> > > > >>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID>
> > wrote:
> > > > >>>
> > > > >>> Hi, Ivan,
> > > > >>>
> > > > >>> Thanks for the reply.
> > > > >>>
> > > > >>> "As I understand, you’re speaking about locally materialized
> > > segments.
> > > > >> They
> > > > >>> will indeed consume some IOPS. See them as a cache that could
> > always
> > > be
> > > > >>> restored from the remote storage. While it’s not ideal, it's
> still
> > OK
> > > > to
> > > > >>> lose data in them due to a machine crash, for example. Because of
> > > this,
> > > > >> we
> > > > >>> can avoid explicit flushing on local materialized segments at all
> > and
> > > > let
> > > > >>> the file system and page cache figure out when to flush
> optimally.
> > > This
> > > > >>> would not eliminate the extra IOPS, but should reduce it
> > > dramatically,
> > > > >>> depending on throughput for each partition. We, of course,
> continue
> > > > >>> flushing the metadata segments as before."
> > > > >>>
> > > > >>> If we have a mix of classic and diskless topics on the same
> broker,
> > > > it's
> > > > >>> important that the classic topics' data is flushed to disk as
> > quickly
> > > > as
> > > > >>> possible. To achieve this, users typically set
> > dirty_expire_centisecs
> > > > in
> > > > >>> the kernel based on the number of available disk IOPS. Once you
> set
> > > > this
> > > > >>> number, it applies to all dirty files, including the cached data
> in
> > > > >>> diskless topics. So, if there are more files actively
> accumulating
> > > > data,
> > > > >>> the flush frequency and therefore RPO is reduced for classic
> > topics.
> > > > >>>
> > > > >>> "We should have mentioned this explicitly, but this step, in
> fact,
> > > > >> remains
> > > > >>> in the form of segments offloading to tiered storage. When we
> > > assemble
> > > > a
> > > > >>> segment and hand it over to RemoteLogManager, we’re effectively
> > doing
> > > > >>> metadata compaction: replacing a big number of pieces of metadata
> > > about
> > > > >>> individual batches with a single record in
> __remote_log_metadata."
> > > > >>>
> > > > >>> The object merging in tier storage typically only kicks in after
> a
> > > few
> > > > >>> hours. The impact is (1) the amount of accumulated metadata is
> > still
> > > > >> quite
> > > > >>> large; (2) there are many small objects, leading to poor read
> > > > >> performance.
> > > > >>> I think we need a more frequent object merging process than tier
> > > > storage
> > > > >>> provides.
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me>
> > > > wrote:
> > > > >>>
> > > > >>>> Hello Jack, Jun, Luke, and all!
> > > > >>>>
> > > > >>>> Thank you for your messages.
> > > > >>>>
> > > > >>>> Let me first address some of Jun’s comments.
> > > > >>>>
> > > > >>>>> First, it degrades the durability.
> > > > >>>>> For each partition, now there are two files being actively
> > written
> > > > at a
> > > > >>>>> given point of time, one for the data and another for the
> > metadata.
> > > > >>>>> Flushing each file requires a separate IO. If the disk has 1K
> > IOPS
> > > > and
> > > > >> we
> > > > >>>>> have 5K partitions in a broker, currently we can afford to
> flush
> > > each
> > > > >>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we
> > > > double
> > > > >>>> the
> > > > >>>>> number of files per partition, we can only flush each partition
> > > every
> > > > >> 10
> > > > >>>>> seconds, which makes RPO twice as bad.
> > > > >>>>
> > > > >>>> As I understand, you’re speaking about locally materialized
> > > segments.
> > > > >> They
> > > > >>>> will indeed consume some IOPS. See them as a cache that could
> > always
> > > > be
> > > > >>>> restored from the remote storage. While it’s not ideal, it's
> still
> > > OK
> > > > to
> > > > >>>> lose data in them due to a machine crash, for example. Because
> of
> > > > this,
> > > > >> we
> > > > >>>> can avoid explicit flushing on local materialized segments at
> all
> > > and
> > > > >> let
> > > > >>>> the file system and page cache figure out when to flush
> optimally.
> > > > This
> > > > >>>> would not eliminate the extra IOPS, but should reduce it
> > > dramatically,
> > > > >>>> depending on throughput for each partition. We, of course,
> > continue
> > > > >>>> flushing the metadata segments as before.
> > > > >>>>
> > > > >>>> It’s worth making a note on caching. I think nobody will
> disagree
> > > that
> > > > >>>> doing direct reads from remote storage every time a batch is
> > > requested
> > > > >> by a
> > > > >>>> consumer will not be practical neither from the performance nor
> > from
> > > > the
> > > > >>>> economy point of view. We need a way to keep the number of GET
> > > > requests
> > > > >>>> down. There are multiple options, for example:
> > > > >>>> 1. Rack-aware distributed in-memory caching.
> > > > >>>> 2. Local in-memory caching. Comes with less network chattiness
> and
> > > > >> works
> > > > >>>> well if we have more or less stable brokers to consume from.
> > > > >>>> 3. Materialization of diskless logs on local disk. Way lower
> > impact
> > > on
> > > > >>>> RAM and also requires stable brokers for consumption (using just
> > > > >> assigned
> > > > >>>> replicas will probably work well).
> > > > >>>>
> > > > >>>> Materialization is one of possible options, but we can choose
> > > another
> > > > >> one.
> > > > >>>> However, we will have this dilemma regardless of whether we have
> > an
> > > > >>>> explicit coordinator or we go “coordinator-less”.
> > > > >>>>
> > > > >>>>> Second, if we ever need this
> > > > >>>>> metadata somewhere else, say in the WAL file manager, the
> > consumer
> > > > >> needs
> > > > >>>> to
> > > > >>>>> subscribe to every partition in the cluster, which is
> > inefficient.
> > > > The
> > > > >>>>> actual benefit of this approach is also questionable. On the
> > > surface,
> > > > >> it
> > > > >>>>> might seem that we could reduce the number of lines that need
> to
> > be
> > > > >>>> changed
> > > > >>>>> for this KIP. However, the changes are quite intrusive to the
> > > classic
> > > > >>>>> partition's code path and will probably make the code base
> harder
> > > to
> > > > >>>>> maintain in the long run. I like the original approach based on
> > the
> > > > >> batch
> > > > >>>>> coordinator much better than this one. We could probably
> refactor
> > > the
> > > > >>>>> producer state code so that it could be reused in the batch
> > > > >> coordinator.
> > > > >>>>
> > > > >>>> It’s hard to disagree with this. The explicit coordinator is
> more
> > a
> > > > side
> > > > >>>> thing, while coordinator-less approach is more about extending
> > > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > > >>>>
> > > > >>>>> Thanks for addressing the concerns on the number of RPCs in the
> > > > produce
> > > > >>>>> path. I agree that with the metadata crafting mechanism, we
> could
> > > > >>>> mitigate
> > > > >>>>> the PRC problem. However, since we now require the metadata to
> be
> > > > >>>>> collocated with the data on the same set of brokers, it's weird
> > > that
> > > > >> they
> > > > >>>>> are now managed by different mechanisms. The data assignment
> now
> > > uses
> > > > >> the
> > > > >>>>> metadata crafting mechanism, but the metadata is stored in the
> > > > classic
> > > > >>>>> partition using its own assignment strategy. It will be
> > complicated
> > > > to
> > > > >>>> keep
> > > > >>>>> them collocated.
> > > > >>>>
> > > > >>>> I would like to note that the metadata crafting is needed only
> to
> > > tell
> > > > >>>> producers which brokers they should send Produce requests to,
> but
> > > data
> > > > >> (as
> > > > >>>> in “locally materialized log”) is located on partition replicas,
> > > i.e.
> > > > >>>> automatically co-located with metadata.
> > > > >>>>
> > > > >>>> As a side note, it would probably be better that instead of
> > > implicitly
> > > > >>>> crafting partition metadata, we extend the metadata protocol so
> > that
> > > > for
> > > > >>>> diskless partitions we return not only the leader and replicas,
> > but
> > > > also
> > > > >>>> some “recommended produce brokers”, selected for optimal
> > performance
> > > > and
> > > > >>>> costs. Producers will pick ones in their racks.
> > > > >>>>
> > > > >>>>> I am also concerned about the removal of the object
> > > > compaction/merging
> > > > >>>>> step.
> > > > >>>>
> > > > >>>> We should have mentioned this explicitly, but this step, in
> fact,
> > > > >> remains
> > > > >>>> in the form of segments offloading to tiered storage. When we
> > > > assemble a
> > > > >>>> segment and hand it over to RemoteLogManager, we’re effectively
> > > doing
> > > > >>>> metadata compaction: replacing a big number of pieces of
> metadata
> > > > about
> > > > >>>> individual batches with a single record in
> __remote_log_metadata.
> > > > >>>>
> > > > >>>> We could create a Diskless-specific merging mechanism instead if
> > > > needed.
> > > > >>>> It’s rather easy with the explicit coordinator approach. With
> the
> > > > >>>> coordinator-less approach, this would probably be a bit more
> > tricky
> > > > >>>> (rewriting the tail of the log by the leader + replicating this
> > > change
> > > > >>>> reliably).
> > > > >>>>
> > > > >>>>> I see a tendency toward primarily optimizing for the fewest
> code
> > > > >> changes
> > > > >>>> in
> > > > >>>>> the KIP. Instead, our primary goal should be a clean design
> that
> > > can
> > > > >> last
> > > > >>>>> for the long term.
> > > > >>>>
> > > > >>>> Yes, totally agree.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Luke,
> > > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > > because
> > > > >> of
> > > > >>>>> leaderless cluster, do you think it will be simpler if we only
> > > focus
> > > > on
> > > > >>>> the
> > > > >>>>> "diskless" design to handle object compaction/merging to/from
> the
> > > > >> remote
> > > > >>>>> storage to save the cross-AZ cost?
> > > > >>>>
> > > > >>>> After some evolution of the original proposal, leaderless is now
> > > > >> limited.
> > > > >>>> We only need to be able to accept Produce requests on more than
> > one
> > > > >> broker
> > > > >>>> to eliminate the cross-AZ costs for producers. Do I get it right
> > > that
> > > > >> you
> > > > >>>> propose to get rid of this? Or do I misunderstand?
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Let’s now look at this problem from a higher level, as Jack
> > > proposed.
> > > > As
> > > > >>>> it was said, the big choice we need to make is whether we 1)
> > create
> > > an
> > > > >>>> explicit batch coordinator; or 2) go for the coordinator-less
> > > > approach,
> > > > >>>> where each diskless partition is managed by its leader as in
> > classic
> > > > >> topics.
> > > > >>>>
> > > > >>>> If we try to compare the two approaches:
> > > > >>>>
> > > > >>>> Pluggability:
> > > > >>>> - Explicit coordinator: Possible. For example, some setups may
> > > benefit
> > > > >>>> from batch metadata being stored in a cloud database (such as
> AWS
> > > > >> DynamoDB
> > > > >>>> or GCP Spanner).
> > > > >>>> - Coordinator-less: Impossible.
> > > > >>>>
> > > > >>>> Scalability and fault tolerance:
> > > > >>>> - Explicit coordinator: Depends on the implementation and it’s
> > also
> > > > >>>> necessary to actively work for it.
> > > > >>>> - Coordinator-less: Closer to classic Kafka topics. Scaling is
> > done
> > > by
> > > > >>>> partition placement, partitions could fail independently.
> > > > >>>>
> > > > >>>> Separation of concerns:
> > > > >>>> - Explicit coordinator: Very good. Diskless remains more
> > independent
> > > > >> from
> > > > >>>> classic topics in terms of code and workflows. For example, the
> > > > >>>> above-mentioned non-tiered storage metadata compaction mechanism
> > > could
> > > > >> be
> > > > >>>> relatively simply implemented with it. As a flip side of this,
> > some
> > > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > > >>>> - Coordinator-less: Less so. It leans to the opposite: bringing
> > > > diskless
> > > > >>>> closer to classic topics. Some code paths and workflows could be
> > > more
> > > > >>>> straightforwardly reused, but they will inevitably have to be
> > > adapted
> > > > to
> > > > >>>> accommodate both topic types as also discussed.
> > > > >>>>
> > > > >>>> Cloud-nativeness. This is a vague concept, also related to the
> > > > previous,
> > > > >>>> but let’s try:
> > > > >>>> - Explicit coordinator: Storing and processing metadata
> separately
> > > > makes
> > > > >>>> it easier for brokers to take different roles, be purely
> stateless
> > > if
> > > > >>>> needed, etc.
> > > > >>>> - Coordinator-less: Less so. Something could be achieved with
> > > creative
> > > > >>>> partition placement, but not much.
> > > > >>>>
> > > > >>>> Both seem to have their pros and cons. However, answering Jack’s
> > > > >> question,
> > > > >>>> the explicit coordinator approach may indeed lead to a more
> > flexible
> > > > >> design.
> > > > >>>>
> > > > >>>>
> > > > >>>> The purpose of this deviation in the discussion was to receive a
> > > > >>>> preliminary community evaluation of the coordinator-less
> approach
> > > > >> without
> > > > >>>> taking on the task of writing a separate KIP and fitting it in
> the
> > > > >> system
> > > > >>>> of KIP-1150 and its children. We’re open to stopping it and
> > getting
> > > > >> back to
> > > > >>>> working out the coordinator design if the community doesn’t
> favor
> > > the
> > > > >>>> proposed approach.
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Ivan and Diskless team
> > > > >>>>
> > > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > > >>>>> Hi Ivan,
> > > > >>>>>
> > > > >>>>> As Jun pointed out, the updated design seems to have some
> > > > shortcomings
> > > > >>>>> although it simplifies the implementation.
> > > > >>>>>
> > > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > > because
> > > > >> of
> > > > >>>>> leaderless cluster, do you think it will be simpler if we only
> > > focus
> > > > on
> > > > >>>> the
> > > > >>>>> "diskless" design to handle object compaction/merging to/from
> the
> > > > >> remote
> > > > >>>>> storage to save the cross-AZ cost?
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> Thank you,
> > > > >>>>> Luke
> > > > >>>>>
> > > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao
> <jun@confluent.io.invalid
> > >
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> Hi, Ivan,
> > > > >>>>>>
> > > > >>>>>> Thanks for the explanation.
> > > > >>>>>>
> > > > >>>>>> "we write the reference to the WAL file with the batch data"
> > > > >>>>>>
> > > > >>>>>> I understand the approach now, but I think it is a hacky one.
> > > There
> > > > >> are
> > > > >>>>>> multiple short comings with this design. First, it degrades
> the
> > > > >>>> durability.
> > > > >>>>>> For each partition, now there are two files being actively
> > written
> > > > at
> > > > >> a
> > > > >>>>>> given point of time, one for the data and another for the
> > > metadata.
> > > > >>>>>> Flushing each file requires a separate IO. If the disk has 1K
> > IOPS
> > > > and
> > > > >>>> we
> > > > >>>>>> have 5K partitions in a broker, currently we can afford to
> flush
> > > > each
> > > > >>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If
> we
> > > > double
> > > > >>>> the
> > > > >>>>>> number of files per partition, we can only flush each
> partition
> > > > every
> > > > >>>> 10
> > > > >>>>>> seconds, which makes RPO twice as bad. Second, if we ever need
> > > this
> > > > >>>>>> metadata somewhere else, say in the WAL file manager, the
> > consumer
> > > > >>>> needs to
> > > > >>>>>> subscribe to every partition in the cluster, which is
> > inefficient.
> > > > The
> > > > >>>>>> actual benefit of this approach is also questionable. On the
> > > > surface,
> > > > >>>> it
> > > > >>>>>> might seem that we could reduce the number of lines that need
> to
> > > be
> > > > >>>> changed
> > > > >>>>>> for this KIP. However, the changes are quite intrusive to the
> > > > classic
> > > > >>>>>> partition's code path and will probably make the code base
> > harder
> > > to
> > > > >>>>>> maintain in the long run. I like the original approach based
> on
> > > the
> > > > >>>> batch
> > > > >>>>>> coordinator much better than this one. We could probably
> > refactor
> > > > the
> > > > >>>>>> producer state code so that it could be reused in the batch
> > > > >>>> coordinator.
> > > > >>>>>>
> > > > >>>>>> Thanks for addressing the concerns on the number of RPCs in
> the
> > > > >> produce
> > > > >>>>>> path. I agree that with the metadata crafting mechanism, we
> > could
> > > > >>>> mitigate
> > > > >>>>>> the PRC problem. However, since we now require the metadata to
> > be
> > > > >>>>>> collocated with the data on the same set of brokers, it's
> weird
> > > that
> > > > >>>> they
> > > > >>>>>> are now managed by different mechanisms. The data assignment
> now
> > > > uses
> > > > >>>> the
> > > > >>>>>> metadata crafting mechanism, but the metadata is stored in the
> > > > classic
> > > > >>>>>> partition using its own assignment strategy. It will be
> > > complicated
> > > > to
> > > > >>>> keep
> > > > >>>>>> them collocated.
> > > > >>>>>>
> > > > >>>>>> I am also concerned about the removal of the object
> > > > compaction/merging
> > > > >>>>>> step. My first concern is on the amount of metadata that need
> to
> > > be
> > > > >>>> kept.
> > > > >>>>>> Without object compcation, the metadata generated in the
> produce
> > > > path
> > > > >>>> can
> > > > >>>>>> only be deleted after remote tiering kicks in. Let's say for
> > every
> > > > >>>> 250ms we
> > > > >>>>>> produce 100 byte of metadata per partition. Let's say remoting
> > > > tiering
> > > > >>>>>> kicks in after 5 hours. In a cluster with 100K partitions, we
> > need
> > > > to
> > > > >>>> keep
> > > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata,
> > quite
> > > > >>>>>> signficant. A second concern is on performance. Every time we
> > need
> > > > to
> > > > >>>>>> rebuild the caching data, we need to read a bunch of small
> > objects
> > > > >>>> from S3,
> > > > >>>>>> slowing down the building process. If a consumer happens to
> need
> > > > such
> > > > >>>> data,
> > > > >>>>>> it could slow down the application.
> > > > >>>>>>
> > > > >>>>>> I see a tendency toward primarily optimizing for the fewest
> code
> > > > >>>> changes in
> > > > >>>>>> the KIP. Instead, our primary goal should be a clean design
> that
> > > can
> > > > >>>> last
> > > > >>>>>> for the long term.
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>>
> > > > >>>>>> Jun
> > > > >>>>>>
> > > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <
> ivan@ivanyu.me
> > >
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi Jun,
> > > > >>>>>>>
> > > > >>>>>>> Thank you for your message. I’m sorry that I failed to
> clearly
> > > > >>>> explain
> > > > >>>>>> the
> > > > >>>>>>> idea. Let me try to fix this.
> > > > >>>>>>>
> > > > >>>>>>>> Does each partition now have a metadata partition and a
> > separate
> > > > >>>> data
> > > > >>>>>>>> partition? If so, I am concerned that it essentially doubles
> > the
> > > > >>>> number
> > > > >>>>>>> of
> > > > >>>>>>>> partitions, which impacts the number of open file
> descriptors
> > > and
> > > > >>>> the
> > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > > separate
> > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > creating
> > > > an
> > > > >>>>>>>> internal topic with an unbounded number of partitions.
> > > > >>>>>>>
> > > > >>>>>>> No. There will be only one physical partition per diskless
> > > > >>>> partition. Let
> > > > >>>>>>> me explain this with an example. Let’s say we have a diskless
> > > > >>>> partition
> > > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We
> > > produce
> > > > >>>> some
> > > > >>>>>>> batches to this partition. The content of the segment file
> will
> > > be
> > > > >>>>>>> something like this (for each batch):
> > > > >>>>>>>
> > > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > > >>>>>>> Length: 123456 (like in classic)
> > > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > > >>>>>>> Magic: like in classic
> > > > >>>>>>> CRC: like in classic
> > > > >>>>>>> Attributes: like in classic
> > > > >>>>>>> LastOffsetDelta: like in classic
> > > > >>>>>>> BaseTimestamp: like in classic
> > > > >>>>>>> MaxTimestamp: like in classic
> > > > >>>>>>> ProducerId: like in classic
> > > > >>>>>>> ProducerEpoch: like in classic
> > > > >>>>>>> BaseSequence: like in classic
> > > > >>>>>>> RecordsCount: like in classic
> > > > >>>>>>> Records:
> > > > >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte
> > > offset
> > > > >>>>>>> 123456
> > > > >>>>>>>
> > > > >>>>>>> It looks very much like classic log entries. The only
> > difference
> > > is
> > > > >>>> that
> > > > >>>>>>> instead of writing real Records, we write the reference to
> the
> > > WAL
> > > > >>>> file
> > > > >>>>>>> with the batch data (I guess we need only the name and the
> byte
> > > > >>>> offset,
> > > > >>>>>>> because the byte length is the standard field above).
> > Otherwise,
> > > > >>>> it’s a
> > > > >>>>>>> normal Kafka log with the leader and replicas.
> > > > >>>>>>>
> > > > >>>>>>> So we have as many partitions for diskless as for classic. As
> > of
> > > > open
> > > > >>>>>> file
> > > > >>>>>>> descriptors, let’s proceed to the following:
> > > > >>>>>>>
> > > > >>>>>>>> Are the metadata and
> > > > >>>>>>>> the data for the same partition always collocated on the
> same
> > > > >>>> broker?
> > > > >>>>>> If
> > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > >>>>>>>
> > > > >>>>>>> The source of truth for the data is still in WAL files on
> > object
> > > > >>>> storage.
> > > > >>>>>>> The source of truth for the metadata is in segment files on
> the
> > > > >>>> brokers
> > > > >>>>>> in
> > > > >>>>>>> the replica set. Two new mechanisms are planned, both
> > independent
> > > > of
> > > > >>>> this
> > > > >>>>>>> new proposal, but I want to present them to give the idea
> that
> > > > only a
> > > > >>>>>>> limited amount of data files will be operated locally:
> > > > >>>>>>> - We want to assemble batches into segment files and offload
> > them
> > > > to
> > > > >>>>>>> tiered storage in order to prevent the unbounded growth of
> > batch
> > > > >>>>>> metadata.
> > > > >>>>>>> For this, we need to open only  a few file descriptors (for
> the
> > > > >>>> segment
> > > > >>>>>>> file itself + the necessary indexes) before the segment is
> > fully
> > > > >>>> written
> > > > >>>>>>> and handed over to RemoteLogManager.
> > > > >>>>>>> - We want to assemble local segment files for caching
> purposes
> > as
> > > > >>>> well,
> > > > >>>>>>> i.e. to speed up fetching. This will not materialize the full
> > > > >>>> content of
> > > > >>>>>>> the log, but only the hot set according to some policy (or
> > > > >>>> configurable
> > > > >>>>>>> policies), i.e. the number of segments and file descriptors
> > will
> > > > >>>> also be
> > > > >>>>>>> limited.
> > > > >>>>>>>
> > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > higher.
> > > > For
> > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> cluster
> > > > >>>> with 100
> > > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > > requests.
> > > > >>>>>> This
> > > > >>>>>>>> will significantly increase the request rate.
> > > > >>>>>>>
> > > > >>>>>>> This is a valid concern that we considered, but this issue
> can
> > be
> > > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > > >>>>>>> The situation with a single broker is trivial: all the commit
> > > > >>>> requests go
> > > > >>>>>>> from the broker to itself.
> > > > >>>>>>> Let’s scale this to a multi-broker cluster, but located in
> the
> > > > single
> > > > >>>>>> rack
> > > > >>>>>>> (AZ). Any broker can accept Produce requests for diskless
> > > > >>>> partitions, but
> > > > >>>>>>> we can tell producers (through metadata) to always send
> Produce
> > > > >>>> requests
> > > > >>>>>> to
> > > > >>>>>>> leaders. For example, broker 0 hosts the leader replicas for
> > > > diskless
> > > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> > > > >>>> requests
> > > > >>>>>> for
> > > > >>>>>>> these partitions in various combinations, but only for them.
> > > > >>>>>>>
> > > > >>>>>>>                   Broker 0
> > > > >>>>>>>             +-----------------+
> > > > >>>>>>>             |    t1-0         |
> > > > >>>>>>>             |    t2-1 <--------------------+
> > > > >>>>>>>             |    t3-0         |            |
> > > > >>>>>>> produce      | +-------------+ |            |
> > > > >>>>>>> requests     | |  diskless   | |            |
> > > > >>>>>>> --------------->|   produce   +--------------+
> > > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > > >>>>>>> partitions   | +-------------+ |    for these partitions
> > > > >>>>>>>             |                 |
> > > > >>>>>>>             +-----------------+
> > > > >>>>>>>
> > > > >>>>>>> The same applies for other brokers in this cluster.
> > Effectively,
> > > > each
> > > > >>>>>>> broker will commit only to itself, which effectively means 1
> > > commit
> > > > >>>>>> request
> > > > >>>>>>> per WAL buffer (this may be 0 physical network calls, if we
> > wish,
> > > > >>>> just a
> > > > >>>>>>> local function call).
> > > > >>>>>>>
> > > > >>>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we
> > > cannot
> > > > >>>> always
> > > > >>>>>>> send Produce requests to the designated leaders of diskless
> > > > >>>> partitions:
> > > > >>>>>>> this would mean inter-AZ network traffic, which we would like
> > to
> > > > >>>> avoid.
> > > > >>>>>> To
> > > > >>>>>>> avoid it, we say that every broker has a “diskless produce
> > > > >>>>>> representative”
> > > > >>>>>>> in every AZ. If we continue our example: when a Produce
> request
> > > for
> > > > >>>> t1-0,
> > > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on
> > broker 0
> > > > >>>> (in the
> > > > >>>>>>> broker’s AZ the representative is the broker itself).
> However,
> > if
> > > > it
> > > > >>>>>> comes
> > > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > > > >>>>>>>
> > > > >>>>>>> |produce requests         |produce requests        |produce
> > > > >>>> requests
> > > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0,
> > > t2-1,
> > > > >>>>>> t3-0
> > > > >>>>>>> |from AZ 0                |from AZ 1               |from AZ 2
> > > > >>>>>>> v                         v                        v
> > > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > > > >>>>>>> +---------------+      +---------------+
> +---------------+
> > > > >>>>>>> |     t1-0      |      |               |      |
>  |
> > > > >>>>>>> |     t2-1      |      |               |      |
>  |
> > > > >>>>>>> |     t3-0      |      |               |      |
>  |
> > > > >>>>>>> +---------------+      +--------+------+
> +--------+------+
> > > > >>>>>>>    ^     ^                    |                      |
> > > > >>>>>>>    |     +--------------------+                      |
> > > > >>>>>>>    |     commit requests for these partitions        |
> > > > >>>>>>>    |                                                 |
> > > > >>>>>>>    +-------------------------------------------------+
> > > > >>>>>>>          commit requests for these partitions
> > > > >>>>>>>
> > > > >>>>>>> All the partitions that broker 0 is the leader of will be
> > > > >>>> “represented”
> > > > >>>>>> by
> > > > >>>>>>> brokers 1 and 2 in their AZs.
> > > > >>>>>>>
> > > > >>>>>>> Of course, this relationship goes both ways between AZs (not
> > > > >>>> necessarily
> > > > >>>>>>> between the same brokers). It means that provided the cluster
> > is
> > > > >>>> balanced
> > > > >>>>>>> by the number of brokers per AZ, each broker will represent
> > > > >>>>>> (number_of_azs
> > > > >>>>>>> - 1) other brokers. This will result in the situation that
> for
> > > the
> > > > >>>>>> majority
> > > > >>>>>>> of commits, each broker will do up to (number_of_azs - 1)
> > network
> > > > >>>> commit
> > > > >>>>>>> requests (plus one local). Cloud regions tend to have 3 AZs,
> > very
> > > > >>>> rarely
> > > > >>>>>>> more. That means, brokers will be doing up to 2 network
> commit
> > > > >>>> requests
> > > > >>>>>> per
> > > > >>>>>>> WAL file.
> > > > >>>>>>>
> > > > >>>>>>> There are the following exceptions:
> > > > >>>>>>> 1. Broker count imbalance between AZs. For example, when we
> > have
> > > 2
> > > > >>>> AZs
> > > > >>>>>> and
> > > > >>>>>>> one has three brokers and another AZ has one. This one broker
> > > will
> > > > do
> > > > >>>>>>> between 1 and 3 commit requests per WAL file. This is not an
> > > > extreme
> > > > >>>>>>> amplification. Such an imbalance is not healthy in most
> > practical
> > > > >>>> setups
> > > > >>>>>>> and should be avoided anyway.
> > > > >>>>>>> 2. Leadership changes and metadata propagation period. When
> the
> > > > >>>> partition
> > > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the
> producers
> > > > will
> > > > >>>> not
> > > > >>>>>>> know this immediately (unless we want to be strict and
> respond
> > > with
> > > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
> > > > >>>> together
> > > > >>>>>> in a
> > > > >>>>>>> WAL buffer on broker 2, it will have to send two commit
> > requests:
> > > > to
> > > > >>>>>> broker
> > > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0.
> This
> > > > >>>> situation
> > > > >>>>>>> is not permanent and as producers update the cluster
> metadata,
> > it
> > > > >>>> will be
> > > > >>>>>>> resolved.
> > > > >>>>>>>
> > > > >>>>>>> This all could be built with the metadata crafting mechanism
> > only
> > > > >>>> (which
> > > > >>>>>>> is anyway needed for Diskless in one way or another to direct
> > > > >>>> producers
> > > > >>>>>> and
> > > > >>>>>>> consumers where we need to avoid inter-AZ traffic), just with
> > the
> > > > >>>> right
> > > > >>>>>>> policy for it (for example, some deterministic hash-based
> > > formula).
> > > > >>>> I.e.
> > > > >>>>>> no
> > > > >>>>>>> explicit support for “produce representative” or anything
> like
> > > this
> > > > >>>> is
> > > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > > >>>>>>>
> > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> places,
> > > > >>>> partition
> > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> truth,
> > > and
> > > > >>>> how
> > > > >>>>>> do
> > > > >>>>>>>> we maintain consistency between the two places?
> > > > >>>>>>>
> > > > >>>>>>> We do only two operations on WAL files that span multiple
> > > diskless
> > > > >>>>>>> partitions: committing and deleting. Commits can be done
> > > > >>>> independently as
> > > > >>>>>>> described above. But deletes are different, because when a
> file
> > > is
> > > > >>>>>> deleted,
> > > > >>>>>>> this affects all the partitions that still have alive batches
> > in
> > > > this
> > > > >>>>>> file
> > > > >>>>>>> (if any).
> > > > >>>>>>>
> > > > >>>>>>> The WAL file manager is a necessary point of coordination to
> > > delete
> > > > >>>> WAL
> > > > >>>>>>> files safely. We can say it is the source of truth about
> files
> > > > >>>>>> themselves,
> > > > >>>>>>> while the partition leaders and their logs hold the truth
> about
> > > > >>>> whether a
> > > > >>>>>>> particular file contains live batches of this particular
> > > partition.
> > > > >>>>>>>
> > > > >>>>>>> The file manager will do this important task: be able to say
> > for
> > > > sure
> > > > >>>>>> that
> > > > >>>>>>> a file does not contain any live batch of any existing
> > partition.
> > > > For
> > > > >>>>>> this,
> > > > >>>>>>> it will have to periodically check against the partition
> > leaders.
> > > > >>>>>>> Considering that batch deletion is irreversible, when we
> > declare
> > > a
> > > > >>>> file
> > > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > > >>>>>>>
> > > > >>>>>>> The file manager has to know about files being committed to
> > start
> > > > >>>> track
> > > > >>>>>>> them and periodically check if they are empty. We can
> consider
> > > > >>>> various
> > > > >>>>>> ways
> > > > >>>>>>> to achieve this:
> > > > >>>>>>> 1. As was proposed in my previous message: best effort commit
> > by
> > > > >>>> brokers
> > > > >>>>>> +
> > > > >>>>>>> periodic prefix scans of object storage to detect files that
> > went
> > > > >>>> below
> > > > >>>>>> the
> > > > >>>>>>> radar due to network issue or the file manager temporary
> > > > >>>> unavailability.
> > > > >>>>>>> We’re speaking about listing the file names only and opening
> > only
> > > > >>>>>>> previously unknown files in order to find the partitions
> > involved
> > > > >>>> with
> > > > >>>>>> them.
> > > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill the list
> of
> > > > files
> > > > >>>>>>> fully asynchronously and in the background. This may be not
> > ideal
> > > > >>>> due to
> > > > >>>>>>> costs and performance of scanning tons of files. However, the
> > > > number
> > > > >>>> of
> > > > >>>>>>> live WAL files should be limited due to tiered storage
> > > offloading +
> > > > >>>> we
> > > > >>>>>> can
> > > > >>>>>>> optimize this if we give files some global soft order in
> their
> > > > names.
> > > > >>>>>>>
> > > > >>>>>>>> I am not sure how this design simplifies the implementation.
> > The
> > > > >>>>>> existing
> > > > >>>>>>>> producer/replication code can't be simply reused. Adjusting
> > both
> > > > >>>> the
> > > > >>>>>>> write
> > > > >>>>>>>> path in the leader and the replication path in the follower
> to
> > > > >>>>>> understand
> > > > >>>>>>>> batch-header only data is quite intrusive to the existing
> > logic.
> > > > >>>>>>>
> > > > >>>>>>> It is true that we’ll have to change LocalLog and UnifiedLog
> in
> > > > >>>> order to
> > > > >>>>>>> support these changes. However, it seems that idempotence,
> > > > >>>> transactions,
> > > > >>>>>>> queues, tiered storage will have to be changed less than with
> > the
> > > > >>>>>> original
> > > > >>>>>>> design. This is because the partition leader state would
> remain
> > > in
> > > > >>>> the
> > > > >>>>>> same
> > > > >>>>>>> place (on brokers) and existing workflows that involve it
> would
> > > > have
> > > > >>>> to
> > > > >>>>>> be
> > > > >>>>>>> changed less compared to the situation where we globalize the
> > > > >>>> partition
> > > > >>>>>>> leader state in the batch coordinator. I admit this is hard
> to
> > > make
> > > > >>>>>>> convincing without both real implementations to hand :)
> > > > >>>>>>>
> > > > >>>>>>>> I am also
> > > > >>>>>>>> not sure how this enables seamless switching the topic modes
> > > > >>>> between
> > > > >>>>>>>> diskless and classic. Could you provide more details on
> those?
> > > > >>>>>>>
> > > > >>>>>>> Let’s consider the scenario of turning a classic topic into
> > > > >>>> diskless. The
> > > > >>>>>>> user sets diskless.enabled=true, the leader receives this
> > > metadata
> > > > >>>> update
> > > > >>>>>>> and does the following:
> > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > >>>>>>> 2. Close the current active segment.
> > > > >>>>>>> 3. Start a new segment that will be written in the diskless
> > > format
> > > > >>>> (i.e.
> > > > >>>>>>> without data).
> > > > >>>>>>> 4. Start accepting diskless commits.
> > > > >>>>>>>
> > > > >>>>>>> Since it’s the same log, the followers will know about that
> > > switch
> > > > >>>>>>> consistently. They will finish replicating the classic
> segments
> > > and
> > > > >>>> start
> > > > >>>>>>> replicating the diskless ones. They will always know where
> each
> > > > >>>> batch is
> > > > >>>>>>> located (either inside a classic segment or referenced by a
> > > > diskless
> > > > >>>>>> one).
> > > > >>>>>>> Switching back should be similar.
> > > > >>>>>>>
> > > > >>>>>>> Doing this with the coordinator is possible, but has some
> > > caveats.
> > > > >>>> The
> > > > >>>>>>> leader must do the following:
> > > > >>>>>>> 1. Stop accepting normal append writes.
> > > > >>>>>>> 2. Close the current active segment.
> > > > >>>>>>> 3. Write a special control segment to persist and replicate
> the
> > > > fact
> > > > >>>> that
> > > > >>>>>>> from offset N the partition is now in the diskless mode.
> > > > >>>>>>> 4. Inform the coordinator about the first offset N of the
> > > “diskless
> > > > >>>> era”.
> > > > >>>>>>> 5. Inform the controller quorum that the transition has
> > finished
> > > > and
> > > > >>>> that
> > > > >>>>>>> brokers now can process diskless writes for this partition.
> > > > >>>>>>> This could fail at some points, so this will probably require
> > > some
> > > > >>>>>>> explicit state machine with replication either in the
> partition
> > > log
> > > > >>>> or in
> > > > >>>>>>> KRaft.
> > > > >>>>>>>
> > > > >>>>>>> It seems that the coordinator-less approach makes this
> simpler
> > > > >>>> because
> > > > >>>>>> the
> > > > >>>>>>> “coordinator” for the partition and the partition leader are
> > the
> > > > >>>> same and
> > > > >>>>>>> they store the partition metadata in the same log, too. While
> > in
> > > > the
> > > > >>>>>>> coordinator approach we have to perform some kind of a
> > > distributed
> > > > >>>> commit
> > > > >>>>>>> to handover metadata management from the classic partition
> > leader
> > > > to
> > > > >>>> the
> > > > >>>>>>> batch coordinator.
> > > > >>>>>>>
> > > > >>>>>>> I hope these explanations help to clarify the idea. Please
> let
> > me
> > > > >>>> know if
> > > > >>>>>>> I should go deeper anywhere.
> > > > >>>>>>>
> > > > >>>>>>> Best,
> > > > >>>>>>> Ivan and the Diskless team
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > >>>>>>>> Hi, Ivan,
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks for the update.
> > > > >>>>>>>>
> > > > >>>>>>>> I am not sure that I fully understand the new design, but it
> > > seems
> > > > >>>> less
> > > > >>>>>>>> clean than before.
> > > > >>>>>>>>
> > > > >>>>>>>> Does each partition now have a metadata partition and a
> > separate
> > > > >>>> data
> > > > >>>>>>>> partition? If so, I am concerned that it essentially doubles
> > the
> > > > >>>> number
> > > > >>>>>>> of
> > > > >>>>>>>> partitions, which impacts the number of open file
> descriptors
> > > and
> > > > >>>> the
> > > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > > separate
> > > > >>>>>>>> partition just to store the metadata. It's as if we are
> > creating
> > > > an
> > > > >>>>>>>> internal topic with an unbounded number of partitions. Are
> the
> > > > >>>> metadata
> > > > >>>>>>> and
> > > > >>>>>>>> the data for the same partition always collocated on the
> same
> > > > >>>> broker?
> > > > >>>>>> If
> > > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > > >>>>>>>>
> > > > >>>>>>>> The number of RPCs in the produce path is significantly
> > higher.
> > > > For
> > > > >>>>>>>> example, if a produce request has 100 partitions, in a
> cluster
> > > > >>>> with 100
> > > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > > requests.
> > > > >>>>>> This
> > > > >>>>>>>> will significantly increase the request rate.
> > > > >>>>>>>>
> > > > >>>>>>>> The same WAL file metadata is now duplicated into two
> places,
> > > > >>>> partition
> > > > >>>>>>>> leader and WAL File Manager. Which one is the source of
> truth,
> > > and
> > > > >>>> how
> > > > >>>>>> do
> > > > >>>>>>>> we maintain consistency between the two places?
> > > > >>>>>>>>
> > > > >>>>>>>> I am not sure how this design simplifies the implementation.
> > The
> > > > >>>>>> existing
> > > > >>>>>>>> producer/replication code can't be simply reused. Adjusting
> > both
> > > > >>>> the
> > > > >>>>>>> write
> > > > >>>>>>>> path in the leader and the replication path in the follower
> to
> > > > >>>>>> understand
> > > > >>>>>>>> batch-header only data is quite intrusive to the existing
> > > logic. I
> > > > >>>> am
> > > > >>>>>>> also
> > > > >>>>>>>> not sure how this enables seamless switching the topic modes
> > > > >>>> between
> > > > >>>>>>>> diskless and classic. Could you provide more details on
> those?
> > > > >>>>>>>>
> > > > >>>>>>>> Jun
> > > > >>>>>>>>
> > > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <
> ivan@ivanyu.me
> > >
> > > > >>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi dear Kafka community,
> > > > >>>>>>>>>
> > > > >>>>>>>>> In the initial Diskless proposal, we proposed to have a
> > > separate
> > > > >>>>>>>>> component, batch/diskless coordinator, whose role would be
> to
> > > > >>>>>> centrally
> > > > >>>>>>>>> manage the batch and WAL file metadata for diskless topics.
> > > This
> > > > >>>>>>> component
> > > > >>>>>>>>> drew many reasonable comments from the community about how
> it
> > > > >>>> would
> > > > >>>>>>> support
> > > > >>>>>>>>> various Kafka features (transactions, queues) and its
> > > > >>>> scalability.
> > > > >>>>>>> While we
> > > > >>>>>>>>> believe we have good answers to all the expressed concerns,
> > we
> > > > >>>> took a
> > > > >>>>>>> step
> > > > >>>>>>>>> back and looked at the problem from a different
> perspective.
> > > > >>>>>>>>>
> > > > >>>>>>>>> We would like to propose an alternative Diskless design
> > > *without
> > > > >>>> a
> > > > >>>>>>>>> centralized coordinator*. We believe this approach has
> > > potential
> > > > >>>> and
> > > > >>>>>>>>> propose to discuss it as it may be more appealing to the
> > > > >>>> community.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Let us explain the idea. Most of the complications with the
> > > > >>>> original
> > > > >>>>>>>>> Diskless approach come from one necessary architecture
> > change:
> > > > >>>>>>> globalizing
> > > > >>>>>>>>> the local state of partition leader in the batch
> coordinator.
> > > > >>>> This
> > > > >>>>>>> causes
> > > > >>>>>>>>> deviations to the established workflows in various features
> > > like
> > > > >>>>>>> produce
> > > > >>>>>>>>> idempotence and transactions, queues, retention, etc. These
> > > > >>>>>> deviations
> > > > >>>>>>> need
> > > > >>>>>>>>> to be carefully considered, designed, and later implemented
> > and
> > > > >>>>>>> tested. In
> > > > >>>>>>>>> the new approach we want to avoid this by making partition
> > > > >>>> leaders
> > > > >>>>>>> again
> > > > >>>>>>>>> responsible for managing their partitions, even in diskless
> > > > >>>> topics.
> > > > >>>>>>>>>
> > > > >>>>>>>>> In classic Kafka topics, batch data and metadata are
> blended
> > > > >>>> together
> > > > >>>>>>> in
> > > > >>>>>>>>> the one partition log. The crux of the Diskless idea is to
> > > > >>>> decouple
> > > > >>>>>>> them
> > > > >>>>>>>>> and move data to the remote storage, while keeping metadata
> > > > >>>> somewhere
> > > > >>>>>>> else.
> > > > >>>>>>>>> Using the central batch coordinator for managing batch
> > metadata
> > > > >>>> is
> > > > >>>>>> one
> > > > >>>>>>> way,
> > > > >>>>>>>>> but not the only.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Let’s now think about managing metadata for each user
> > partition
> > > > >>>>>>>>> independently. Generally partitions are independent and
> don’t
> > > > >>>> share
> > > > >>>>>>>>> anything apart from that their data are mixed in WAL files.
> > If
> > > we
> > > > >>>>>>> figure
> > > > >>>>>>>>> out how to commit and later delete WAL files safely, we
> will
> > > > >>>> achieve
> > > > >>>>>>> the
> > > > >>>>>>>>> necessary autonomy that allows us to get rid of the central
> > > batch
> > > > >>>>>>>>> coordinator. Instead, *each diskless user partition will be
> > > > >>>> managed
> > > > >>>>>> by
> > > > >>>>>>> its
> > > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in classic
> > > > >>>> topics, the
> > > > >>>>>>>>> leader uses the partition log as the way to persist batch
> > > > >>>> metadata,
> > > > >>>>>>> i.e.
> > > > >>>>>>>>> the regular batch header + the information about how to
> find
> > > this
> > > > >>>>>>> batch on
> > > > >>>>>>>>> remote storage. In contrast to classic topics, batch data
> is
> > in
> > > > >>>>>> remote
> > > > >>>>>>>>> storage.
> > > > >>>>>>>>>
> > > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > > >>>>>>>>> • Classic topics:
> > > > >>>>>>>>>  • Data and metadata are co-located in the partition log.
> > > > >>>>>>>>>  • The partition log content: [Batch header
> (metadata)|Batch
> > > > >>>> data].
> > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > >>>> metadata.
> > > > >>>>>>>>> • Original Diskless:
> > > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on remote
> > > > >>>> storage.
> > > > >>>>>>>>>  • The partition state is global in the batch coordinator.
> > > > >>>>>>>>> • New Diskless:
> > > > >>>>>>>>>  • Metadata is in the partition log, data is on remote
> > storage.
> > > > >>>>>>>>>  • Partition log content: [Batch header (metadata)|Batch
> > > > >>>>>> coordinates
> > > > >>>>>>> on
> > > > >>>>>>>>> remote storage].
> > > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > > >>>>>>>>>  • The replicas and leader have local state built from
> > > > >>>> metadata.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Let’s consider the produce path. Here’s the reminder of the
> > > > >>>> original
> > > > >>>>>>>>> Diskless design:
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> The new approach could be depicted as the following:
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> As you can see, the main difference is that now instead of
> a
> > > > >>>> single
> > > > >>>>>>> commit
> > > > >>>>>>>>> request to the batch coordinator, we send multiple parallel
> > > > >>>> commit
> > > > >>>>>>> requests
> > > > >>>>>>>>> to all the leaders of each partition involved in the WAL
> > file.
> > > > >>>> Each
> > > > >>>>>> of
> > > > >>>>>>> them
> > > > >>>>>>>>> will commit its batches independently, without coordinating
> > > with
> > > > >>>>>> other
> > > > >>>>>>>>> leaders and any other components. Batch data is addressed
> by
> > > the
> > > > >>>> WAL
> > > > >>>>>>> file
> > > > >>>>>>>>> name, the byte offset and size, which allows partitions to
> > know
> > > > >>>>>> nothing
> > > > >>>>>>>>> about other partitions to access their data in shared WAL
> > > files.
> > > > >>>>>>>>>
> > > > >>>>>>>>> The number of partitions involved in a single WAL file may
> be
> > > > >>>> quite
> > > > >>>>>>> large,
> > > > >>>>>>>>> e.g. a hundred. A hundred network requests to commit one
> WAL
> > > > >>>> file is
> > > > >>>>>>> very
> > > > >>>>>>>>> impractical. However, there are ways to reduce this number:
> > > > >>>>>>>>> 1. Partition leaders are located on brokers. Requests to
> > > > >>>> leaders on
> > > > >>>>>>> one
> > > > >>>>>>>>> broker could be grouped together into a single physical
> > network
> > > > >>>>>> request
> > > > >>>>>>>>> (resembling the normal Produce request that may carry
> batches
> > > for
> > > > >>>>>> many
> > > > >>>>>>>>> partitions inside). This will cap the number of network
> > > requests
> > > > >>>> to
> > > > >>>>>> the
> > > > >>>>>>>>> number of brokers in the cluster.
> > > > >>>>>>>>> 2. If we craft the cluster metadata to make producers send
> > > their
> > > > >>>>>>> requests
> > > > >>>>>>>>> to the right brokers (with respect to AZs), we may achieve
> > the
> > > > >>>> higher
> > > > >>>>>>>>> concentration of logical commit requests in physical
> network
> > > > >>>> requests
> > > > >>>>>>>>> reducing the number of the latter ones even further,
> ideally
> > to
> > > > >>>> one.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Obviously, out of multiple commit requests some may fail or
> > > time
> > > > >>>> out
> > > > >>>>>>> for a
> > > > >>>>>>>>> variety of reasons. This is fine. Some producers will
> receive
> > > > >>>> totally
> > > > >>>>>>> or
> > > > >>>>>>>>> partially failed responses to their Produce requests,
> similar
> > > to
> > > > >>>> what
> > > > >>>>>>> they
> > > > >>>>>>>>> would have received when appending to a classic topic fails
> > or
> > > > >>>> times
> > > > >>>>>>> out.
> > > > >>>>>>>>> If a partition experiences problems, other partitions will
> > not
> > > be
> > > > >>>>>>> affected
> > > > >>>>>>>>> (again, like in classic topics). Of course, the uncommitted
> > > data
> > > > >>>> will
> > > > >>>>>>> be
> > > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived
> (batches
> > > are
> > > > >>>>>>> constantly
> > > > >>>>>>>>> assembled into segments and offloaded to tiered storage),
> so
> > > this
> > > > >>>>>>> garbage
> > > > >>>>>>>>> will be eventually deleted.
> > > > >>>>>>>>>
> > > > >>>>>>>>> For safely deleting WAL files we now need to centrally
> manage
> > > > >>>> them,
> > > > >>>>>> as
> > > > >>>>>>>>> this is the only state and logic that spans multiple
> > > partitions.
> > > > >>>> On
> > > > >>>>>> the
> > > > >>>>>>>>> diagram, you can see another commit request called “Commit
> > file
> > > > >>>> (best
> > > > >>>>>>>>> effort)” going to the WAL File Manager. This manager will
> be
> > > > >>>>>>> responsible
> > > > >>>>>>>>> for the following:
> > > > >>>>>>>>> 1. Collecting (by requests from brokers) and persisting
> > > > >>>> information
> > > > >>>>>>> about
> > > > >>>>>>>>> committed WAL files.
> > > > >>>>>>>>> 2. To handle potential failures in file information
> delivery,
> > > it
> > > > >>>>>> will
> > > > >>>>>>> be
> > > > >>>>>>>>> doing prefix scan on the remote storage periodically to
> find
> > > and
> > > > >>>>>>> register
> > > > >>>>>>>>> unknown files. The period of this scan will be configurable
> > and
> > > > >>>>>> ideally
> > > > >>>>>>>>> should be quite long.
> > > > >>>>>>>>> 3. Checking with the relevant partition leaders (after a
> > grace
> > > > >>>>>>> period) if
> > > > >>>>>>>>> they still have batches in a particular file.
> > > > >>>>>>>>> 4. Physically deleting files when they aren’t anymore
> > referred
> > > > >>>> to by
> > > > >>>>>>> any
> > > > >>>>>>>>> partition.
> > > > >>>>>>>>>
> > > > >>>>>>>>> This new design offers the following advantages:
> > > > >>>>>>>>> 1. It simplifies the implementation of many Kafka features
> > such
> > > > >>>> as
> > > > >>>>>>>>> idempotence, transactions, queues, tiered storage,
> retention.
> > > > >>>> Now we
> > > > >>>>>>> don’t
> > > > >>>>>>>>> need to abstract away and reuse the code from partition
> > leaders
> > > > >>>> in
> > > > >>>>>> the
> > > > >>>>>>>>> batch coordinator. Instead, we will literally use the same
> > code
> > > > >>>> paths
> > > > >>>>>>> in
> > > > >>>>>>>>> leaders, with little adaptation. Workflows from classic
> > topics
> > > > >>>> mostly
> > > > >>>>>>>>> remain unchanged.
> > > > >>>>>>>>> For example, it seems that
> > > > >>>>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> > and
> > > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for transaction
> > > > >>>> support
> > > > >>>>>> on
> > > > >>>>>>> the
> > > > >>>>>>>>> partition leader side could be used for diskless topics
> with
> > > > >>>> little
> > > > >>>>>>>>> adaptation. ProducerStateManager, needed for both
> idempotent
> > > > >>>> produce
> > > > >>>>>>> and
> > > > >>>>>>>>> transactions, would be reused.
> > > > >>>>>>>>> Another example is share groups support, where the share
> > > > >>>> partition
> > > > >>>>>>> leader,
> > > > >>>>>>>>> being co-located with the partition leader, would execute
> the
> > > > >>>> same
> > > > >>>>>>> logic
> > > > >>>>>>>>> for both diskless and classic topics.
> > > > >>>>>>>>> 2. It returns to the familiar partition-based scaling
> model,
> > > > >>>> where
> > > > >>>>>>>>> partitions are independent.
> > > > >>>>>>>>> 3. It makes the operation and failure patterns closer to
> the
> > > > >>>>>> familiar
> > > > >>>>>>>>> ones from classic topics.
> > > > >>>>>>>>> 4. It opens a straightforward path to seamless switching
> the
> > > > >>>> topics
> > > > >>>>>>> modes
> > > > >>>>>>>>> between diskless and classic.
> > > > >>>>>>>>>
> > > > >>>>>>>>> The rest of the things remain unchanged compared to the
> > > previous
> > > > >>>>>>> Diskless
> > > > >>>>>>>>> design (after all previous discussions). Such things as
> local
> > > > >>>> segment
> > > > >>>>>>>>> materialization by replicas, the consume path, tiered
> storage
> > > > >>>>>>> integration,
> > > > >>>>>>>>> etc.
> > > > >>>>>>>>>
> > > > >>>>>>>>> If the community finds this design more suitable, we will
> > > update
> > > > >>>> the
> > > > >>>>>>>>> KIP(s) accordingly and continue working on it. Please let
> us
> > > know
> > > > >>>>>> what
> > > > >>>>>>> you
> > > > >>>>>>>>> think.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Best regards,
> > > > >>>>>>>>> Ivan and Diskless team
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > >>>>>>>>>> Hi Justine,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Yes, you're right. We need to track the aborted
> transactions
> > > > >>>> for in
> > > > >>>>>>> the
> > > > >>>>>>>>> diskless coordinator for as long as the corresponding
> offsets
> > > are
> > > > >>>>>>> there.
> > > > >>>>>>>>> With the tiered storage unification Greg mentioned earlier,
> > > this
> > > > >>>> will
> > > > >>>>>>> be
> > > > >>>>>>>>> finite time even for infinite data retention.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best,
> > > > >>>>>>>>>> Ivan
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > >>>>>>>>>>> Hey Ivan,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks for the response. I think most of what you said
> made
> > > > >>>>>> sense,
> > > > >>>>>>> but
> > > > >>>>>>>>> I
> > > > >>>>>>>>>>> did have some questions about this part:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > > >>>> topics
> > > > >>>>>>> forgets
> > > > >>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> > > > >>>> it). The
> > > > >>>>>>>>>>> transaction coordinator acts like the main guardian,
> > allowing
> > > > >>>>>>> partition
> > > > >>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > > > >>>> wrong. We
> > > > >>>>>>> think
> > > > >>>>>>>>>>> about relying on this with the batch coordinator and
> delete
> > > > >>>> the
> > > > >>>>>>>>> information
> > > > >>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > > >>>> replication
> > > > >>>>>>> and
> > > > >>>>>>>>> HWM
> > > > >>>>>>>>>>> advances immediately).
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I didn't quite understand this. In classic topics, we
> have
> > > > >>>> maps
> > > > >>>>>> for
> > > > >>>>>>>>> ongoing
> > > > >>>>>>>>>>> transactions which remove state when the transaction is
> > > > >>>> completed
> > > > >>>>>>> and
> > > > >>>>>>>>> an
> > > > >>>>>>>>>>> aborted transactions index which is retained for much
> > longer.
> > > > >>>>>> Once
> > > > >>>>>>> the
> > > > >>>>>>>>>>> transaction is completed, the coordinator is no longer
> > > > >>>> involved
> > > > >>>>>> in
> > > > >>>>>>>>>>> maintaining this partition side state, and it is subject
> to
> > > > >>>>>>> compaction
> > > > >>>>>>>>> etc.
> > > > >>>>>>>>>>> Looking back at the outline provided above, I didn't see
> > much
> > > > >>>>>>> about the
> > > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit
> further.
> > I
> > > > >>>> saw
> > > > >>>>>>> the
> > > > >>>>>>>>>>> following in a response:
> > > > >>>>>>>>>>>> When the broker constructs a fully valid local segment,
> > > > >>>> all the
> > > > >>>>>>>>> necessary
> > > > >>>>>>>>>>> control batches will be inserted and indices, including
> the
> > > > >>>>>>> transaction
> > > > >>>>>>>>>>> index will be built to serve FetchRequests exactly as
> they
> > > > >>>> are
> > > > >>>>>>> today.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Based on this, it seems like we need to retain the
> > > > >>>> information
> > > > >>>>>>> about
> > > > >>>>>>>>>>> aborted txns for longer.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>> Justine
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > > >>>> ivan@ivanyu.me>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi Justine and all,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thank you for your questions!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> > > > >>>> with
> > > > >>>>>>>>> producer ID
> > > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > > >>>> cached
> > > > >>>>>>>>> locally
> > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > >>>>>> transactions
> > > > >>>>>>> can
> > > > >>>>>>>>> be
> > > > >>>>>>>>>>>> used
> > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > > >>>> with
> > > > >>>>>>>>> producer id +
> > > > >>>>>>>>>>>>> epoch
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> You’re right that we (probably unintentionally) focused
> > > > >>>> only on
> > > > >>>>>>>>> version 2.
> > > > >>>>>>>>>>>> We can either limit the support to version 2 or consider
> > > > >>>> using
> > > > >>>>>>> some
> > > > >>>>>>>>>>>> surrogates to support version 1.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> transactional
> > > > >>>>>>> checks
> > > > >>>>>>>>> of the
> > > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > > >>>> like the
> > > > >>>>>>>>> partition
> > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > > >>>>>> checking
> > > > >>>>>>> if
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is
> ongoing,
> > > > >>>> and
> > > > >>>>>> of
> > > > >>>>>>>>> course
> > > > >>>>>>>>>>>> the normal idempotence checks. What the partition leader
> > > > >>>> in the
> > > > >>>>>>>>> classic
> > > > >>>>>>>>>>>> topics does before appending a batch to the local log
> > > > >>>> (e.g. in
> > > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In
> Diskless,
> > > > >>>> we
> > > > >>>>>>>>> unfortunately
> > > > >>>>>>>>>>>> cannot do these checks before appending the data to the
> > WAL
> > > > >>>>>>> segment
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>> uploading it, but we can “tombstone” these batches in
> the
> > > > >>>> batch
> > > > >>>>>>>>> coordinator
> > > > >>>>>>>>>>>> during the final commit.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Is there state about ongoing
> > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> > > > >>>> state
> > > > >>>>>>>>> mentioned
> > > > >>>>>>>>>>>> in
> > > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > > >>>> what
> > > > >>>>>>> state is
> > > > >>>>>>>>>>>> stored
> > > > >>>>>>>>>>>>> and when it is stored.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Right, this should have been more explicit. As the
> > > > >>>> partition
> > > > >>>>>>> leader
> > > > >>>>>>>>> tracks
> > > > >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> > > > >>>> coordinator
> > > > >>>>>>> has
> > > > >>>>>>>>> to as
> > > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > > >>>> transaction
> > > > >>>>>>>>> coordinator
> > > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > > >>>> perhaps
> > > > >>>>>>> that
> > > > >>>>>>>>> would
> > > > >>>>>>>>>>>> be
> > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Yes. This could be deduced from the committed batches
> and
> > > > >>>> other
> > > > >>>>>>>>>>>> information, but for the sake of performance we’d better
> > > > >>>> store
> > > > >>>>>> it
> > > > >>>>>>>>>>>> explicitly.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> transactional
> > > > >>>>>>> state is
> > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> > > > >>>>>> cleaned
> > > > >>>>>>> up?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > > >>>> topics
> > > > >>>>>>> forgets
> > > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> > > > >>>> it).
> > > > >>>>>> The
> > > > >>>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > > >>>> allowing
> > > > >>>>>>>>> partition
> > > > >>>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > > > >>>> wrong.
> > > > >>>>>> We
> > > > >>>>>>>>> think
> > > > >>>>>>>>>>>> about relying on this with the batch coordinator and
> > > > >>>> delete the
> > > > >>>>>>>>> information
> > > > >>>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > > >>>>>> replication
> > > > >>>>>>>>> and HWM
> > > > >>>>>>>>>>>> advances immediately).
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>> Ivan
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > >>>>>>>>>>>>> Hey folks,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Excited to see some updates related to transactions!
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I had a few questions.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> > > > >>>> with
> > > > >>>>>>>>> producer ID
> > > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > > >>>> cached
> > > > >>>>>>>>> locally
> > > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > > >>>>>> transactions
> > > > >>>>>>> can
> > > > >>>>>>>>> be
> > > > >>>>>>>>>>>> used
> > > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > > >>>> with
> > > > >>>>>>>>> producer id +
> > > > >>>>>>>>>>>>> epoch
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final
> transactional
> > > > >>>>>>> checks
> > > > >>>>>>>>> of the
> > > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > > >>>> like the
> > > > >>>>>>>>> partition
> > > > >>>>>>>>>>>>> leader in classic topics would do.
> > > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > > >>>>>> checking
> > > > >>>>>>> if
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>> transaction was still ongoing for example? Is there
> state
> > > > >>>>>> about
> > > > >>>>>>>>> ongoing
> > > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> > > > >>>> state
> > > > >>>>>>>>> mentioned
> > > > >>>>>>>>>>>> in
> > > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > > >>>> what
> > > > >>>>>>> state is
> > > > >>>>>>>>>>>> stored
> > > > >>>>>>>>>>>>> and when it is stored.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > > >>>> perhaps
> > > > >>>>>>> that
> > > > >>>>>>>>> would
> > > > >>>>>>>>>>>> be
> > > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long
> transactional
> > > > >>>>>>> state is
> > > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> > > > >>>>>> cleaned
> > > > >>>>>>> up?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > >>>>>>> <ju...@confluent.io.invalid>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> > > > >>>>>> segments,
> > > > >>>>>>>>> making
> > > > >>>>>>>>>>>> use of
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and zero-copy
> > > > >>>>>>>>> functionality
> > > > >>>>>>>>>>>> already
> > > > >>>>>>>>>>>>>> built into classic topics."
> > > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
> > > > >>>> each
> > > > >>>>>>>>> partition in
> > > > >>>>>>>>>>>>>> diskless topics?
> > > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
> > > > >>>> need
> > > > >>>>>> to
> > > > >>>>>>> know
> > > > >>>>>>>>>>>> aborted
> > > > >>>>>>>>>>>>>> records. How is that achieved?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > > >>>> transaction is
> > > > >>>>>>>>> finished
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> also inserts the control batches in the corresponding
> > > > >>>> logs
> > > > >>>>>>> of the
> > > > >>>>>>>>>>>> involved
> > > > >>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
> > > > >>>> level,
> > > > >>>>>> no
> > > > >>>>>>>>> actual
> > > > >>>>>>>>>>>> control
> > > > >>>>>>>>>>>>>> batches are written to any file. "
> > > > >>>>>>>>>>>>>> A fetch response could include multiple transactional
> > > > >>>>>>> batches.
> > > > >>>>>>>>> How
> > > > >>>>>>>>>>>> does the
> > > > >>>>>>>>>>>>>> broker obtain the information about the ending control
> > > > >>>>>> batch
> > > > >>>>>>> for
> > > > >>>>>>>>> each
> > > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response needs to
> be
> > > > >>>>>>> built by
> > > > >>>>>>>>>>>>>> stitching record batches and generated control batches
> > > > >>>>>>> together?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition leader
> > > > >>>> that
> > > > >>>>>>> all
> > > > >>>>>>>>>>>> consumers
> > > > >>>>>>>>>>>>>> are routed to?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this or
> > > > >>>> it's
> > > > >>>>>>> too
> > > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to include
> > > > >>>>>> enough
> > > > >>>>>>>>> details
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
> > > > >>>>>> producer
> > > > >>>>>>>>> contain a
> > > > >>>>>>>>>>>>>> sequence number, which is ordered. If a producer sends
> > > > >>>>>> every
> > > > >>>>>>>>> Produce
> > > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests could
> > > > >>>> reach
> > > > >>>>>>> the
> > > > >>>>>>>>> batch
> > > > >>>>>>>>>>>>>> coordinator in different order and lead to rejection
> > > > >>>> of the
> > > > >>>>>>>>> produce
> > > > >>>>>>>>>>>>>> requests.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > > >>>>>>> ivan@ivanyu.me>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> We have also thought in a bit more details about
> > > > >>>>>>> transactions
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>> queues,
> > > > >>>>>>>>>>>>>>> here's the plan.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> *Transactions*
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> The support for transactions in *classic topics* is
> > > > >>>> based
> > > > >>>>>>> on
> > > > >>>>>>>>> precise
> > > > >>>>>>>>>>>>>>> interactions between three actors: clients (mostly
> > > > >>>>>>> producers,
> > > > >>>>>>>>> but
> > > > >>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > > >>>> classes),
> > > > >>>>>> and
> > > > >>>>>>>>>>>> transaction
> > > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders with
> > > > >>>>>> their
> > > > >>>>>>>>> local
> > > > >>>>>>>>>>>> state
> > > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> The high level (some details skipped) workflow is the
> > > > >>>>>>>>> following.
> > > > >>>>>>>>>>>> When a
> > > > >>>>>>>>>>>>>>> transactional Produce request is received by the
> > > > >>>> broker:
> > > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader checks
> > > > >>>> if a
> > > > >>>>>>>>> non-empty
> > > > >>>>>>>>>>>>>>> transaction is running for this partition. This is
> > > > >>>> done
> > > > >>>>>>> using
> > > > >>>>>>>>> its
> > > > >>>>>>>>>>>> local
> > > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > > >>>>>> (ProducerStateManager,
> > > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed about all
> > > > >>>> the
> > > > >>>>>>>>> partitions
> > > > >>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> > > > >>>>>>>>>>>>>>> 3. The partition leaders do additional transactional
> > > > >>>>>>> checks.
> > > > >>>>>>>>>>>>>>> 4. The partition leaders append the transactional
> > > > >>>> data to
> > > > >>>>>>>>> their logs
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>> update some of their state (for example, log the fact
> > > > >>>>>> that
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>>> transaction
> > > > >>>>>>>>>>>>>>> is running for the partition and its first offset).
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> > > > >>>>>>> directly
> > > > >>>>>>>>> with
> > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> > > > >>>> or
> > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > > >>>>>>> WriteTxnMarkersRequest to
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>> leaders
> > > > >>>>>>>>>>>>>>> of the involved partitions.
> > > > >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> > > > >>>> markers to
> > > > >>>>>>>>> their logs
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > > >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction state
> > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
> > > > >>>> of
> > > > >>>>>>>>> important
> > > > >>>>>>>>>>>> state
> > > > >>>>>>>>>>>>>>> necessary for supporting this workflow is local. The
> > > > >>>> main
> > > > >>>>>>>>> challenge
> > > > >>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
> > > > >>>> are no
> > > > >>>>>>>>> partition
> > > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
> > > > >>>> to be
> > > > >>>>>>>>> globalized
> > > > >>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> > > > >>>> support
> > > > >>>>>>>>> idempotent
> > > > >>>>>>>>>>>>>> produce.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics* would
> > > > >>>> look
> > > > >>>>>>> very
> > > > >>>>>>>>>>>> similar:
> > > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > > >>>> non-empty
> > > > >>>>>>>>> transaction
> > > > >>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>> running for this partition. In contrast to classic
> > > > >>>>>> topics,
> > > > >>>>>>>>> this is
> > > > >>>>>>>>>>>>>> checked
> > > > >>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
> > > > >>>> Since a
> > > > >>>>>>>>> transaction
> > > > >>>>>>>>>>>>>> could
> > > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
> > > > >>>> the
> > > > >>>>>>> positive
> > > > >>>>>>>>>>>> result of
> > > > >>>>>>>>>>>>>>> this check could be cached locally (for the double
> > > > >>>>>>> configured
> > > > >>>>>>>>>>>> duration
> > > > >>>>>>>>>>>>>> of a
> > > > >>>>>>>>>>>>>>> transaction, for example).
> > > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is informed
> > > > >>>>>> about
> > > > >>>>>>> all
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
> > > > >>>> include
> > > > >>>>>>> them.
> > > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
> > > > >>>> side.
> > > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data to the
> > > > >>>>>> current
> > > > >>>>>>>>> shared
> > > > >>>>>>>>>>>> WAL
> > > > >>>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
> > > > >>>> state
> > > > >>>>>>> for
> > > > >>>>>>>>> Diskless
> > > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > > >>>> coordinator
> > > > >>>>>>> like
> > > > >>>>>>>>> in the
> > > > >>>>>>>>>>>>>>> normal produce flow.
> > > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final transactional
> > > > >>>>>>> checks
> > > > >>>>>>>>> of the
> > > > >>>>>>>>>>>>>>> batches. This procedure would output the same errors
> > > > >>>> like
> > > > >>>>>>> the
> > > > >>>>>>>>>>>> partition
> > > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some batches
> > > > >>>>>> could
> > > > >>>>>>> be
> > > > >>>>>>>>>>>> rejected.
> > > > >>>>>>>>>>>>>>> This means, there will potentially be garbage in the
> > > > >>>> WAL
> > > > >>>>>>>>> segment
> > > > >>>>>>>>>>>> file in
> > > > >>>>>>>>>>>>>>> case of transactional errors. This is preferable to
> > > > >>>> doing
> > > > >>>>>>> more
> > > > >>>>>>>>>>>> network
> > > > >>>>>>>>>>>>>>> round trips, especially considering the WAL segments
> > > > >>>> will
> > > > >>>>>>> be
> > > > >>>>>>>>>>>> relatively
> > > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> > > > >>>>>>> directly
> > > > >>>>>>>>> with
> > > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> > > > >>>> or
> > > > >>>>>>>>>>>> PREPARE_ABORT to
> > > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
> > > > >>>> batch
> > > > >>>>>>>>> coordinator
> > > > >>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>> the transaction is finished.
> > > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> > > > >>>>>>> transaction is
> > > > >>>>>>>>>>>> finished
> > > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > > >>>> corresponding
> > > > >>>>>>> logs
> > > > >>>>>>>>> of the
> > > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only on the
> > > > >>>>>> metadata
> > > > >>>>>>>>> level, no
> > > > >>>>>>>>>>>>>>> actual control batches are written to any file. They
> > > > >>>> will
> > > > >>>>>>> be
> > > > >>>>>>>>>>>> dynamically
> > > > >>>>>>>>>>>>>>> created on Fetch and other read operations. We could
> > > > >>>>>>>>> technically
> > > > >>>>>>>>>>>> write
> > > > >>>>>>>>>>>>>>> these control batches for real, but this would mean
> > > > >>>> extra
> > > > >>>>>>>>> produce
> > > > >>>>>>>>>>>>>> latency,
> > > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > > >>>> coordinator
> > > > >>>>>>> and
> > > > >>>>>>>>> save
> > > > >>>>>>>>>>>> these
> > > > >>>>>>>>>>>>>>> milliseconds.
> > > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > > >>>>>>> WriteTxnMarkersRequest to
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>> leaders
> > > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
> > > > >>>> topics
> > > > >>>>>>> now.
> > > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics write the
> > > > >>>>>>>>> transaction
> > > > >>>>>>>>>>>> markers
> > > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > > >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction state
> > > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
> > > > >>>> get:
> > > > >>>>>>>>>>>>>>> 1. An extra network round trip between brokers and
> > > > >>>> the
> > > > >>>>>>> batch
> > > > >>>>>>>>>>>> coordinator
> > > > >>>>>>>>>>>>>>> when a new partition appear in the transaction. To
> > > > >>>>>>> mitigate the
> > > > >>>>>>>>>>>> impact of
> > > > >>>>>>>>>>>>>>> them:
> > > > >>>>>>>>>>>>>>> - The results will be cached.
> > > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one Produce
> > > > >>>>>>> request
> > > > >>>>>>>>> will be
> > > > >>>>>>>>>>>>>>> grouped.
> > > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> > > > >>>> fast
> > > > >>>>>>>>> response to
> > > > >>>>>>>>>>>> these
> > > > >>>>>>>>>>>>>>> RPCs.
> > > > >>>>>>>>>>>>>>> - The fact that a single producer normally will
> > > > >>>>>>> communicate
> > > > >>>>>>>>> with a
> > > > >>>>>>>>>>>>>>> single broker for the duration of the transaction
> > > > >>>> further
> > > > >>>>>>>>> reduces the
> > > > >>>>>>>>>>>>>>> expected number of round trips.
> > > > >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> > > > >>>>>> coordinator
> > > > >>>>>>> and
> > > > >>>>>>>>> batch
> > > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> With this proposal, transactions will also be able to
> > > > >>>>>> span
> > > > >>>>>>> both
> > > > >>>>>>>>>>>> classic
> > > > >>>>>>>>>>>>>>> and Diskless topics.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> *Queues*
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> The share group coordination and management is a
> > > > >>>> side job
> > > > >>>>>>> that
> > > > >>>>>>>>>>>> doesn't
> > > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > > >>>> replicas,
> > > > >>>>>>> physical
> > > > >>>>>>>>>>>> storage
> > > > >>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and consumers
> > > > >>>>>>> (Fetch and
> > > > >>>>>>>>>>>> Produce
> > > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not affected.)
> > > > >>>> We
> > > > >>>>>>> don't
> > > > >>>>>>>>> see any
> > > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
> > > > >>>> with
> > > > >>>>>>> share
> > > > >>>>>>>>>>>> groups the
> > > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the code
> > > > >>>> level,
> > > > >>>>>> we
> > > > >>>>>>>>> don't
> > > > >>>>>>>>>>>> expect
> > > > >>>>>>>>>>>>>> any
> > > > >>>>>>>>>>>>>>> serious refactoring: the same reading routines are
> > > > >>>> used
> > > > >>>>>>> that
> > > > >>>>>>>>> are
> > > > >>>>>>>>>>>> used for
> > > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
> > > > >>>> too
> > > > >>>>>>>>>>>>>>> implementation-focused?
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>> Ivan
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > >>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thank you all for your questions and design input
> > > > >>>> on
> > > > >>>>>>>>> KIP-1150.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
> > > > >>>> new
> > > > >>>>>>>>> design. To
> > > > >>>>>>>>>>>>>>> summarize
> > > > >>>>>>>>>>>>>>>> the changes:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> > > > >>>> existing
> > > > >>>>>>>>> KIP-405
> > > > >>>>>>>>>>>> Tiered
> > > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
> > > > >>>>>>> Diskless
> > > > >>>>>>>>> topic
> > > > >>>>>>>>>>>> to be
> > > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for the
> > > > >>>> Batch
> > > > >>>>>>>>> Coordinator
> > > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> > > > >>>> Tiered
> > > > >>>>>>> Storage
> > > > >>>>>>>>>>>> plugin
> > > > >>>>>>>>>>>>>>>> features such as encryption and alternative data
> > > > >>>>>> formats.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> > > > >>>> segments,
> > > > >>>>>>>>> making use
> > > > >>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > > >>>> zero-copy
> > > > >>>>>>>>> functionality
> > > > >>>>>>>>>>>>>>> already
> > > > >>>>>>>>>>>>>>>> built into classic topics.
> > > > >>>>>>>>>>>>>>>> However, local segments are now considered cache
> > > > >>>>>>> elements,
> > > > >>>>>>>>> do not
> > > > >>>>>>>>>>>> need
> > > > >>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > > >>>> contacting
> > > > >>>>>>> any
> > > > >>>>>>>>> other
> > > > >>>>>>>>>>>>>>> replicas.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 3. The design has been simplified substantially, by
> > > > >>>>>>> removing
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>> previous
> > > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > > >>>> component, and
> > > > >>>>>>>>> "object
> > > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > > >>>> enabled by
> > > > >>>>>>> the
> > > > >>>>>>>>> Batch
> > > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
> > > > >>>>>> earlier
> > > > >>>>>>>>> design,
> > > > >>>>>>>>>>>> while
> > > > >>>>>>>>>>>>>>>> being simpler and integrating better into the
> > > > >>>> existing
> > > > >>>>>>>>> ecosystem.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
> > > > >>>> the
> > > > >>>>>> new
> > > > >>>>>>>>> design.
> > > > >>>>>>>>>>>>>>>> Greg Harris
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> > > > >>>> of
> > > > >>>>>> all
> > > > >>>>>>>>> existing
> > > > >>>>>>>>>>>>>> client
> > > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > > >>>> supporting
> > > > >>>>>> APIs
> > > > >>>>>>>>> like
> > > > >>>>>>>>>>>>>>> transactions
> > > > >>>>>>>>>>>>>>>>> and queues.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> > > > >>>>>>> status of
> > > > >>>>>>>>> this
> > > > >>>>>>>>>>>> KIP?
> > > > >>>>>>>>>>>>>> I
> > > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> > > > >>>>>>> months,
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>> vote got
> > > > >>>>>>>>>>>>>>>>> vetoed
> > > > >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> > > > >>>>>> answered
> > > > >>>>>>>>> since
> > > > >>>>>>>>>>>> then.
> > > > >>>>>>>>>>>>>>> KIP-1183
> > > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > > >>>> activity
> > > > >>>>>>> since
> > > > >>>>>>>>> May.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> > > > >>>>>> choices
> > > > >>>>>>>>> that can
> > > > >>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> > > > >>>> by
> > > > >>>>>> far
> > > > >>>>>>> the
> > > > >>>>>>>>>>>> dominant
> > > > >>>>>>>>>>>>>> one
> > > > >>>>>>>>>>>>>>>>> when
> > > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
> > > > >>>>>>>>> standalone
> > > > >>>>>>>>>>>>>> products.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> > > > >>>> hate to
> > > > >>>>>>> see a
> > > > >>>>>>>>>>>> stall. So
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> > > > >>>>>> McCabe -
> > > > >>>>>>>>> have your
> > > > >>>>>>>>>>>>>>> questions
> > > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> > > > >>>> do
> > > > >>>>>> to
> > > > >>>>>>>>> help? I'm
> > > > >>>>>>>>>>>>>> deeply
> > > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > > >>>> written
> > > > >>>>>>> about
> > > > >>>>>>>>> both?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > > >>>>>>>>>>>>>>>>>> Jan
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > > >>>>>> Kozlovski
> > > > >>>>>>> <
> > > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > > >>>> main
> > > > >>>>>> one
> > > > >>>>>>>>> (just a
> > > > >>>>>>>>>>>> bunch
> > > > >>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>>> links
> > > > >>>>>>>>>>>>>>>>>>> to everything)
> > > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > > >>>> example, I
> > > > >>>>>>> was
> > > > >>>>>>>>>>>> searching
> > > > >>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> > > > >>>> far
> > > > >>>>>> as
> > > > >>>>>>> I
> > > > >>>>>>>>> can
> > > > >>>>>>>>>>>> tell, it
> > > > >>>>>>>>>>>>>>>>> doesn't
> > > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> > > > >>>> having
> > > > >>>>>>> the
> > > > >>>>>>>>> root
> > > > >>>>>>>>>>>> KIP be
> > > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> > > > >>>> to
> > > > >>>>>>>>> sub-KIPs
> > > > >>>>>>>>>>>> where
> > > > >>>>>>>>>>>>>> were
> > > > >>>>>>>>>>>>>>>>> aimed
> > > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> > > > >>>> would
> > > > >>>>>>> be
> > > > >>>>>>>>> good to
> > > > >>>>>>>>>>>>>> create
> > > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>> Stan
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> > > > >>>>>>>>> introducing a
> > > > >>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>> type of
> > > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > > >>>> Storage as
> > > > >>>>>>> the
> > > > >>>>>>>>> primary
> > > > >>>>>>>>>>>>>>> source of
> > > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > > >>>> decided
> > > > >>>>>>> to
> > > > >>>>>>>>> split it
> > > > >>>>>>>>>>>>>> into
> > > > >>>>>>>>>>>>>>>>>> multiple
> > > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > >>>>>>>>>>>>>>>>>>> )
> > > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > > >>>> should aim
> > > > >>>>>>> to
> > > > >>>>>>>>> have
> > > > >>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>> type of
> > > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > > >>>>>> details
> > > > >>>>>>> on
> > > > >>>>>>>>> how to
> > > > >>>>>>>>>>>>>>> implement
> > > > >>>>>>>>>>>>>>>>>> it.
> > > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
> > > > >>>>>>> discussed
> > > > >>>>>>>>>>>> KRaft.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> > > > >>>> hard
> > > > >>>>>>> to
> > > > >>>>>>>>> discuss
> > > > >>>>>>>>>>>> on
> > > > >>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>> meta
> > > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > > >>>> (linked
> > > > >>>>>>> in
> > > > >>>>>>>>>>>> KIP-1150)
> > > > >>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>> offer
> > > > >>>>>>>>>>>>>>>>>>> an
> > > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> > > > >>>>>>> threads
> > > > >>>>>>>>> for
> > > > >>>>>>>>>>>> each
> > > > >>>>>>>>>>>>>>> type of
> > > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > > >>>> whether
> > > > >>>>>>> Apache
> > > > >>>>>>>>> Kafka
> > > > >>>>>>>>>>>> wants
> > > > >>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> have
> > > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > > >>>> authors
> > > > >>>>>> of
> > > > >>>>>>>>> this KIP.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> > > > >>>>>>> aiven.io
> > > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > >>>>>>> Valtonen,
> > > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>


-- 
Anatolii Popov
Senior Software Developer, *Aiven OY*
m: +358505126242
w: aiven.io  e: anatolii.popov@aiven.io
<https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Thomas Thornton via dev <de...@kafka.apache.org>]

Hi all,

We (the team at Slack) have been following the recent discussion regarding
the scope and timeline of KIP-1150. We agree with the community sentiment
that Diskless Topics represents the right long-term architecture for Kafka
in the cloud, but we also recognize the valid concerns raised regarding the
engineering resources required to deliver such an ambitious change.

To help address these concerns and accelerate the timeline, we are happy to
announce that we are partnering with the KIP-1150 authors to co-develop
this feature.

We previously proposed KIP-1176: Tiered Storage for Active Log Segments to
solve similar problems. However, rather than fragmenting the community's
efforts across competing designs, we have decided to withdraw KIP-1176 and
consolidate our engineering resources behind KIP-1150.

To start, we plan to take ownership of Compaction for Tiered Storage. This
has long been a missing feature in KIP-405, and it becomes critical in a
Diskless architecture where long-lived data is obligated to be tiered. By
driving this related prerequisite feature, we hope to allow for faster
delivery of KIP-1150.

We are excited to collaborate on this to ensure a robust and timely
delivery for the community.

Best,
Tom & Henry

On Fri, Nov 14, 2025 at 8:35 AM Luke Chen <sh...@gmail.com> wrote:

> Hi Greg,
>
> Thanks for sharing the meeting notes.
> I agree we should keep polishing the contents of 1150 & high level design
> in 1163 to prepare for a vote.
>
>
> Thanks.
> Luke
>
> On Fri, Nov 14, 2025 at 3:54 AM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hi all,
> >
> > There was a video call between myself, Ivan Yurchenko, Jun Rao, and
> Andrew
> > Schofield pertaining to KIP-1150. Here are the notes from that meeting:
> >
> > Ivan: What is the future state of Kafka in this area, in 5 years?
> > Jun: Do we want something more cloud native? Yes, started with Tiered
> > Storage. If there’s a better way, we should explore it. In the long term
> > this will be useful
> > Because Kafka is used so widely, we need to make sure everything we add
> is
> > for the long term and for everyone, not just for a single company.
> > When we add TS, it doesn’t just solve Uber’s use-case. We want something
> > that’s high quality/lasts/maintainable, and can work with all existing
> > capabilities.
> > If both 1150 and 1176 proceed at the same time, it’s confusing. They
> > overlap, but Diskless is more ambitious.
> > If both KIPs are being seriously worked on, then we don’t really need
> both,
> > because Diskless clearly is better. Having multiple will confuse people.
> It
> > will duplicate some of the effort.
> > If we want diskless ultimately, what is the short term strategy, to get
> > some early wins first?
> > Ivan: Andrew, do you want a more revolutionary approach?
> > Andrew: Eventually the architecture will change substantially, it may not
> > be necessary to put all of that bill onto Diskless at once.
> > Greg: We all agree on having a high quality feature merged upstream, and
> > supporting all APIs
> > Jun: We should try and keep things simple, but there is some minimum
> > complexity needed.
> > When doing the short term changes (1176), it doesn’t really progress in
> > changing to a more modern architecture.
> > Greg: Was TS+Compaction the only feature miss we’ve had so far?
> > Jun: The danger of only applying changes to some part of the API, you set
> > the precedence that you only have to implement part of the API.
> Supporting
> > the full API set should be a minimum requirement.
> > Andrew: When we started Kraft, how much did we know the design?
> > Jun: For Kraft we didn’t really know much about the migration, but the
> > high-level was clear.
> > Greg: Is 1150 votable in its current state?
> > Jun: 1150 should promise to support all APIs. It doesn’t have to have all
> > the details/apis/etc. KIP-500 didn’t have it.
> > We do need some high-level design enough to give confidence that the
> > promise is able to be fulfilled.
> > Greg: Is the draft version in 1163 enough detail or is more needed?
> > Jun: We need to agree on the core design, such as leaderless etc. And how
> > the APIs will be supported.
> > Greg: Okay we can include these things, and provide a sketch of how the
> > other leader-based features operate.
> > Jun: Yeah if at a high level the sketch appears to work, we can approve
> > that functionality.
> > Are you committed to doing the more involved and big project?
> > Greg: Yes, we’re committed to the 1163 design and can’t really accept
> 1176.
> > Jun: TS was slow because of Uber resourcing problems
> > Greg: We’ll push internally for resources, and use the community
> sentiment
> > to motivate Aiven.
> > How far into the future should we look? What sort of scale?
> > Jun: As long as there’s a path forward, and we’re not closing off future
> > improvements, we can figure out how to handle a larger scale when it
> > arises.
> > Greg: Random replica placement is very harmful, can we recommend users to
> > use an external tool like CruiseControl?
> > Jun: Not everyone uses CruiseControl, we would probably need some
> solution
> > for this out of the box
> > Ivan: Should the Batch Coordinator be pluggable?
> > Jun: Out-of-box experience should be good, good to allow other
> > implementations
> > Greg: But it could hurt Kafka feature/upgrade velocity when we wait for
> > plugin providers to implement it
> > Ivan: We imagined that maybe cloud hyperscalers could implement it with
> > e.g. dynamodb
> > Greg: Could we bake more details of the different providers into Kafka,
> or
> > does it still make sense for it to be pluggable?
> > Jun: Make it whatever is easiest to roll out and add new clients
> > Andrew: What happens next? Do you want to get KIP-1150 voted?
> > Ivan: The vote is already open, we’re not too pressed for time. We’ll go
> > improve the 1163 design and communication.
> > Is 1176 a competing design? Someone will ask.
> > Jun: If we are seriously working on something more ambitious, yeah we
> > shouldn’t do the stop-gap solution.
> > It’s diverting review resources. If we can get the short term thing in
> 1yr
> > but Diskless solution is 2y it makes sense to go for Diskless. If it’s
> 5yr,
> > that’s different and maybe the stop-gap solution is needed.
> > Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should we
> > explicitly exclude 1176?
> > Andrew: Put your arms around the feature set you actually want, and use
> > that to rule out 1176.
> > Probably don’t need -1 votes, most likely KIPs just don’t receive votes.
> > Ivan: Should we have sync meetings like tiered storage did?
> > Jun: Satish posted meeting notes regularly, we should do the same.
> >
> > To summarize, we will be polishing the contents of 1150 & high level
> design
> > in 1163 to prepare for a vote.
> > We believe that the community should select the feature set of 1150 to
> > fully eliminate producer cross-zone costs, and make the investment in a
> > high quality Diskless Topics implementation rather than in stop-gap
> > solutions.
> >
> > Thanks,
> > Greg
> >
> > On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:
> >
> > > This may be a tangent, but we needed to offload storage off of Kafka
> into
> > > S3. We are keeping Kafka not as a source of truth, but as a mostly
> > > ephemeral broker that can come and go as it pleases. Be that scaling or
> > > outage. Disks can be destroyed and recreated at will, we still retain
> > data
> > > and use broker for just that, brokering messages. Not only that, we
> > reduced
> > > the requirement on the actual Kafka resources by reducing the size of a
> > > payload via a claim check pattern. Maybe this is an anti–pattern, but
> it
> > is
> > > super fast and highly cost efficient. We reworked ProducerRequest to
> > allow
> > > plugins. We added a custom http plugin that submits every request via a
> > > persisted connection to a microservice. Microservice stores the payload
> > and
> > > returns a tiny json metadata object,a claim check, that can be used to
> > find
> > > the actual data. Think of it as zipping the payload. This claim check
> > > metadata traverses the pipelines with consumers using the urls in
> > metadata
> > > to pull what they need. Think unzipping. This allowed us to also pull
> > ONLY
> > > the data that we need in graphql like manner. So if you have a 100K
> json
> > > payload and you need only a subsection, you can pull that by jmespath.
> > When
> > > you have multiple consumer groups yanking down huge payloads it is
> > > cumbersome on the broker. When you have the same consumer groups
> yanking
> > > down a claim check, and then going out of band directly to the source
> of
> > > truth, the broker has some breathing room. Obviously our microservice
> > does
> > > not go directly to the cloud storage, as that would be too slow. It
> > stores
> > > the payload in high speed memory cache and returns asap. That memory is
> > > eventually persisted into S3. The retrieval goest against the cache
> > first,
> > > then against the S3. Overall a rather cheappy and zippy solution. I
> tried
> > > proposing the KIP for this, but there was no excitement. Check this
> out:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> > >
> > >
> > > > On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID>
> wrote:
> > > >
> > > > Hi, Andrew,
> > > >
> > > > If we want to focus only on reducing cross-zone replication costs,
> > there
> > > is
> > > > an alternative design in the KIP-1176 discussion thread that seems
> > > simpler
> > > > than the proposal here. I am copying the outline of that approach
> > below.
> > > >
> > > > 1. A new leader is elected.
> > > > 2. Leader maintains a first tiered offset, which is initialized to
> log
> > > end
> > > > offset.
> > > > 3. Leader writes produced data from the client to local log.
> > > > 4. Leader uploads produced data from all local logs as a combined
> > object
> > > > 5. Leader stores the metadata for the combined object in memory.
> > > > 6. If a follower fetch request has an offset >= first tiered offset,
> > the
> > > > metadata for the corresponding combined object is returned.
> Otherwise,
> > > the
> > > > local data is returned.
> > > > 7. Leader periodically advances first tiered offset.
> > > >
> > > > It's still a bit unnatural, but it could work.
> > > >
> > > > Hi, Ivan,
> > > >
> > > > Are you still committed to proceeding with the original design of
> > > KIP-1150?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > > andrew_schofield@live.com>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >> I’ve been following KIP-1150 and friends for a while. I’m going to
> > jump
> > > >> into the discussions too.
> > > >>
> > > >> Looking back at Jack Vanlightly’s message, I am not quite so
> convinced
> > > >> that it’s a kind of fork in the road. The primary aim of the effort
> is
> > > to
> > > >> reduce cross-zone replication costs so Apache Kafka is not
> > prohibitively
> > > >> expensive to use on cloud storage. I think it would be entirely
> viable
> > > to
> > > >> prioritise code reuse for an initial implementation of diskless
> > topics,
> > > and
> > > >> we could still have a more cloud-native design in the future. It’s
> > hard
> > > to
> > > >> predict what the community will prioritise in the future.
> > > >>
> > > >> Of the three major revisions, I’m in the rev3 camp. We can support
> > > >> leaderless produce requests, first writing WAL segments into object
> > > >> storage, and then using the regular partition leaders to sequence
> the
> > > >> records. The active log segment for a diskless topic will initially
> > > contain
> > > >> batch coordinates rather than record batches. The batch coordinates
> > can
> > > be
> > > >> resolved from WAL segments for consumers, and also in order to
> prepare
> > > log
> > > >> segments for uploading to tiered storage. Jun is probably correct
> that
> > > we
> > > >> need a more frequent object merging process than tiered storage
> > > provides.
> > > >> This is just the transition from write-optimised WAL segments to
> > > >> read-optimised tiered segments, and all of the object storage-based
> > > >> implementations of Kafka that I’m aware of do this rearrangement.
> But
> > > >> perhaps this more frequent object merging is a pre-GA improvement,
> > > rather
> > > >> than a strict requirement for an initial implementation for early
> > access
> > > >> use.
> > > >>
> > > >> For zone-aligned share consumers, the share group assignor is
> intended
> > > to
> > > >> be rack-aware. Consumers should be assigned to partitions with
> leaders
> > > in
> > > >> their zone. The simple assignor is not rack-aware, but it easily
> could
> > > be
> > > >> or we could have a rack-aware assignor.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >>
> > > >>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID>
> wrote:
> > > >>>
> > > >>> Hi, Ivan,
> > > >>>
> > > >>> Thanks for the reply.
> > > >>>
> > > >>> "As I understand, you’re speaking about locally materialized
> > segments.
> > > >> They
> > > >>> will indeed consume some IOPS. See them as a cache that could
> always
> > be
> > > >>> restored from the remote storage. While it’s not ideal, it's still
> OK
> > > to
> > > >>> lose data in them due to a machine crash, for example. Because of
> > this,
> > > >> we
> > > >>> can avoid explicit flushing on local materialized segments at all
> and
> > > let
> > > >>> the file system and page cache figure out when to flush optimally.
> > This
> > > >>> would not eliminate the extra IOPS, but should reduce it
> > dramatically,
> > > >>> depending on throughput for each partition. We, of course, continue
> > > >>> flushing the metadata segments as before."
> > > >>>
> > > >>> If we have a mix of classic and diskless topics on the same broker,
> > > it's
> > > >>> important that the classic topics' data is flushed to disk as
> quickly
> > > as
> > > >>> possible. To achieve this, users typically set
> dirty_expire_centisecs
> > > in
> > > >>> the kernel based on the number of available disk IOPS. Once you set
> > > this
> > > >>> number, it applies to all dirty files, including the cached data in
> > > >>> diskless topics. So, if there are more files actively accumulating
> > > data,
> > > >>> the flush frequency and therefore RPO is reduced for classic
> topics.
> > > >>>
> > > >>> "We should have mentioned this explicitly, but this step, in fact,
> > > >> remains
> > > >>> in the form of segments offloading to tiered storage. When we
> > assemble
> > > a
> > > >>> segment and hand it over to RemoteLogManager, we’re effectively
> doing
> > > >>> metadata compaction: replacing a big number of pieces of metadata
> > about
> > > >>> individual batches with a single record in __remote_log_metadata."
> > > >>>
> > > >>> The object merging in tier storage typically only kicks in after a
> > few
> > > >>> hours. The impact is (1) the amount of accumulated metadata is
> still
> > > >> quite
> > > >>> large; (2) there are many small objects, leading to poor read
> > > >> performance.
> > > >>> I think we need a more frequent object merging process than tier
> > > storage
> > > >>> provides.
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>>
> > > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me>
> > > wrote:
> > > >>>
> > > >>>> Hello Jack, Jun, Luke, and all!
> > > >>>>
> > > >>>> Thank you for your messages.
> > > >>>>
> > > >>>> Let me first address some of Jun’s comments.
> > > >>>>
> > > >>>>> First, it degrades the durability.
> > > >>>>> For each partition, now there are two files being actively
> written
> > > at a
> > > >>>>> given point of time, one for the data and another for the
> metadata.
> > > >>>>> Flushing each file requires a separate IO. If the disk has 1K
> IOPS
> > > and
> > > >> we
> > > >>>>> have 5K partitions in a broker, currently we can afford to flush
> > each
> > > >>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we
> > > double
> > > >>>> the
> > > >>>>> number of files per partition, we can only flush each partition
> > every
> > > >> 10
> > > >>>>> seconds, which makes RPO twice as bad.
> > > >>>>
> > > >>>> As I understand, you’re speaking about locally materialized
> > segments.
> > > >> They
> > > >>>> will indeed consume some IOPS. See them as a cache that could
> always
> > > be
> > > >>>> restored from the remote storage. While it’s not ideal, it's still
> > OK
> > > to
> > > >>>> lose data in them due to a machine crash, for example. Because of
> > > this,
> > > >> we
> > > >>>> can avoid explicit flushing on local materialized segments at all
> > and
> > > >> let
> > > >>>> the file system and page cache figure out when to flush optimally.
> > > This
> > > >>>> would not eliminate the extra IOPS, but should reduce it
> > dramatically,
> > > >>>> depending on throughput for each partition. We, of course,
> continue
> > > >>>> flushing the metadata segments as before.
> > > >>>>
> > > >>>> It’s worth making a note on caching. I think nobody will disagree
> > that
> > > >>>> doing direct reads from remote storage every time a batch is
> > requested
> > > >> by a
> > > >>>> consumer will not be practical neither from the performance nor
> from
> > > the
> > > >>>> economy point of view. We need a way to keep the number of GET
> > > requests
> > > >>>> down. There are multiple options, for example:
> > > >>>> 1. Rack-aware distributed in-memory caching.
> > > >>>> 2. Local in-memory caching. Comes with less network chattiness and
> > > >> works
> > > >>>> well if we have more or less stable brokers to consume from.
> > > >>>> 3. Materialization of diskless logs on local disk. Way lower
> impact
> > on
> > > >>>> RAM and also requires stable brokers for consumption (using just
> > > >> assigned
> > > >>>> replicas will probably work well).
> > > >>>>
> > > >>>> Materialization is one of possible options, but we can choose
> > another
> > > >> one.
> > > >>>> However, we will have this dilemma regardless of whether we have
> an
> > > >>>> explicit coordinator or we go “coordinator-less”.
> > > >>>>
> > > >>>>> Second, if we ever need this
> > > >>>>> metadata somewhere else, say in the WAL file manager, the
> consumer
> > > >> needs
> > > >>>> to
> > > >>>>> subscribe to every partition in the cluster, which is
> inefficient.
> > > The
> > > >>>>> actual benefit of this approach is also questionable. On the
> > surface,
> > > >> it
> > > >>>>> might seem that we could reduce the number of lines that need to
> be
> > > >>>> changed
> > > >>>>> for this KIP. However, the changes are quite intrusive to the
> > classic
> > > >>>>> partition's code path and will probably make the code base harder
> > to
> > > >>>>> maintain in the long run. I like the original approach based on
> the
> > > >> batch
> > > >>>>> coordinator much better than this one. We could probably refactor
> > the
> > > >>>>> producer state code so that it could be reused in the batch
> > > >> coordinator.
> > > >>>>
> > > >>>> It’s hard to disagree with this. The explicit coordinator is more
> a
> > > side
> > > >>>> thing, while coordinator-less approach is more about extending
> > > >>>> ReplicaManager, UnifiedLog and others substantially.
> > > >>>>
> > > >>>>> Thanks for addressing the concerns on the number of RPCs in the
> > > produce
> > > >>>>> path. I agree that with the metadata crafting mechanism, we could
> > > >>>> mitigate
> > > >>>>> the PRC problem. However, since we now require the metadata to be
> > > >>>>> collocated with the data on the same set of brokers, it's weird
> > that
> > > >> they
> > > >>>>> are now managed by different mechanisms. The data assignment now
> > uses
> > > >> the
> > > >>>>> metadata crafting mechanism, but the metadata is stored in the
> > > classic
> > > >>>>> partition using its own assignment strategy. It will be
> complicated
> > > to
> > > >>>> keep
> > > >>>>> them collocated.
> > > >>>>
> > > >>>> I would like to note that the metadata crafting is needed only to
> > tell
> > > >>>> producers which brokers they should send Produce requests to, but
> > data
> > > >> (as
> > > >>>> in “locally materialized log”) is located on partition replicas,
> > i.e.
> > > >>>> automatically co-located with metadata.
> > > >>>>
> > > >>>> As a side note, it would probably be better that instead of
> > implicitly
> > > >>>> crafting partition metadata, we extend the metadata protocol so
> that
> > > for
> > > >>>> diskless partitions we return not only the leader and replicas,
> but
> > > also
> > > >>>> some “recommended produce brokers”, selected for optimal
> performance
> > > and
> > > >>>> costs. Producers will pick ones in their racks.
> > > >>>>
> > > >>>>> I am also concerned about the removal of the object
> > > compaction/merging
> > > >>>>> step.
> > > >>>>
> > > >>>> We should have mentioned this explicitly, but this step, in fact,
> > > >> remains
> > > >>>> in the form of segments offloading to tiered storage. When we
> > > assemble a
> > > >>>> segment and hand it over to RemoteLogManager, we’re effectively
> > doing
> > > >>>> metadata compaction: replacing a big number of pieces of metadata
> > > about
> > > >>>> individual batches with a single record in __remote_log_metadata.
> > > >>>>
> > > >>>> We could create a Diskless-specific merging mechanism instead if
> > > needed.
> > > >>>> It’s rather easy with the explicit coordinator approach. With the
> > > >>>> coordinator-less approach, this would probably be a bit more
> tricky
> > > >>>> (rewriting the tail of the log by the leader + replicating this
> > change
> > > >>>> reliably).
> > > >>>>
> > > >>>>> I see a tendency toward primarily optimizing for the fewest code
> > > >> changes
> > > >>>> in
> > > >>>>> the KIP. Instead, our primary goal should be a clean design that
> > can
> > > >> last
> > > >>>>> for the long term.
> > > >>>>
> > > >>>> Yes, totally agree.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Luke,
> > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > because
> > > >> of
> > > >>>>> leaderless cluster, do you think it will be simpler if we only
> > focus
> > > on
> > > >>>> the
> > > >>>>> "diskless" design to handle object compaction/merging to/from the
> > > >> remote
> > > >>>>> storage to save the cross-AZ cost?
> > > >>>>
> > > >>>> After some evolution of the original proposal, leaderless is now
> > > >> limited.
> > > >>>> We only need to be able to accept Produce requests on more than
> one
> > > >> broker
> > > >>>> to eliminate the cross-AZ costs for producers. Do I get it right
> > that
> > > >> you
> > > >>>> propose to get rid of this? Or do I misunderstand?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Let’s now look at this problem from a higher level, as Jack
> > proposed.
> > > As
> > > >>>> it was said, the big choice we need to make is whether we 1)
> create
> > an
> > > >>>> explicit batch coordinator; or 2) go for the coordinator-less
> > > approach,
> > > >>>> where each diskless partition is managed by its leader as in
> classic
> > > >> topics.
> > > >>>>
> > > >>>> If we try to compare the two approaches:
> > > >>>>
> > > >>>> Pluggability:
> > > >>>> - Explicit coordinator: Possible. For example, some setups may
> > benefit
> > > >>>> from batch metadata being stored in a cloud database (such as AWS
> > > >> DynamoDB
> > > >>>> or GCP Spanner).
> > > >>>> - Coordinator-less: Impossible.
> > > >>>>
> > > >>>> Scalability and fault tolerance:
> > > >>>> - Explicit coordinator: Depends on the implementation and it’s
> also
> > > >>>> necessary to actively work for it.
> > > >>>> - Coordinator-less: Closer to classic Kafka topics. Scaling is
> done
> > by
> > > >>>> partition placement, partitions could fail independently.
> > > >>>>
> > > >>>> Separation of concerns:
> > > >>>> - Explicit coordinator: Very good. Diskless remains more
> independent
> > > >> from
> > > >>>> classic topics in terms of code and workflows. For example, the
> > > >>>> above-mentioned non-tiered storage metadata compaction mechanism
> > could
> > > >> be
> > > >>>> relatively simply implemented with it. As a flip side of this,
> some
> > > >>>> workflows (e.g. transactions) will have to be adapted.
> > > >>>> - Coordinator-less: Less so. It leans to the opposite: bringing
> > > diskless
> > > >>>> closer to classic topics. Some code paths and workflows could be
> > more
> > > >>>> straightforwardly reused, but they will inevitably have to be
> > adapted
> > > to
> > > >>>> accommodate both topic types as also discussed.
> > > >>>>
> > > >>>> Cloud-nativeness. This is a vague concept, also related to the
> > > previous,
> > > >>>> but let’s try:
> > > >>>> - Explicit coordinator: Storing and processing metadata separately
> > > makes
> > > >>>> it easier for brokers to take different roles, be purely stateless
> > if
> > > >>>> needed, etc.
> > > >>>> - Coordinator-less: Less so. Something could be achieved with
> > creative
> > > >>>> partition placement, but not much.
> > > >>>>
> > > >>>> Both seem to have their pros and cons. However, answering Jack’s
> > > >> question,
> > > >>>> the explicit coordinator approach may indeed lead to a more
> flexible
> > > >> design.
> > > >>>>
> > > >>>>
> > > >>>> The purpose of this deviation in the discussion was to receive a
> > > >>>> preliminary community evaluation of the coordinator-less approach
> > > >> without
> > > >>>> taking on the task of writing a separate KIP and fitting it in the
> > > >> system
> > > >>>> of KIP-1150 and its children. We’re open to stopping it and
> getting
> > > >> back to
> > > >>>> working out the coordinator design if the community doesn’t favor
> > the
> > > >>>> proposed approach.
> > > >>>>
> > > >>>> Best,
> > > >>>> Ivan and Diskless team
> > > >>>>
> > > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > > >>>>> Hi Ivan,
> > > >>>>>
> > > >>>>> As Jun pointed out, the updated design seems to have some
> > > shortcomings
> > > >>>>> although it simplifies the implementation.
> > > >>>>>
> > > >>>>> I'm wondering if the complexity of designing txn and queue is
> > because
> > > >> of
> > > >>>>> leaderless cluster, do you think it will be simpler if we only
> > focus
> > > on
> > > >>>> the
> > > >>>>> "diskless" design to handle object compaction/merging to/from the
> > > >> remote
> > > >>>>> storage to save the cross-AZ cost?
> > > >>>>>
> > > >>>>>
> > > >>>>> Thank you,
> > > >>>>> Luke
> > > >>>>>
> > > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <jun@confluent.io.invalid
> >
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> Hi, Ivan,
> > > >>>>>>
> > > >>>>>> Thanks for the explanation.
> > > >>>>>>
> > > >>>>>> "we write the reference to the WAL file with the batch data"
> > > >>>>>>
> > > >>>>>> I understand the approach now, but I think it is a hacky one.
> > There
> > > >> are
> > > >>>>>> multiple short comings with this design. First, it degrades the
> > > >>>> durability.
> > > >>>>>> For each partition, now there are two files being actively
> written
> > > at
> > > >> a
> > > >>>>>> given point of time, one for the data and another for the
> > metadata.
> > > >>>>>> Flushing each file requires a separate IO. If the disk has 1K
> IOPS
> > > and
> > > >>>> we
> > > >>>>>> have 5K partitions in a broker, currently we can afford to flush
> > > each
> > > >>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we
> > > double
> > > >>>> the
> > > >>>>>> number of files per partition, we can only flush each partition
> > > every
> > > >>>> 10
> > > >>>>>> seconds, which makes RPO twice as bad. Second, if we ever need
> > this
> > > >>>>>> metadata somewhere else, say in the WAL file manager, the
> consumer
> > > >>>> needs to
> > > >>>>>> subscribe to every partition in the cluster, which is
> inefficient.
> > > The
> > > >>>>>> actual benefit of this approach is also questionable. On the
> > > surface,
> > > >>>> it
> > > >>>>>> might seem that we could reduce the number of lines that need to
> > be
> > > >>>> changed
> > > >>>>>> for this KIP. However, the changes are quite intrusive to the
> > > classic
> > > >>>>>> partition's code path and will probably make the code base
> harder
> > to
> > > >>>>>> maintain in the long run. I like the original approach based on
> > the
> > > >>>> batch
> > > >>>>>> coordinator much better than this one. We could probably
> refactor
> > > the
> > > >>>>>> producer state code so that it could be reused in the batch
> > > >>>> coordinator.
> > > >>>>>>
> > > >>>>>> Thanks for addressing the concerns on the number of RPCs in the
> > > >> produce
> > > >>>>>> path. I agree that with the metadata crafting mechanism, we
> could
> > > >>>> mitigate
> > > >>>>>> the PRC problem. However, since we now require the metadata to
> be
> > > >>>>>> collocated with the data on the same set of brokers, it's weird
> > that
> > > >>>> they
> > > >>>>>> are now managed by different mechanisms. The data assignment now
> > > uses
> > > >>>> the
> > > >>>>>> metadata crafting mechanism, but the metadata is stored in the
> > > classic
> > > >>>>>> partition using its own assignment strategy. It will be
> > complicated
> > > to
> > > >>>> keep
> > > >>>>>> them collocated.
> > > >>>>>>
> > > >>>>>> I am also concerned about the removal of the object
> > > compaction/merging
> > > >>>>>> step. My first concern is on the amount of metadata that need to
> > be
> > > >>>> kept.
> > > >>>>>> Without object compcation, the metadata generated in the produce
> > > path
> > > >>>> can
> > > >>>>>> only be deleted after remote tiering kicks in. Let's say for
> every
> > > >>>> 250ms we
> > > >>>>>> produce 100 byte of metadata per partition. Let's say remoting
> > > tiering
> > > >>>>>> kicks in after 5 hours. In a cluster with 100K partitions, we
> need
> > > to
> > > >>>> keep
> > > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata,
> quite
> > > >>>>>> signficant. A second concern is on performance. Every time we
> need
> > > to
> > > >>>>>> rebuild the caching data, we need to read a bunch of small
> objects
> > > >>>> from S3,
> > > >>>>>> slowing down the building process. If a consumer happens to need
> > > such
> > > >>>> data,
> > > >>>>>> it could slow down the application.
> > > >>>>>>
> > > >>>>>> I see a tendency toward primarily optimizing for the fewest code
> > > >>>> changes in
> > > >>>>>> the KIP. Instead, our primary goal should be a clean design that
> > can
> > > >>>> last
> > > >>>>>> for the long term.
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>>
> > > >>>>>> Jun
> > > >>>>>>
> > > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <ivan@ivanyu.me
> >
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi Jun,
> > > >>>>>>>
> > > >>>>>>> Thank you for your message. I’m sorry that I failed to clearly
> > > >>>> explain
> > > >>>>>> the
> > > >>>>>>> idea. Let me try to fix this.
> > > >>>>>>>
> > > >>>>>>>> Does each partition now have a metadata partition and a
> separate
> > > >>>> data
> > > >>>>>>>> partition? If so, I am concerned that it essentially doubles
> the
> > > >>>> number
> > > >>>>>>> of
> > > >>>>>>>> partitions, which impacts the number of open file descriptors
> > and
> > > >>>> the
> > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > separate
> > > >>>>>>>> partition just to store the metadata. It's as if we are
> creating
> > > an
> > > >>>>>>>> internal topic with an unbounded number of partitions.
> > > >>>>>>>
> > > >>>>>>> No. There will be only one physical partition per diskless
> > > >>>> partition. Let
> > > >>>>>>> me explain this with an example. Let’s say we have a diskless
> > > >>>> partition
> > > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We
> > produce
> > > >>>> some
> > > >>>>>>> batches to this partition. The content of the segment file will
> > be
> > > >>>>>>> something like this (for each batch):
> > > >>>>>>>
> > > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > > >>>>>>> Length: 123456 (like in classic)
> > > >>>>>>> PartitionLeaderEpoch: like in classic
> > > >>>>>>> Magic: like in classic
> > > >>>>>>> CRC: like in classic
> > > >>>>>>> Attributes: like in classic
> > > >>>>>>> LastOffsetDelta: like in classic
> > > >>>>>>> BaseTimestamp: like in classic
> > > >>>>>>> MaxTimestamp: like in classic
> > > >>>>>>> ProducerId: like in classic
> > > >>>>>>> ProducerEpoch: like in classic
> > > >>>>>>> BaseSequence: like in classic
> > > >>>>>>> RecordsCount: like in classic
> > > >>>>>>> Records:
> > > >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte
> > offset
> > > >>>>>>> 123456
> > > >>>>>>>
> > > >>>>>>> It looks very much like classic log entries. The only
> difference
> > is
> > > >>>> that
> > > >>>>>>> instead of writing real Records, we write the reference to the
> > WAL
> > > >>>> file
> > > >>>>>>> with the batch data (I guess we need only the name and the byte
> > > >>>> offset,
> > > >>>>>>> because the byte length is the standard field above).
> Otherwise,
> > > >>>> it’s a
> > > >>>>>>> normal Kafka log with the leader and replicas.
> > > >>>>>>>
> > > >>>>>>> So we have as many partitions for diskless as for classic. As
> of
> > > open
> > > >>>>>> file
> > > >>>>>>> descriptors, let’s proceed to the following:
> > > >>>>>>>
> > > >>>>>>>> Are the metadata and
> > > >>>>>>>> the data for the same partition always collocated on the same
> > > >>>> broker?
> > > >>>>>> If
> > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > >>>>>>>
> > > >>>>>>> The source of truth for the data is still in WAL files on
> object
> > > >>>> storage.
> > > >>>>>>> The source of truth for the metadata is in segment files on the
> > > >>>> brokers
> > > >>>>>> in
> > > >>>>>>> the replica set. Two new mechanisms are planned, both
> independent
> > > of
> > > >>>> this
> > > >>>>>>> new proposal, but I want to present them to give the idea that
> > > only a
> > > >>>>>>> limited amount of data files will be operated locally:
> > > >>>>>>> - We want to assemble batches into segment files and offload
> them
> > > to
> > > >>>>>>> tiered storage in order to prevent the unbounded growth of
> batch
> > > >>>>>> metadata.
> > > >>>>>>> For this, we need to open only  a few file descriptors (for the
> > > >>>> segment
> > > >>>>>>> file itself + the necessary indexes) before the segment is
> fully
> > > >>>> written
> > > >>>>>>> and handed over to RemoteLogManager.
> > > >>>>>>> - We want to assemble local segment files for caching purposes
> as
> > > >>>> well,
> > > >>>>>>> i.e. to speed up fetching. This will not materialize the full
> > > >>>> content of
> > > >>>>>>> the log, but only the hot set according to some policy (or
> > > >>>> configurable
> > > >>>>>>> policies), i.e. the number of segments and file descriptors
> will
> > > >>>> also be
> > > >>>>>>> limited.
> > > >>>>>>>
> > > >>>>>>>> The number of RPCs in the produce path is significantly
> higher.
> > > For
> > > >>>>>>>> example, if a produce request has 100 partitions, in a cluster
> > > >>>> with 100
> > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > requests.
> > > >>>>>> This
> > > >>>>>>>> will significantly increase the request rate.
> > > >>>>>>>
> > > >>>>>>> This is a valid concern that we considered, but this issue can
> be
> > > >>>>>>> mitigated. I’ll try to explain the approach.
> > > >>>>>>> The situation with a single broker is trivial: all the commit
> > > >>>> requests go
> > > >>>>>>> from the broker to itself.
> > > >>>>>>> Let’s scale this to a multi-broker cluster, but located in the
> > > single
> > > >>>>>> rack
> > > >>>>>>> (AZ). Any broker can accept Produce requests for diskless
> > > >>>> partitions, but
> > > >>>>>>> we can tell producers (through metadata) to always send Produce
> > > >>>> requests
> > > >>>>>> to
> > > >>>>>>> leaders. For example, broker 0 hosts the leader replicas for
> > > diskless
> > > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> > > >>>> requests
> > > >>>>>> for
> > > >>>>>>> these partitions in various combinations, but only for them.
> > > >>>>>>>
> > > >>>>>>>                   Broker 0
> > > >>>>>>>             +-----------------+
> > > >>>>>>>             |    t1-0         |
> > > >>>>>>>             |    t2-1 <--------------------+
> > > >>>>>>>             |    t3-0         |            |
> > > >>>>>>> produce      | +-------------+ |            |
> > > >>>>>>> requests     | |  diskless   | |            |
> > > >>>>>>> --------------->|   produce   +--------------+
> > > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > > >>>>>>> partitions   | +-------------+ |    for these partitions
> > > >>>>>>>             |                 |
> > > >>>>>>>             +-----------------+
> > > >>>>>>>
> > > >>>>>>> The same applies for other brokers in this cluster.
> Effectively,
> > > each
> > > >>>>>>> broker will commit only to itself, which effectively means 1
> > commit
> > > >>>>>> request
> > > >>>>>>> per WAL buffer (this may be 0 physical network calls, if we
> wish,
> > > >>>> just a
> > > >>>>>>> local function call).
> > > >>>>>>>
> > > >>>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we
> > cannot
> > > >>>> always
> > > >>>>>>> send Produce requests to the designated leaders of diskless
> > > >>>> partitions:
> > > >>>>>>> this would mean inter-AZ network traffic, which we would like
> to
> > > >>>> avoid.
> > > >>>>>> To
> > > >>>>>>> avoid it, we say that every broker has a “diskless produce
> > > >>>>>> representative”
> > > >>>>>>> in every AZ. If we continue our example: when a Produce request
> > for
> > > >>>> t1-0,
> > > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on
> broker 0
> > > >>>> (in the
> > > >>>>>>> broker’s AZ the representative is the broker itself). However,
> if
> > > it
> > > >>>>>> comes
> > > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > > >>>>>>>
> > > >>>>>>> |produce requests         |produce requests        |produce
> > > >>>> requests
> > > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0,
> > t2-1,
> > > >>>>>> t3-0
> > > >>>>>>> |from AZ 0                |from AZ 1               |from AZ 2
> > > >>>>>>> v                         v                        v
> > > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > > >>>>>>> +---------------+      +---------------+      +---------------+
> > > >>>>>>> |     t1-0      |      |               |      |               |
> > > >>>>>>> |     t2-1      |      |               |      |               |
> > > >>>>>>> |     t3-0      |      |               |      |               |
> > > >>>>>>> +---------------+      +--------+------+      +--------+------+
> > > >>>>>>>    ^     ^                    |                      |
> > > >>>>>>>    |     +--------------------+                      |
> > > >>>>>>>    |     commit requests for these partitions        |
> > > >>>>>>>    |                                                 |
> > > >>>>>>>    +-------------------------------------------------+
> > > >>>>>>>          commit requests for these partitions
> > > >>>>>>>
> > > >>>>>>> All the partitions that broker 0 is the leader of will be
> > > >>>> “represented”
> > > >>>>>> by
> > > >>>>>>> brokers 1 and 2 in their AZs.
> > > >>>>>>>
> > > >>>>>>> Of course, this relationship goes both ways between AZs (not
> > > >>>> necessarily
> > > >>>>>>> between the same brokers). It means that provided the cluster
> is
> > > >>>> balanced
> > > >>>>>>> by the number of brokers per AZ, each broker will represent
> > > >>>>>> (number_of_azs
> > > >>>>>>> - 1) other brokers. This will result in the situation that for
> > the
> > > >>>>>> majority
> > > >>>>>>> of commits, each broker will do up to (number_of_azs - 1)
> network
> > > >>>> commit
> > > >>>>>>> requests (plus one local). Cloud regions tend to have 3 AZs,
> very
> > > >>>> rarely
> > > >>>>>>> more. That means, brokers will be doing up to 2 network commit
> > > >>>> requests
> > > >>>>>> per
> > > >>>>>>> WAL file.
> > > >>>>>>>
> > > >>>>>>> There are the following exceptions:
> > > >>>>>>> 1. Broker count imbalance between AZs. For example, when we
> have
> > 2
> > > >>>> AZs
> > > >>>>>> and
> > > >>>>>>> one has three brokers and another AZ has one. This one broker
> > will
> > > do
> > > >>>>>>> between 1 and 3 commit requests per WAL file. This is not an
> > > extreme
> > > >>>>>>> amplification. Such an imbalance is not healthy in most
> practical
> > > >>>> setups
> > > >>>>>>> and should be avoided anyway.
> > > >>>>>>> 2. Leadership changes and metadata propagation period. When the
> > > >>>> partition
> > > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the producers
> > > will
> > > >>>> not
> > > >>>>>>> know this immediately (unless we want to be strict and respond
> > with
> > > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
> > > >>>> together
> > > >>>>>> in a
> > > >>>>>>> WAL buffer on broker 2, it will have to send two commit
> requests:
> > > to
> > > >>>>>> broker
> > > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
> > > >>>> situation
> > > >>>>>>> is not permanent and as producers update the cluster metadata,
> it
> > > >>>> will be
> > > >>>>>>> resolved.
> > > >>>>>>>
> > > >>>>>>> This all could be built with the metadata crafting mechanism
> only
> > > >>>> (which
> > > >>>>>>> is anyway needed for Diskless in one way or another to direct
> > > >>>> producers
> > > >>>>>> and
> > > >>>>>>> consumers where we need to avoid inter-AZ traffic), just with
> the
> > > >>>> right
> > > >>>>>>> policy for it (for example, some deterministic hash-based
> > formula).
> > > >>>> I.e.
> > > >>>>>> no
> > > >>>>>>> explicit support for “produce representative” or anything like
> > this
> > > >>>> is
> > > >>>>>>> needed on the cluster level, in KRaft, etc.
> > > >>>>>>>
> > > >>>>>>>> The same WAL file metadata is now duplicated into two places,
> > > >>>> partition
> > > >>>>>>>> leader and WAL File Manager. Which one is the source of truth,
> > and
> > > >>>> how
> > > >>>>>> do
> > > >>>>>>>> we maintain consistency between the two places?
> > > >>>>>>>
> > > >>>>>>> We do only two operations on WAL files that span multiple
> > diskless
> > > >>>>>>> partitions: committing and deleting. Commits can be done
> > > >>>> independently as
> > > >>>>>>> described above. But deletes are different, because when a file
> > is
> > > >>>>>> deleted,
> > > >>>>>>> this affects all the partitions that still have alive batches
> in
> > > this
> > > >>>>>> file
> > > >>>>>>> (if any).
> > > >>>>>>>
> > > >>>>>>> The WAL file manager is a necessary point of coordination to
> > delete
> > > >>>> WAL
> > > >>>>>>> files safely. We can say it is the source of truth about files
> > > >>>>>> themselves,
> > > >>>>>>> while the partition leaders and their logs hold the truth about
> > > >>>> whether a
> > > >>>>>>> particular file contains live batches of this particular
> > partition.
> > > >>>>>>>
> > > >>>>>>> The file manager will do this important task: be able to say
> for
> > > sure
> > > >>>>>> that
> > > >>>>>>> a file does not contain any live batch of any existing
> partition.
> > > For
> > > >>>>>> this,
> > > >>>>>>> it will have to periodically check against the partition
> leaders.
> > > >>>>>>> Considering that batch deletion is irreversible, when we
> declare
> > a
> > > >>>> file
> > > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > > >>>>>>>
> > > >>>>>>> The file manager has to know about files being committed to
> start
> > > >>>> track
> > > >>>>>>> them and periodically check if they are empty. We can consider
> > > >>>> various
> > > >>>>>> ways
> > > >>>>>>> to achieve this:
> > > >>>>>>> 1. As was proposed in my previous message: best effort commit
> by
> > > >>>> brokers
> > > >>>>>> +
> > > >>>>>>> periodic prefix scans of object storage to detect files that
> went
> > > >>>> below
> > > >>>>>> the
> > > >>>>>>> radar due to network issue or the file manager temporary
> > > >>>> unavailability.
> > > >>>>>>> We’re speaking about listing the file names only and opening
> only
> > > >>>>>>> previously unknown files in order to find the partitions
> involved
> > > >>>> with
> > > >>>>>> them.
> > > >>>>>>> 2. Only do scans without explicit commit, i.e. fill the list of
> > > files
> > > >>>>>>> fully asynchronously and in the background. This may be not
> ideal
> > > >>>> due to
> > > >>>>>>> costs and performance of scanning tons of files. However, the
> > > number
> > > >>>> of
> > > >>>>>>> live WAL files should be limited due to tiered storage
> > offloading +
> > > >>>> we
> > > >>>>>> can
> > > >>>>>>> optimize this if we give files some global soft order in their
> > > names.
> > > >>>>>>>
> > > >>>>>>>> I am not sure how this design simplifies the implementation.
> The
> > > >>>>>> existing
> > > >>>>>>>> producer/replication code can't be simply reused. Adjusting
> both
> > > >>>> the
> > > >>>>>>> write
> > > >>>>>>>> path in the leader and the replication path in the follower to
> > > >>>>>> understand
> > > >>>>>>>> batch-header only data is quite intrusive to the existing
> logic.
> > > >>>>>>>
> > > >>>>>>> It is true that we’ll have to change LocalLog and UnifiedLog in
> > > >>>> order to
> > > >>>>>>> support these changes. However, it seems that idempotence,
> > > >>>> transactions,
> > > >>>>>>> queues, tiered storage will have to be changed less than with
> the
> > > >>>>>> original
> > > >>>>>>> design. This is because the partition leader state would remain
> > in
> > > >>>> the
> > > >>>>>> same
> > > >>>>>>> place (on brokers) and existing workflows that involve it would
> > > have
> > > >>>> to
> > > >>>>>> be
> > > >>>>>>> changed less compared to the situation where we globalize the
> > > >>>> partition
> > > >>>>>>> leader state in the batch coordinator. I admit this is hard to
> > make
> > > >>>>>>> convincing without both real implementations to hand :)
> > > >>>>>>>
> > > >>>>>>>> I am also
> > > >>>>>>>> not sure how this enables seamless switching the topic modes
> > > >>>> between
> > > >>>>>>>> diskless and classic. Could you provide more details on those?
> > > >>>>>>>
> > > >>>>>>> Let’s consider the scenario of turning a classic topic into
> > > >>>> diskless. The
> > > >>>>>>> user sets diskless.enabled=true, the leader receives this
> > metadata
> > > >>>> update
> > > >>>>>>> and does the following:
> > > >>>>>>> 1. Stop accepting normal append writes.
> > > >>>>>>> 2. Close the current active segment.
> > > >>>>>>> 3. Start a new segment that will be written in the diskless
> > format
> > > >>>> (i.e.
> > > >>>>>>> without data).
> > > >>>>>>> 4. Start accepting diskless commits.
> > > >>>>>>>
> > > >>>>>>> Since it’s the same log, the followers will know about that
> > switch
> > > >>>>>>> consistently. They will finish replicating the classic segments
> > and
> > > >>>> start
> > > >>>>>>> replicating the diskless ones. They will always know where each
> > > >>>> batch is
> > > >>>>>>> located (either inside a classic segment or referenced by a
> > > diskless
> > > >>>>>> one).
> > > >>>>>>> Switching back should be similar.
> > > >>>>>>>
> > > >>>>>>> Doing this with the coordinator is possible, but has some
> > caveats.
> > > >>>> The
> > > >>>>>>> leader must do the following:
> > > >>>>>>> 1. Stop accepting normal append writes.
> > > >>>>>>> 2. Close the current active segment.
> > > >>>>>>> 3. Write a special control segment to persist and replicate the
> > > fact
> > > >>>> that
> > > >>>>>>> from offset N the partition is now in the diskless mode.
> > > >>>>>>> 4. Inform the coordinator about the first offset N of the
> > “diskless
> > > >>>> era”.
> > > >>>>>>> 5. Inform the controller quorum that the transition has
> finished
> > > and
> > > >>>> that
> > > >>>>>>> brokers now can process diskless writes for this partition.
> > > >>>>>>> This could fail at some points, so this will probably require
> > some
> > > >>>>>>> explicit state machine with replication either in the partition
> > log
> > > >>>> or in
> > > >>>>>>> KRaft.
> > > >>>>>>>
> > > >>>>>>> It seems that the coordinator-less approach makes this simpler
> > > >>>> because
> > > >>>>>> the
> > > >>>>>>> “coordinator” for the partition and the partition leader are
> the
> > > >>>> same and
> > > >>>>>>> they store the partition metadata in the same log, too. While
> in
> > > the
> > > >>>>>>> coordinator approach we have to perform some kind of a
> > distributed
> > > >>>> commit
> > > >>>>>>> to handover metadata management from the classic partition
> leader
> > > to
> > > >>>> the
> > > >>>>>>> batch coordinator.
> > > >>>>>>>
> > > >>>>>>> I hope these explanations help to clarify the idea. Please let
> me
> > > >>>> know if
> > > >>>>>>> I should go deeper anywhere.
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Ivan and the Diskless team
> > > >>>>>>>
> > > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > >>>>>>>> Hi, Ivan,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the update.
> > > >>>>>>>>
> > > >>>>>>>> I am not sure that I fully understand the new design, but it
> > seems
> > > >>>> less
> > > >>>>>>>> clean than before.
> > > >>>>>>>>
> > > >>>>>>>> Does each partition now have a metadata partition and a
> separate
> > > >>>> data
> > > >>>>>>>> partition? If so, I am concerned that it essentially doubles
> the
> > > >>>> number
> > > >>>>>>> of
> > > >>>>>>>> partitions, which impacts the number of open file descriptors
> > and
> > > >>>> the
> > > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > > separate
> > > >>>>>>>> partition just to store the metadata. It's as if we are
> creating
> > > an
> > > >>>>>>>> internal topic with an unbounded number of partitions. Are the
> > > >>>> metadata
> > > >>>>>>> and
> > > >>>>>>>> the data for the same partition always collocated on the same
> > > >>>> broker?
> > > >>>>>> If
> > > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > > >>>>>>>>
> > > >>>>>>>> The number of RPCs in the produce path is significantly
> higher.
> > > For
> > > >>>>>>>> example, if a produce request has 100 partitions, in a cluster
> > > >>>> with 100
> > > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > > requests.
> > > >>>>>> This
> > > >>>>>>>> will significantly increase the request rate.
> > > >>>>>>>>
> > > >>>>>>>> The same WAL file metadata is now duplicated into two places,
> > > >>>> partition
> > > >>>>>>>> leader and WAL File Manager. Which one is the source of truth,
> > and
> > > >>>> how
> > > >>>>>> do
> > > >>>>>>>> we maintain consistency between the two places?
> > > >>>>>>>>
> > > >>>>>>>> I am not sure how this design simplifies the implementation.
> The
> > > >>>>>> existing
> > > >>>>>>>> producer/replication code can't be simply reused. Adjusting
> both
> > > >>>> the
> > > >>>>>>> write
> > > >>>>>>>> path in the leader and the replication path in the follower to
> > > >>>>>> understand
> > > >>>>>>>> batch-header only data is quite intrusive to the existing
> > logic. I
> > > >>>> am
> > > >>>>>>> also
> > > >>>>>>>> not sure how this enables seamless switching the topic modes
> > > >>>> between
> > > >>>>>>>> diskless and classic. Could you provide more details on those?
> > > >>>>>>>>
> > > >>>>>>>> Jun
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <ivan@ivanyu.me
> >
> > > >>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi dear Kafka community,
> > > >>>>>>>>>
> > > >>>>>>>>> In the initial Diskless proposal, we proposed to have a
> > separate
> > > >>>>>>>>> component, batch/diskless coordinator, whose role would be to
> > > >>>>>> centrally
> > > >>>>>>>>> manage the batch and WAL file metadata for diskless topics.
> > This
> > > >>>>>>> component
> > > >>>>>>>>> drew many reasonable comments from the community about how it
> > > >>>> would
> > > >>>>>>> support
> > > >>>>>>>>> various Kafka features (transactions, queues) and its
> > > >>>> scalability.
> > > >>>>>>> While we
> > > >>>>>>>>> believe we have good answers to all the expressed concerns,
> we
> > > >>>> took a
> > > >>>>>>> step
> > > >>>>>>>>> back and looked at the problem from a different perspective.
> > > >>>>>>>>>
> > > >>>>>>>>> We would like to propose an alternative Diskless design
> > *without
> > > >>>> a
> > > >>>>>>>>> centralized coordinator*. We believe this approach has
> > potential
> > > >>>> and
> > > >>>>>>>>> propose to discuss it as it may be more appealing to the
> > > >>>> community.
> > > >>>>>>>>>
> > > >>>>>>>>> Let us explain the idea. Most of the complications with the
> > > >>>> original
> > > >>>>>>>>> Diskless approach come from one necessary architecture
> change:
> > > >>>>>>> globalizing
> > > >>>>>>>>> the local state of partition leader in the batch coordinator.
> > > >>>> This
> > > >>>>>>> causes
> > > >>>>>>>>> deviations to the established workflows in various features
> > like
> > > >>>>>>> produce
> > > >>>>>>>>> idempotence and transactions, queues, retention, etc. These
> > > >>>>>> deviations
> > > >>>>>>> need
> > > >>>>>>>>> to be carefully considered, designed, and later implemented
> and
> > > >>>>>>> tested. In
> > > >>>>>>>>> the new approach we want to avoid this by making partition
> > > >>>> leaders
> > > >>>>>>> again
> > > >>>>>>>>> responsible for managing their partitions, even in diskless
> > > >>>> topics.
> > > >>>>>>>>>
> > > >>>>>>>>> In classic Kafka topics, batch data and metadata are blended
> > > >>>> together
> > > >>>>>>> in
> > > >>>>>>>>> the one partition log. The crux of the Diskless idea is to
> > > >>>> decouple
> > > >>>>>>> them
> > > >>>>>>>>> and move data to the remote storage, while keeping metadata
> > > >>>> somewhere
> > > >>>>>>> else.
> > > >>>>>>>>> Using the central batch coordinator for managing batch
> metadata
> > > >>>> is
> > > >>>>>> one
> > > >>>>>>> way,
> > > >>>>>>>>> but not the only.
> > > >>>>>>>>>
> > > >>>>>>>>> Let’s now think about managing metadata for each user
> partition
> > > >>>>>>>>> independently. Generally partitions are independent and don’t
> > > >>>> share
> > > >>>>>>>>> anything apart from that their data are mixed in WAL files.
> If
> > we
> > > >>>>>>> figure
> > > >>>>>>>>> out how to commit and later delete WAL files safely, we will
> > > >>>> achieve
> > > >>>>>>> the
> > > >>>>>>>>> necessary autonomy that allows us to get rid of the central
> > batch
> > > >>>>>>>>> coordinator. Instead, *each diskless user partition will be
> > > >>>> managed
> > > >>>>>> by
> > > >>>>>>> its
> > > >>>>>>>>> leader*, as in classic Kafka topics. Also like in classic
> > > >>>> topics, the
> > > >>>>>>>>> leader uses the partition log as the way to persist batch
> > > >>>> metadata,
> > > >>>>>>> i.e.
> > > >>>>>>>>> the regular batch header + the information about how to find
> > this
> > > >>>>>>> batch on
> > > >>>>>>>>> remote storage. In contrast to classic topics, batch data is
> in
> > > >>>>>> remote
> > > >>>>>>>>> storage.
> > > >>>>>>>>>
> > > >>>>>>>>> For clarity, let’s compare the three designs:
> > > >>>>>>>>> • Classic topics:
> > > >>>>>>>>>  • Data and metadata are co-located in the partition log.
> > > >>>>>>>>>  • The partition log content: [Batch header (metadata)|Batch
> > > >>>> data].
> > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > >>>>>>>>>  • The replicas and leader have local state built from
> > > >>>> metadata.
> > > >>>>>>>>> • Original Diskless:
> > > >>>>>>>>>  • Metadata is in the batch coordinator, data is on remote
> > > >>>> storage.
> > > >>>>>>>>>  • The partition state is global in the batch coordinator.
> > > >>>>>>>>> • New Diskless:
> > > >>>>>>>>>  • Metadata is in the partition log, data is on remote
> storage.
> > > >>>>>>>>>  • Partition log content: [Batch header (metadata)|Batch
> > > >>>>>> coordinates
> > > >>>>>>> on
> > > >>>>>>>>> remote storage].
> > > >>>>>>>>>  • The partition log is replicated to the followers.
> > > >>>>>>>>>  • The replicas and leader have local state built from
> > > >>>> metadata.
> > > >>>>>>>>>
> > > >>>>>>>>> Let’s consider the produce path. Here’s the reminder of the
> > > >>>> original
> > > >>>>>>>>> Diskless design:
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> The new approach could be depicted as the following:
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> As you can see, the main difference is that now instead of a
> > > >>>> single
> > > >>>>>>> commit
> > > >>>>>>>>> request to the batch coordinator, we send multiple parallel
> > > >>>> commit
> > > >>>>>>> requests
> > > >>>>>>>>> to all the leaders of each partition involved in the WAL
> file.
> > > >>>> Each
> > > >>>>>> of
> > > >>>>>>> them
> > > >>>>>>>>> will commit its batches independently, without coordinating
> > with
> > > >>>>>> other
> > > >>>>>>>>> leaders and any other components. Batch data is addressed by
> > the
> > > >>>> WAL
> > > >>>>>>> file
> > > >>>>>>>>> name, the byte offset and size, which allows partitions to
> know
> > > >>>>>> nothing
> > > >>>>>>>>> about other partitions to access their data in shared WAL
> > files.
> > > >>>>>>>>>
> > > >>>>>>>>> The number of partitions involved in a single WAL file may be
> > > >>>> quite
> > > >>>>>>> large,
> > > >>>>>>>>> e.g. a hundred. A hundred network requests to commit one WAL
> > > >>>> file is
> > > >>>>>>> very
> > > >>>>>>>>> impractical. However, there are ways to reduce this number:
> > > >>>>>>>>> 1. Partition leaders are located on brokers. Requests to
> > > >>>> leaders on
> > > >>>>>>> one
> > > >>>>>>>>> broker could be grouped together into a single physical
> network
> > > >>>>>> request
> > > >>>>>>>>> (resembling the normal Produce request that may carry batches
> > for
> > > >>>>>> many
> > > >>>>>>>>> partitions inside). This will cap the number of network
> > requests
> > > >>>> to
> > > >>>>>> the
> > > >>>>>>>>> number of brokers in the cluster.
> > > >>>>>>>>> 2. If we craft the cluster metadata to make producers send
> > their
> > > >>>>>>> requests
> > > >>>>>>>>> to the right brokers (with respect to AZs), we may achieve
> the
> > > >>>> higher
> > > >>>>>>>>> concentration of logical commit requests in physical network
> > > >>>> requests
> > > >>>>>>>>> reducing the number of the latter ones even further, ideally
> to
> > > >>>> one.
> > > >>>>>>>>>
> > > >>>>>>>>> Obviously, out of multiple commit requests some may fail or
> > time
> > > >>>> out
> > > >>>>>>> for a
> > > >>>>>>>>> variety of reasons. This is fine. Some producers will receive
> > > >>>> totally
> > > >>>>>>> or
> > > >>>>>>>>> partially failed responses to their Produce requests, similar
> > to
> > > >>>> what
> > > >>>>>>> they
> > > >>>>>>>>> would have received when appending to a classic topic fails
> or
> > > >>>> times
> > > >>>>>>> out.
> > > >>>>>>>>> If a partition experiences problems, other partitions will
> not
> > be
> > > >>>>>>> affected
> > > >>>>>>>>> (again, like in classic topics). Of course, the uncommitted
> > data
> > > >>>> will
> > > >>>>>>> be
> > > >>>>>>>>> garbage in WAL files. But WAL files are short-lived (batches
> > are
> > > >>>>>>> constantly
> > > >>>>>>>>> assembled into segments and offloaded to tiered storage), so
> > this
> > > >>>>>>> garbage
> > > >>>>>>>>> will be eventually deleted.
> > > >>>>>>>>>
> > > >>>>>>>>> For safely deleting WAL files we now need to centrally manage
> > > >>>> them,
> > > >>>>>> as
> > > >>>>>>>>> this is the only state and logic that spans multiple
> > partitions.
> > > >>>> On
> > > >>>>>> the
> > > >>>>>>>>> diagram, you can see another commit request called “Commit
> file
> > > >>>> (best
> > > >>>>>>>>> effort)” going to the WAL File Manager. This manager will be
> > > >>>>>>> responsible
> > > >>>>>>>>> for the following:
> > > >>>>>>>>> 1. Collecting (by requests from brokers) and persisting
> > > >>>> information
> > > >>>>>>> about
> > > >>>>>>>>> committed WAL files.
> > > >>>>>>>>> 2. To handle potential failures in file information delivery,
> > it
> > > >>>>>> will
> > > >>>>>>> be
> > > >>>>>>>>> doing prefix scan on the remote storage periodically to find
> > and
> > > >>>>>>> register
> > > >>>>>>>>> unknown files. The period of this scan will be configurable
> and
> > > >>>>>> ideally
> > > >>>>>>>>> should be quite long.
> > > >>>>>>>>> 3. Checking with the relevant partition leaders (after a
> grace
> > > >>>>>>> period) if
> > > >>>>>>>>> they still have batches in a particular file.
> > > >>>>>>>>> 4. Physically deleting files when they aren’t anymore
> referred
> > > >>>> to by
> > > >>>>>>> any
> > > >>>>>>>>> partition.
> > > >>>>>>>>>
> > > >>>>>>>>> This new design offers the following advantages:
> > > >>>>>>>>> 1. It simplifies the implementation of many Kafka features
> such
> > > >>>> as
> > > >>>>>>>>> idempotence, transactions, queues, tiered storage, retention.
> > > >>>> Now we
> > > >>>>>>> don’t
> > > >>>>>>>>> need to abstract away and reuse the code from partition
> leaders
> > > >>>> in
> > > >>>>>> the
> > > >>>>>>>>> batch coordinator. Instead, we will literally use the same
> code
> > > >>>> paths
> > > >>>>>>> in
> > > >>>>>>>>> leaders, with little adaptation. Workflows from classic
> topics
> > > >>>> mostly
> > > >>>>>>>>> remain unchanged.
> > > >>>>>>>>> For example, it seems that
> > > >>>>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator
> and
> > > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for transaction
> > > >>>> support
> > > >>>>>> on
> > > >>>>>>> the
> > > >>>>>>>>> partition leader side could be used for diskless topics with
> > > >>>> little
> > > >>>>>>>>> adaptation. ProducerStateManager, needed for both idempotent
> > > >>>> produce
> > > >>>>>>> and
> > > >>>>>>>>> transactions, would be reused.
> > > >>>>>>>>> Another example is share groups support, where the share
> > > >>>> partition
> > > >>>>>>> leader,
> > > >>>>>>>>> being co-located with the partition leader, would execute the
> > > >>>> same
> > > >>>>>>> logic
> > > >>>>>>>>> for both diskless and classic topics.
> > > >>>>>>>>> 2. It returns to the familiar partition-based scaling model,
> > > >>>> where
> > > >>>>>>>>> partitions are independent.
> > > >>>>>>>>> 3. It makes the operation and failure patterns closer to the
> > > >>>>>> familiar
> > > >>>>>>>>> ones from classic topics.
> > > >>>>>>>>> 4. It opens a straightforward path to seamless switching the
> > > >>>> topics
> > > >>>>>>> modes
> > > >>>>>>>>> between diskless and classic.
> > > >>>>>>>>>
> > > >>>>>>>>> The rest of the things remain unchanged compared to the
> > previous
> > > >>>>>>> Diskless
> > > >>>>>>>>> design (after all previous discussions). Such things as local
> > > >>>> segment
> > > >>>>>>>>> materialization by replicas, the consume path, tiered storage
> > > >>>>>>> integration,
> > > >>>>>>>>> etc.
> > > >>>>>>>>>
> > > >>>>>>>>> If the community finds this design more suitable, we will
> > update
> > > >>>> the
> > > >>>>>>>>> KIP(s) accordingly and continue working on it. Please let us
> > know
> > > >>>>>> what
> > > >>>>>>> you
> > > >>>>>>>>> think.
> > > >>>>>>>>>
> > > >>>>>>>>> Best regards,
> > > >>>>>>>>> Ivan and Diskless team
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > >>>>>>>>>> Hi Justine,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Yes, you're right. We need to track the aborted transactions
> > > >>>> for in
> > > >>>>>>> the
> > > >>>>>>>>> diskless coordinator for as long as the corresponding offsets
> > are
> > > >>>>>>> there.
> > > >>>>>>>>> With the tiered storage unification Greg mentioned earlier,
> > this
> > > >>>> will
> > > >>>>>>> be
> > > >>>>>>>>> finite time even for infinite data retention.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best,
> > > >>>>>>>>>> Ivan
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > >>>>>>>>>>> Hey Ivan,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for the response. I think most of what you said made
> > > >>>>>> sense,
> > > >>>>>>> but
> > > >>>>>>>>> I
> > > >>>>>>>>>>> did have some questions about this part:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > >>>> topics
> > > >>>>>>> forgets
> > > >>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> > > >>>> it). The
> > > >>>>>>>>>>> transaction coordinator acts like the main guardian,
> allowing
> > > >>>>>>> partition
> > > >>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > > >>>> wrong. We
> > > >>>>>>> think
> > > >>>>>>>>>>> about relying on this with the batch coordinator and delete
> > > >>>> the
> > > >>>>>>>>> information
> > > >>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > >>>> replication
> > > >>>>>>> and
> > > >>>>>>>>> HWM
> > > >>>>>>>>>>> advances immediately).
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I didn't quite understand this. In classic topics, we have
> > > >>>> maps
> > > >>>>>> for
> > > >>>>>>>>> ongoing
> > > >>>>>>>>>>> transactions which remove state when the transaction is
> > > >>>> completed
> > > >>>>>>> and
> > > >>>>>>>>> an
> > > >>>>>>>>>>> aborted transactions index which is retained for much
> longer.
> > > >>>>>> Once
> > > >>>>>>> the
> > > >>>>>>>>>>> transaction is completed, the coordinator is no longer
> > > >>>> involved
> > > >>>>>> in
> > > >>>>>>>>>>> maintaining this partition side state, and it is subject to
> > > >>>>>>> compaction
> > > >>>>>>>>> etc.
> > > >>>>>>>>>>> Looking back at the outline provided above, I didn't see
> much
> > > >>>>>>> about the
> > > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit further.
> I
> > > >>>> saw
> > > >>>>>>> the
> > > >>>>>>>>>>> following in a response:
> > > >>>>>>>>>>>> When the broker constructs a fully valid local segment,
> > > >>>> all the
> > > >>>>>>>>> necessary
> > > >>>>>>>>>>> control batches will be inserted and indices, including the
> > > >>>>>>> transaction
> > > >>>>>>>>>>> index will be built to serve FetchRequests exactly as they
> > > >>>> are
> > > >>>>>>> today.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Based on this, it seems like we need to retain the
> > > >>>> information
> > > >>>>>>> about
> > > >>>>>>>>>>> aborted txns for longer.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>> Justine
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > > >>>> ivan@ivanyu.me>
> > > >>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Justine and all,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thank you for your questions!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> > > >>>> with
> > > >>>>>>>>> producer ID
> > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > >>>> cached
> > > >>>>>>>>> locally
> > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > >>>>>> transactions
> > > >>>>>>> can
> > > >>>>>>>>> be
> > > >>>>>>>>>>>> used
> > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > >>>> with
> > > >>>>>>>>> producer id +
> > > >>>>>>>>>>>>> epoch
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> You’re right that we (probably unintentionally) focused
> > > >>>> only on
> > > >>>>>>>>> version 2.
> > > >>>>>>>>>>>> We can either limit the support to version 2 or consider
> > > >>>> using
> > > >>>>>>> some
> > > >>>>>>>>>>>> surrogates to support version 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> > > >>>>>>> checks
> > > >>>>>>>>> of the
> > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > >>>> like the
> > > >>>>>>>>> partition
> > > >>>>>>>>>>>>> leader in classic topics would do.
> > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > >>>>>> checking
> > > >>>>>>> if
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is ongoing,
> > > >>>> and
> > > >>>>>> of
> > > >>>>>>>>> course
> > > >>>>>>>>>>>> the normal idempotence checks. What the partition leader
> > > >>>> in the
> > > >>>>>>>>> classic
> > > >>>>>>>>>>>> topics does before appending a batch to the local log
> > > >>>> (e.g. in
> > > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In Diskless,
> > > >>>> we
> > > >>>>>>>>> unfortunately
> > > >>>>>>>>>>>> cannot do these checks before appending the data to the
> WAL
> > > >>>>>>> segment
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> uploading it, but we can “tombstone” these batches in the
> > > >>>> batch
> > > >>>>>>>>> coordinator
> > > >>>>>>>>>>>> during the final commit.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Is there state about ongoing
> > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> > > >>>> state
> > > >>>>>>>>> mentioned
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > >>>> what
> > > >>>>>>> state is
> > > >>>>>>>>>>>> stored
> > > >>>>>>>>>>>>> and when it is stored.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Right, this should have been more explicit. As the
> > > >>>> partition
> > > >>>>>>> leader
> > > >>>>>>>>> tracks
> > > >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> > > >>>> coordinator
> > > >>>>>>> has
> > > >>>>>>>>> to as
> > > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > > >>>> transaction
> > > >>>>>>>>> coordinator
> > > >>>>>>>>>>>> must inform the batch coordinator about this.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > >>>> perhaps
> > > >>>>>>> that
> > > >>>>>>>>> would
> > > >>>>>>>>>>>> be
> > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Yes. This could be deduced from the committed batches and
> > > >>>> other
> > > >>>>>>>>>>>> information, but for the sake of performance we’d better
> > > >>>> store
> > > >>>>>> it
> > > >>>>>>>>>>>> explicitly.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> > > >>>>>>> state is
> > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> > > >>>>>> cleaned
> > > >>>>>>> up?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > > >>>> topics
> > > >>>>>>> forgets
> > > >>>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> > > >>>> it).
> > > >>>>>> The
> > > >>>>>>>>>>>> transaction coordinator acts like the main guardian,
> > > >>>> allowing
> > > >>>>>>>>> partition
> > > >>>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > > >>>> wrong.
> > > >>>>>> We
> > > >>>>>>>>> think
> > > >>>>>>>>>>>> about relying on this with the batch coordinator and
> > > >>>> delete the
> > > >>>>>>>>> information
> > > >>>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > > >>>>>> replication
> > > >>>>>>>>> and HWM
> > > >>>>>>>>>>>> advances immediately).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Ivan
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > >>>>>>>>>>>>> Hey folks,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Excited to see some updates related to transactions!
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I had a few questions.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> > > >>>> with
> > > >>>>>>>>> producer ID
> > > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > > >>>> cached
> > > >>>>>>>>> locally
> > > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > > >>>>>> transactions
> > > >>>>>>> can
> > > >>>>>>>>> be
> > > >>>>>>>>>>>> used
> > > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > > >>>> with
> > > >>>>>>>>> producer id +
> > > >>>>>>>>>>>>> epoch
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> > > >>>>>>> checks
> > > >>>>>>>>> of the
> > > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > > >>>> like the
> > > >>>>>>>>> partition
> > > >>>>>>>>>>>>> leader in classic topics would do.
> > > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > > >>>>>> checking
> > > >>>>>>> if
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>> transaction was still ongoing for example? Is there state
> > > >>>>>> about
> > > >>>>>>>>> ongoing
> > > >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> > > >>>> state
> > > >>>>>>>>> mentioned
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > > >>>> what
> > > >>>>>>> state is
> > > >>>>>>>>>>>> stored
> > > >>>>>>>>>>>>> and when it is stored.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > > >>>> perhaps
> > > >>>>>>> that
> > > >>>>>>>>> would
> > > >>>>>>>>>>>> be
> > > >>>>>>>>>>>>> stored in the batch coordinator?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> > > >>>>>>> state is
> > > >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> > > >>>>>> cleaned
> > > >>>>>>> up?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > >>>>>>> <ju...@confluent.io.invalid>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> > > >>>>>> segments,
> > > >>>>>>>>> making
> > > >>>>>>>>>>>> use of
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and zero-copy
> > > >>>>>>>>> functionality
> > > >>>>>>>>>>>> already
> > > >>>>>>>>>>>>>> built into classic topics."
> > > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
> > > >>>> each
> > > >>>>>>>>> partition in
> > > >>>>>>>>>>>>>> diskless topics?
> > > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
> > > >>>> need
> > > >>>>>> to
> > > >>>>>>> know
> > > >>>>>>>>>>>> aborted
> > > >>>>>>>>>>>>>> records. How is that achieved?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > > >>>> transaction is
> > > >>>>>>>>> finished
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> also inserts the control batches in the corresponding
> > > >>>> logs
> > > >>>>>>> of the
> > > >>>>>>>>>>>> involved
> > > >>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
> > > >>>> level,
> > > >>>>>> no
> > > >>>>>>>>> actual
> > > >>>>>>>>>>>> control
> > > >>>>>>>>>>>>>> batches are written to any file. "
> > > >>>>>>>>>>>>>> A fetch response could include multiple transactional
> > > >>>>>>> batches.
> > > >>>>>>>>> How
> > > >>>>>>>>>>>> does the
> > > >>>>>>>>>>>>>> broker obtain the information about the ending control
> > > >>>>>> batch
> > > >>>>>>> for
> > > >>>>>>>>> each
> > > >>>>>>>>>>>>>> batch? Does that mean that a fetch response needs to be
> > > >>>>>>> built by
> > > >>>>>>>>>>>>>> stitching record batches and generated control batches
> > > >>>>>>> together?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition leader
> > > >>>> that
> > > >>>>>>> all
> > > >>>>>>>>>>>> consumers
> > > >>>>>>>>>>>>>> are routed to?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this or
> > > >>>> it's
> > > >>>>>>> too
> > > >>>>>>>>>>>>>> implementation-focused?" It would be useful to include
> > > >>>>>> enough
> > > >>>>>>>>> details
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> understand correctness and performance impact.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
> > > >>>>>> producer
> > > >>>>>>>>> contain a
> > > >>>>>>>>>>>>>> sequence number, which is ordered. If a producer sends
> > > >>>>>> every
> > > >>>>>>>>> Produce
> > > >>>>>>>>>>>>>> request to an arbitrary broker, those requests could
> > > >>>> reach
> > > >>>>>>> the
> > > >>>>>>>>> batch
> > > >>>>>>>>>>>>>> coordinator in different order and lead to rejection
> > > >>>> of the
> > > >>>>>>>>> produce
> > > >>>>>>>>>>>>>> requests.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > >>>>>>> ivan@ivanyu.me>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We have also thought in a bit more details about
> > > >>>>>>> transactions
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> queues,
> > > >>>>>>>>>>>>>>> here's the plan.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *Transactions*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> The support for transactions in *classic topics* is
> > > >>>> based
> > > >>>>>>> on
> > > >>>>>>>>> precise
> > > >>>>>>>>>>>>>>> interactions between three actors: clients (mostly
> > > >>>>>>> producers,
> > > >>>>>>>>> but
> > > >>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > > >>>> classes),
> > > >>>>>> and
> > > >>>>>>>>>>>> transaction
> > > >>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders with
> > > >>>>>> their
> > > >>>>>>>>> local
> > > >>>>>>>>>>>> state
> > > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> The high level (some details skipped) workflow is the
> > > >>>>>>>>> following.
> > > >>>>>>>>>>>> When a
> > > >>>>>>>>>>>>>>> transactional Produce request is received by the
> > > >>>> broker:
> > > >>>>>>>>>>>>>>> 1. For each partition, the partition leader checks
> > > >>>> if a
> > > >>>>>>>>> non-empty
> > > >>>>>>>>>>>>>>> transaction is running for this partition. This is
> > > >>>> done
> > > >>>>>>> using
> > > >>>>>>>>> its
> > > >>>>>>>>>>>> local
> > > >>>>>>>>>>>>>>> state derived from the log metadata
> > > >>>>>> (ProducerStateManager,
> > > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed about all
> > > >>>> the
> > > >>>>>>>>> partitions
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> > > >>>>>>>>>>>>>>> 3. The partition leaders do additional transactional
> > > >>>>>>> checks.
> > > >>>>>>>>>>>>>>> 4. The partition leaders append the transactional
> > > >>>> data to
> > > >>>>>>>>> their logs
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>> update some of their state (for example, log the fact
> > > >>>>>> that
> > > >>>>>>> the
> > > >>>>>>>>>>>>>> transaction
> > > >>>>>>>>>>>>>>> is running for the partition and its first offset).
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> > > >>>>>>> directly
> > > >>>>>>>>> with
> > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> > > >>>> or
> > > >>>>>>>>>>>> PREPARE_ABORT to
> > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > > >>>>>>> WriteTxnMarkersRequest to
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>> leaders
> > > >>>>>>>>>>>>>>> of the involved partitions.
> > > >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> > > >>>> markers to
> > > >>>>>>>>> their logs
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>> respond to the coordinator.
> > > >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction state
> > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
> > > >>>> of
> > > >>>>>>>>> important
> > > >>>>>>>>>>>> state
> > > >>>>>>>>>>>>>>> necessary for supporting this workflow is local. The
> > > >>>> main
> > > >>>>>>>>> challenge
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
> > > >>>> are no
> > > >>>>>>>>> partition
> > > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
> > > >>>> to be
> > > >>>>>>>>> globalized
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> > > >>>> support
> > > >>>>>>>>> idempotent
> > > >>>>>>>>>>>>>> produce.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> The high level workflow for *diskless topics* would
> > > >>>> look
> > > >>>>>>> very
> > > >>>>>>>>>>>> similar:
> > > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > > >>>> non-empty
> > > >>>>>>>>> transaction
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>> running for this partition. In contrast to classic
> > > >>>>>> topics,
> > > >>>>>>>>> this is
> > > >>>>>>>>>>>>>> checked
> > > >>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
> > > >>>> Since a
> > > >>>>>>>>> transaction
> > > >>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
> > > >>>> the
> > > >>>>>>> positive
> > > >>>>>>>>>>>> result of
> > > >>>>>>>>>>>>>>> this check could be cached locally (for the double
> > > >>>>>>> configured
> > > >>>>>>>>>>>> duration
> > > >>>>>>>>>>>>>> of a
> > > >>>>>>>>>>>>>>> transaction, for example).
> > > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is informed
> > > >>>>>> about
> > > >>>>>>> all
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
> > > >>>> include
> > > >>>>>>> them.
> > > >>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
> > > >>>> side.
> > > >>>>>>>>>>>>>>> 4. The broker appends the transactional data to the
> > > >>>>>> current
> > > >>>>>>>>> shared
> > > >>>>>>>>>>>> WAL
> > > >>>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
> > > >>>> state
> > > >>>>>>> for
> > > >>>>>>>>> Diskless
> > > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > > >>>> coordinator
> > > >>>>>>> like
> > > >>>>>>>>> in the
> > > >>>>>>>>>>>>>>> normal produce flow.
> > > >>>>>>>>>>>>>>> 6. The batch coordinator does the final transactional
> > > >>>>>>> checks
> > > >>>>>>>>> of the
> > > >>>>>>>>>>>>>>> batches. This procedure would output the same errors
> > > >>>> like
> > > >>>>>>> the
> > > >>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some batches
> > > >>>>>> could
> > > >>>>>>> be
> > > >>>>>>>>>>>> rejected.
> > > >>>>>>>>>>>>>>> This means, there will potentially be garbage in the
> > > >>>> WAL
> > > >>>>>>>>> segment
> > > >>>>>>>>>>>> file in
> > > >>>>>>>>>>>>>>> case of transactional errors. This is preferable to
> > > >>>> doing
> > > >>>>>>> more
> > > >>>>>>>>>>>> network
> > > >>>>>>>>>>>>>>> round trips, especially considering the WAL segments
> > > >>>> will
> > > >>>>>>> be
> > > >>>>>>>>>>>> relatively
> > > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > > >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> > > >>>>>>> directly
> > > >>>>>>>>> with
> > > >>>>>>>>>>>>>>> EndTxnRequest.
> > > >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> > > >>>> or
> > > >>>>>>>>>>>> PREPARE_ABORT to
> > > >>>>>>>>>>>>>>> its log and responds to the producer.
> > > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
> > > >>>> batch
> > > >>>>>>>>> coordinator
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> the transaction is finished.
> > > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> > > >>>>>>> transaction is
> > > >>>>>>>>>>>> finished
> > > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > > >>>> corresponding
> > > >>>>>>> logs
> > > >>>>>>>>> of the
> > > >>>>>>>>>>>>>>> involved Diskless topics. This happens only on the
> > > >>>>>> metadata
> > > >>>>>>>>> level, no
> > > >>>>>>>>>>>>>>> actual control batches are written to any file. They
> > > >>>> will
> > > >>>>>>> be
> > > >>>>>>>>>>>> dynamically
> > > >>>>>>>>>>>>>>> created on Fetch and other read operations. We could
> > > >>>>>>>>> technically
> > > >>>>>>>>>>>> write
> > > >>>>>>>>>>>>>>> these control batches for real, but this would mean
> > > >>>> extra
> > > >>>>>>>>> produce
> > > >>>>>>>>>>>>>> latency,
> > > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > > >>>> coordinator
> > > >>>>>>> and
> > > >>>>>>>>> save
> > > >>>>>>>>>>>> these
> > > >>>>>>>>>>>>>>> milliseconds.
> > > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > > >>>>>>> WriteTxnMarkersRequest to
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>> leaders
> > > >>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
> > > >>>> topics
> > > >>>>>>> now.
> > > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics write the
> > > >>>>>>>>> transaction
> > > >>>>>>>>>>>> markers
> > > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > > >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction state
> > > >>>>>>>>>>>> COMPLETE_COMMIT or
> > > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
> > > >>>> get:
> > > >>>>>>>>>>>>>>> 1. An extra network round trip between brokers and
> > > >>>> the
> > > >>>>>>> batch
> > > >>>>>>>>>>>> coordinator
> > > >>>>>>>>>>>>>>> when a new partition appear in the transaction. To
> > > >>>>>>> mitigate the
> > > >>>>>>>>>>>> impact of
> > > >>>>>>>>>>>>>>> them:
> > > >>>>>>>>>>>>>>> - The results will be cached.
> > > >>>>>>>>>>>>>>> - The calls for multiple partitions in one Produce
> > > >>>>>>> request
> > > >>>>>>>>> will be
> > > >>>>>>>>>>>>>>> grouped.
> > > >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> > > >>>> fast
> > > >>>>>>>>> response to
> > > >>>>>>>>>>>> these
> > > >>>>>>>>>>>>>>> RPCs.
> > > >>>>>>>>>>>>>>> - The fact that a single producer normally will
> > > >>>>>>> communicate
> > > >>>>>>>>> with a
> > > >>>>>>>>>>>>>>> single broker for the duration of the transaction
> > > >>>> further
> > > >>>>>>>>> reduces the
> > > >>>>>>>>>>>>>>> expected number of round trips.
> > > >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> > > >>>>>> coordinator
> > > >>>>>>> and
> > > >>>>>>>>> batch
> > > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> With this proposal, transactions will also be able to
> > > >>>>>> span
> > > >>>>>>> both
> > > >>>>>>>>>>>> classic
> > > >>>>>>>>>>>>>>> and Diskless topics.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *Queues*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> The share group coordination and management is a
> > > >>>> side job
> > > >>>>>>> that
> > > >>>>>>>>>>>> doesn't
> > > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > > >>>> replicas,
> > > >>>>>>> physical
> > > >>>>>>>>>>>> storage
> > > >>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and consumers
> > > >>>>>>> (Fetch and
> > > >>>>>>>>>>>> Produce
> > > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not affected.)
> > > >>>> We
> > > >>>>>>> don't
> > > >>>>>>>>> see any
> > > >>>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
> > > >>>> with
> > > >>>>>>> share
> > > >>>>>>>>>>>> groups the
> > > >>>>>>>>>>>>>>> same way as classic topics are. Even on the code
> > > >>>> level,
> > > >>>>>> we
> > > >>>>>>>>> don't
> > > >>>>>>>>>>>> expect
> > > >>>>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>> serious refactoring: the same reading routines are
> > > >>>> used
> > > >>>>>>> that
> > > >>>>>>>>> are
> > > >>>>>>>>>>>> used for
> > > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
> > > >>>> too
> > > >>>>>>>>>>>>>>> implementation-focused?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>> Ivan
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > >>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thank you all for your questions and design input
> > > >>>> on
> > > >>>>>>>>> KIP-1150.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
> > > >>>> new
> > > >>>>>>>>> design. To
> > > >>>>>>>>>>>>>>> summarize
> > > >>>>>>>>>>>>>>>> the changes:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> > > >>>> existing
> > > >>>>>>>>> KIP-405
> > > >>>>>>>>>>>> Tiered
> > > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
> > > >>>>>>> Diskless
> > > >>>>>>>>> topic
> > > >>>>>>>>>>>> to be
> > > >>>>>>>>>>>>>>>> moved to tiered storage.
> > > >>>>>>>>>>>>>>>> This lowers the scalability requirements for the
> > > >>>> Batch
> > > >>>>>>>>> Coordinator
> > > >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> > > >>>> Tiered
> > > >>>>>>> Storage
> > > >>>>>>>>>>>> plugin
> > > >>>>>>>>>>>>>>>> features such as encryption and alternative data
> > > >>>>>> formats.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> > > >>>> segments,
> > > >>>>>>>>> making use
> > > >>>>>>>>>>>> of
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > > >>>> zero-copy
> > > >>>>>>>>> functionality
> > > >>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>> built into classic topics.
> > > >>>>>>>>>>>>>>>> However, local segments are now considered cache
> > > >>>>>>> elements,
> > > >>>>>>>>> do not
> > > >>>>>>>>>>>> need
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > > >>>> contacting
> > > >>>>>>> any
> > > >>>>>>>>> other
> > > >>>>>>>>>>>>>>> replicas.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 3. The design has been simplified substantially, by
> > > >>>>>>> removing
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>> previous
> > > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > > >>>> component, and
> > > >>>>>>>>> "object
> > > >>>>>>>>>>>>>>>> compaction/merging" step.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > > >>>> enabled by
> > > >>>>>>> the
> > > >>>>>>>>> Batch
> > > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
> > > >>>>>> earlier
> > > >>>>>>>>> design,
> > > >>>>>>>>>>>> while
> > > >>>>>>>>>>>>>>>> being simpler and integrating better into the
> > > >>>> existing
> > > >>>>>>>>> ecosystem.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
> > > >>>> the
> > > >>>>>> new
> > > >>>>>>>>> design.
> > > >>>>>>>>>>>>>>>> Greg Harris
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > >>>>>>>>> <ju...@confluent.io.invalid>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi, Jan,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> > > >>>> of
> > > >>>>>> all
> > > >>>>>>>>> existing
> > > >>>>>>>>>>>>>> client
> > > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > > >>>> supporting
> > > >>>>>> APIs
> > > >>>>>>>>> like
> > > >>>>>>>>>>>>>>> transactions
> > > >>>>>>>>>>>>>>>>> and queues.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> > > >>>>>>> status of
> > > >>>>>>>>> this
> > > >>>>>>>>>>>> KIP?
> > > >>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> > > >>>>>>> months,
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> vote got
> > > >>>>>>>>>>>>>>>>> vetoed
> > > >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> > > >>>>>> answered
> > > >>>>>>>>> since
> > > >>>>>>>>>>>> then.
> > > >>>>>>>>>>>>>>> KIP-1183
> > > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > > >>>> activity
> > > >>>>>>> since
> > > >>>>>>>>> May.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> > > >>>>>> choices
> > > >>>>>>>>> that can
> > > >>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> > > >>>> by
> > > >>>>>> far
> > > >>>>>>> the
> > > >>>>>>>>>>>> dominant
> > > >>>>>>>>>>>>>> one
> > > >>>>>>>>>>>>>>>>> when
> > > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
> > > >>>>>>>>> standalone
> > > >>>>>>>>>>>>>> products.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> > > >>>> hate to
> > > >>>>>>> see a
> > > >>>>>>>>>>>> stall. So
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> > > >>>>>> McCabe -
> > > >>>>>>>>> have your
> > > >>>>>>>>>>>>>>> questions
> > > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> > > >>>> do
> > > >>>>>> to
> > > >>>>>>>>> help? I'm
> > > >>>>>>>>>>>>>> deeply
> > > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > > >>>> written
> > > >>>>>>> about
> > > >>>>>>>>> both?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Kind regards,
> > > >>>>>>>>>>>>>>>>>> Jan
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > >>>>>> Kozlovski
> > > >>>>>>> <
> > > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > > >>>> main
> > > >>>>>> one
> > > >>>>>>>>> (just a
> > > >>>>>>>>>>>> bunch
> > > >>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>> links
> > > >>>>>>>>>>>>>>>>>>> to everything)
> > > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > > >>>> example, I
> > > >>>>>>> was
> > > >>>>>>>>>>>> searching
> > > >>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> > > >>>> far
> > > >>>>>> as
> > > >>>>>>> I
> > > >>>>>>>>> can
> > > >>>>>>>>>>>> tell, it
> > > >>>>>>>>>>>>>>>>> doesn't
> > > >>>>>>>>>>>>>>>>>>> exist yet.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> > > >>>> having
> > > >>>>>>> the
> > > >>>>>>>>> root
> > > >>>>>>>>>>>> KIP be
> > > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> > > >>>> to
> > > >>>>>>>>> sub-KIPs
> > > >>>>>>>>>>>> where
> > > >>>>>>>>>>>>>> were
> > > >>>>>>>>>>>>>>>>> aimed
> > > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> > > >>>> would
> > > >>>>>>> be
> > > >>>>>>>>> good to
> > > >>>>>>>>>>>>>> create
> > > >>>>>>>>>>>>>>>>>> DISCUSS
> > > >>>>>>>>>>>>>>>>>>> threads for them all.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>> Stan
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> > > >>>>>>>>> introducing a
> > > >>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>> type of
> > > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > > >>>> Storage as
> > > >>>>>>> the
> > > >>>>>>>>> primary
> > > >>>>>>>>>>>>>>> source of
> > > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > > >>>> decided
> > > >>>>>>> to
> > > >>>>>>>>> split it
> > > >>>>>>>>>>>>>> into
> > > >>>>>>>>>>>>>>>>>> multiple
> > > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > >>>>>>>>>>>>>>>>>>> )
> > > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > > >>>> should aim
> > > >>>>>>> to
> > > >>>>>>>>> have
> > > >>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> type of
> > > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > > >>>>>> details
> > > >>>>>>> on
> > > >>>>>>>>> how to
> > > >>>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>>> it.
> > > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
> > > >>>>>>> discussed
> > > >>>>>>>>>>>> KRaft.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> > > >>>> hard
> > > >>>>>>> to
> > > >>>>>>>>> discuss
> > > >>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> meta
> > > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > > >>>> (linked
> > > >>>>>>> in
> > > >>>>>>>>>>>> KIP-1150)
> > > >>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> offer
> > > >>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> > > >>>>>>> threads
> > > >>>>>>>>> for
> > > >>>>>>>>>>>> each
> > > >>>>>>>>>>>>>>> type of
> > > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > > >>>> whether
> > > >>>>>>> Apache
> > > >>>>>>>>> Kafka
> > > >>>>>>>>>>>> wants
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> have
> > > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > > >>>> authors
> > > >>>>>> of
> > > >>>>>>>>> this KIP.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> ------------------
> > > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> > > >>>>>>> aiven.io
> > > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > > >>>>>>> Valtonen,
> > > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Luke Chen <sh...@gmail.com>]

Hi Greg,

Thanks for sharing the meeting notes.
I agree we should keep polishing the contents of 1150 & high level design
in 1163 to prepare for a vote.


Thanks.
Luke

On Fri, Nov 14, 2025 at 3:54 AM Greg Harris <gr...@aiven.io.invalid>
wrote:

> Hi all,
>
> There was a video call between myself, Ivan Yurchenko, Jun Rao, and Andrew
> Schofield pertaining to KIP-1150. Here are the notes from that meeting:
>
> Ivan: What is the future state of Kafka in this area, in 5 years?
> Jun: Do we want something more cloud native? Yes, started with Tiered
> Storage. If there’s a better way, we should explore it. In the long term
> this will be useful
> Because Kafka is used so widely, we need to make sure everything we add is
> for the long term and for everyone, not just for a single company.
> When we add TS, it doesn’t just solve Uber’s use-case. We want something
> that’s high quality/lasts/maintainable, and can work with all existing
> capabilities.
> If both 1150 and 1176 proceed at the same time, it’s confusing. They
> overlap, but Diskless is more ambitious.
> If both KIPs are being seriously worked on, then we don’t really need both,
> because Diskless clearly is better. Having multiple will confuse people. It
> will duplicate some of the effort.
> If we want diskless ultimately, what is the short term strategy, to get
> some early wins first?
> Ivan: Andrew, do you want a more revolutionary approach?
> Andrew: Eventually the architecture will change substantially, it may not
> be necessary to put all of that bill onto Diskless at once.
> Greg: We all agree on having a high quality feature merged upstream, and
> supporting all APIs
> Jun: We should try and keep things simple, but there is some minimum
> complexity needed.
> When doing the short term changes (1176), it doesn’t really progress in
> changing to a more modern architecture.
> Greg: Was TS+Compaction the only feature miss we’ve had so far?
> Jun: The danger of only applying changes to some part of the API, you set
> the precedence that you only have to implement part of the API. Supporting
> the full API set should be a minimum requirement.
> Andrew: When we started Kraft, how much did we know the design?
> Jun: For Kraft we didn’t really know much about the migration, but the
> high-level was clear.
> Greg: Is 1150 votable in its current state?
> Jun: 1150 should promise to support all APIs. It doesn’t have to have all
> the details/apis/etc. KIP-500 didn’t have it.
> We do need some high-level design enough to give confidence that the
> promise is able to be fulfilled.
> Greg: Is the draft version in 1163 enough detail or is more needed?
> Jun: We need to agree on the core design, such as leaderless etc. And how
> the APIs will be supported.
> Greg: Okay we can include these things, and provide a sketch of how the
> other leader-based features operate.
> Jun: Yeah if at a high level the sketch appears to work, we can approve
> that functionality.
> Are you committed to doing the more involved and big project?
> Greg: Yes, we’re committed to the 1163 design and can’t really accept 1176.
> Jun: TS was slow because of Uber resourcing problems
> Greg: We’ll push internally for resources, and use the community sentiment
> to motivate Aiven.
> How far into the future should we look? What sort of scale?
> Jun: As long as there’s a path forward, and we’re not closing off future
> improvements, we can figure out how to handle a larger scale when it
> arises.
> Greg: Random replica placement is very harmful, can we recommend users to
> use an external tool like CruiseControl?
> Jun: Not everyone uses CruiseControl, we would probably need some solution
> for this out of the box
> Ivan: Should the Batch Coordinator be pluggable?
> Jun: Out-of-box experience should be good, good to allow other
> implementations
> Greg: But it could hurt Kafka feature/upgrade velocity when we wait for
> plugin providers to implement it
> Ivan: We imagined that maybe cloud hyperscalers could implement it with
> e.g. dynamodb
> Greg: Could we bake more details of the different providers into Kafka, or
> does it still make sense for it to be pluggable?
> Jun: Make it whatever is easiest to roll out and add new clients
> Andrew: What happens next? Do you want to get KIP-1150 voted?
> Ivan: The vote is already open, we’re not too pressed for time. We’ll go
> improve the 1163 design and communication.
> Is 1176 a competing design? Someone will ask.
> Jun: If we are seriously working on something more ambitious, yeah we
> shouldn’t do the stop-gap solution.
> It’s diverting review resources. If we can get the short term thing in 1yr
> but Diskless solution is 2y it makes sense to go for Diskless. If it’s 5yr,
> that’s different and maybe the stop-gap solution is needed.
> Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should we
> explicitly exclude 1176?
> Andrew: Put your arms around the feature set you actually want, and use
> that to rule out 1176.
> Probably don’t need -1 votes, most likely KIPs just don’t receive votes.
> Ivan: Should we have sync meetings like tiered storage did?
> Jun: Satish posted meeting notes regularly, we should do the same.
>
> To summarize, we will be polishing the contents of 1150 & high level design
> in 1163 to prepare for a vote.
> We believe that the community should select the feature set of 1150 to
> fully eliminate producer cross-zone costs, and make the investment in a
> high quality Diskless Topics implementation rather than in stop-gap
> solutions.
>
> Thanks,
> Greg
>
> On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:
>
> > This may be a tangent, but we needed to offload storage off of Kafka into
> > S3. We are keeping Kafka not as a source of truth, but as a mostly
> > ephemeral broker that can come and go as it pleases. Be that scaling or
> > outage. Disks can be destroyed and recreated at will, we still retain
> data
> > and use broker for just that, brokering messages. Not only that, we
> reduced
> > the requirement on the actual Kafka resources by reducing the size of a
> > payload via a claim check pattern. Maybe this is an anti–pattern, but it
> is
> > super fast and highly cost efficient. We reworked ProducerRequest to
> allow
> > plugins. We added a custom http plugin that submits every request via a
> > persisted connection to a microservice. Microservice stores the payload
> and
> > returns a tiny json metadata object,a claim check, that can be used to
> find
> > the actual data. Think of it as zipping the payload. This claim check
> > metadata traverses the pipelines with consumers using the urls in
> metadata
> > to pull what they need. Think unzipping. This allowed us to also pull
> ONLY
> > the data that we need in graphql like manner. So if you have a 100K json
> > payload and you need only a subsection, you can pull that by jmespath.
> When
> > you have multiple consumer groups yanking down huge payloads it is
> > cumbersome on the broker. When you have the same consumer groups yanking
> > down a claim check, and then going out of band directly to the source of
> > truth, the broker has some breathing room. Obviously our microservice
> does
> > not go directly to the cloud storage, as that would be too slow. It
> stores
> > the payload in high speed memory cache and returns asap. That memory is
> > eventually persisted into S3. The retrieval goest against the cache
> first,
> > then against the S3. Overall a rather cheappy and zippy solution. I tried
> > proposing the KIP for this, but there was no excitement. Check this out:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
> >
> >
> > > On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID> wrote:
> > >
> > > Hi, Andrew,
> > >
> > > If we want to focus only on reducing cross-zone replication costs,
> there
> > is
> > > an alternative design in the KIP-1176 discussion thread that seems
> > simpler
> > > than the proposal here. I am copying the outline of that approach
> below.
> > >
> > > 1. A new leader is elected.
> > > 2. Leader maintains a first tiered offset, which is initialized to log
> > end
> > > offset.
> > > 3. Leader writes produced data from the client to local log.
> > > 4. Leader uploads produced data from all local logs as a combined
> object
> > > 5. Leader stores the metadata for the combined object in memory.
> > > 6. If a follower fetch request has an offset >= first tiered offset,
> the
> > > metadata for the corresponding combined object is returned. Otherwise,
> > the
> > > local data is returned.
> > > 7. Leader periodically advances first tiered offset.
> > >
> > > It's still a bit unnatural, but it could work.
> > >
> > > Hi, Ivan,
> > >
> > > Are you still committed to proceeding with the original design of
> > KIP-1150?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> > andrew_schofield@live.com>
> > > wrote:
> > >
> > >> Hi,
> > >> I’ve been following KIP-1150 and friends for a while. I’m going to
> jump
> > >> into the discussions too.
> > >>
> > >> Looking back at Jack Vanlightly’s message, I am not quite so convinced
> > >> that it’s a kind of fork in the road. The primary aim of the effort is
> > to
> > >> reduce cross-zone replication costs so Apache Kafka is not
> prohibitively
> > >> expensive to use on cloud storage. I think it would be entirely viable
> > to
> > >> prioritise code reuse for an initial implementation of diskless
> topics,
> > and
> > >> we could still have a more cloud-native design in the future. It’s
> hard
> > to
> > >> predict what the community will prioritise in the future.
> > >>
> > >> Of the three major revisions, I’m in the rev3 camp. We can support
> > >> leaderless produce requests, first writing WAL segments into object
> > >> storage, and then using the regular partition leaders to sequence the
> > >> records. The active log segment for a diskless topic will initially
> > contain
> > >> batch coordinates rather than record batches. The batch coordinates
> can
> > be
> > >> resolved from WAL segments for consumers, and also in order to prepare
> > log
> > >> segments for uploading to tiered storage. Jun is probably correct that
> > we
> > >> need a more frequent object merging process than tiered storage
> > provides.
> > >> This is just the transition from write-optimised WAL segments to
> > >> read-optimised tiered segments, and all of the object storage-based
> > >> implementations of Kafka that I’m aware of do this rearrangement. But
> > >> perhaps this more frequent object merging is a pre-GA improvement,
> > rather
> > >> than a strict requirement for an initial implementation for early
> access
> > >> use.
> > >>
> > >> For zone-aligned share consumers, the share group assignor is intended
> > to
> > >> be rack-aware. Consumers should be assigned to partitions with leaders
> > in
> > >> their zone. The simple assignor is not rack-aware, but it easily could
> > be
> > >> or we could have a rack-aware assignor.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>
> > >>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID> wrote:
> > >>>
> > >>> Hi, Ivan,
> > >>>
> > >>> Thanks for the reply.
> > >>>
> > >>> "As I understand, you’re speaking about locally materialized
> segments.
> > >> They
> > >>> will indeed consume some IOPS. See them as a cache that could always
> be
> > >>> restored from the remote storage. While it’s not ideal, it's still OK
> > to
> > >>> lose data in them due to a machine crash, for example. Because of
> this,
> > >> we
> > >>> can avoid explicit flushing on local materialized segments at all and
> > let
> > >>> the file system and page cache figure out when to flush optimally.
> This
> > >>> would not eliminate the extra IOPS, but should reduce it
> dramatically,
> > >>> depending on throughput for each partition. We, of course, continue
> > >>> flushing the metadata segments as before."
> > >>>
> > >>> If we have a mix of classic and diskless topics on the same broker,
> > it's
> > >>> important that the classic topics' data is flushed to disk as quickly
> > as
> > >>> possible. To achieve this, users typically set dirty_expire_centisecs
> > in
> > >>> the kernel based on the number of available disk IOPS. Once you set
> > this
> > >>> number, it applies to all dirty files, including the cached data in
> > >>> diskless topics. So, if there are more files actively accumulating
> > data,
> > >>> the flush frequency and therefore RPO is reduced for classic topics.
> > >>>
> > >>> "We should have mentioned this explicitly, but this step, in fact,
> > >> remains
> > >>> in the form of segments offloading to tiered storage. When we
> assemble
> > a
> > >>> segment and hand it over to RemoteLogManager, we’re effectively doing
> > >>> metadata compaction: replacing a big number of pieces of metadata
> about
> > >>> individual batches with a single record in __remote_log_metadata."
> > >>>
> > >>> The object merging in tier storage typically only kicks in after a
> few
> > >>> hours. The impact is (1) the amount of accumulated metadata is still
> > >> quite
> > >>> large; (2) there are many small objects, leading to poor read
> > >> performance.
> > >>> I think we need a more frequent object merging process than tier
> > storage
> > >>> provides.
> > >>>
> > >>> Jun
> > >>>
> > >>>
> > >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me>
> > wrote:
> > >>>
> > >>>> Hello Jack, Jun, Luke, and all!
> > >>>>
> > >>>> Thank you for your messages.
> > >>>>
> > >>>> Let me first address some of Jun’s comments.
> > >>>>
> > >>>>> First, it degrades the durability.
> > >>>>> For each partition, now there are two files being actively written
> > at a
> > >>>>> given point of time, one for the data and another for the metadata.
> > >>>>> Flushing each file requires a separate IO. If the disk has 1K IOPS
> > and
> > >> we
> > >>>>> have 5K partitions in a broker, currently we can afford to flush
> each
> > >>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we
> > double
> > >>>> the
> > >>>>> number of files per partition, we can only flush each partition
> every
> > >> 10
> > >>>>> seconds, which makes RPO twice as bad.
> > >>>>
> > >>>> As I understand, you’re speaking about locally materialized
> segments.
> > >> They
> > >>>> will indeed consume some IOPS. See them as a cache that could always
> > be
> > >>>> restored from the remote storage. While it’s not ideal, it's still
> OK
> > to
> > >>>> lose data in them due to a machine crash, for example. Because of
> > this,
> > >> we
> > >>>> can avoid explicit flushing on local materialized segments at all
> and
> > >> let
> > >>>> the file system and page cache figure out when to flush optimally.
> > This
> > >>>> would not eliminate the extra IOPS, but should reduce it
> dramatically,
> > >>>> depending on throughput for each partition. We, of course, continue
> > >>>> flushing the metadata segments as before.
> > >>>>
> > >>>> It’s worth making a note on caching. I think nobody will disagree
> that
> > >>>> doing direct reads from remote storage every time a batch is
> requested
> > >> by a
> > >>>> consumer will not be practical neither from the performance nor from
> > the
> > >>>> economy point of view. We need a way to keep the number of GET
> > requests
> > >>>> down. There are multiple options, for example:
> > >>>> 1. Rack-aware distributed in-memory caching.
> > >>>> 2. Local in-memory caching. Comes with less network chattiness and
> > >> works
> > >>>> well if we have more or less stable brokers to consume from.
> > >>>> 3. Materialization of diskless logs on local disk. Way lower impact
> on
> > >>>> RAM and also requires stable brokers for consumption (using just
> > >> assigned
> > >>>> replicas will probably work well).
> > >>>>
> > >>>> Materialization is one of possible options, but we can choose
> another
> > >> one.
> > >>>> However, we will have this dilemma regardless of whether we have an
> > >>>> explicit coordinator or we go “coordinator-less”.
> > >>>>
> > >>>>> Second, if we ever need this
> > >>>>> metadata somewhere else, say in the WAL file manager, the consumer
> > >> needs
> > >>>> to
> > >>>>> subscribe to every partition in the cluster, which is inefficient.
> > The
> > >>>>> actual benefit of this approach is also questionable. On the
> surface,
> > >> it
> > >>>>> might seem that we could reduce the number of lines that need to be
> > >>>> changed
> > >>>>> for this KIP. However, the changes are quite intrusive to the
> classic
> > >>>>> partition's code path and will probably make the code base harder
> to
> > >>>>> maintain in the long run. I like the original approach based on the
> > >> batch
> > >>>>> coordinator much better than this one. We could probably refactor
> the
> > >>>>> producer state code so that it could be reused in the batch
> > >> coordinator.
> > >>>>
> > >>>> It’s hard to disagree with this. The explicit coordinator is more a
> > side
> > >>>> thing, while coordinator-less approach is more about extending
> > >>>> ReplicaManager, UnifiedLog and others substantially.
> > >>>>
> > >>>>> Thanks for addressing the concerns on the number of RPCs in the
> > produce
> > >>>>> path. I agree that with the metadata crafting mechanism, we could
> > >>>> mitigate
> > >>>>> the PRC problem. However, since we now require the metadata to be
> > >>>>> collocated with the data on the same set of brokers, it's weird
> that
> > >> they
> > >>>>> are now managed by different mechanisms. The data assignment now
> uses
> > >> the
> > >>>>> metadata crafting mechanism, but the metadata is stored in the
> > classic
> > >>>>> partition using its own assignment strategy. It will be complicated
> > to
> > >>>> keep
> > >>>>> them collocated.
> > >>>>
> > >>>> I would like to note that the metadata crafting is needed only to
> tell
> > >>>> producers which brokers they should send Produce requests to, but
> data
> > >> (as
> > >>>> in “locally materialized log”) is located on partition replicas,
> i.e.
> > >>>> automatically co-located with metadata.
> > >>>>
> > >>>> As a side note, it would probably be better that instead of
> implicitly
> > >>>> crafting partition metadata, we extend the metadata protocol so that
> > for
> > >>>> diskless partitions we return not only the leader and replicas, but
> > also
> > >>>> some “recommended produce brokers”, selected for optimal performance
> > and
> > >>>> costs. Producers will pick ones in their racks.
> > >>>>
> > >>>>> I am also concerned about the removal of the object
> > compaction/merging
> > >>>>> step.
> > >>>>
> > >>>> We should have mentioned this explicitly, but this step, in fact,
> > >> remains
> > >>>> in the form of segments offloading to tiered storage. When we
> > assemble a
> > >>>> segment and hand it over to RemoteLogManager, we’re effectively
> doing
> > >>>> metadata compaction: replacing a big number of pieces of metadata
> > about
> > >>>> individual batches with a single record in __remote_log_metadata.
> > >>>>
> > >>>> We could create a Diskless-specific merging mechanism instead if
> > needed.
> > >>>> It’s rather easy with the explicit coordinator approach. With the
> > >>>> coordinator-less approach, this would probably be a bit more tricky
> > >>>> (rewriting the tail of the log by the leader + replicating this
> change
> > >>>> reliably).
> > >>>>
> > >>>>> I see a tendency toward primarily optimizing for the fewest code
> > >> changes
> > >>>> in
> > >>>>> the KIP. Instead, our primary goal should be a clean design that
> can
> > >> last
> > >>>>> for the long term.
> > >>>>
> > >>>> Yes, totally agree.
> > >>>>
> > >>>>
> > >>>>
> > >>>> Luke,
> > >>>>> I'm wondering if the complexity of designing txn and queue is
> because
> > >> of
> > >>>>> leaderless cluster, do you think it will be simpler if we only
> focus
> > on
> > >>>> the
> > >>>>> "diskless" design to handle object compaction/merging to/from the
> > >> remote
> > >>>>> storage to save the cross-AZ cost?
> > >>>>
> > >>>> After some evolution of the original proposal, leaderless is now
> > >> limited.
> > >>>> We only need to be able to accept Produce requests on more than one
> > >> broker
> > >>>> to eliminate the cross-AZ costs for producers. Do I get it right
> that
> > >> you
> > >>>> propose to get rid of this? Or do I misunderstand?
> > >>>>
> > >>>>
> > >>>>
> > >>>> Let’s now look at this problem from a higher level, as Jack
> proposed.
> > As
> > >>>> it was said, the big choice we need to make is whether we 1) create
> an
> > >>>> explicit batch coordinator; or 2) go for the coordinator-less
> > approach,
> > >>>> where each diskless partition is managed by its leader as in classic
> > >> topics.
> > >>>>
> > >>>> If we try to compare the two approaches:
> > >>>>
> > >>>> Pluggability:
> > >>>> - Explicit coordinator: Possible. For example, some setups may
> benefit
> > >>>> from batch metadata being stored in a cloud database (such as AWS
> > >> DynamoDB
> > >>>> or GCP Spanner).
> > >>>> - Coordinator-less: Impossible.
> > >>>>
> > >>>> Scalability and fault tolerance:
> > >>>> - Explicit coordinator: Depends on the implementation and it’s also
> > >>>> necessary to actively work for it.
> > >>>> - Coordinator-less: Closer to classic Kafka topics. Scaling is done
> by
> > >>>> partition placement, partitions could fail independently.
> > >>>>
> > >>>> Separation of concerns:
> > >>>> - Explicit coordinator: Very good. Diskless remains more independent
> > >> from
> > >>>> classic topics in terms of code and workflows. For example, the
> > >>>> above-mentioned non-tiered storage metadata compaction mechanism
> could
> > >> be
> > >>>> relatively simply implemented with it. As a flip side of this, some
> > >>>> workflows (e.g. transactions) will have to be adapted.
> > >>>> - Coordinator-less: Less so. It leans to the opposite: bringing
> > diskless
> > >>>> closer to classic topics. Some code paths and workflows could be
> more
> > >>>> straightforwardly reused, but they will inevitably have to be
> adapted
> > to
> > >>>> accommodate both topic types as also discussed.
> > >>>>
> > >>>> Cloud-nativeness. This is a vague concept, also related to the
> > previous,
> > >>>> but let’s try:
> > >>>> - Explicit coordinator: Storing and processing metadata separately
> > makes
> > >>>> it easier for brokers to take different roles, be purely stateless
> if
> > >>>> needed, etc.
> > >>>> - Coordinator-less: Less so. Something could be achieved with
> creative
> > >>>> partition placement, but not much.
> > >>>>
> > >>>> Both seem to have their pros and cons. However, answering Jack’s
> > >> question,
> > >>>> the explicit coordinator approach may indeed lead to a more flexible
> > >> design.
> > >>>>
> > >>>>
> > >>>> The purpose of this deviation in the discussion was to receive a
> > >>>> preliminary community evaluation of the coordinator-less approach
> > >> without
> > >>>> taking on the task of writing a separate KIP and fitting it in the
> > >> system
> > >>>> of KIP-1150 and its children. We’re open to stopping it and getting
> > >> back to
> > >>>> working out the coordinator design if the community doesn’t favor
> the
> > >>>> proposed approach.
> > >>>>
> > >>>> Best,
> > >>>> Ivan and Diskless team
> > >>>>
> > >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > >>>>> Hi Ivan,
> > >>>>>
> > >>>>> As Jun pointed out, the updated design seems to have some
> > shortcomings
> > >>>>> although it simplifies the implementation.
> > >>>>>
> > >>>>> I'm wondering if the complexity of designing txn and queue is
> because
> > >> of
> > >>>>> leaderless cluster, do you think it will be simpler if we only
> focus
> > on
> > >>>> the
> > >>>>> "diskless" design to handle object compaction/merging to/from the
> > >> remote
> > >>>>> storage to save the cross-AZ cost?
> > >>>>>
> > >>>>>
> > >>>>> Thank you,
> > >>>>> Luke
> > >>>>>
> > >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Hi, Ivan,
> > >>>>>>
> > >>>>>> Thanks for the explanation.
> > >>>>>>
> > >>>>>> "we write the reference to the WAL file with the batch data"
> > >>>>>>
> > >>>>>> I understand the approach now, but I think it is a hacky one.
> There
> > >> are
> > >>>>>> multiple short comings with this design. First, it degrades the
> > >>>> durability.
> > >>>>>> For each partition, now there are two files being actively written
> > at
> > >> a
> > >>>>>> given point of time, one for the data and another for the
> metadata.
> > >>>>>> Flushing each file requires a separate IO. If the disk has 1K IOPS
> > and
> > >>>> we
> > >>>>>> have 5K partitions in a broker, currently we can afford to flush
> > each
> > >>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we
> > double
> > >>>> the
> > >>>>>> number of files per partition, we can only flush each partition
> > every
> > >>>> 10
> > >>>>>> seconds, which makes RPO twice as bad. Second, if we ever need
> this
> > >>>>>> metadata somewhere else, say in the WAL file manager, the consumer
> > >>>> needs to
> > >>>>>> subscribe to every partition in the cluster, which is inefficient.
> > The
> > >>>>>> actual benefit of this approach is also questionable. On the
> > surface,
> > >>>> it
> > >>>>>> might seem that we could reduce the number of lines that need to
> be
> > >>>> changed
> > >>>>>> for this KIP. However, the changes are quite intrusive to the
> > classic
> > >>>>>> partition's code path and will probably make the code base harder
> to
> > >>>>>> maintain in the long run. I like the original approach based on
> the
> > >>>> batch
> > >>>>>> coordinator much better than this one. We could probably refactor
> > the
> > >>>>>> producer state code so that it could be reused in the batch
> > >>>> coordinator.
> > >>>>>>
> > >>>>>> Thanks for addressing the concerns on the number of RPCs in the
> > >> produce
> > >>>>>> path. I agree that with the metadata crafting mechanism, we could
> > >>>> mitigate
> > >>>>>> the PRC problem. However, since we now require the metadata to be
> > >>>>>> collocated with the data on the same set of brokers, it's weird
> that
> > >>>> they
> > >>>>>> are now managed by different mechanisms. The data assignment now
> > uses
> > >>>> the
> > >>>>>> metadata crafting mechanism, but the metadata is stored in the
> > classic
> > >>>>>> partition using its own assignment strategy. It will be
> complicated
> > to
> > >>>> keep
> > >>>>>> them collocated.
> > >>>>>>
> > >>>>>> I am also concerned about the removal of the object
> > compaction/merging
> > >>>>>> step. My first concern is on the amount of metadata that need to
> be
> > >>>> kept.
> > >>>>>> Without object compcation, the metadata generated in the produce
> > path
> > >>>> can
> > >>>>>> only be deleted after remote tiering kicks in. Let's say for every
> > >>>> 250ms we
> > >>>>>> produce 100 byte of metadata per partition. Let's say remoting
> > tiering
> > >>>>>> kicks in after 5 hours. In a cluster with 100K partitions, we need
> > to
> > >>>> keep
> > >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
> > >>>>>> signficant. A second concern is on performance. Every time we need
> > to
> > >>>>>> rebuild the caching data, we need to read a bunch of small objects
> > >>>> from S3,
> > >>>>>> slowing down the building process. If a consumer happens to need
> > such
> > >>>> data,
> > >>>>>> it could slow down the application.
> > >>>>>>
> > >>>>>> I see a tendency toward primarily optimizing for the fewest code
> > >>>> changes in
> > >>>>>> the KIP. Instead, our primary goal should be a clean design that
> can
> > >>>> last
> > >>>>>> for the long term.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Jun
> > >>>>>>
> > >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Jun,
> > >>>>>>>
> > >>>>>>> Thank you for your message. I’m sorry that I failed to clearly
> > >>>> explain
> > >>>>>> the
> > >>>>>>> idea. Let me try to fix this.
> > >>>>>>>
> > >>>>>>>> Does each partition now have a metadata partition and a separate
> > >>>> data
> > >>>>>>>> partition? If so, I am concerned that it essentially doubles the
> > >>>> number
> > >>>>>>> of
> > >>>>>>>> partitions, which impacts the number of open file descriptors
> and
> > >>>> the
> > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > separate
> > >>>>>>>> partition just to store the metadata. It's as if we are creating
> > an
> > >>>>>>>> internal topic with an unbounded number of partitions.
> > >>>>>>>
> > >>>>>>> No. There will be only one physical partition per diskless
> > >>>> partition. Let
> > >>>>>>> me explain this with an example. Let’s say we have a diskless
> > >>>> partition
> > >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We
> produce
> > >>>> some
> > >>>>>>> batches to this partition. The content of the segment file will
> be
> > >>>>>>> something like this (for each batch):
> > >>>>>>>
> > >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> > >>>>>>> Length: 123456 (like in classic)
> > >>>>>>> PartitionLeaderEpoch: like in classic
> > >>>>>>> Magic: like in classic
> > >>>>>>> CRC: like in classic
> > >>>>>>> Attributes: like in classic
> > >>>>>>> LastOffsetDelta: like in classic
> > >>>>>>> BaseTimestamp: like in classic
> > >>>>>>> MaxTimestamp: like in classic
> > >>>>>>> ProducerId: like in classic
> > >>>>>>> ProducerEpoch: like in classic
> > >>>>>>> BaseSequence: like in classic
> > >>>>>>> RecordsCount: like in classic
> > >>>>>>> Records:
> > >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte
> offset
> > >>>>>>> 123456
> > >>>>>>>
> > >>>>>>> It looks very much like classic log entries. The only difference
> is
> > >>>> that
> > >>>>>>> instead of writing real Records, we write the reference to the
> WAL
> > >>>> file
> > >>>>>>> with the batch data (I guess we need only the name and the byte
> > >>>> offset,
> > >>>>>>> because the byte length is the standard field above). Otherwise,
> > >>>> it’s a
> > >>>>>>> normal Kafka log with the leader and replicas.
> > >>>>>>>
> > >>>>>>> So we have as many partitions for diskless as for classic. As of
> > open
> > >>>>>> file
> > >>>>>>> descriptors, let’s proceed to the following:
> > >>>>>>>
> > >>>>>>>> Are the metadata and
> > >>>>>>>> the data for the same partition always collocated on the same
> > >>>> broker?
> > >>>>>> If
> > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > >>>>>>>
> > >>>>>>> The source of truth for the data is still in WAL files on object
> > >>>> storage.
> > >>>>>>> The source of truth for the metadata is in segment files on the
> > >>>> brokers
> > >>>>>> in
> > >>>>>>> the replica set. Two new mechanisms are planned, both independent
> > of
> > >>>> this
> > >>>>>>> new proposal, but I want to present them to give the idea that
> > only a
> > >>>>>>> limited amount of data files will be operated locally:
> > >>>>>>> - We want to assemble batches into segment files and offload them
> > to
> > >>>>>>> tiered storage in order to prevent the unbounded growth of batch
> > >>>>>> metadata.
> > >>>>>>> For this, we need to open only  a few file descriptors (for the
> > >>>> segment
> > >>>>>>> file itself + the necessary indexes) before the segment is fully
> > >>>> written
> > >>>>>>> and handed over to RemoteLogManager.
> > >>>>>>> - We want to assemble local segment files for caching purposes as
> > >>>> well,
> > >>>>>>> i.e. to speed up fetching. This will not materialize the full
> > >>>> content of
> > >>>>>>> the log, but only the hot set according to some policy (or
> > >>>> configurable
> > >>>>>>> policies), i.e. the number of segments and file descriptors will
> > >>>> also be
> > >>>>>>> limited.
> > >>>>>>>
> > >>>>>>>> The number of RPCs in the produce path is significantly higher.
> > For
> > >>>>>>>> example, if a produce request has 100 partitions, in a cluster
> > >>>> with 100
> > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > requests.
> > >>>>>> This
> > >>>>>>>> will significantly increase the request rate.
> > >>>>>>>
> > >>>>>>> This is a valid concern that we considered, but this issue can be
> > >>>>>>> mitigated. I’ll try to explain the approach.
> > >>>>>>> The situation with a single broker is trivial: all the commit
> > >>>> requests go
> > >>>>>>> from the broker to itself.
> > >>>>>>> Let’s scale this to a multi-broker cluster, but located in the
> > single
> > >>>>>> rack
> > >>>>>>> (AZ). Any broker can accept Produce requests for diskless
> > >>>> partitions, but
> > >>>>>>> we can tell producers (through metadata) to always send Produce
> > >>>> requests
> > >>>>>> to
> > >>>>>>> leaders. For example, broker 0 hosts the leader replicas for
> > diskless
> > >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> > >>>> requests
> > >>>>>> for
> > >>>>>>> these partitions in various combinations, but only for them.
> > >>>>>>>
> > >>>>>>>                   Broker 0
> > >>>>>>>             +-----------------+
> > >>>>>>>             |    t1-0         |
> > >>>>>>>             |    t2-1 <--------------------+
> > >>>>>>>             |    t3-0         |            |
> > >>>>>>> produce      | +-------------+ |            |
> > >>>>>>> requests     | |  diskless   | |            |
> > >>>>>>> --------------->|   produce   +--------------+
> > >>>>>>> for these    | | WAL buffer  | |    commit requests
> > >>>>>>> partitions   | +-------------+ |    for these partitions
> > >>>>>>>             |                 |
> > >>>>>>>             +-----------------+
> > >>>>>>>
> > >>>>>>> The same applies for other brokers in this cluster. Effectively,
> > each
> > >>>>>>> broker will commit only to itself, which effectively means 1
> commit
> > >>>>>> request
> > >>>>>>> per WAL buffer (this may be 0 physical network calls, if we wish,
> > >>>> just a
> > >>>>>>> local function call).
> > >>>>>>>
> > >>>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we
> cannot
> > >>>> always
> > >>>>>>> send Produce requests to the designated leaders of diskless
> > >>>> partitions:
> > >>>>>>> this would mean inter-AZ network traffic, which we would like to
> > >>>> avoid.
> > >>>>>> To
> > >>>>>>> avoid it, we say that every broker has a “diskless produce
> > >>>>>> representative”
> > >>>>>>> in every AZ. If we continue our example: when a Produce request
> for
> > >>>> t1-0,
> > >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0
> > >>>> (in the
> > >>>>>>> broker’s AZ the representative is the broker itself). However, if
> > it
> > >>>>>> comes
> > >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > >>>>>>>
> > >>>>>>> |produce requests         |produce requests        |produce
> > >>>> requests
> > >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0,
> t2-1,
> > >>>>>> t3-0
> > >>>>>>> |from AZ 0                |from AZ 1               |from AZ 2
> > >>>>>>> v                         v                        v
> > >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > >>>>>>> +---------------+      +---------------+      +---------------+
> > >>>>>>> |     t1-0      |      |               |      |               |
> > >>>>>>> |     t2-1      |      |               |      |               |
> > >>>>>>> |     t3-0      |      |               |      |               |
> > >>>>>>> +---------------+      +--------+------+      +--------+------+
> > >>>>>>>    ^     ^                    |                      |
> > >>>>>>>    |     +--------------------+                      |
> > >>>>>>>    |     commit requests for these partitions        |
> > >>>>>>>    |                                                 |
> > >>>>>>>    +-------------------------------------------------+
> > >>>>>>>          commit requests for these partitions
> > >>>>>>>
> > >>>>>>> All the partitions that broker 0 is the leader of will be
> > >>>> “represented”
> > >>>>>> by
> > >>>>>>> brokers 1 and 2 in their AZs.
> > >>>>>>>
> > >>>>>>> Of course, this relationship goes both ways between AZs (not
> > >>>> necessarily
> > >>>>>>> between the same brokers). It means that provided the cluster is
> > >>>> balanced
> > >>>>>>> by the number of brokers per AZ, each broker will represent
> > >>>>>> (number_of_azs
> > >>>>>>> - 1) other brokers. This will result in the situation that for
> the
> > >>>>>> majority
> > >>>>>>> of commits, each broker will do up to (number_of_azs - 1) network
> > >>>> commit
> > >>>>>>> requests (plus one local). Cloud regions tend to have 3 AZs, very
> > >>>> rarely
> > >>>>>>> more. That means, brokers will be doing up to 2 network commit
> > >>>> requests
> > >>>>>> per
> > >>>>>>> WAL file.
> > >>>>>>>
> > >>>>>>> There are the following exceptions:
> > >>>>>>> 1. Broker count imbalance between AZs. For example, when we have
> 2
> > >>>> AZs
> > >>>>>> and
> > >>>>>>> one has three brokers and another AZ has one. This one broker
> will
> > do
> > >>>>>>> between 1 and 3 commit requests per WAL file. This is not an
> > extreme
> > >>>>>>> amplification. Such an imbalance is not healthy in most practical
> > >>>> setups
> > >>>>>>> and should be avoided anyway.
> > >>>>>>> 2. Leadership changes and metadata propagation period. When the
> > >>>> partition
> > >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the producers
> > will
> > >>>> not
> > >>>>>>> know this immediately (unless we want to be strict and respond
> with
> > >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
> > >>>> together
> > >>>>>> in a
> > >>>>>>> WAL buffer on broker 2, it will have to send two commit requests:
> > to
> > >>>>>> broker
> > >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
> > >>>> situation
> > >>>>>>> is not permanent and as producers update the cluster metadata, it
> > >>>> will be
> > >>>>>>> resolved.
> > >>>>>>>
> > >>>>>>> This all could be built with the metadata crafting mechanism only
> > >>>> (which
> > >>>>>>> is anyway needed for Diskless in one way or another to direct
> > >>>> producers
> > >>>>>> and
> > >>>>>>> consumers where we need to avoid inter-AZ traffic), just with the
> > >>>> right
> > >>>>>>> policy for it (for example, some deterministic hash-based
> formula).
> > >>>> I.e.
> > >>>>>> no
> > >>>>>>> explicit support for “produce representative” or anything like
> this
> > >>>> is
> > >>>>>>> needed on the cluster level, in KRaft, etc.
> > >>>>>>>
> > >>>>>>>> The same WAL file metadata is now duplicated into two places,
> > >>>> partition
> > >>>>>>>> leader and WAL File Manager. Which one is the source of truth,
> and
> > >>>> how
> > >>>>>> do
> > >>>>>>>> we maintain consistency between the two places?
> > >>>>>>>
> > >>>>>>> We do only two operations on WAL files that span multiple
> diskless
> > >>>>>>> partitions: committing and deleting. Commits can be done
> > >>>> independently as
> > >>>>>>> described above. But deletes are different, because when a file
> is
> > >>>>>> deleted,
> > >>>>>>> this affects all the partitions that still have alive batches in
> > this
> > >>>>>> file
> > >>>>>>> (if any).
> > >>>>>>>
> > >>>>>>> The WAL file manager is a necessary point of coordination to
> delete
> > >>>> WAL
> > >>>>>>> files safely. We can say it is the source of truth about files
> > >>>>>> themselves,
> > >>>>>>> while the partition leaders and their logs hold the truth about
> > >>>> whether a
> > >>>>>>> particular file contains live batches of this particular
> partition.
> > >>>>>>>
> > >>>>>>> The file manager will do this important task: be able to say for
> > sure
> > >>>>>> that
> > >>>>>>> a file does not contain any live batch of any existing partition.
> > For
> > >>>>>> this,
> > >>>>>>> it will have to periodically check against the partition leaders.
> > >>>>>>> Considering that batch deletion is irreversible, when we declare
> a
> > >>>> file
> > >>>>>>> “empty”, this is guaranteed to be and stay so.
> > >>>>>>>
> > >>>>>>> The file manager has to know about files being committed to start
> > >>>> track
> > >>>>>>> them and periodically check if they are empty. We can consider
> > >>>> various
> > >>>>>> ways
> > >>>>>>> to achieve this:
> > >>>>>>> 1. As was proposed in my previous message: best effort commit by
> > >>>> brokers
> > >>>>>> +
> > >>>>>>> periodic prefix scans of object storage to detect files that went
> > >>>> below
> > >>>>>> the
> > >>>>>>> radar due to network issue or the file manager temporary
> > >>>> unavailability.
> > >>>>>>> We’re speaking about listing the file names only and opening only
> > >>>>>>> previously unknown files in order to find the partitions involved
> > >>>> with
> > >>>>>> them.
> > >>>>>>> 2. Only do scans without explicit commit, i.e. fill the list of
> > files
> > >>>>>>> fully asynchronously and in the background. This may be not ideal
> > >>>> due to
> > >>>>>>> costs and performance of scanning tons of files. However, the
> > number
> > >>>> of
> > >>>>>>> live WAL files should be limited due to tiered storage
> offloading +
> > >>>> we
> > >>>>>> can
> > >>>>>>> optimize this if we give files some global soft order in their
> > names.
> > >>>>>>>
> > >>>>>>>> I am not sure how this design simplifies the implementation. The
> > >>>>>> existing
> > >>>>>>>> producer/replication code can't be simply reused. Adjusting both
> > >>>> the
> > >>>>>>> write
> > >>>>>>>> path in the leader and the replication path in the follower to
> > >>>>>> understand
> > >>>>>>>> batch-header only data is quite intrusive to the existing logic.
> > >>>>>>>
> > >>>>>>> It is true that we’ll have to change LocalLog and UnifiedLog in
> > >>>> order to
> > >>>>>>> support these changes. However, it seems that idempotence,
> > >>>> transactions,
> > >>>>>>> queues, tiered storage will have to be changed less than with the
> > >>>>>> original
> > >>>>>>> design. This is because the partition leader state would remain
> in
> > >>>> the
> > >>>>>> same
> > >>>>>>> place (on brokers) and existing workflows that involve it would
> > have
> > >>>> to
> > >>>>>> be
> > >>>>>>> changed less compared to the situation where we globalize the
> > >>>> partition
> > >>>>>>> leader state in the batch coordinator. I admit this is hard to
> make
> > >>>>>>> convincing without both real implementations to hand :)
> > >>>>>>>
> > >>>>>>>> I am also
> > >>>>>>>> not sure how this enables seamless switching the topic modes
> > >>>> between
> > >>>>>>>> diskless and classic. Could you provide more details on those?
> > >>>>>>>
> > >>>>>>> Let’s consider the scenario of turning a classic topic into
> > >>>> diskless. The
> > >>>>>>> user sets diskless.enabled=true, the leader receives this
> metadata
> > >>>> update
> > >>>>>>> and does the following:
> > >>>>>>> 1. Stop accepting normal append writes.
> > >>>>>>> 2. Close the current active segment.
> > >>>>>>> 3. Start a new segment that will be written in the diskless
> format
> > >>>> (i.e.
> > >>>>>>> without data).
> > >>>>>>> 4. Start accepting diskless commits.
> > >>>>>>>
> > >>>>>>> Since it’s the same log, the followers will know about that
> switch
> > >>>>>>> consistently. They will finish replicating the classic segments
> and
> > >>>> start
> > >>>>>>> replicating the diskless ones. They will always know where each
> > >>>> batch is
> > >>>>>>> located (either inside a classic segment or referenced by a
> > diskless
> > >>>>>> one).
> > >>>>>>> Switching back should be similar.
> > >>>>>>>
> > >>>>>>> Doing this with the coordinator is possible, but has some
> caveats.
> > >>>> The
> > >>>>>>> leader must do the following:
> > >>>>>>> 1. Stop accepting normal append writes.
> > >>>>>>> 2. Close the current active segment.
> > >>>>>>> 3. Write a special control segment to persist and replicate the
> > fact
> > >>>> that
> > >>>>>>> from offset N the partition is now in the diskless mode.
> > >>>>>>> 4. Inform the coordinator about the first offset N of the
> “diskless
> > >>>> era”.
> > >>>>>>> 5. Inform the controller quorum that the transition has finished
> > and
> > >>>> that
> > >>>>>>> brokers now can process diskless writes for this partition.
> > >>>>>>> This could fail at some points, so this will probably require
> some
> > >>>>>>> explicit state machine with replication either in the partition
> log
> > >>>> or in
> > >>>>>>> KRaft.
> > >>>>>>>
> > >>>>>>> It seems that the coordinator-less approach makes this simpler
> > >>>> because
> > >>>>>> the
> > >>>>>>> “coordinator” for the partition and the partition leader are the
> > >>>> same and
> > >>>>>>> they store the partition metadata in the same log, too. While in
> > the
> > >>>>>>> coordinator approach we have to perform some kind of a
> distributed
> > >>>> commit
> > >>>>>>> to handover metadata management from the classic partition leader
> > to
> > >>>> the
> > >>>>>>> batch coordinator.
> > >>>>>>>
> > >>>>>>> I hope these explanations help to clarify the idea. Please let me
> > >>>> know if
> > >>>>>>> I should go deeper anywhere.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Ivan and the Diskless team
> > >>>>>>>
> > >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > >>>>>>>> Hi, Ivan,
> > >>>>>>>>
> > >>>>>>>> Thanks for the update.
> > >>>>>>>>
> > >>>>>>>> I am not sure that I fully understand the new design, but it
> seems
> > >>>> less
> > >>>>>>>> clean than before.
> > >>>>>>>>
> > >>>>>>>> Does each partition now have a metadata partition and a separate
> > >>>> data
> > >>>>>>>> partition? If so, I am concerned that it essentially doubles the
> > >>>> number
> > >>>>>>> of
> > >>>>>>>> partitions, which impacts the number of open file descriptors
> and
> > >>>> the
> > >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> > separate
> > >>>>>>>> partition just to store the metadata. It's as if we are creating
> > an
> > >>>>>>>> internal topic with an unbounded number of partitions. Are the
> > >>>> metadata
> > >>>>>>> and
> > >>>>>>>> the data for the same partition always collocated on the same
> > >>>> broker?
> > >>>>>> If
> > >>>>>>>> so, how do we enforce that when replicas are reassigned?
> > >>>>>>>>
> > >>>>>>>> The number of RPCs in the produce path is significantly higher.
> > For
> > >>>>>>>> example, if a produce request has 100 partitions, in a cluster
> > >>>> with 100
> > >>>>>>>> brokers, each produce request could generate 100 more RPC
> > requests.
> > >>>>>> This
> > >>>>>>>> will significantly increase the request rate.
> > >>>>>>>>
> > >>>>>>>> The same WAL file metadata is now duplicated into two places,
> > >>>> partition
> > >>>>>>>> leader and WAL File Manager. Which one is the source of truth,
> and
> > >>>> how
> > >>>>>> do
> > >>>>>>>> we maintain consistency between the two places?
> > >>>>>>>>
> > >>>>>>>> I am not sure how this design simplifies the implementation. The
> > >>>>>> existing
> > >>>>>>>> producer/replication code can't be simply reused. Adjusting both
> > >>>> the
> > >>>>>>> write
> > >>>>>>>> path in the leader and the replication path in the follower to
> > >>>>>> understand
> > >>>>>>>> batch-header only data is quite intrusive to the existing
> logic. I
> > >>>> am
> > >>>>>>> also
> > >>>>>>>> not sure how this enables seamless switching the topic modes
> > >>>> between
> > >>>>>>>> diskless and classic. Could you provide more details on those?
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me>
> > >>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi dear Kafka community,
> > >>>>>>>>>
> > >>>>>>>>> In the initial Diskless proposal, we proposed to have a
> separate
> > >>>>>>>>> component, batch/diskless coordinator, whose role would be to
> > >>>>>> centrally
> > >>>>>>>>> manage the batch and WAL file metadata for diskless topics.
> This
> > >>>>>>> component
> > >>>>>>>>> drew many reasonable comments from the community about how it
> > >>>> would
> > >>>>>>> support
> > >>>>>>>>> various Kafka features (transactions, queues) and its
> > >>>> scalability.
> > >>>>>>> While we
> > >>>>>>>>> believe we have good answers to all the expressed concerns, we
> > >>>> took a
> > >>>>>>> step
> > >>>>>>>>> back and looked at the problem from a different perspective.
> > >>>>>>>>>
> > >>>>>>>>> We would like to propose an alternative Diskless design
> *without
> > >>>> a
> > >>>>>>>>> centralized coordinator*. We believe this approach has
> potential
> > >>>> and
> > >>>>>>>>> propose to discuss it as it may be more appealing to the
> > >>>> community.
> > >>>>>>>>>
> > >>>>>>>>> Let us explain the idea. Most of the complications with the
> > >>>> original
> > >>>>>>>>> Diskless approach come from one necessary architecture change:
> > >>>>>>> globalizing
> > >>>>>>>>> the local state of partition leader in the batch coordinator.
> > >>>> This
> > >>>>>>> causes
> > >>>>>>>>> deviations to the established workflows in various features
> like
> > >>>>>>> produce
> > >>>>>>>>> idempotence and transactions, queues, retention, etc. These
> > >>>>>> deviations
> > >>>>>>> need
> > >>>>>>>>> to be carefully considered, designed, and later implemented and
> > >>>>>>> tested. In
> > >>>>>>>>> the new approach we want to avoid this by making partition
> > >>>> leaders
> > >>>>>>> again
> > >>>>>>>>> responsible for managing their partitions, even in diskless
> > >>>> topics.
> > >>>>>>>>>
> > >>>>>>>>> In classic Kafka topics, batch data and metadata are blended
> > >>>> together
> > >>>>>>> in
> > >>>>>>>>> the one partition log. The crux of the Diskless idea is to
> > >>>> decouple
> > >>>>>>> them
> > >>>>>>>>> and move data to the remote storage, while keeping metadata
> > >>>> somewhere
> > >>>>>>> else.
> > >>>>>>>>> Using the central batch coordinator for managing batch metadata
> > >>>> is
> > >>>>>> one
> > >>>>>>> way,
> > >>>>>>>>> but not the only.
> > >>>>>>>>>
> > >>>>>>>>> Let’s now think about managing metadata for each user partition
> > >>>>>>>>> independently. Generally partitions are independent and don’t
> > >>>> share
> > >>>>>>>>> anything apart from that their data are mixed in WAL files. If
> we
> > >>>>>>> figure
> > >>>>>>>>> out how to commit and later delete WAL files safely, we will
> > >>>> achieve
> > >>>>>>> the
> > >>>>>>>>> necessary autonomy that allows us to get rid of the central
> batch
> > >>>>>>>>> coordinator. Instead, *each diskless user partition will be
> > >>>> managed
> > >>>>>> by
> > >>>>>>> its
> > >>>>>>>>> leader*, as in classic Kafka topics. Also like in classic
> > >>>> topics, the
> > >>>>>>>>> leader uses the partition log as the way to persist batch
> > >>>> metadata,
> > >>>>>>> i.e.
> > >>>>>>>>> the regular batch header + the information about how to find
> this
> > >>>>>>> batch on
> > >>>>>>>>> remote storage. In contrast to classic topics, batch data is in
> > >>>>>> remote
> > >>>>>>>>> storage.
> > >>>>>>>>>
> > >>>>>>>>> For clarity, let’s compare the three designs:
> > >>>>>>>>> • Classic topics:
> > >>>>>>>>>  • Data and metadata are co-located in the partition log.
> > >>>>>>>>>  • The partition log content: [Batch header (metadata)|Batch
> > >>>> data].
> > >>>>>>>>>  • The partition log is replicated to the followers.
> > >>>>>>>>>  • The replicas and leader have local state built from
> > >>>> metadata.
> > >>>>>>>>> • Original Diskless:
> > >>>>>>>>>  • Metadata is in the batch coordinator, data is on remote
> > >>>> storage.
> > >>>>>>>>>  • The partition state is global in the batch coordinator.
> > >>>>>>>>> • New Diskless:
> > >>>>>>>>>  • Metadata is in the partition log, data is on remote storage.
> > >>>>>>>>>  • Partition log content: [Batch header (metadata)|Batch
> > >>>>>> coordinates
> > >>>>>>> on
> > >>>>>>>>> remote storage].
> > >>>>>>>>>  • The partition log is replicated to the followers.
> > >>>>>>>>>  • The replicas and leader have local state built from
> > >>>> metadata.
> > >>>>>>>>>
> > >>>>>>>>> Let’s consider the produce path. Here’s the reminder of the
> > >>>> original
> > >>>>>>>>> Diskless design:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> The new approach could be depicted as the following:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> As you can see, the main difference is that now instead of a
> > >>>> single
> > >>>>>>> commit
> > >>>>>>>>> request to the batch coordinator, we send multiple parallel
> > >>>> commit
> > >>>>>>> requests
> > >>>>>>>>> to all the leaders of each partition involved in the WAL file.
> > >>>> Each
> > >>>>>> of
> > >>>>>>> them
> > >>>>>>>>> will commit its batches independently, without coordinating
> with
> > >>>>>> other
> > >>>>>>>>> leaders and any other components. Batch data is addressed by
> the
> > >>>> WAL
> > >>>>>>> file
> > >>>>>>>>> name, the byte offset and size, which allows partitions to know
> > >>>>>> nothing
> > >>>>>>>>> about other partitions to access their data in shared WAL
> files.
> > >>>>>>>>>
> > >>>>>>>>> The number of partitions involved in a single WAL file may be
> > >>>> quite
> > >>>>>>> large,
> > >>>>>>>>> e.g. a hundred. A hundred network requests to commit one WAL
> > >>>> file is
> > >>>>>>> very
> > >>>>>>>>> impractical. However, there are ways to reduce this number:
> > >>>>>>>>> 1. Partition leaders are located on brokers. Requests to
> > >>>> leaders on
> > >>>>>>> one
> > >>>>>>>>> broker could be grouped together into a single physical network
> > >>>>>> request
> > >>>>>>>>> (resembling the normal Produce request that may carry batches
> for
> > >>>>>> many
> > >>>>>>>>> partitions inside). This will cap the number of network
> requests
> > >>>> to
> > >>>>>> the
> > >>>>>>>>> number of brokers in the cluster.
> > >>>>>>>>> 2. If we craft the cluster metadata to make producers send
> their
> > >>>>>>> requests
> > >>>>>>>>> to the right brokers (with respect to AZs), we may achieve the
> > >>>> higher
> > >>>>>>>>> concentration of logical commit requests in physical network
> > >>>> requests
> > >>>>>>>>> reducing the number of the latter ones even further, ideally to
> > >>>> one.
> > >>>>>>>>>
> > >>>>>>>>> Obviously, out of multiple commit requests some may fail or
> time
> > >>>> out
> > >>>>>>> for a
> > >>>>>>>>> variety of reasons. This is fine. Some producers will receive
> > >>>> totally
> > >>>>>>> or
> > >>>>>>>>> partially failed responses to their Produce requests, similar
> to
> > >>>> what
> > >>>>>>> they
> > >>>>>>>>> would have received when appending to a classic topic fails or
> > >>>> times
> > >>>>>>> out.
> > >>>>>>>>> If a partition experiences problems, other partitions will not
> be
> > >>>>>>> affected
> > >>>>>>>>> (again, like in classic topics). Of course, the uncommitted
> data
> > >>>> will
> > >>>>>>> be
> > >>>>>>>>> garbage in WAL files. But WAL files are short-lived (batches
> are
> > >>>>>>> constantly
> > >>>>>>>>> assembled into segments and offloaded to tiered storage), so
> this
> > >>>>>>> garbage
> > >>>>>>>>> will be eventually deleted.
> > >>>>>>>>>
> > >>>>>>>>> For safely deleting WAL files we now need to centrally manage
> > >>>> them,
> > >>>>>> as
> > >>>>>>>>> this is the only state and logic that spans multiple
> partitions.
> > >>>> On
> > >>>>>> the
> > >>>>>>>>> diagram, you can see another commit request called “Commit file
> > >>>> (best
> > >>>>>>>>> effort)” going to the WAL File Manager. This manager will be
> > >>>>>>> responsible
> > >>>>>>>>> for the following:
> > >>>>>>>>> 1. Collecting (by requests from brokers) and persisting
> > >>>> information
> > >>>>>>> about
> > >>>>>>>>> committed WAL files.
> > >>>>>>>>> 2. To handle potential failures in file information delivery,
> it
> > >>>>>> will
> > >>>>>>> be
> > >>>>>>>>> doing prefix scan on the remote storage periodically to find
> and
> > >>>>>>> register
> > >>>>>>>>> unknown files. The period of this scan will be configurable and
> > >>>>>> ideally
> > >>>>>>>>> should be quite long.
> > >>>>>>>>> 3. Checking with the relevant partition leaders (after a grace
> > >>>>>>> period) if
> > >>>>>>>>> they still have batches in a particular file.
> > >>>>>>>>> 4. Physically deleting files when they aren’t anymore referred
> > >>>> to by
> > >>>>>>> any
> > >>>>>>>>> partition.
> > >>>>>>>>>
> > >>>>>>>>> This new design offers the following advantages:
> > >>>>>>>>> 1. It simplifies the implementation of many Kafka features such
> > >>>> as
> > >>>>>>>>> idempotence, transactions, queues, tiered storage, retention.
> > >>>> Now we
> > >>>>>>> don’t
> > >>>>>>>>> need to abstract away and reuse the code from partition leaders
> > >>>> in
> > >>>>>> the
> > >>>>>>>>> batch coordinator. Instead, we will literally use the same code
> > >>>> paths
> > >>>>>>> in
> > >>>>>>>>> leaders, with little adaptation. Workflows from classic topics
> > >>>> mostly
> > >>>>>>>>> remain unchanged.
> > >>>>>>>>> For example, it seems that
> > >>>>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for transaction
> > >>>> support
> > >>>>>> on
> > >>>>>>> the
> > >>>>>>>>> partition leader side could be used for diskless topics with
> > >>>> little
> > >>>>>>>>> adaptation. ProducerStateManager, needed for both idempotent
> > >>>> produce
> > >>>>>>> and
> > >>>>>>>>> transactions, would be reused.
> > >>>>>>>>> Another example is share groups support, where the share
> > >>>> partition
> > >>>>>>> leader,
> > >>>>>>>>> being co-located with the partition leader, would execute the
> > >>>> same
> > >>>>>>> logic
> > >>>>>>>>> for both diskless and classic topics.
> > >>>>>>>>> 2. It returns to the familiar partition-based scaling model,
> > >>>> where
> > >>>>>>>>> partitions are independent.
> > >>>>>>>>> 3. It makes the operation and failure patterns closer to the
> > >>>>>> familiar
> > >>>>>>>>> ones from classic topics.
> > >>>>>>>>> 4. It opens a straightforward path to seamless switching the
> > >>>> topics
> > >>>>>>> modes
> > >>>>>>>>> between diskless and classic.
> > >>>>>>>>>
> > >>>>>>>>> The rest of the things remain unchanged compared to the
> previous
> > >>>>>>> Diskless
> > >>>>>>>>> design (after all previous discussions). Such things as local
> > >>>> segment
> > >>>>>>>>> materialization by replicas, the consume path, tiered storage
> > >>>>>>> integration,
> > >>>>>>>>> etc.
> > >>>>>>>>>
> > >>>>>>>>> If the community finds this design more suitable, we will
> update
> > >>>> the
> > >>>>>>>>> KIP(s) accordingly and continue working on it. Please let us
> know
> > >>>>>> what
> > >>>>>>> you
> > >>>>>>>>> think.
> > >>>>>>>>>
> > >>>>>>>>> Best regards,
> > >>>>>>>>> Ivan and Diskless team
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > >>>>>>>>>> Hi Justine,
> > >>>>>>>>>>
> > >>>>>>>>>> Yes, you're right. We need to track the aborted transactions
> > >>>> for in
> > >>>>>>> the
> > >>>>>>>>> diskless coordinator for as long as the corresponding offsets
> are
> > >>>>>>> there.
> > >>>>>>>>> With the tiered storage unification Greg mentioned earlier,
> this
> > >>>> will
> > >>>>>>> be
> > >>>>>>>>> finite time even for infinite data retention.
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Ivan
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > >>>>>>>>>>> Hey Ivan,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the response. I think most of what you said made
> > >>>>>> sense,
> > >>>>>>> but
> > >>>>>>>>> I
> > >>>>>>>>>>> did have some questions about this part:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > >>>> topics
> > >>>>>>> forgets
> > >>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> > >>>> it). The
> > >>>>>>>>>>> transaction coordinator acts like the main guardian, allowing
> > >>>>>>> partition
> > >>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > >>>> wrong. We
> > >>>>>>> think
> > >>>>>>>>>>> about relying on this with the batch coordinator and delete
> > >>>> the
> > >>>>>>>>> information
> > >>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > >>>> replication
> > >>>>>>> and
> > >>>>>>>>> HWM
> > >>>>>>>>>>> advances immediately).
> > >>>>>>>>>>>
> > >>>>>>>>>>> I didn't quite understand this. In classic topics, we have
> > >>>> maps
> > >>>>>> for
> > >>>>>>>>> ongoing
> > >>>>>>>>>>> transactions which remove state when the transaction is
> > >>>> completed
> > >>>>>>> and
> > >>>>>>>>> an
> > >>>>>>>>>>> aborted transactions index which is retained for much longer.
> > >>>>>> Once
> > >>>>>>> the
> > >>>>>>>>>>> transaction is completed, the coordinator is no longer
> > >>>> involved
> > >>>>>> in
> > >>>>>>>>>>> maintaining this partition side state, and it is subject to
> > >>>>>>> compaction
> > >>>>>>>>> etc.
> > >>>>>>>>>>> Looking back at the outline provided above, I didn't see much
> > >>>>>>> about the
> > >>>>>>>>>>> fetch path, so maybe that could be expanded a bit further. I
> > >>>> saw
> > >>>>>>> the
> > >>>>>>>>>>> following in a response:
> > >>>>>>>>>>>> When the broker constructs a fully valid local segment,
> > >>>> all the
> > >>>>>>>>> necessary
> > >>>>>>>>>>> control batches will be inserted and indices, including the
> > >>>>>>> transaction
> > >>>>>>>>>>> index will be built to serve FetchRequests exactly as they
> > >>>> are
> > >>>>>>> today.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Based on this, it seems like we need to retain the
> > >>>> information
> > >>>>>>> about
> > >>>>>>>>>>> aborted txns for longer.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> Justine
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> > >>>> ivan@ivanyu.me>
> > >>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi Justine and all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thank you for your questions!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> > >>>> with
> > >>>>>>>>> producer ID
> > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > >>>> cached
> > >>>>>>>>> locally
> > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > >>>>>> transactions
> > >>>>>>> can
> > >>>>>>>>> be
> > >>>>>>>>>>>> used
> > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > >>>> with
> > >>>>>>>>> producer id +
> > >>>>>>>>>>>>> epoch
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> You’re right that we (probably unintentionally) focused
> > >>>> only on
> > >>>>>>>>> version 2.
> > >>>>>>>>>>>> We can either limit the support to version 2 or consider
> > >>>> using
> > >>>>>>> some
> > >>>>>>>>>>>> surrogates to support version 1.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> > >>>>>>> checks
> > >>>>>>>>> of the
> > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > >>>> like the
> > >>>>>>>>> partition
> > >>>>>>>>>>>>> leader in classic topics would do.
> > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > >>>>>> checking
> > >>>>>>> if
> > >>>>>>>>> the
> > >>>>>>>>>>>>> transaction was still ongoing for example?* *
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Yes, the producer epoch, that the transaction is ongoing,
> > >>>> and
> > >>>>>> of
> > >>>>>>>>> course
> > >>>>>>>>>>>> the normal idempotence checks. What the partition leader
> > >>>> in the
> > >>>>>>>>> classic
> > >>>>>>>>>>>> topics does before appending a batch to the local log
> > >>>> (e.g. in
> > >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> > >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In Diskless,
> > >>>> we
> > >>>>>>>>> unfortunately
> > >>>>>>>>>>>> cannot do these checks before appending the data to the WAL
> > >>>>>>> segment
> > >>>>>>>>> and
> > >>>>>>>>>>>> uploading it, but we can “tombstone” these batches in the
> > >>>> batch
> > >>>>>>>>> coordinator
> > >>>>>>>>>>>> during the final commit.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Is there state about ongoing
> > >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> > >>>> state
> > >>>>>>>>> mentioned
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > >>>> what
> > >>>>>>> state is
> > >>>>>>>>>>>> stored
> > >>>>>>>>>>>>> and when it is stored.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Right, this should have been more explicit. As the
> > >>>> partition
> > >>>>>>> leader
> > >>>>>>>>> tracks
> > >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> > >>>> coordinator
> > >>>>>>> has
> > >>>>>>>>> to as
> > >>>>>>>>>>>> well. So when a transaction starts and ends, the
> > >>>> transaction
> > >>>>>>>>> coordinator
> > >>>>>>>>>>>> must inform the batch coordinator about this.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > >>>> perhaps
> > >>>>>>> that
> > >>>>>>>>> would
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>> stored in the batch coordinator?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Yes. This could be deduced from the committed batches and
> > >>>> other
> > >>>>>>>>>>>> information, but for the sake of performance we’d better
> > >>>> store
> > >>>>>> it
> > >>>>>>>>>>>> explicitly.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> > >>>>>>> state is
> > >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> > >>>>>> cleaned
> > >>>>>>> up?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> As we understand this, the partition leader in classic
> > >>>> topics
> > >>>>>>> forgets
> > >>>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> > >>>> it).
> > >>>>>> The
> > >>>>>>>>>>>> transaction coordinator acts like the main guardian,
> > >>>> allowing
> > >>>>>>>>> partition
> > >>>>>>>>>>>> leaders to do this safely. Please correct me if this is
> > >>>> wrong.
> > >>>>>> We
> > >>>>>>>>> think
> > >>>>>>>>>>>> about relying on this with the batch coordinator and
> > >>>> delete the
> > >>>>>>>>> information
> > >>>>>>>>>>>> about a transaction once it’s finished (as there’s no
> > >>>>>> replication
> > >>>>>>>>> and HWM
> > >>>>>>>>>>>> advances immediately).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Ivan
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > >>>>>>>>>>>>> Hey folks,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Excited to see some updates related to transactions!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I had a few questions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> > >>>> with
> > >>>>>>>>> producer ID
> > >>>>>>>>>>>>> and epoch, the positive result of this check could be
> > >>>> cached
> > >>>>>>>>> locally
> > >>>>>>>>>>>>> Are we saying that only new transaction version 2
> > >>>>>> transactions
> > >>>>>>> can
> > >>>>>>>>> be
> > >>>>>>>>>>>> used
> > >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> > >>>> with
> > >>>>>>>>> producer id +
> > >>>>>>>>>>>>> epoch
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> > >>>>>>> checks
> > >>>>>>>>> of the
> > >>>>>>>>>>>>> batches. This procedure would output the same errors
> > >>>> like the
> > >>>>>>>>> partition
> > >>>>>>>>>>>>> leader in classic topics would do.
> > >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> > >>>>>> checking
> > >>>>>>> if
> > >>>>>>>>> the
> > >>>>>>>>>>>>> transaction was still ongoing for example? Is there state
> > >>>>>> about
> > >>>>>>>>> ongoing
> > >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> > >>>> state
> > >>>>>>>>> mentioned
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>> the End transaction section, but it's not super clear
> > >>>> what
> > >>>>>>> state is
> > >>>>>>>>>>>> stored
> > >>>>>>>>>>>>> and when it is stored.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> > >>>> perhaps
> > >>>>>>> that
> > >>>>>>>>> would
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>> stored in the batch coordinator?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> > >>>>>>> state is
> > >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> > >>>>>> cleaned
> > >>>>>>> up?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > >>>>>>> <ju...@confluent.io.invalid>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi, Greg and Ivan,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the update. A few comments.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> > >>>>>> segments,
> > >>>>>>>>> making
> > >>>>>>>>>>>> use of
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> indexes, page cache, request purgatory, and zero-copy
> > >>>>>>>>> functionality
> > >>>>>>>>>>>> already
> > >>>>>>>>>>>>>> built into classic topics."
> > >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
> > >>>> each
> > >>>>>>>>> partition in
> > >>>>>>>>>>>>>> diskless topics?
> > >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
> > >>>> need
> > >>>>>> to
> > >>>>>>> know
> > >>>>>>>>>>>> aborted
> > >>>>>>>>>>>>>> records. How is that achieved?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> > >>>> transaction is
> > >>>>>>>>> finished
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>> also inserts the control batches in the corresponding
> > >>>> logs
> > >>>>>>> of the
> > >>>>>>>>>>>> involved
> > >>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
> > >>>> level,
> > >>>>>> no
> > >>>>>>>>> actual
> > >>>>>>>>>>>> control
> > >>>>>>>>>>>>>> batches are written to any file. "
> > >>>>>>>>>>>>>> A fetch response could include multiple transactional
> > >>>>>>> batches.
> > >>>>>>>>> How
> > >>>>>>>>>>>> does the
> > >>>>>>>>>>>>>> broker obtain the information about the ending control
> > >>>>>> batch
> > >>>>>>> for
> > >>>>>>>>> each
> > >>>>>>>>>>>>>> batch? Does that mean that a fetch response needs to be
> > >>>>>>> built by
> > >>>>>>>>>>>>>> stitching record batches and generated control batches
> > >>>>>>> together?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition leader
> > >>>> that
> > >>>>>>> all
> > >>>>>>>>>>>> consumers
> > >>>>>>>>>>>>>> are routed to?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this or
> > >>>> it's
> > >>>>>>> too
> > >>>>>>>>>>>>>> implementation-focused?" It would be useful to include
> > >>>>>> enough
> > >>>>>>>>> details
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> understand correctness and performance impact.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
> > >>>>>> producer
> > >>>>>>>>> contain a
> > >>>>>>>>>>>>>> sequence number, which is ordered. If a producer sends
> > >>>>>> every
> > >>>>>>>>> Produce
> > >>>>>>>>>>>>>> request to an arbitrary broker, those requests could
> > >>>> reach
> > >>>>>>> the
> > >>>>>>>>> batch
> > >>>>>>>>>>>>>> coordinator in different order and lead to rejection
> > >>>> of the
> > >>>>>>>>> produce
> > >>>>>>>>>>>>>> requests.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > >>>>>>> ivan@ivanyu.me>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We have also thought in a bit more details about
> > >>>>>>> transactions
> > >>>>>>>>> and
> > >>>>>>>>>>>> queues,
> > >>>>>>>>>>>>>>> here's the plan.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *Transactions*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> The support for transactions in *classic topics* is
> > >>>> based
> > >>>>>>> on
> > >>>>>>>>> precise
> > >>>>>>>>>>>>>>> interactions between three actors: clients (mostly
> > >>>>>>> producers,
> > >>>>>>>>> but
> > >>>>>>>>>>>> also
> > >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> > >>>> classes),
> > >>>>>> and
> > >>>>>>>>>>>> transaction
> > >>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders with
> > >>>>>> their
> > >>>>>>>>> local
> > >>>>>>>>>>>> state
> > >>>>>>>>>>>>>>> (ProducerStateManager and others).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> The high level (some details skipped) workflow is the
> > >>>>>>>>> following.
> > >>>>>>>>>>>> When a
> > >>>>>>>>>>>>>>> transactional Produce request is received by the
> > >>>> broker:
> > >>>>>>>>>>>>>>> 1. For each partition, the partition leader checks
> > >>>> if a
> > >>>>>>>>> non-empty
> > >>>>>>>>>>>>>>> transaction is running for this partition. This is
> > >>>> done
> > >>>>>>> using
> > >>>>>>>>> its
> > >>>>>>>>>>>> local
> > >>>>>>>>>>>>>>> state derived from the log metadata
> > >>>>>> (ProducerStateManager,
> > >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> > >>>>>>>>>>>>>>> 2. The transaction coordinator is informed about all
> > >>>> the
> > >>>>>>>>> partitions
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> > >>>>>>>>>>>>>>> 3. The partition leaders do additional transactional
> > >>>>>>> checks.
> > >>>>>>>>>>>>>>> 4. The partition leaders append the transactional
> > >>>> data to
> > >>>>>>>>> their logs
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>> update some of their state (for example, log the fact
> > >>>>>> that
> > >>>>>>> the
> > >>>>>>>>>>>>>> transaction
> > >>>>>>>>>>>>>>> is running for the partition and its first offset).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> > >>>>>>> directly
> > >>>>>>>>> with
> > >>>>>>>>>>>>>>> EndTxnRequest.
> > >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> > >>>> or
> > >>>>>>>>>>>> PREPARE_ABORT to
> > >>>>>>>>>>>>>>> its log and responds to the producer.
> > >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> > >>>>>>> WriteTxnMarkersRequest to
> > >>>>>>>>> the
> > >>>>>>>>>>>>>> leaders
> > >>>>>>>>>>>>>>> of the involved partitions.
> > >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> > >>>> markers to
> > >>>>>>>>> their logs
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>> respond to the coordinator.
> > >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction state
> > >>>>>>>>>>>> COMPLETE_COMMIT or
> > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
> > >>>> of
> > >>>>>>>>> important
> > >>>>>>>>>>>> state
> > >>>>>>>>>>>>>>> necessary for supporting this workflow is local. The
> > >>>> main
> > >>>>>>>>> challenge
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
> > >>>> are no
> > >>>>>>>>> partition
> > >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
> > >>>> to be
> > >>>>>>>>> globalized
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> > >>>> support
> > >>>>>>>>> idempotent
> > >>>>>>>>>>>>>> produce.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> The high level workflow for *diskless topics* would
> > >>>> look
> > >>>>>>> very
> > >>>>>>>>>>>> similar:
> > >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> > >>>> non-empty
> > >>>>>>>>> transaction
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>>> running for this partition. In contrast to classic
> > >>>>>> topics,
> > >>>>>>>>> this is
> > >>>>>>>>>>>>>> checked
> > >>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
> > >>>> Since a
> > >>>>>>>>> transaction
> > >>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
> > >>>> the
> > >>>>>>> positive
> > >>>>>>>>>>>> result of
> > >>>>>>>>>>>>>>> this check could be cached locally (for the double
> > >>>>>>> configured
> > >>>>>>>>>>>> duration
> > >>>>>>>>>>>>>> of a
> > >>>>>>>>>>>>>>> transaction, for example).
> > >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is informed
> > >>>>>> about
> > >>>>>>> all
> > >>>>>>>>> the
> > >>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
> > >>>> include
> > >>>>>>> them.
> > >>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
> > >>>> side.
> > >>>>>>>>>>>>>>> 4. The broker appends the transactional data to the
> > >>>>>> current
> > >>>>>>>>> shared
> > >>>>>>>>>>>> WAL
> > >>>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
> > >>>> state
> > >>>>>>> for
> > >>>>>>>>> Diskless
> > >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> > >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> > >>>> coordinator
> > >>>>>>> like
> > >>>>>>>>> in the
> > >>>>>>>>>>>>>>> normal produce flow.
> > >>>>>>>>>>>>>>> 6. The batch coordinator does the final transactional
> > >>>>>>> checks
> > >>>>>>>>> of the
> > >>>>>>>>>>>>>>> batches. This procedure would output the same errors
> > >>>> like
> > >>>>>>> the
> > >>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some batches
> > >>>>>> could
> > >>>>>>> be
> > >>>>>>>>>>>> rejected.
> > >>>>>>>>>>>>>>> This means, there will potentially be garbage in the
> > >>>> WAL
> > >>>>>>>>> segment
> > >>>>>>>>>>>> file in
> > >>>>>>>>>>>>>>> case of transactional errors. This is preferable to
> > >>>> doing
> > >>>>>>> more
> > >>>>>>>>>>>> network
> > >>>>>>>>>>>>>>> round trips, especially considering the WAL segments
> > >>>> will
> > >>>>>>> be
> > >>>>>>>>>>>> relatively
> > >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> > >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> > >>>>>>> directly
> > >>>>>>>>> with
> > >>>>>>>>>>>>>>> EndTxnRequest.
> > >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> > >>>> or
> > >>>>>>>>>>>> PREPARE_ABORT to
> > >>>>>>>>>>>>>>> its log and responds to the producer.
> > >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
> > >>>> batch
> > >>>>>>>>> coordinator
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> the transaction is finished.
> > >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> > >>>>>>> transaction is
> > >>>>>>>>>>>> finished
> > >>>>>>>>>>>>>>> and also inserts the control batches in the
> > >>>> corresponding
> > >>>>>>> logs
> > >>>>>>>>> of the
> > >>>>>>>>>>>>>>> involved Diskless topics. This happens only on the
> > >>>>>> metadata
> > >>>>>>>>> level, no
> > >>>>>>>>>>>>>>> actual control batches are written to any file. They
> > >>>> will
> > >>>>>>> be
> > >>>>>>>>>>>> dynamically
> > >>>>>>>>>>>>>>> created on Fetch and other read operations. We could
> > >>>>>>>>> technically
> > >>>>>>>>>>>> write
> > >>>>>>>>>>>>>>> these control batches for real, but this would mean
> > >>>> extra
> > >>>>>>>>> produce
> > >>>>>>>>>>>>>> latency,
> > >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> > >>>> coordinator
> > >>>>>>> and
> > >>>>>>>>> save
> > >>>>>>>>>>>> these
> > >>>>>>>>>>>>>>> milliseconds.
> > >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> > >>>>>>> WriteTxnMarkersRequest to
> > >>>>>>>>> the
> > >>>>>>>>>>>>>> leaders
> > >>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
> > >>>> topics
> > >>>>>>> now.
> > >>>>>>>>>>>>>>> 6. The partition leaders of classic topics write the
> > >>>>>>>>> transaction
> > >>>>>>>>>>>> markers
> > >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> > >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction state
> > >>>>>>>>>>>> COMPLETE_COMMIT or
> > >>>>>>>>>>>>>>> COMPLETE_ABORT.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
> > >>>> get:
> > >>>>>>>>>>>>>>> 1. An extra network round trip between brokers and
> > >>>> the
> > >>>>>>> batch
> > >>>>>>>>>>>> coordinator
> > >>>>>>>>>>>>>>> when a new partition appear in the transaction. To
> > >>>>>>> mitigate the
> > >>>>>>>>>>>> impact of
> > >>>>>>>>>>>>>>> them:
> > >>>>>>>>>>>>>>> - The results will be cached.
> > >>>>>>>>>>>>>>> - The calls for multiple partitions in one Produce
> > >>>>>>> request
> > >>>>>>>>> will be
> > >>>>>>>>>>>>>>> grouped.
> > >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> > >>>> fast
> > >>>>>>>>> response to
> > >>>>>>>>>>>> these
> > >>>>>>>>>>>>>>> RPCs.
> > >>>>>>>>>>>>>>> - The fact that a single producer normally will
> > >>>>>>> communicate
> > >>>>>>>>> with a
> > >>>>>>>>>>>>>>> single broker for the duration of the transaction
> > >>>> further
> > >>>>>>>>> reduces the
> > >>>>>>>>>>>>>>> expected number of round trips.
> > >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> > >>>>>> coordinator
> > >>>>>>> and
> > >>>>>>>>> batch
> > >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> With this proposal, transactions will also be able to
> > >>>>>> span
> > >>>>>>> both
> > >>>>>>>>>>>> classic
> > >>>>>>>>>>>>>>> and Diskless topics.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *Queues*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> The share group coordination and management is a
> > >>>> side job
> > >>>>>>> that
> > >>>>>>>>>>>> doesn't
> > >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> > >>>> replicas,
> > >>>>>>> physical
> > >>>>>>>>>>>> storage
> > >>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>> records, etc.) and non-queue producers and consumers
> > >>>>>>> (Fetch and
> > >>>>>>>>>>>> Produce
> > >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not affected.)
> > >>>> We
> > >>>>>>> don't
> > >>>>>>>>> see any
> > >>>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
> > >>>> with
> > >>>>>>> share
> > >>>>>>>>>>>> groups the
> > >>>>>>>>>>>>>>> same way as classic topics are. Even on the code
> > >>>> level,
> > >>>>>> we
> > >>>>>>>>> don't
> > >>>>>>>>>>>> expect
> > >>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>> serious refactoring: the same reading routines are
> > >>>> used
> > >>>>>>> that
> > >>>>>>>>> are
> > >>>>>>>>>>>> used for
> > >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
> > >>>> too
> > >>>>>>>>>>>>>>> implementation-focused?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>> Ivan
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > >>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thank you all for your questions and design input
> > >>>> on
> > >>>>>>>>> KIP-1150.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
> > >>>> new
> > >>>>>>>>> design. To
> > >>>>>>>>>>>>>>> summarize
> > >>>>>>>>>>>>>>>> the changes:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> > >>>> existing
> > >>>>>>>>> KIP-405
> > >>>>>>>>>>>> Tiered
> > >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
> > >>>>>>> Diskless
> > >>>>>>>>> topic
> > >>>>>>>>>>>> to be
> > >>>>>>>>>>>>>>>> moved to tiered storage.
> > >>>>>>>>>>>>>>>> This lowers the scalability requirements for the
> > >>>> Batch
> > >>>>>>>>> Coordinator
> > >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> > >>>> Tiered
> > >>>>>>> Storage
> > >>>>>>>>>>>> plugin
> > >>>>>>>>>>>>>>>> features such as encryption and alternative data
> > >>>>>> formats.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> > >>>> segments,
> > >>>>>>>>> making use
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> > >>>> zero-copy
> > >>>>>>>>> functionality
> > >>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>> built into classic topics.
> > >>>>>>>>>>>>>>>> However, local segments are now considered cache
> > >>>>>>> elements,
> > >>>>>>>>> do not
> > >>>>>>>>>>>> need
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> be durably stored, and can be built without
> > >>>> contacting
> > >>>>>>> any
> > >>>>>>>>> other
> > >>>>>>>>>>>>>>> replicas.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 3. The design has been simplified substantially, by
> > >>>>>>> removing
> > >>>>>>>>> the
> > >>>>>>>>>>>>>> previous
> > >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> > >>>> component, and
> > >>>>>>>>> "object
> > >>>>>>>>>>>>>>>> compaction/merging" step.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> > >>>> enabled by
> > >>>>>>> the
> > >>>>>>>>> Batch
> > >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
> > >>>>>> earlier
> > >>>>>>>>> design,
> > >>>>>>>>>>>> while
> > >>>>>>>>>>>>>>>> being simpler and integrating better into the
> > >>>> existing
> > >>>>>>>>> ecosystem.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
> > >>>> the
> > >>>>>> new
> > >>>>>>>>> design.
> > >>>>>>>>>>>>>>>> Greg Harris
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > >>>>>>>>> <ju...@confluent.io.invalid>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi, Jan,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> > >>>> of
> > >>>>>> all
> > >>>>>>>>> existing
> > >>>>>>>>>>>>>> client
> > >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> > >>>> supporting
> > >>>>>> APIs
> > >>>>>>>>> like
> > >>>>>>>>>>>>>>> transactions
> > >>>>>>>>>>>>>>>>> and queues.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> > >>>>>>> status of
> > >>>>>>>>> this
> > >>>>>>>>>>>> KIP?
> > >>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> > >>>>>>> months,
> > >>>>>>>>> the
> > >>>>>>>>>>>> vote got
> > >>>>>>>>>>>>>>>>> vetoed
> > >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> > >>>>>> answered
> > >>>>>>>>> since
> > >>>>>>>>>>>> then.
> > >>>>>>>>>>>>>>> KIP-1183
> > >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> > >>>> activity
> > >>>>>>> since
> > >>>>>>>>> May.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> > >>>>>> choices
> > >>>>>>>>> that can
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> > >>>> by
> > >>>>>> far
> > >>>>>>> the
> > >>>>>>>>>>>> dominant
> > >>>>>>>>>>>>>> one
> > >>>>>>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
> > >>>>>>>>> standalone
> > >>>>>>>>>>>>>> products.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> > >>>> hate to
> > >>>>>>> see a
> > >>>>>>>>>>>> stall. So
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> question is: can we get an update?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> > >>>>>> McCabe -
> > >>>>>>>>> have your
> > >>>>>>>>>>>>>>> questions
> > >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> > >>>> do
> > >>>>>> to
> > >>>>>>>>> help? I'm
> > >>>>>>>>>>>>>> deeply
> > >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> > >>>> written
> > >>>>>>> about
> > >>>>>>>>> both?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Kind regards,
> > >>>>>>>>>>>>>>>>>> Jan
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > >>>>>> Kozlovski
> > >>>>>>> <
> > >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> > >>>> main
> > >>>>>> one
> > >>>>>>>>> (just a
> > >>>>>>>>>>>> bunch
> > >>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>> links
> > >>>>>>>>>>>>>>>>>>> to everything)
> > >>>>>>>>>>>>>>>>>>> b) create the email threads
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> > >>>> example, I
> > >>>>>>> was
> > >>>>>>>>>>>> searching
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> > >>>> far
> > >>>>>> as
> > >>>>>>> I
> > >>>>>>>>> can
> > >>>>>>>>>>>> tell, it
> > >>>>>>>>>>>>>>>>> doesn't
> > >>>>>>>>>>>>>>>>>>> exist yet.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> > >>>> having
> > >>>>>>> the
> > >>>>>>>>> root
> > >>>>>>>>>>>> KIP be
> > >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> > >>>> to
> > >>>>>>>>> sub-KIPs
> > >>>>>>>>>>>> where
> > >>>>>>>>>>>>>> were
> > >>>>>>>>>>>>>>>>> aimed
> > >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> > >>>> would
> > >>>>>>> be
> > >>>>>>>>> good to
> > >>>>>>>>>>>>>> create
> > >>>>>>>>>>>>>>>>>> DISCUSS
> > >>>>>>>>>>>>>>>>>>> threads for them all.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>> Stan
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> > >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> > >>>>>>>>> introducing a
> > >>>>>>>>>>>> new
> > >>>>>>>>>>>>>>> type of
> > >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> > >>>> Storage as
> > >>>>>>> the
> > >>>>>>>>> primary
> > >>>>>>>>>>>>>>> source of
> > >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> > >>>> decided
> > >>>>>>> to
> > >>>>>>>>> split it
> > >>>>>>>>>>>>>> into
> > >>>>>>>>>>>>>>>>>> multiple
> > >>>>>>>>>>>>>>>>>>>> related KIPs.
> > >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > >>>>>>>>>>>>>>>>>>> )
> > >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> > >>>> should aim
> > >>>>>>> to
> > >>>>>>>>> have
> > >>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> type of
> > >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> > >>>>>> details
> > >>>>>>> on
> > >>>>>>>>> how to
> > >>>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>>> it.
> > >>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
> > >>>>>>> discussed
> > >>>>>>>>>>>> KRaft.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> > >>>> hard
> > >>>>>>> to
> > >>>>>>>>> discuss
> > >>>>>>>>>>>> on
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> meta
> > >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> > >>>> (linked
> > >>>>>>> in
> > >>>>>>>>>>>> KIP-1150)
> > >>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> offer
> > >>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> > >>>>>>> threads
> > >>>>>>>>> for
> > >>>>>>>>>>>> each
> > >>>>>>>>>>>>>>> type of
> > >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> > >>>> whether
> > >>>>>>> Apache
> > >>>>>>>>> Kafka
> > >>>>>>>>>>>> wants
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>>>> this feature or not.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> > >>>> authors
> > >>>>>> of
> > >>>>>>>>> this KIP.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> ------------------
> > >>>>>>>>>>>>>>>>>>>> Josep Prat
> > >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> > >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> > >>>>>>> aiven.io
> > >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> > >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> > >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> > >>>>>>> Valtonen,
> > >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> > >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Greg Harris <gr...@aiven.io.INVALID>]

Hi all,

There was a video call between myself, Ivan Yurchenko, Jun Rao, and Andrew
Schofield pertaining to KIP-1150. Here are the notes from that meeting:

Ivan: What is the future state of Kafka in this area, in 5 years?
Jun: Do we want something more cloud native? Yes, started with Tiered
Storage. If there’s a better way, we should explore it. In the long term
this will be useful
Because Kafka is used so widely, we need to make sure everything we add is
for the long term and for everyone, not just for a single company.
When we add TS, it doesn’t just solve Uber’s use-case. We want something
that’s high quality/lasts/maintainable, and can work with all existing
capabilities.
If both 1150 and 1176 proceed at the same time, it’s confusing. They
overlap, but Diskless is more ambitious.
If both KIPs are being seriously worked on, then we don’t really need both,
because Diskless clearly is better. Having multiple will confuse people. It
will duplicate some of the effort.
If we want diskless ultimately, what is the short term strategy, to get
some early wins first?
Ivan: Andrew, do you want a more revolutionary approach?
Andrew: Eventually the architecture will change substantially, it may not
be necessary to put all of that bill onto Diskless at once.
Greg: We all agree on having a high quality feature merged upstream, and
supporting all APIs
Jun: We should try and keep things simple, but there is some minimum
complexity needed.
When doing the short term changes (1176), it doesn’t really progress in
changing to a more modern architecture.
Greg: Was TS+Compaction the only feature miss we’ve had so far?
Jun: The danger of only applying changes to some part of the API, you set
the precedence that you only have to implement part of the API. Supporting
the full API set should be a minimum requirement.
Andrew: When we started Kraft, how much did we know the design?
Jun: For Kraft we didn’t really know much about the migration, but the
high-level was clear.
Greg: Is 1150 votable in its current state?
Jun: 1150 should promise to support all APIs. It doesn’t have to have all
the details/apis/etc. KIP-500 didn’t have it.
We do need some high-level design enough to give confidence that the
promise is able to be fulfilled.
Greg: Is the draft version in 1163 enough detail or is more needed?
Jun: We need to agree on the core design, such as leaderless etc. And how
the APIs will be supported.
Greg: Okay we can include these things, and provide a sketch of how the
other leader-based features operate.
Jun: Yeah if at a high level the sketch appears to work, we can approve
that functionality.
Are you committed to doing the more involved and big project?
Greg: Yes, we’re committed to the 1163 design and can’t really accept 1176.
Jun: TS was slow because of Uber resourcing problems
Greg: We’ll push internally for resources, and use the community sentiment
to motivate Aiven.
How far into the future should we look? What sort of scale?
Jun: As long as there’s a path forward, and we’re not closing off future
improvements, we can figure out how to handle a larger scale when it arises.
Greg: Random replica placement is very harmful, can we recommend users to
use an external tool like CruiseControl?
Jun: Not everyone uses CruiseControl, we would probably need some solution
for this out of the box
Ivan: Should the Batch Coordinator be pluggable?
Jun: Out-of-box experience should be good, good to allow other
implementations
Greg: But it could hurt Kafka feature/upgrade velocity when we wait for
plugin providers to implement it
Ivan: We imagined that maybe cloud hyperscalers could implement it with
e.g. dynamodb
Greg: Could we bake more details of the different providers into Kafka, or
does it still make sense for it to be pluggable?
Jun: Make it whatever is easiest to roll out and add new clients
Andrew: What happens next? Do you want to get KIP-1150 voted?
Ivan: The vote is already open, we’re not too pressed for time. We’ll go
improve the 1163 design and communication.
Is 1176 a competing design? Someone will ask.
Jun: If we are seriously working on something more ambitious, yeah we
shouldn’t do the stop-gap solution.
It’s diverting review resources. If we can get the short term thing in 1yr
but Diskless solution is 2y it makes sense to go for Diskless. If it’s 5yr,
that’s different and maybe the stop-gap solution is needed.
Greg: I’m biased but I believe we’re in the 1yr/2yr case. Should we
explicitly exclude 1176?
Andrew: Put your arms around the feature set you actually want, and use
that to rule out 1176.
Probably don’t need -1 votes, most likely KIPs just don’t receive votes.
Ivan: Should we have sync meetings like tiered storage did?
Jun: Satish posted meeting notes regularly, we should do the same.

To summarize, we will be polishing the contents of 1150 & high level design
in 1163 to prepare for a vote.
We believe that the community should select the feature set of 1150 to
fully eliminate producer cross-zone costs, and make the investment in a
high quality Diskless Topics implementation rather than in stop-gap
solutions.

Thanks,
Greg

On Fri, Nov 7, 2025 at 9:19 PM Max fortun <ma...@maxf.net> wrote:

> This may be a tangent, but we needed to offload storage off of Kafka into
> S3. We are keeping Kafka not as a source of truth, but as a mostly
> ephemeral broker that can come and go as it pleases. Be that scaling or
> outage. Disks can be destroyed and recreated at will, we still retain data
> and use broker for just that, brokering messages. Not only that, we reduced
> the requirement on the actual Kafka resources by reducing the size of a
> payload via a claim check pattern. Maybe this is an anti–pattern, but it is
> super fast and highly cost efficient. We reworked ProducerRequest to allow
> plugins. We added a custom http plugin that submits every request via a
> persisted connection to a microservice. Microservice stores the payload and
> returns a tiny json metadata object,a claim check, that can be used to find
> the actual data. Think of it as zipping the payload. This claim check
> metadata traverses the pipelines with consumers using the urls in metadata
> to pull what they need. Think unzipping. This allowed us to also pull ONLY
> the data that we need in graphql like manner. So if you have a 100K json
> payload and you need only a subsection, you can pull that by jmespath. When
> you have multiple consumer groups yanking down huge payloads it is
> cumbersome on the broker. When you have the same consumer groups yanking
> down a claim check, and then going out of band directly to the source of
> truth, the broker has some breathing room. Obviously our microservice does
> not go directly to the cloud storage, as that would be too slow. It stores
> the payload in high speed memory cache and returns asap. That memory is
> eventually persisted into S3. The retrieval goest against the cache first,
> then against the S3. Overall a rather cheappy and zippy solution. I tried
> proposing the KIP for this, but there was no excitement. Check this out:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528
>
>
> > On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >
> > Hi, Andrew,
> >
> > If we want to focus only on reducing cross-zone replication costs, there
> is
> > an alternative design in the KIP-1176 discussion thread that seems
> simpler
> > than the proposal here. I am copying the outline of that approach below.
> >
> > 1. A new leader is elected.
> > 2. Leader maintains a first tiered offset, which is initialized to log
> end
> > offset.
> > 3. Leader writes produced data from the client to local log.
> > 4. Leader uploads produced data from all local logs as a combined object
> > 5. Leader stores the metadata for the combined object in memory.
> > 6. If a follower fetch request has an offset >= first tiered offset, the
> > metadata for the corresponding combined object is returned. Otherwise,
> the
> > local data is returned.
> > 7. Leader periodically advances first tiered offset.
> >
> > It's still a bit unnatural, but it could work.
> >
> > Hi, Ivan,
> >
> > Are you still committed to proceeding with the original design of
> KIP-1150?
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <
> andrew_schofield@live.com>
> > wrote:
> >
> >> Hi,
> >> I’ve been following KIP-1150 and friends for a while. I’m going to jump
> >> into the discussions too.
> >>
> >> Looking back at Jack Vanlightly’s message, I am not quite so convinced
> >> that it’s a kind of fork in the road. The primary aim of the effort is
> to
> >> reduce cross-zone replication costs so Apache Kafka is not prohibitively
> >> expensive to use on cloud storage. I think it would be entirely viable
> to
> >> prioritise code reuse for an initial implementation of diskless topics,
> and
> >> we could still have a more cloud-native design in the future. It’s hard
> to
> >> predict what the community will prioritise in the future.
> >>
> >> Of the three major revisions, I’m in the rev3 camp. We can support
> >> leaderless produce requests, first writing WAL segments into object
> >> storage, and then using the regular partition leaders to sequence the
> >> records. The active log segment for a diskless topic will initially
> contain
> >> batch coordinates rather than record batches. The batch coordinates can
> be
> >> resolved from WAL segments for consumers, and also in order to prepare
> log
> >> segments for uploading to tiered storage. Jun is probably correct that
> we
> >> need a more frequent object merging process than tiered storage
> provides.
> >> This is just the transition from write-optimised WAL segments to
> >> read-optimised tiered segments, and all of the object storage-based
> >> implementations of Kafka that I’m aware of do this rearrangement. But
> >> perhaps this more frequent object merging is a pre-GA improvement,
> rather
> >> than a strict requirement for an initial implementation for early access
> >> use.
> >>
> >> For zone-aligned share consumers, the share group assignor is intended
> to
> >> be rack-aware. Consumers should be assigned to partitions with leaders
> in
> >> their zone. The simple assignor is not rack-aware, but it easily could
> be
> >> or we could have a rack-aware assignor.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>
> >>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >>>
> >>> Hi, Ivan,
> >>>
> >>> Thanks for the reply.
> >>>
> >>> "As I understand, you’re speaking about locally materialized segments.
> >> They
> >>> will indeed consume some IOPS. See them as a cache that could always be
> >>> restored from the remote storage. While it’s not ideal, it's still OK
> to
> >>> lose data in them due to a machine crash, for example. Because of this,
> >> we
> >>> can avoid explicit flushing on local materialized segments at all and
> let
> >>> the file system and page cache figure out when to flush optimally. This
> >>> would not eliminate the extra IOPS, but should reduce it dramatically,
> >>> depending on throughput for each partition. We, of course, continue
> >>> flushing the metadata segments as before."
> >>>
> >>> If we have a mix of classic and diskless topics on the same broker,
> it's
> >>> important that the classic topics' data is flushed to disk as quickly
> as
> >>> possible. To achieve this, users typically set dirty_expire_centisecs
> in
> >>> the kernel based on the number of available disk IOPS. Once you set
> this
> >>> number, it applies to all dirty files, including the cached data in
> >>> diskless topics. So, if there are more files actively accumulating
> data,
> >>> the flush frequency and therefore RPO is reduced for classic topics.
> >>>
> >>> "We should have mentioned this explicitly, but this step, in fact,
> >> remains
> >>> in the form of segments offloading to tiered storage. When we assemble
> a
> >>> segment and hand it over to RemoteLogManager, we’re effectively doing
> >>> metadata compaction: replacing a big number of pieces of metadata about
> >>> individual batches with a single record in __remote_log_metadata."
> >>>
> >>> The object merging in tier storage typically only kicks in after a few
> >>> hours. The impact is (1) the amount of accumulated metadata is still
> >> quite
> >>> large; (2) there are many small objects, leading to poor read
> >> performance.
> >>> I think we need a more frequent object merging process than tier
> storage
> >>> provides.
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> >>>
> >>>> Hello Jack, Jun, Luke, and all!
> >>>>
> >>>> Thank you for your messages.
> >>>>
> >>>> Let me first address some of Jun’s comments.
> >>>>
> >>>>> First, it degrades the durability.
> >>>>> For each partition, now there are two files being actively written
> at a
> >>>>> given point of time, one for the data and another for the metadata.
> >>>>> Flushing each file requires a separate IO. If the disk has 1K IOPS
> and
> >> we
> >>>>> have 5K partitions in a broker, currently we can afford to flush each
> >>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we
> double
> >>>> the
> >>>>> number of files per partition, we can only flush each partition every
> >> 10
> >>>>> seconds, which makes RPO twice as bad.
> >>>>
> >>>> As I understand, you’re speaking about locally materialized segments.
> >> They
> >>>> will indeed consume some IOPS. See them as a cache that could always
> be
> >>>> restored from the remote storage. While it’s not ideal, it's still OK
> to
> >>>> lose data in them due to a machine crash, for example. Because of
> this,
> >> we
> >>>> can avoid explicit flushing on local materialized segments at all and
> >> let
> >>>> the file system and page cache figure out when to flush optimally.
> This
> >>>> would not eliminate the extra IOPS, but should reduce it dramatically,
> >>>> depending on throughput for each partition. We, of course, continue
> >>>> flushing the metadata segments as before.
> >>>>
> >>>> It’s worth making a note on caching. I think nobody will disagree that
> >>>> doing direct reads from remote storage every time a batch is requested
> >> by a
> >>>> consumer will not be practical neither from the performance nor from
> the
> >>>> economy point of view. We need a way to keep the number of GET
> requests
> >>>> down. There are multiple options, for example:
> >>>> 1. Rack-aware distributed in-memory caching.
> >>>> 2. Local in-memory caching. Comes with less network chattiness and
> >> works
> >>>> well if we have more or less stable brokers to consume from.
> >>>> 3. Materialization of diskless logs on local disk. Way lower impact on
> >>>> RAM and also requires stable brokers for consumption (using just
> >> assigned
> >>>> replicas will probably work well).
> >>>>
> >>>> Materialization is one of possible options, but we can choose another
> >> one.
> >>>> However, we will have this dilemma regardless of whether we have an
> >>>> explicit coordinator or we go “coordinator-less”.
> >>>>
> >>>>> Second, if we ever need this
> >>>>> metadata somewhere else, say in the WAL file manager, the consumer
> >> needs
> >>>> to
> >>>>> subscribe to every partition in the cluster, which is inefficient.
> The
> >>>>> actual benefit of this approach is also questionable. On the surface,
> >> it
> >>>>> might seem that we could reduce the number of lines that need to be
> >>>> changed
> >>>>> for this KIP. However, the changes are quite intrusive to the classic
> >>>>> partition's code path and will probably make the code base harder to
> >>>>> maintain in the long run. I like the original approach based on the
> >> batch
> >>>>> coordinator much better than this one. We could probably refactor the
> >>>>> producer state code so that it could be reused in the batch
> >> coordinator.
> >>>>
> >>>> It’s hard to disagree with this. The explicit coordinator is more a
> side
> >>>> thing, while coordinator-less approach is more about extending
> >>>> ReplicaManager, UnifiedLog and others substantially.
> >>>>
> >>>>> Thanks for addressing the concerns on the number of RPCs in the
> produce
> >>>>> path. I agree that with the metadata crafting mechanism, we could
> >>>> mitigate
> >>>>> the PRC problem. However, since we now require the metadata to be
> >>>>> collocated with the data on the same set of brokers, it's weird that
> >> they
> >>>>> are now managed by different mechanisms. The data assignment now uses
> >> the
> >>>>> metadata crafting mechanism, but the metadata is stored in the
> classic
> >>>>> partition using its own assignment strategy. It will be complicated
> to
> >>>> keep
> >>>>> them collocated.
> >>>>
> >>>> I would like to note that the metadata crafting is needed only to tell
> >>>> producers which brokers they should send Produce requests to, but data
> >> (as
> >>>> in “locally materialized log”) is located on partition replicas, i.e.
> >>>> automatically co-located with metadata.
> >>>>
> >>>> As a side note, it would probably be better that instead of implicitly
> >>>> crafting partition metadata, we extend the metadata protocol so that
> for
> >>>> diskless partitions we return not only the leader and replicas, but
> also
> >>>> some “recommended produce brokers”, selected for optimal performance
> and
> >>>> costs. Producers will pick ones in their racks.
> >>>>
> >>>>> I am also concerned about the removal of the object
> compaction/merging
> >>>>> step.
> >>>>
> >>>> We should have mentioned this explicitly, but this step, in fact,
> >> remains
> >>>> in the form of segments offloading to tiered storage. When we
> assemble a
> >>>> segment and hand it over to RemoteLogManager, we’re effectively doing
> >>>> metadata compaction: replacing a big number of pieces of metadata
> about
> >>>> individual batches with a single record in __remote_log_metadata.
> >>>>
> >>>> We could create a Diskless-specific merging mechanism instead if
> needed.
> >>>> It’s rather easy with the explicit coordinator approach. With the
> >>>> coordinator-less approach, this would probably be a bit more tricky
> >>>> (rewriting the tail of the log by the leader + replicating this change
> >>>> reliably).
> >>>>
> >>>>> I see a tendency toward primarily optimizing for the fewest code
> >> changes
> >>>> in
> >>>>> the KIP. Instead, our primary goal should be a clean design that can
> >> last
> >>>>> for the long term.
> >>>>
> >>>> Yes, totally agree.
> >>>>
> >>>>
> >>>>
> >>>> Luke,
> >>>>> I'm wondering if the complexity of designing txn and queue is because
> >> of
> >>>>> leaderless cluster, do you think it will be simpler if we only focus
> on
> >>>> the
> >>>>> "diskless" design to handle object compaction/merging to/from the
> >> remote
> >>>>> storage to save the cross-AZ cost?
> >>>>
> >>>> After some evolution of the original proposal, leaderless is now
> >> limited.
> >>>> We only need to be able to accept Produce requests on more than one
> >> broker
> >>>> to eliminate the cross-AZ costs for producers. Do I get it right that
> >> you
> >>>> propose to get rid of this? Or do I misunderstand?
> >>>>
> >>>>
> >>>>
> >>>> Let’s now look at this problem from a higher level, as Jack proposed.
> As
> >>>> it was said, the big choice we need to make is whether we 1) create an
> >>>> explicit batch coordinator; or 2) go for the coordinator-less
> approach,
> >>>> where each diskless partition is managed by its leader as in classic
> >> topics.
> >>>>
> >>>> If we try to compare the two approaches:
> >>>>
> >>>> Pluggability:
> >>>> - Explicit coordinator: Possible. For example, some setups may benefit
> >>>> from batch metadata being stored in a cloud database (such as AWS
> >> DynamoDB
> >>>> or GCP Spanner).
> >>>> - Coordinator-less: Impossible.
> >>>>
> >>>> Scalability and fault tolerance:
> >>>> - Explicit coordinator: Depends on the implementation and it’s also
> >>>> necessary to actively work for it.
> >>>> - Coordinator-less: Closer to classic Kafka topics. Scaling is done by
> >>>> partition placement, partitions could fail independently.
> >>>>
> >>>> Separation of concerns:
> >>>> - Explicit coordinator: Very good. Diskless remains more independent
> >> from
> >>>> classic topics in terms of code and workflows. For example, the
> >>>> above-mentioned non-tiered storage metadata compaction mechanism could
> >> be
> >>>> relatively simply implemented with it. As a flip side of this, some
> >>>> workflows (e.g. transactions) will have to be adapted.
> >>>> - Coordinator-less: Less so. It leans to the opposite: bringing
> diskless
> >>>> closer to classic topics. Some code paths and workflows could be more
> >>>> straightforwardly reused, but they will inevitably have to be adapted
> to
> >>>> accommodate both topic types as also discussed.
> >>>>
> >>>> Cloud-nativeness. This is a vague concept, also related to the
> previous,
> >>>> but let’s try:
> >>>> - Explicit coordinator: Storing and processing metadata separately
> makes
> >>>> it easier for brokers to take different roles, be purely stateless if
> >>>> needed, etc.
> >>>> - Coordinator-less: Less so. Something could be achieved with creative
> >>>> partition placement, but not much.
> >>>>
> >>>> Both seem to have their pros and cons. However, answering Jack’s
> >> question,
> >>>> the explicit coordinator approach may indeed lead to a more flexible
> >> design.
> >>>>
> >>>>
> >>>> The purpose of this deviation in the discussion was to receive a
> >>>> preliminary community evaluation of the coordinator-less approach
> >> without
> >>>> taking on the task of writing a separate KIP and fitting it in the
> >> system
> >>>> of KIP-1150 and its children. We’re open to stopping it and getting
> >> back to
> >>>> working out the coordinator design if the community doesn’t favor the
> >>>> proposed approach.
> >>>>
> >>>> Best,
> >>>> Ivan and Diskless team
> >>>>
> >>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> >>>>> Hi Ivan,
> >>>>>
> >>>>> As Jun pointed out, the updated design seems to have some
> shortcomings
> >>>>> although it simplifies the implementation.
> >>>>>
> >>>>> I'm wondering if the complexity of designing txn and queue is because
> >> of
> >>>>> leaderless cluster, do you think it will be simpler if we only focus
> on
> >>>> the
> >>>>> "diskless" design to handle object compaction/merging to/from the
> >> remote
> >>>>> storage to save the cross-AZ cost?
> >>>>>
> >>>>>
> >>>>> Thank you,
> >>>>> Luke
> >>>>>
> >>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid>
> >>>> wrote:
> >>>>>
> >>>>>> Hi, Ivan,
> >>>>>>
> >>>>>> Thanks for the explanation.
> >>>>>>
> >>>>>> "we write the reference to the WAL file with the batch data"
> >>>>>>
> >>>>>> I understand the approach now, but I think it is a hacky one. There
> >> are
> >>>>>> multiple short comings with this design. First, it degrades the
> >>>> durability.
> >>>>>> For each partition, now there are two files being actively written
> at
> >> a
> >>>>>> given point of time, one for the data and another for the metadata.
> >>>>>> Flushing each file requires a separate IO. If the disk has 1K IOPS
> and
> >>>> we
> >>>>>> have 5K partitions in a broker, currently we can afford to flush
> each
> >>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we
> double
> >>>> the
> >>>>>> number of files per partition, we can only flush each partition
> every
> >>>> 10
> >>>>>> seconds, which makes RPO twice as bad. Second, if we ever need this
> >>>>>> metadata somewhere else, say in the WAL file manager, the consumer
> >>>> needs to
> >>>>>> subscribe to every partition in the cluster, which is inefficient.
> The
> >>>>>> actual benefit of this approach is also questionable. On the
> surface,
> >>>> it
> >>>>>> might seem that we could reduce the number of lines that need to be
> >>>> changed
> >>>>>> for this KIP. However, the changes are quite intrusive to the
> classic
> >>>>>> partition's code path and will probably make the code base harder to
> >>>>>> maintain in the long run. I like the original approach based on the
> >>>> batch
> >>>>>> coordinator much better than this one. We could probably refactor
> the
> >>>>>> producer state code so that it could be reused in the batch
> >>>> coordinator.
> >>>>>>
> >>>>>> Thanks for addressing the concerns on the number of RPCs in the
> >> produce
> >>>>>> path. I agree that with the metadata crafting mechanism, we could
> >>>> mitigate
> >>>>>> the PRC problem. However, since we now require the metadata to be
> >>>>>> collocated with the data on the same set of brokers, it's weird that
> >>>> they
> >>>>>> are now managed by different mechanisms. The data assignment now
> uses
> >>>> the
> >>>>>> metadata crafting mechanism, but the metadata is stored in the
> classic
> >>>>>> partition using its own assignment strategy. It will be complicated
> to
> >>>> keep
> >>>>>> them collocated.
> >>>>>>
> >>>>>> I am also concerned about the removal of the object
> compaction/merging
> >>>>>> step. My first concern is on the amount of metadata that need to be
> >>>> kept.
> >>>>>> Without object compcation, the metadata generated in the produce
> path
> >>>> can
> >>>>>> only be deleted after remote tiering kicks in. Let's say for every
> >>>> 250ms we
> >>>>>> produce 100 byte of metadata per partition. Let's say remoting
> tiering
> >>>>>> kicks in after 5 hours. In a cluster with 100K partitions, we need
> to
> >>>> keep
> >>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
> >>>>>> signficant. A second concern is on performance. Every time we need
> to
> >>>>>> rebuild the caching data, we need to read a bunch of small objects
> >>>> from S3,
> >>>>>> slowing down the building process. If a consumer happens to need
> such
> >>>> data,
> >>>>>> it could slow down the application.
> >>>>>>
> >>>>>> I see a tendency toward primarily optimizing for the fewest code
> >>>> changes in
> >>>>>> the KIP. Instead, our primary goal should be a clean design that can
> >>>> last
> >>>>>> for the long term.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me>
> >>>> wrote:
> >>>>>>
> >>>>>>> Hi Jun,
> >>>>>>>
> >>>>>>> Thank you for your message. I’m sorry that I failed to clearly
> >>>> explain
> >>>>>> the
> >>>>>>> idea. Let me try to fix this.
> >>>>>>>
> >>>>>>>> Does each partition now have a metadata partition and a separate
> >>>> data
> >>>>>>>> partition? If so, I am concerned that it essentially doubles the
> >>>> number
> >>>>>>> of
> >>>>>>>> partitions, which impacts the number of open file descriptors and
> >>>> the
> >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> separate
> >>>>>>>> partition just to store the metadata. It's as if we are creating
> an
> >>>>>>>> internal topic with an unbounded number of partitions.
> >>>>>>>
> >>>>>>> No. There will be only one physical partition per diskless
> >>>> partition. Let
> >>>>>>> me explain this with an example. Let’s say we have a diskless
> >>>> partition
> >>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce
> >>>> some
> >>>>>>> batches to this partition. The content of the segment file will be
> >>>>>>> something like this (for each batch):
> >>>>>>>
> >>>>>>> BaseOffset: 00000000000000000000 (like in classic)
> >>>>>>> Length: 123456 (like in classic)
> >>>>>>> PartitionLeaderEpoch: like in classic
> >>>>>>> Magic: like in classic
> >>>>>>> CRC: like in classic
> >>>>>>> Attributes: like in classic
> >>>>>>> LastOffsetDelta: like in classic
> >>>>>>> BaseTimestamp: like in classic
> >>>>>>> MaxTimestamp: like in classic
> >>>>>>> ProducerId: like in classic
> >>>>>>> ProducerEpoch: like in classic
> >>>>>>> BaseSequence: like in classic
> >>>>>>> RecordsCount: like in classic
> >>>>>>> Records:
> >>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> >>>>>>> 123456
> >>>>>>>
> >>>>>>> It looks very much like classic log entries. The only difference is
> >>>> that
> >>>>>>> instead of writing real Records, we write the reference to the WAL
> >>>> file
> >>>>>>> with the batch data (I guess we need only the name and the byte
> >>>> offset,
> >>>>>>> because the byte length is the standard field above). Otherwise,
> >>>> it’s a
> >>>>>>> normal Kafka log with the leader and replicas.
> >>>>>>>
> >>>>>>> So we have as many partitions for diskless as for classic. As of
> open
> >>>>>> file
> >>>>>>> descriptors, let’s proceed to the following:
> >>>>>>>
> >>>>>>>> Are the metadata and
> >>>>>>>> the data for the same partition always collocated on the same
> >>>> broker?
> >>>>>> If
> >>>>>>>> so, how do we enforce that when replicas are reassigned?
> >>>>>>>
> >>>>>>> The source of truth for the data is still in WAL files on object
> >>>> storage.
> >>>>>>> The source of truth for the metadata is in segment files on the
> >>>> brokers
> >>>>>> in
> >>>>>>> the replica set. Two new mechanisms are planned, both independent
> of
> >>>> this
> >>>>>>> new proposal, but I want to present them to give the idea that
> only a
> >>>>>>> limited amount of data files will be operated locally:
> >>>>>>> - We want to assemble batches into segment files and offload them
> to
> >>>>>>> tiered storage in order to prevent the unbounded growth of batch
> >>>>>> metadata.
> >>>>>>> For this, we need to open only  a few file descriptors (for the
> >>>> segment
> >>>>>>> file itself + the necessary indexes) before the segment is fully
> >>>> written
> >>>>>>> and handed over to RemoteLogManager.
> >>>>>>> - We want to assemble local segment files for caching purposes as
> >>>> well,
> >>>>>>> i.e. to speed up fetching. This will not materialize the full
> >>>> content of
> >>>>>>> the log, but only the hot set according to some policy (or
> >>>> configurable
> >>>>>>> policies), i.e. the number of segments and file descriptors will
> >>>> also be
> >>>>>>> limited.
> >>>>>>>
> >>>>>>>> The number of RPCs in the produce path is significantly higher.
> For
> >>>>>>>> example, if a produce request has 100 partitions, in a cluster
> >>>> with 100
> >>>>>>>> brokers, each produce request could generate 100 more RPC
> requests.
> >>>>>> This
> >>>>>>>> will significantly increase the request rate.
> >>>>>>>
> >>>>>>> This is a valid concern that we considered, but this issue can be
> >>>>>>> mitigated. I’ll try to explain the approach.
> >>>>>>> The situation with a single broker is trivial: all the commit
> >>>> requests go
> >>>>>>> from the broker to itself.
> >>>>>>> Let’s scale this to a multi-broker cluster, but located in the
> single
> >>>>>> rack
> >>>>>>> (AZ). Any broker can accept Produce requests for diskless
> >>>> partitions, but
> >>>>>>> we can tell producers (through metadata) to always send Produce
> >>>> requests
> >>>>>> to
> >>>>>>> leaders. For example, broker 0 hosts the leader replicas for
> diskless
> >>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> >>>> requests
> >>>>>> for
> >>>>>>> these partitions in various combinations, but only for them.
> >>>>>>>
> >>>>>>>                   Broker 0
> >>>>>>>             +-----------------+
> >>>>>>>             |    t1-0         |
> >>>>>>>             |    t2-1 <--------------------+
> >>>>>>>             |    t3-0         |            |
> >>>>>>> produce      | +-------------+ |            |
> >>>>>>> requests     | |  diskless   | |            |
> >>>>>>> --------------->|   produce   +--------------+
> >>>>>>> for these    | | WAL buffer  | |    commit requests
> >>>>>>> partitions   | +-------------+ |    for these partitions
> >>>>>>>             |                 |
> >>>>>>>             +-----------------+
> >>>>>>>
> >>>>>>> The same applies for other brokers in this cluster. Effectively,
> each
> >>>>>>> broker will commit only to itself, which effectively means 1 commit
> >>>>>> request
> >>>>>>> per WAL buffer (this may be 0 physical network calls, if we wish,
> >>>> just a
> >>>>>>> local function call).
> >>>>>>>
> >>>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we cannot
> >>>> always
> >>>>>>> send Produce requests to the designated leaders of diskless
> >>>> partitions:
> >>>>>>> this would mean inter-AZ network traffic, which we would like to
> >>>> avoid.
> >>>>>> To
> >>>>>>> avoid it, we say that every broker has a “diskless produce
> >>>>>> representative”
> >>>>>>> in every AZ. If we continue our example: when a Produce request for
> >>>> t1-0,
> >>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0
> >>>> (in the
> >>>>>>> broker’s AZ the representative is the broker itself). However, if
> it
> >>>>>> comes
> >>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> >>>>>>>
> >>>>>>> |produce requests         |produce requests        |produce
> >>>> requests
> >>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
> >>>>>> t3-0
> >>>>>>> |from AZ 0                |from AZ 1               |from AZ 2
> >>>>>>> v                         v                        v
> >>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> >>>>>>> +---------------+      +---------------+      +---------------+
> >>>>>>> |     t1-0      |      |               |      |               |
> >>>>>>> |     t2-1      |      |               |      |               |
> >>>>>>> |     t3-0      |      |               |      |               |
> >>>>>>> +---------------+      +--------+------+      +--------+------+
> >>>>>>>    ^     ^                    |                      |
> >>>>>>>    |     +--------------------+                      |
> >>>>>>>    |     commit requests for these partitions        |
> >>>>>>>    |                                                 |
> >>>>>>>    +-------------------------------------------------+
> >>>>>>>          commit requests for these partitions
> >>>>>>>
> >>>>>>> All the partitions that broker 0 is the leader of will be
> >>>> “represented”
> >>>>>> by
> >>>>>>> brokers 1 and 2 in their AZs.
> >>>>>>>
> >>>>>>> Of course, this relationship goes both ways between AZs (not
> >>>> necessarily
> >>>>>>> between the same brokers). It means that provided the cluster is
> >>>> balanced
> >>>>>>> by the number of brokers per AZ, each broker will represent
> >>>>>> (number_of_azs
> >>>>>>> - 1) other brokers. This will result in the situation that for the
> >>>>>> majority
> >>>>>>> of commits, each broker will do up to (number_of_azs - 1) network
> >>>> commit
> >>>>>>> requests (plus one local). Cloud regions tend to have 3 AZs, very
> >>>> rarely
> >>>>>>> more. That means, brokers will be doing up to 2 network commit
> >>>> requests
> >>>>>> per
> >>>>>>> WAL file.
> >>>>>>>
> >>>>>>> There are the following exceptions:
> >>>>>>> 1. Broker count imbalance between AZs. For example, when we have 2
> >>>> AZs
> >>>>>> and
> >>>>>>> one has three brokers and another AZ has one. This one broker will
> do
> >>>>>>> between 1 and 3 commit requests per WAL file. This is not an
> extreme
> >>>>>>> amplification. Such an imbalance is not healthy in most practical
> >>>> setups
> >>>>>>> and should be avoided anyway.
> >>>>>>> 2. Leadership changes and metadata propagation period. When the
> >>>> partition
> >>>>>>> t3-0 is relocated from broker 0 to some broker 3, the producers
> will
> >>>> not
> >>>>>>> know this immediately (unless we want to be strict and respond with
> >>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
> >>>> together
> >>>>>> in a
> >>>>>>> WAL buffer on broker 2, it will have to send two commit requests:
> to
> >>>>>> broker
> >>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
> >>>> situation
> >>>>>>> is not permanent and as producers update the cluster metadata, it
> >>>> will be
> >>>>>>> resolved.
> >>>>>>>
> >>>>>>> This all could be built with the metadata crafting mechanism only
> >>>> (which
> >>>>>>> is anyway needed for Diskless in one way or another to direct
> >>>> producers
> >>>>>> and
> >>>>>>> consumers where we need to avoid inter-AZ traffic), just with the
> >>>> right
> >>>>>>> policy for it (for example, some deterministic hash-based formula).
> >>>> I.e.
> >>>>>> no
> >>>>>>> explicit support for “produce representative” or anything like this
> >>>> is
> >>>>>>> needed on the cluster level, in KRaft, etc.
> >>>>>>>
> >>>>>>>> The same WAL file metadata is now duplicated into two places,
> >>>> partition
> >>>>>>>> leader and WAL File Manager. Which one is the source of truth, and
> >>>> how
> >>>>>> do
> >>>>>>>> we maintain consistency between the two places?
> >>>>>>>
> >>>>>>> We do only two operations on WAL files that span multiple diskless
> >>>>>>> partitions: committing and deleting. Commits can be done
> >>>> independently as
> >>>>>>> described above. But deletes are different, because when a file is
> >>>>>> deleted,
> >>>>>>> this affects all the partitions that still have alive batches in
> this
> >>>>>> file
> >>>>>>> (if any).
> >>>>>>>
> >>>>>>> The WAL file manager is a necessary point of coordination to delete
> >>>> WAL
> >>>>>>> files safely. We can say it is the source of truth about files
> >>>>>> themselves,
> >>>>>>> while the partition leaders and their logs hold the truth about
> >>>> whether a
> >>>>>>> particular file contains live batches of this particular partition.
> >>>>>>>
> >>>>>>> The file manager will do this important task: be able to say for
> sure
> >>>>>> that
> >>>>>>> a file does not contain any live batch of any existing partition.
> For
> >>>>>> this,
> >>>>>>> it will have to periodically check against the partition leaders.
> >>>>>>> Considering that batch deletion is irreversible, when we declare a
> >>>> file
> >>>>>>> “empty”, this is guaranteed to be and stay so.
> >>>>>>>
> >>>>>>> The file manager has to know about files being committed to start
> >>>> track
> >>>>>>> them and periodically check if they are empty. We can consider
> >>>> various
> >>>>>> ways
> >>>>>>> to achieve this:
> >>>>>>> 1. As was proposed in my previous message: best effort commit by
> >>>> brokers
> >>>>>> +
> >>>>>>> periodic prefix scans of object storage to detect files that went
> >>>> below
> >>>>>> the
> >>>>>>> radar due to network issue or the file manager temporary
> >>>> unavailability.
> >>>>>>> We’re speaking about listing the file names only and opening only
> >>>>>>> previously unknown files in order to find the partitions involved
> >>>> with
> >>>>>> them.
> >>>>>>> 2. Only do scans without explicit commit, i.e. fill the list of
> files
> >>>>>>> fully asynchronously and in the background. This may be not ideal
> >>>> due to
> >>>>>>> costs and performance of scanning tons of files. However, the
> number
> >>>> of
> >>>>>>> live WAL files should be limited due to tiered storage offloading +
> >>>> we
> >>>>>> can
> >>>>>>> optimize this if we give files some global soft order in their
> names.
> >>>>>>>
> >>>>>>>> I am not sure how this design simplifies the implementation. The
> >>>>>> existing
> >>>>>>>> producer/replication code can't be simply reused. Adjusting both
> >>>> the
> >>>>>>> write
> >>>>>>>> path in the leader and the replication path in the follower to
> >>>>>> understand
> >>>>>>>> batch-header only data is quite intrusive to the existing logic.
> >>>>>>>
> >>>>>>> It is true that we’ll have to change LocalLog and UnifiedLog in
> >>>> order to
> >>>>>>> support these changes. However, it seems that idempotence,
> >>>> transactions,
> >>>>>>> queues, tiered storage will have to be changed less than with the
> >>>>>> original
> >>>>>>> design. This is because the partition leader state would remain in
> >>>> the
> >>>>>> same
> >>>>>>> place (on brokers) and existing workflows that involve it would
> have
> >>>> to
> >>>>>> be
> >>>>>>> changed less compared to the situation where we globalize the
> >>>> partition
> >>>>>>> leader state in the batch coordinator. I admit this is hard to make
> >>>>>>> convincing without both real implementations to hand :)
> >>>>>>>
> >>>>>>>> I am also
> >>>>>>>> not sure how this enables seamless switching the topic modes
> >>>> between
> >>>>>>>> diskless and classic. Could you provide more details on those?
> >>>>>>>
> >>>>>>> Let’s consider the scenario of turning a classic topic into
> >>>> diskless. The
> >>>>>>> user sets diskless.enabled=true, the leader receives this metadata
> >>>> update
> >>>>>>> and does the following:
> >>>>>>> 1. Stop accepting normal append writes.
> >>>>>>> 2. Close the current active segment.
> >>>>>>> 3. Start a new segment that will be written in the diskless format
> >>>> (i.e.
> >>>>>>> without data).
> >>>>>>> 4. Start accepting diskless commits.
> >>>>>>>
> >>>>>>> Since it’s the same log, the followers will know about that switch
> >>>>>>> consistently. They will finish replicating the classic segments and
> >>>> start
> >>>>>>> replicating the diskless ones. They will always know where each
> >>>> batch is
> >>>>>>> located (either inside a classic segment or referenced by a
> diskless
> >>>>>> one).
> >>>>>>> Switching back should be similar.
> >>>>>>>
> >>>>>>> Doing this with the coordinator is possible, but has some caveats.
> >>>> The
> >>>>>>> leader must do the following:
> >>>>>>> 1. Stop accepting normal append writes.
> >>>>>>> 2. Close the current active segment.
> >>>>>>> 3. Write a special control segment to persist and replicate the
> fact
> >>>> that
> >>>>>>> from offset N the partition is now in the diskless mode.
> >>>>>>> 4. Inform the coordinator about the first offset N of the “diskless
> >>>> era”.
> >>>>>>> 5. Inform the controller quorum that the transition has finished
> and
> >>>> that
> >>>>>>> brokers now can process diskless writes for this partition.
> >>>>>>> This could fail at some points, so this will probably require some
> >>>>>>> explicit state machine with replication either in the partition log
> >>>> or in
> >>>>>>> KRaft.
> >>>>>>>
> >>>>>>> It seems that the coordinator-less approach makes this simpler
> >>>> because
> >>>>>> the
> >>>>>>> “coordinator” for the partition and the partition leader are the
> >>>> same and
> >>>>>>> they store the partition metadata in the same log, too. While in
> the
> >>>>>>> coordinator approach we have to perform some kind of a distributed
> >>>> commit
> >>>>>>> to handover metadata management from the classic partition leader
> to
> >>>> the
> >>>>>>> batch coordinator.
> >>>>>>>
> >>>>>>> I hope these explanations help to clarify the idea. Please let me
> >>>> know if
> >>>>>>> I should go deeper anywhere.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Ivan and the Diskless team
> >>>>>>>
> >>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> >>>>>>>> Hi, Ivan,
> >>>>>>>>
> >>>>>>>> Thanks for the update.
> >>>>>>>>
> >>>>>>>> I am not sure that I fully understand the new design, but it seems
> >>>> less
> >>>>>>>> clean than before.
> >>>>>>>>
> >>>>>>>> Does each partition now have a metadata partition and a separate
> >>>> data
> >>>>>>>> partition? If so, I am concerned that it essentially doubles the
> >>>> number
> >>>>>>> of
> >>>>>>>> partitions, which impacts the number of open file descriptors and
> >>>> the
> >>>>>>>> required IOPS, and so on. It also seems wasteful to have a
> separate
> >>>>>>>> partition just to store the metadata. It's as if we are creating
> an
> >>>>>>>> internal topic with an unbounded number of partitions. Are the
> >>>> metadata
> >>>>>>> and
> >>>>>>>> the data for the same partition always collocated on the same
> >>>> broker?
> >>>>>> If
> >>>>>>>> so, how do we enforce that when replicas are reassigned?
> >>>>>>>>
> >>>>>>>> The number of RPCs in the produce path is significantly higher.
> For
> >>>>>>>> example, if a produce request has 100 partitions, in a cluster
> >>>> with 100
> >>>>>>>> brokers, each produce request could generate 100 more RPC
> requests.
> >>>>>> This
> >>>>>>>> will significantly increase the request rate.
> >>>>>>>>
> >>>>>>>> The same WAL file metadata is now duplicated into two places,
> >>>> partition
> >>>>>>>> leader and WAL File Manager. Which one is the source of truth, and
> >>>> how
> >>>>>> do
> >>>>>>>> we maintain consistency between the two places?
> >>>>>>>>
> >>>>>>>> I am not sure how this design simplifies the implementation. The
> >>>>>> existing
> >>>>>>>> producer/replication code can't be simply reused. Adjusting both
> >>>> the
> >>>>>>> write
> >>>>>>>> path in the leader and the replication path in the follower to
> >>>>>> understand
> >>>>>>>> batch-header only data is quite intrusive to the existing logic. I
> >>>> am
> >>>>>>> also
> >>>>>>>> not sure how this enables seamless switching the topic modes
> >>>> between
> >>>>>>>> diskless and classic. Could you provide more details on those?
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi dear Kafka community,
> >>>>>>>>>
> >>>>>>>>> In the initial Diskless proposal, we proposed to have a separate
> >>>>>>>>> component, batch/diskless coordinator, whose role would be to
> >>>>>> centrally
> >>>>>>>>> manage the batch and WAL file metadata for diskless topics. This
> >>>>>>> component
> >>>>>>>>> drew many reasonable comments from the community about how it
> >>>> would
> >>>>>>> support
> >>>>>>>>> various Kafka features (transactions, queues) and its
> >>>> scalability.
> >>>>>>> While we
> >>>>>>>>> believe we have good answers to all the expressed concerns, we
> >>>> took a
> >>>>>>> step
> >>>>>>>>> back and looked at the problem from a different perspective.
> >>>>>>>>>
> >>>>>>>>> We would like to propose an alternative Diskless design *without
> >>>> a
> >>>>>>>>> centralized coordinator*. We believe this approach has potential
> >>>> and
> >>>>>>>>> propose to discuss it as it may be more appealing to the
> >>>> community.
> >>>>>>>>>
> >>>>>>>>> Let us explain the idea. Most of the complications with the
> >>>> original
> >>>>>>>>> Diskless approach come from one necessary architecture change:
> >>>>>>> globalizing
> >>>>>>>>> the local state of partition leader in the batch coordinator.
> >>>> This
> >>>>>>> causes
> >>>>>>>>> deviations to the established workflows in various features like
> >>>>>>> produce
> >>>>>>>>> idempotence and transactions, queues, retention, etc. These
> >>>>>> deviations
> >>>>>>> need
> >>>>>>>>> to be carefully considered, designed, and later implemented and
> >>>>>>> tested. In
> >>>>>>>>> the new approach we want to avoid this by making partition
> >>>> leaders
> >>>>>>> again
> >>>>>>>>> responsible for managing their partitions, even in diskless
> >>>> topics.
> >>>>>>>>>
> >>>>>>>>> In classic Kafka topics, batch data and metadata are blended
> >>>> together
> >>>>>>> in
> >>>>>>>>> the one partition log. The crux of the Diskless idea is to
> >>>> decouple
> >>>>>>> them
> >>>>>>>>> and move data to the remote storage, while keeping metadata
> >>>> somewhere
> >>>>>>> else.
> >>>>>>>>> Using the central batch coordinator for managing batch metadata
> >>>> is
> >>>>>> one
> >>>>>>> way,
> >>>>>>>>> but not the only.
> >>>>>>>>>
> >>>>>>>>> Let’s now think about managing metadata for each user partition
> >>>>>>>>> independently. Generally partitions are independent and don’t
> >>>> share
> >>>>>>>>> anything apart from that their data are mixed in WAL files. If we
> >>>>>>> figure
> >>>>>>>>> out how to commit and later delete WAL files safely, we will
> >>>> achieve
> >>>>>>> the
> >>>>>>>>> necessary autonomy that allows us to get rid of the central batch
> >>>>>>>>> coordinator. Instead, *each diskless user partition will be
> >>>> managed
> >>>>>> by
> >>>>>>> its
> >>>>>>>>> leader*, as in classic Kafka topics. Also like in classic
> >>>> topics, the
> >>>>>>>>> leader uses the partition log as the way to persist batch
> >>>> metadata,
> >>>>>>> i.e.
> >>>>>>>>> the regular batch header + the information about how to find this
> >>>>>>> batch on
> >>>>>>>>> remote storage. In contrast to classic topics, batch data is in
> >>>>>> remote
> >>>>>>>>> storage.
> >>>>>>>>>
> >>>>>>>>> For clarity, let’s compare the three designs:
> >>>>>>>>> • Classic topics:
> >>>>>>>>>  • Data and metadata are co-located in the partition log.
> >>>>>>>>>  • The partition log content: [Batch header (metadata)|Batch
> >>>> data].
> >>>>>>>>>  • The partition log is replicated to the followers.
> >>>>>>>>>  • The replicas and leader have local state built from
> >>>> metadata.
> >>>>>>>>> • Original Diskless:
> >>>>>>>>>  • Metadata is in the batch coordinator, data is on remote
> >>>> storage.
> >>>>>>>>>  • The partition state is global in the batch coordinator.
> >>>>>>>>> • New Diskless:
> >>>>>>>>>  • Metadata is in the partition log, data is on remote storage.
> >>>>>>>>>  • Partition log content: [Batch header (metadata)|Batch
> >>>>>> coordinates
> >>>>>>> on
> >>>>>>>>> remote storage].
> >>>>>>>>>  • The partition log is replicated to the followers.
> >>>>>>>>>  • The replicas and leader have local state built from
> >>>> metadata.
> >>>>>>>>>
> >>>>>>>>> Let’s consider the produce path. Here’s the reminder of the
> >>>> original
> >>>>>>>>> Diskless design:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The new approach could be depicted as the following:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> As you can see, the main difference is that now instead of a
> >>>> single
> >>>>>>> commit
> >>>>>>>>> request to the batch coordinator, we send multiple parallel
> >>>> commit
> >>>>>>> requests
> >>>>>>>>> to all the leaders of each partition involved in the WAL file.
> >>>> Each
> >>>>>> of
> >>>>>>> them
> >>>>>>>>> will commit its batches independently, without coordinating with
> >>>>>> other
> >>>>>>>>> leaders and any other components. Batch data is addressed by the
> >>>> WAL
> >>>>>>> file
> >>>>>>>>> name, the byte offset and size, which allows partitions to know
> >>>>>> nothing
> >>>>>>>>> about other partitions to access their data in shared WAL files.
> >>>>>>>>>
> >>>>>>>>> The number of partitions involved in a single WAL file may be
> >>>> quite
> >>>>>>> large,
> >>>>>>>>> e.g. a hundred. A hundred network requests to commit one WAL
> >>>> file is
> >>>>>>> very
> >>>>>>>>> impractical. However, there are ways to reduce this number:
> >>>>>>>>> 1. Partition leaders are located on brokers. Requests to
> >>>> leaders on
> >>>>>>> one
> >>>>>>>>> broker could be grouped together into a single physical network
> >>>>>> request
> >>>>>>>>> (resembling the normal Produce request that may carry batches for
> >>>>>> many
> >>>>>>>>> partitions inside). This will cap the number of network requests
> >>>> to
> >>>>>> the
> >>>>>>>>> number of brokers in the cluster.
> >>>>>>>>> 2. If we craft the cluster metadata to make producers send their
> >>>>>>> requests
> >>>>>>>>> to the right brokers (with respect to AZs), we may achieve the
> >>>> higher
> >>>>>>>>> concentration of logical commit requests in physical network
> >>>> requests
> >>>>>>>>> reducing the number of the latter ones even further, ideally to
> >>>> one.
> >>>>>>>>>
> >>>>>>>>> Obviously, out of multiple commit requests some may fail or time
> >>>> out
> >>>>>>> for a
> >>>>>>>>> variety of reasons. This is fine. Some producers will receive
> >>>> totally
> >>>>>>> or
> >>>>>>>>> partially failed responses to their Produce requests, similar to
> >>>> what
> >>>>>>> they
> >>>>>>>>> would have received when appending to a classic topic fails or
> >>>> times
> >>>>>>> out.
> >>>>>>>>> If a partition experiences problems, other partitions will not be
> >>>>>>> affected
> >>>>>>>>> (again, like in classic topics). Of course, the uncommitted data
> >>>> will
> >>>>>>> be
> >>>>>>>>> garbage in WAL files. But WAL files are short-lived (batches are
> >>>>>>> constantly
> >>>>>>>>> assembled into segments and offloaded to tiered storage), so this
> >>>>>>> garbage
> >>>>>>>>> will be eventually deleted.
> >>>>>>>>>
> >>>>>>>>> For safely deleting WAL files we now need to centrally manage
> >>>> them,
> >>>>>> as
> >>>>>>>>> this is the only state and logic that spans multiple partitions.
> >>>> On
> >>>>>> the
> >>>>>>>>> diagram, you can see another commit request called “Commit file
> >>>> (best
> >>>>>>>>> effort)” going to the WAL File Manager. This manager will be
> >>>>>>> responsible
> >>>>>>>>> for the following:
> >>>>>>>>> 1. Collecting (by requests from brokers) and persisting
> >>>> information
> >>>>>>> about
> >>>>>>>>> committed WAL files.
> >>>>>>>>> 2. To handle potential failures in file information delivery, it
> >>>>>> will
> >>>>>>> be
> >>>>>>>>> doing prefix scan on the remote storage periodically to find and
> >>>>>>> register
> >>>>>>>>> unknown files. The period of this scan will be configurable and
> >>>>>> ideally
> >>>>>>>>> should be quite long.
> >>>>>>>>> 3. Checking with the relevant partition leaders (after a grace
> >>>>>>> period) if
> >>>>>>>>> they still have batches in a particular file.
> >>>>>>>>> 4. Physically deleting files when they aren’t anymore referred
> >>>> to by
> >>>>>>> any
> >>>>>>>>> partition.
> >>>>>>>>>
> >>>>>>>>> This new design offers the following advantages:
> >>>>>>>>> 1. It simplifies the implementation of many Kafka features such
> >>>> as
> >>>>>>>>> idempotence, transactions, queues, tiered storage, retention.
> >>>> Now we
> >>>>>>> don’t
> >>>>>>>>> need to abstract away and reuse the code from partition leaders
> >>>> in
> >>>>>> the
> >>>>>>>>> batch coordinator. Instead, we will literally use the same code
> >>>> paths
> >>>>>>> in
> >>>>>>>>> leaders, with little adaptation. Workflows from classic topics
> >>>> mostly
> >>>>>>>>> remain unchanged.
> >>>>>>>>> For example, it seems that
> >>>>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> >>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for transaction
> >>>> support
> >>>>>> on
> >>>>>>> the
> >>>>>>>>> partition leader side could be used for diskless topics with
> >>>> little
> >>>>>>>>> adaptation. ProducerStateManager, needed for both idempotent
> >>>> produce
> >>>>>>> and
> >>>>>>>>> transactions, would be reused.
> >>>>>>>>> Another example is share groups support, where the share
> >>>> partition
> >>>>>>> leader,
> >>>>>>>>> being co-located with the partition leader, would execute the
> >>>> same
> >>>>>>> logic
> >>>>>>>>> for both diskless and classic topics.
> >>>>>>>>> 2. It returns to the familiar partition-based scaling model,
> >>>> where
> >>>>>>>>> partitions are independent.
> >>>>>>>>> 3. It makes the operation and failure patterns closer to the
> >>>>>> familiar
> >>>>>>>>> ones from classic topics.
> >>>>>>>>> 4. It opens a straightforward path to seamless switching the
> >>>> topics
> >>>>>>> modes
> >>>>>>>>> between diskless and classic.
> >>>>>>>>>
> >>>>>>>>> The rest of the things remain unchanged compared to the previous
> >>>>>>> Diskless
> >>>>>>>>> design (after all previous discussions). Such things as local
> >>>> segment
> >>>>>>>>> materialization by replicas, the consume path, tiered storage
> >>>>>>> integration,
> >>>>>>>>> etc.
> >>>>>>>>>
> >>>>>>>>> If the community finds this design more suitable, we will update
> >>>> the
> >>>>>>>>> KIP(s) accordingly and continue working on it. Please let us know
> >>>>>> what
> >>>>>>> you
> >>>>>>>>> think.
> >>>>>>>>>
> >>>>>>>>> Best regards,
> >>>>>>>>> Ivan and Diskless team
> >>>>>>>>>
> >>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> >>>>>>>>>> Hi Justine,
> >>>>>>>>>>
> >>>>>>>>>> Yes, you're right. We need to track the aborted transactions
> >>>> for in
> >>>>>>> the
> >>>>>>>>> diskless coordinator for as long as the corresponding offsets are
> >>>>>>> there.
> >>>>>>>>> With the tiered storage unification Greg mentioned earlier, this
> >>>> will
> >>>>>>> be
> >>>>>>>>> finite time even for infinite data retention.
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Ivan
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> >>>>>>>>>>> Hey Ivan,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the response. I think most of what you said made
> >>>>>> sense,
> >>>>>>> but
> >>>>>>>>> I
> >>>>>>>>>>> did have some questions about this part:
> >>>>>>>>>>>
> >>>>>>>>>>>> As we understand this, the partition leader in classic
> >>>> topics
> >>>>>>> forgets
> >>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> >>>> it). The
> >>>>>>>>>>> transaction coordinator acts like the main guardian, allowing
> >>>>>>> partition
> >>>>>>>>>>> leaders to do this safely. Please correct me if this is
> >>>> wrong. We
> >>>>>>> think
> >>>>>>>>>>> about relying on this with the batch coordinator and delete
> >>>> the
> >>>>>>>>> information
> >>>>>>>>>>> about a transaction once it’s finished (as there’s no
> >>>> replication
> >>>>>>> and
> >>>>>>>>> HWM
> >>>>>>>>>>> advances immediately).
> >>>>>>>>>>>
> >>>>>>>>>>> I didn't quite understand this. In classic topics, we have
> >>>> maps
> >>>>>> for
> >>>>>>>>> ongoing
> >>>>>>>>>>> transactions which remove state when the transaction is
> >>>> completed
> >>>>>>> and
> >>>>>>>>> an
> >>>>>>>>>>> aborted transactions index which is retained for much longer.
> >>>>>> Once
> >>>>>>> the
> >>>>>>>>>>> transaction is completed, the coordinator is no longer
> >>>> involved
> >>>>>> in
> >>>>>>>>>>> maintaining this partition side state, and it is subject to
> >>>>>>> compaction
> >>>>>>>>> etc.
> >>>>>>>>>>> Looking back at the outline provided above, I didn't see much
> >>>>>>> about the
> >>>>>>>>>>> fetch path, so maybe that could be expanded a bit further. I
> >>>> saw
> >>>>>>> the
> >>>>>>>>>>> following in a response:
> >>>>>>>>>>>> When the broker constructs a fully valid local segment,
> >>>> all the
> >>>>>>>>> necessary
> >>>>>>>>>>> control batches will be inserted and indices, including the
> >>>>>>> transaction
> >>>>>>>>>>> index will be built to serve FetchRequests exactly as they
> >>>> are
> >>>>>>> today.
> >>>>>>>>>>>
> >>>>>>>>>>> Based on this, it seems like we need to retain the
> >>>> information
> >>>>>>> about
> >>>>>>>>>>> aborted txns for longer.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Justine
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> >>>> ivan@ivanyu.me>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Justine and all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thank you for your questions!
> >>>>>>>>>>>>
> >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> >>>> with
> >>>>>>>>> producer ID
> >>>>>>>>>>>>> and epoch, the positive result of this check could be
> >>>> cached
> >>>>>>>>> locally
> >>>>>>>>>>>>> Are we saying that only new transaction version 2
> >>>>>> transactions
> >>>>>>> can
> >>>>>>>>> be
> >>>>>>>>>>>> used
> >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> >>>> with
> >>>>>>>>> producer id +
> >>>>>>>>>>>>> epoch
> >>>>>>>>>>>>
> >>>>>>>>>>>> You’re right that we (probably unintentionally) focused
> >>>> only on
> >>>>>>>>> version 2.
> >>>>>>>>>>>> We can either limit the support to version 2 or consider
> >>>> using
> >>>>>>> some
> >>>>>>>>>>>> surrogates to support version 1.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> >>>>>>> checks
> >>>>>>>>> of the
> >>>>>>>>>>>>> batches. This procedure would output the same errors
> >>>> like the
> >>>>>>>>> partition
> >>>>>>>>>>>>> leader in classic topics would do.
> >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> >>>>>> checking
> >>>>>>> if
> >>>>>>>>> the
> >>>>>>>>>>>>> transaction was still ongoing for example?* *
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes, the producer epoch, that the transaction is ongoing,
> >>>> and
> >>>>>> of
> >>>>>>>>> course
> >>>>>>>>>>>> the normal idempotence checks. What the partition leader
> >>>> in the
> >>>>>>>>> classic
> >>>>>>>>>>>> topics does before appending a batch to the local log
> >>>> (e.g. in
> >>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> >>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In Diskless,
> >>>> we
> >>>>>>>>> unfortunately
> >>>>>>>>>>>> cannot do these checks before appending the data to the WAL
> >>>>>>> segment
> >>>>>>>>> and
> >>>>>>>>>>>> uploading it, but we can “tombstone” these batches in the
> >>>> batch
> >>>>>>>>> coordinator
> >>>>>>>>>>>> during the final commit.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Is there state about ongoing
> >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> >>>> state
> >>>>>>>>> mentioned
> >>>>>>>>>>>> in
> >>>>>>>>>>>>> the End transaction section, but it's not super clear
> >>>> what
> >>>>>>> state is
> >>>>>>>>>>>> stored
> >>>>>>>>>>>>> and when it is stored.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Right, this should have been more explicit. As the
> >>>> partition
> >>>>>>> leader
> >>>>>>>>> tracks
> >>>>>>>>>>>> ongoing transactions for classic topics, the batch
> >>>> coordinator
> >>>>>>> has
> >>>>>>>>> to as
> >>>>>>>>>>>> well. So when a transaction starts and ends, the
> >>>> transaction
> >>>>>>>>> coordinator
> >>>>>>>>>>>> must inform the batch coordinator about this.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> >>>> perhaps
> >>>>>>> that
> >>>>>>>>> would
> >>>>>>>>>>>> be
> >>>>>>>>>>>>> stored in the batch coordinator?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes. This could be deduced from the committed batches and
> >>>> other
> >>>>>>>>>>>> information, but for the sake of performance we’d better
> >>>> store
> >>>>>> it
> >>>>>>>>>>>> explicitly.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> >>>>>>> state is
> >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> >>>>>> cleaned
> >>>>>>> up?
> >>>>>>>>>>>>
> >>>>>>>>>>>> As we understand this, the partition leader in classic
> >>>> topics
> >>>>>>> forgets
> >>>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> >>>> it).
> >>>>>> The
> >>>>>>>>>>>> transaction coordinator acts like the main guardian,
> >>>> allowing
> >>>>>>>>> partition
> >>>>>>>>>>>> leaders to do this safely. Please correct me if this is
> >>>> wrong.
> >>>>>> We
> >>>>>>>>> think
> >>>>>>>>>>>> about relying on this with the batch coordinator and
> >>>> delete the
> >>>>>>>>> information
> >>>>>>>>>>>> about a transaction once it’s finished (as there’s no
> >>>>>> replication
> >>>>>>>>> and HWM
> >>>>>>>>>>>> advances immediately).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Ivan
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> >>>>>>>>>>>>> Hey folks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Excited to see some updates related to transactions!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I had a few questions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> >>>> with
> >>>>>>>>> producer ID
> >>>>>>>>>>>>> and epoch, the positive result of this check could be
> >>>> cached
> >>>>>>>>> locally
> >>>>>>>>>>>>> Are we saying that only new transaction version 2
> >>>>>> transactions
> >>>>>>> can
> >>>>>>>>> be
> >>>>>>>>>>>> used
> >>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
> >>>> with
> >>>>>>>>> producer id +
> >>>>>>>>>>>>> epoch
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> >>>>>>> checks
> >>>>>>>>> of the
> >>>>>>>>>>>>> batches. This procedure would output the same errors
> >>>> like the
> >>>>>>>>> partition
> >>>>>>>>>>>>> leader in classic topics would do.
> >>>>>>>>>>>>> Can you expand on what these checks are? Would you be
> >>>>>> checking
> >>>>>>> if
> >>>>>>>>> the
> >>>>>>>>>>>>> transaction was still ongoing for example? Is there state
> >>>>>> about
> >>>>>>>>> ongoing
> >>>>>>>>>>>>> transactions in the batch coordinator? I see some other
> >>>> state
> >>>>>>>>> mentioned
> >>>>>>>>>>>> in
> >>>>>>>>>>>>> the End transaction section, but it's not super clear
> >>>> what
> >>>>>>> state is
> >>>>>>>>>>>> stored
> >>>>>>>>>>>>> and when it is stored.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> >>>> perhaps
> >>>>>>> that
> >>>>>>>>> would
> >>>>>>>>>>>> be
> >>>>>>>>>>>>> stored in the batch coordinator?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> >>>>>>> state is
> >>>>>>>>>>>>> maintained in the batch coordinator and how it will be
> >>>>>> cleaned
> >>>>>>> up?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> >>>>>>> <ju...@confluent.io.invalid>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Greg and Ivan,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the update. A few comments.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> >>>>>> segments,
> >>>>>>>>> making
> >>>>>>>>>>>> use of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> indexes, page cache, request purgatory, and zero-copy
> >>>>>>>>> functionality
> >>>>>>>>>>>> already
> >>>>>>>>>>>>>> built into classic topics."
> >>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
> >>>> each
> >>>>>>>>> partition in
> >>>>>>>>>>>>>> diskless topics?
> >>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
> >>>> need
> >>>>>> to
> >>>>>>> know
> >>>>>>>>>>>> aborted
> >>>>>>>>>>>>>> records. How is that achieved?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> >>>> transaction is
> >>>>>>>>> finished
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>> also inserts the control batches in the corresponding
> >>>> logs
> >>>>>>> of the
> >>>>>>>>>>>> involved
> >>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
> >>>> level,
> >>>>>> no
> >>>>>>>>> actual
> >>>>>>>>>>>> control
> >>>>>>>>>>>>>> batches are written to any file. "
> >>>>>>>>>>>>>> A fetch response could include multiple transactional
> >>>>>>> batches.
> >>>>>>>>> How
> >>>>>>>>>>>> does the
> >>>>>>>>>>>>>> broker obtain the information about the ending control
> >>>>>> batch
> >>>>>>> for
> >>>>>>>>> each
> >>>>>>>>>>>>>> batch? Does that mean that a fetch response needs to be
> >>>>>>> built by
> >>>>>>>>>>>>>> stitching record batches and generated control batches
> >>>>>>> together?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition leader
> >>>> that
> >>>>>>> all
> >>>>>>>>>>>> consumers
> >>>>>>>>>>>>>> are routed to?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this or
> >>>> it's
> >>>>>>> too
> >>>>>>>>>>>>>> implementation-focused?" It would be useful to include
> >>>>>> enough
> >>>>>>>>> details
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> understand correctness and performance impact.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
> >>>>>> producer
> >>>>>>>>> contain a
> >>>>>>>>>>>>>> sequence number, which is ordered. If a producer sends
> >>>>>> every
> >>>>>>>>> Produce
> >>>>>>>>>>>>>> request to an arbitrary broker, those requests could
> >>>> reach
> >>>>>>> the
> >>>>>>>>> batch
> >>>>>>>>>>>>>> coordinator in different order and lead to rejection
> >>>> of the
> >>>>>>>>> produce
> >>>>>>>>>>>>>> requests.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> >>>>>>> ivan@ivanyu.me>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We have also thought in a bit more details about
> >>>>>>> transactions
> >>>>>>>>> and
> >>>>>>>>>>>> queues,
> >>>>>>>>>>>>>>> here's the plan.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *Transactions*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The support for transactions in *classic topics* is
> >>>> based
> >>>>>>> on
> >>>>>>>>> precise
> >>>>>>>>>>>>>>> interactions between three actors: clients (mostly
> >>>>>>> producers,
> >>>>>>>>> but
> >>>>>>>>>>>> also
> >>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> >>>> classes),
> >>>>>> and
> >>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders with
> >>>>>> their
> >>>>>>>>> local
> >>>>>>>>>>>> state
> >>>>>>>>>>>>>>> (ProducerStateManager and others).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The high level (some details skipped) workflow is the
> >>>>>>>>> following.
> >>>>>>>>>>>> When a
> >>>>>>>>>>>>>>> transactional Produce request is received by the
> >>>> broker:
> >>>>>>>>>>>>>>> 1. For each partition, the partition leader checks
> >>>> if a
> >>>>>>>>> non-empty
> >>>>>>>>>>>>>>> transaction is running for this partition. This is
> >>>> done
> >>>>>>> using
> >>>>>>>>> its
> >>>>>>>>>>>> local
> >>>>>>>>>>>>>>> state derived from the log metadata
> >>>>>> (ProducerStateManager,
> >>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> >>>>>>>>>>>>>>> 2. The transaction coordinator is informed about all
> >>>> the
> >>>>>>>>> partitions
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>> aren’t part of the transaction to include them.
> >>>>>>>>>>>>>>> 3. The partition leaders do additional transactional
> >>>>>>> checks.
> >>>>>>>>>>>>>>> 4. The partition leaders append the transactional
> >>>> data to
> >>>>>>>>> their logs
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> update some of their state (for example, log the fact
> >>>>>> that
> >>>>>>> the
> >>>>>>>>>>>>>> transaction
> >>>>>>>>>>>>>>> is running for the partition and its first offset).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> >>>>>>> directly
> >>>>>>>>> with
> >>>>>>>>>>>>>>> EndTxnRequest.
> >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> >>>> or
> >>>>>>>>>>>> PREPARE_ABORT to
> >>>>>>>>>>>>>>> its log and responds to the producer.
> >>>>>>>>>>>>>>> 3. The transaction coordinator sends
> >>>>>>> WriteTxnMarkersRequest to
> >>>>>>>>> the
> >>>>>>>>>>>>>> leaders
> >>>>>>>>>>>>>>> of the involved partitions.
> >>>>>>>>>>>>>>> 4. The partition leaders write the transaction
> >>>> markers to
> >>>>>>>>> their logs
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> respond to the coordinator.
> >>>>>>>>>>>>>>> 5. The coordinator writes the final transaction state
> >>>>>>>>>>>> COMPLETE_COMMIT or
> >>>>>>>>>>>>>>> COMPLETE_ABORT.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
> >>>> of
> >>>>>>>>> important
> >>>>>>>>>>>> state
> >>>>>>>>>>>>>>> necessary for supporting this workflow is local. The
> >>>> main
> >>>>>>>>> challenge
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
> >>>> are no
> >>>>>>>>> partition
> >>>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
> >>>> to be
> >>>>>>>>> globalized
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> batch coordinator. We are already doing this to
> >>>> support
> >>>>>>>>> idempotent
> >>>>>>>>>>>>>> produce.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The high level workflow for *diskless topics* would
> >>>> look
> >>>>>>> very
> >>>>>>>>>>>> similar:
> >>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
> >>>> non-empty
> >>>>>>>>> transaction
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>> running for this partition. In contrast to classic
> >>>>>> topics,
> >>>>>>>>> this is
> >>>>>>>>>>>>>> checked
> >>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
> >>>> Since a
> >>>>>>>>> transaction
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
> >>>> the
> >>>>>>> positive
> >>>>>>>>>>>> result of
> >>>>>>>>>>>>>>> this check could be cached locally (for the double
> >>>>>>> configured
> >>>>>>>>>>>> duration
> >>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>> transaction, for example).
> >>>>>>>>>>>>>>> 2. The same: The transaction coordinator is informed
> >>>>>> about
> >>>>>>> all
> >>>>>>>>> the
> >>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
> >>>> include
> >>>>>>> them.
> >>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
> >>>> side.
> >>>>>>>>>>>>>>> 4. The broker appends the transactional data to the
> >>>>>> current
> >>>>>>>>> shared
> >>>>>>>>>>>> WAL
> >>>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
> >>>> state
> >>>>>>> for
> >>>>>>>>> Diskless
> >>>>>>>>>>>>>>> topics, because it doesn’t have any.
> >>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> >>>> coordinator
> >>>>>>> like
> >>>>>>>>> in the
> >>>>>>>>>>>>>>> normal produce flow.
> >>>>>>>>>>>>>>> 6. The batch coordinator does the final transactional
> >>>>>>> checks
> >>>>>>>>> of the
> >>>>>>>>>>>>>>> batches. This procedure would output the same errors
> >>>> like
> >>>>>>> the
> >>>>>>>>>>>> partition
> >>>>>>>>>>>>>>> leader in classic topics would do. I.e. some batches
> >>>>>> could
> >>>>>>> be
> >>>>>>>>>>>> rejected.
> >>>>>>>>>>>>>>> This means, there will potentially be garbage in the
> >>>> WAL
> >>>>>>>>> segment
> >>>>>>>>>>>> file in
> >>>>>>>>>>>>>>> case of transactional errors. This is preferable to
> >>>> doing
> >>>>>>> more
> >>>>>>>>>>>> network
> >>>>>>>>>>>>>>> round trips, especially considering the WAL segments
> >>>> will
> >>>>>>> be
> >>>>>>>>>>>> relatively
> >>>>>>>>>>>>>>> short-living (see the Greg's update above).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> When the transaction is committed or aborted:
> >>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> >>>>>>> directly
> >>>>>>>>> with
> >>>>>>>>>>>>>>> EndTxnRequest.
> >>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> >>>> or
> >>>>>>>>>>>> PREPARE_ABORT to
> >>>>>>>>>>>>>>> its log and responds to the producer.
> >>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
> >>>> batch
> >>>>>>>>> coordinator
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>> the transaction is finished.
> >>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> >>>>>>> transaction is
> >>>>>>>>>>>> finished
> >>>>>>>>>>>>>>> and also inserts the control batches in the
> >>>> corresponding
> >>>>>>> logs
> >>>>>>>>> of the
> >>>>>>>>>>>>>>> involved Diskless topics. This happens only on the
> >>>>>> metadata
> >>>>>>>>> level, no
> >>>>>>>>>>>>>>> actual control batches are written to any file. They
> >>>> will
> >>>>>>> be
> >>>>>>>>>>>> dynamically
> >>>>>>>>>>>>>>> created on Fetch and other read operations. We could
> >>>>>>>>> technically
> >>>>>>>>>>>> write
> >>>>>>>>>>>>>>> these control batches for real, but this would mean
> >>>> extra
> >>>>>>>>> produce
> >>>>>>>>>>>>>> latency,
> >>>>>>>>>>>>>>> so it's better just to mark them in the batch
> >>>> coordinator
> >>>>>>> and
> >>>>>>>>> save
> >>>>>>>>>>>> these
> >>>>>>>>>>>>>>> milliseconds.
> >>>>>>>>>>>>>>> 5. The transaction coordinator sends
> >>>>>>> WriteTxnMarkersRequest to
> >>>>>>>>> the
> >>>>>>>>>>>>>> leaders
> >>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
> >>>> topics
> >>>>>>> now.
> >>>>>>>>>>>>>>> 6. The partition leaders of classic topics write the
> >>>>>>>>> transaction
> >>>>>>>>>>>> markers
> >>>>>>>>>>>>>>> to their logs and respond to the coordinator.
> >>>>>>>>>>>>>>> 7. The coordinator writes the final transaction state
> >>>>>>>>>>>> COMPLETE_COMMIT or
> >>>>>>>>>>>>>>> COMPLETE_ABORT.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
> >>>> get:
> >>>>>>>>>>>>>>> 1. An extra network round trip between brokers and
> >>>> the
> >>>>>>> batch
> >>>>>>>>>>>> coordinator
> >>>>>>>>>>>>>>> when a new partition appear in the transaction. To
> >>>>>>> mitigate the
> >>>>>>>>>>>> impact of
> >>>>>>>>>>>>>>> them:
> >>>>>>>>>>>>>>> - The results will be cached.
> >>>>>>>>>>>>>>> - The calls for multiple partitions in one Produce
> >>>>>>> request
> >>>>>>>>> will be
> >>>>>>>>>>>>>>> grouped.
> >>>>>>>>>>>>>>> - The batch coordinator should be optimized for
> >>>> fast
> >>>>>>>>> response to
> >>>>>>>>>>>> these
> >>>>>>>>>>>>>>> RPCs.
> >>>>>>>>>>>>>>> - The fact that a single producer normally will
> >>>>>>> communicate
> >>>>>>>>> with a
> >>>>>>>>>>>>>>> single broker for the duration of the transaction
> >>>> further
> >>>>>>>>> reduces the
> >>>>>>>>>>>>>>> expected number of round trips.
> >>>>>>>>>>>>>>> 2. An extra round trip between the transaction
> >>>>>> coordinator
> >>>>>>> and
> >>>>>>>>> batch
> >>>>>>>>>>>>>>> coordinator when a transaction is finished.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> With this proposal, transactions will also be able to
> >>>>>> span
> >>>>>>> both
> >>>>>>>>>>>> classic
> >>>>>>>>>>>>>>> and Diskless topics.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *Queues*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The share group coordination and management is a
> >>>> side job
> >>>>>>> that
> >>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>> interfere with the topic itself (leadership,
> >>>> replicas,
> >>>>>>> physical
> >>>>>>>>>>>> storage
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> records, etc.) and non-queue producers and consumers
> >>>>>>> (Fetch and
> >>>>>>>>>>>> Produce
> >>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not affected.)
> >>>> We
> >>>>>>> don't
> >>>>>>>>> see any
> >>>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
> >>>> with
> >>>>>>> share
> >>>>>>>>>>>> groups the
> >>>>>>>>>>>>>>> same way as classic topics are. Even on the code
> >>>> level,
> >>>>>> we
> >>>>>>>>> don't
> >>>>>>>>>>>> expect
> >>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>> serious refactoring: the same reading routines are
> >>>> used
> >>>>>>> that
> >>>>>>>>> are
> >>>>>>>>>>>> used for
> >>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
> >>>> too
> >>>>>>>>>>>>>>> implementation-focused?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Ivan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> >>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you all for your questions and design input
> >>>> on
> >>>>>>>>> KIP-1150.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
> >>>> new
> >>>>>>>>> design. To
> >>>>>>>>>>>>>>> summarize
> >>>>>>>>>>>>>>>> the changes:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> >>>> existing
> >>>>>>>>> KIP-405
> >>>>>>>>>>>> Tiered
> >>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
> >>>>>>> Diskless
> >>>>>>>>> topic
> >>>>>>>>>>>> to be
> >>>>>>>>>>>>>>>> moved to tiered storage.
> >>>>>>>>>>>>>>>> This lowers the scalability requirements for the
> >>>> Batch
> >>>>>>>>> Coordinator
> >>>>>>>>>>>>>>>> component, and allows Diskless to compose with
> >>>> Tiered
> >>>>>>> Storage
> >>>>>>>>>>>> plugin
> >>>>>>>>>>>>>>>> features such as encryption and alternative data
> >>>>>> formats.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> >>>> segments,
> >>>>>>>>> making use
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> >>>> zero-copy
> >>>>>>>>> functionality
> >>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>> built into classic topics.
> >>>>>>>>>>>>>>>> However, local segments are now considered cache
> >>>>>>> elements,
> >>>>>>>>> do not
> >>>>>>>>>>>> need
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> be durably stored, and can be built without
> >>>> contacting
> >>>>>>> any
> >>>>>>>>> other
> >>>>>>>>>>>>>>> replicas.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 3. The design has been simplified substantially, by
> >>>>>>> removing
> >>>>>>>>> the
> >>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
> >>>> component, and
> >>>>>>>>> "object
> >>>>>>>>>>>>>>>> compaction/merging" step.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The design maintains leaderless produces as
> >>>> enabled by
> >>>>>>> the
> >>>>>>>>> Batch
> >>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
> >>>>>> earlier
> >>>>>>>>> design,
> >>>>>>>>>>>> while
> >>>>>>>>>>>>>>>> being simpler and integrating better into the
> >>>> existing
> >>>>>>>>> ecosystem.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
> >>>> the
> >>>>>> new
> >>>>>>>>> design.
> >>>>>>>>>>>>>>>> Greg Harris
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> >>>>>>>>> <ju...@confluent.io.invalid>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi, Jan,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> >>>> of
> >>>>>> all
> >>>>>>>>> existing
> >>>>>>>>>>>>>> client
> >>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
> >>>> supporting
> >>>>>> APIs
> >>>>>>>>> like
> >>>>>>>>>>>>>>> transactions
> >>>>>>>>>>>>>>>>> and queues.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> >>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> >>>>>>> status of
> >>>>>>>>> this
> >>>>>>>>>>>> KIP?
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> >>>>>>> months,
> >>>>>>>>> the
> >>>>>>>>>>>> vote got
> >>>>>>>>>>>>>>>>> vetoed
> >>>>>>>>>>>>>>>>>> but I think the pending questions have been
> >>>>>> answered
> >>>>>>>>> since
> >>>>>>>>>>>> then.
> >>>>>>>>>>>>>>> KIP-1183
> >>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> >>>> activity
> >>>>>>> since
> >>>>>>>>> May.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> >>>>>> choices
> >>>>>>>>> that can
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> >>>> by
> >>>>>> far
> >>>>>>> the
> >>>>>>>>>>>> dominant
> >>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
> >>>>>>>>> standalone
> >>>>>>>>>>>>>> products.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> >>>> hate to
> >>>>>>> see a
> >>>>>>>>>>>> stall. So
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> question is: can we get an update?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> >>>>>> McCabe -
> >>>>>>>>> have your
> >>>>>>>>>>>>>>> questions
> >>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> >>>> do
> >>>>>> to
> >>>>>>>>> help? I'm
> >>>>>>>>>>>>>> deeply
> >>>>>>>>>>>>>>>>>> familiar with both architectures and have
> >>>> written
> >>>>>>> about
> >>>>>>>>> both?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>> Jan
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> >>>>>> Kozlovski
> >>>>>>> <
> >>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> >>>> main
> >>>>>> one
> >>>>>>>>> (just a
> >>>>>>>>>>>> bunch
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> links
> >>>>>>>>>>>>>>>>>>> to everything)
> >>>>>>>>>>>>>>>>>>> b) create the email threads
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> >>>> example, I
> >>>>>>> was
> >>>>>>>>>>>> searching
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> >>>> far
> >>>>>> as
> >>>>>>> I
> >>>>>>>>> can
> >>>>>>>>>>>> tell, it
> >>>>>>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>>>> exist yet.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> >>>> having
> >>>>>>> the
> >>>>>>>>> root
> >>>>>>>>>>>> KIP be
> >>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> >>>> to
> >>>>>>>>> sub-KIPs
> >>>>>>>>>>>> where
> >>>>>>>>>>>>>> were
> >>>>>>>>>>>>>>>>> aimed
> >>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> >>>> would
> >>>>>>> be
> >>>>>>>>> good to
> >>>>>>>>>>>>>> create
> >>>>>>>>>>>>>>>>>> DISCUSS
> >>>>>>>>>>>>>>>>>>> threads for them all.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>> Stan
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> >>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> >>>>>>>>> introducing a
> >>>>>>>>>>>> new
> >>>>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>>>> topics that would make use of Object
> >>>> Storage as
> >>>>>>> the
> >>>>>>>>> primary
> >>>>>>>>>>>>>>> source of
> >>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> >>>> decided
> >>>>>>> to
> >>>>>>>>> split it
> >>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>> multiple
> >>>>>>>>>>>>>>>>>>>> related KIPs.
> >>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> >>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> >>>> should aim
> >>>>>>> to
> >>>>>>>>> have
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> >>>>>> details
> >>>>>>> on
> >>>>>>>>> how to
> >>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
> >>>>>>> discussed
> >>>>>>>>>>>> KRaft.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> >>>> hard
> >>>>>>> to
> >>>>>>>>> discuss
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> meta
> >>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> >>>> (linked
> >>>>>>> in
> >>>>>>>>>>>> KIP-1150)
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> offer
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> implementation of this feature.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> >>>>>>> threads
> >>>>>>>>> for
> >>>>>>>>>>>> each
> >>>>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> >>>> whether
> >>>>>>> Apache
> >>>>>>>>> Kafka
> >>>>>>>>>>>> wants
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>> this feature or not.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> >>>> authors
> >>>>>> of
> >>>>>>>>> this KIP.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> ------------------
> >>>>>>>>>>>>>>>>>>>> Josep Prat
> >>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> >>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> >>>>>>> aiven.io
> >>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> >>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> >>>>>>> Valtonen,
> >>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> >>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Max fortun <ma...@maxf.net>]

This may be a tangent, but we needed to offload storage off of Kafka into S3. We are keeping Kafka not as a source of truth, but as a mostly ephemeral broker that can come and go as it pleases. Be that scaling or outage. Disks can be destroyed and recreated at will, we still retain data and use broker for just that, brokering messages. Not only that, we reduced the requirement on the actual Kafka resources by reducing the size of a payload via a claim check pattern. Maybe this is an anti–pattern, but it is super fast and highly cost efficient. We reworked ProducerRequest to allow plugins. We added a custom http plugin that submits every request via a persisted connection to a microservice. Microservice stores the payload and returns a tiny json metadata object,a claim check, that can be used to find the actual data. Think of it as zipping the payload. This claim check metadata traverses the pipelines with consumers using the urls in metadata to pull what they need. Think unzipping. This allowed us to also pull ONLY the data that we need in graphql like manner. So if you have a 100K json payload and you need only a subsection, you can pull that by jmespath. When you have multiple consumer groups yanking down huge payloads it is cumbersome on the broker. When you have the same consumer groups yanking down a claim check, and then going out of band directly to the source of truth, the broker has some breathing room. Obviously our microservice does not go directly to the cloud storage, as that would be too slow. It stores the payload in high speed memory cache and returns asap. That memory is eventually persisted into S3. The retrieval goest against the cache first, then against the S3. Overall a rather cheappy and zippy solution. I tried proposing the KIP for this, but there was no excitement. Check this out:
 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528


> On Nov 7, 2025, at 5:49 PM, Jun Rao <ju...@confluent.io.INVALID> wrote:
> 
> Hi, Andrew,
> 
> If we want to focus only on reducing cross-zone replication costs, there is
> an alternative design in the KIP-1176 discussion thread that seems simpler
> than the proposal here. I am copying the outline of that approach below.
> 
> 1. A new leader is elected.
> 2. Leader maintains a first tiered offset, which is initialized to log end
> offset.
> 3. Leader writes produced data from the client to local log.
> 4. Leader uploads produced data from all local logs as a combined object
> 5. Leader stores the metadata for the combined object in memory.
> 6. If a follower fetch request has an offset >= first tiered offset, the
> metadata for the corresponding combined object is returned. Otherwise, the
> local data is returned.
> 7. Leader periodically advances first tiered offset.
> 
> It's still a bit unnatural, but it could work.
> 
> Hi, Ivan,
> 
> Are you still committed to proceeding with the original design of KIP-1150?
> 
> Thanks,
> 
> Jun
> 
> On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <an...@live.com>
> wrote:
> 
>> Hi,
>> I’ve been following KIP-1150 and friends for a while. I’m going to jump
>> into the discussions too.
>> 
>> Looking back at Jack Vanlightly’s message, I am not quite so convinced
>> that it’s a kind of fork in the road. The primary aim of the effort is to
>> reduce cross-zone replication costs so Apache Kafka is not prohibitively
>> expensive to use on cloud storage. I think it would be entirely viable to
>> prioritise code reuse for an initial implementation of diskless topics, and
>> we could still have a more cloud-native design in the future. It’s hard to
>> predict what the community will prioritise in the future.
>> 
>> Of the three major revisions, I’m in the rev3 camp. We can support
>> leaderless produce requests, first writing WAL segments into object
>> storage, and then using the regular partition leaders to sequence the
>> records. The active log segment for a diskless topic will initially contain
>> batch coordinates rather than record batches. The batch coordinates can be
>> resolved from WAL segments for consumers, and also in order to prepare log
>> segments for uploading to tiered storage. Jun is probably correct that we
>> need a more frequent object merging process than tiered storage provides.
>> This is just the transition from write-optimised WAL segments to
>> read-optimised tiered segments, and all of the object storage-based
>> implementations of Kafka that I’m aware of do this rearrangement. But
>> perhaps this more frequent object merging is a pre-GA improvement, rather
>> than a strict requirement for an initial implementation for early access
>> use.
>> 
>> For zone-aligned share consumers, the share group assignor is intended to
>> be rack-aware. Consumers should be assigned to partitions with leaders in
>> their zone. The simple assignor is not rack-aware, but it easily could be
>> or we could have a rack-aware assignor.
>> 
>> Thanks,
>> Andrew
>> 
>> 
>>> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID> wrote:
>>> 
>>> Hi, Ivan,
>>> 
>>> Thanks for the reply.
>>> 
>>> "As I understand, you’re speaking about locally materialized segments.
>> They
>>> will indeed consume some IOPS. See them as a cache that could always be
>>> restored from the remote storage. While it’s not ideal, it's still OK to
>>> lose data in them due to a machine crash, for example. Because of this,
>> we
>>> can avoid explicit flushing on local materialized segments at all and let
>>> the file system and page cache figure out when to flush optimally. This
>>> would not eliminate the extra IOPS, but should reduce it dramatically,
>>> depending on throughput for each partition. We, of course, continue
>>> flushing the metadata segments as before."
>>> 
>>> If we have a mix of classic and diskless topics on the same broker, it's
>>> important that the classic topics' data is flushed to disk as quickly as
>>> possible. To achieve this, users typically set dirty_expire_centisecs in
>>> the kernel based on the number of available disk IOPS. Once you set this
>>> number, it applies to all dirty files, including the cached data in
>>> diskless topics. So, if there are more files actively accumulating data,
>>> the flush frequency and therefore RPO is reduced for classic topics.
>>> 
>>> "We should have mentioned this explicitly, but this step, in fact,
>> remains
>>> in the form of segments offloading to tiered storage. When we assemble a
>>> segment and hand it over to RemoteLogManager, we’re effectively doing
>>> metadata compaction: replacing a big number of pieces of metadata about
>>> individual batches with a single record in __remote_log_metadata."
>>> 
>>> The object merging in tier storage typically only kicks in after a few
>>> hours. The impact is (1) the amount of accumulated metadata is still
>> quite
>>> large; (2) there are many small objects, leading to poor read
>> performance.
>>> I think we need a more frequent object merging process than tier storage
>>> provides.
>>> 
>>> Jun
>>> 
>>> 
>>> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>>> 
>>>> Hello Jack, Jun, Luke, and all!
>>>> 
>>>> Thank you for your messages.
>>>> 
>>>> Let me first address some of Jun’s comments.
>>>> 
>>>>> First, it degrades the durability.
>>>>> For each partition, now there are two files being actively written at a
>>>>> given point of time, one for the data and another for the metadata.
>>>>> Flushing each file requires a separate IO. If the disk has 1K IOPS and
>> we
>>>>> have 5K partitions in a broker, currently we can afford to flush each
>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we double
>>>> the
>>>>> number of files per partition, we can only flush each partition every
>> 10
>>>>> seconds, which makes RPO twice as bad.
>>>> 
>>>> As I understand, you’re speaking about locally materialized segments.
>> They
>>>> will indeed consume some IOPS. See them as a cache that could always be
>>>> restored from the remote storage. While it’s not ideal, it's still OK to
>>>> lose data in them due to a machine crash, for example. Because of this,
>> we
>>>> can avoid explicit flushing on local materialized segments at all and
>> let
>>>> the file system and page cache figure out when to flush optimally. This
>>>> would not eliminate the extra IOPS, but should reduce it dramatically,
>>>> depending on throughput for each partition. We, of course, continue
>>>> flushing the metadata segments as before.
>>>> 
>>>> It’s worth making a note on caching. I think nobody will disagree that
>>>> doing direct reads from remote storage every time a batch is requested
>> by a
>>>> consumer will not be practical neither from the performance nor from the
>>>> economy point of view. We need a way to keep the number of GET requests
>>>> down. There are multiple options, for example:
>>>> 1. Rack-aware distributed in-memory caching.
>>>> 2. Local in-memory caching. Comes with less network chattiness and
>> works
>>>> well if we have more or less stable brokers to consume from.
>>>> 3. Materialization of diskless logs on local disk. Way lower impact on
>>>> RAM and also requires stable brokers for consumption (using just
>> assigned
>>>> replicas will probably work well).
>>>> 
>>>> Materialization is one of possible options, but we can choose another
>> one.
>>>> However, we will have this dilemma regardless of whether we have an
>>>> explicit coordinator or we go “coordinator-less”.
>>>> 
>>>>> Second, if we ever need this
>>>>> metadata somewhere else, say in the WAL file manager, the consumer
>> needs
>>>> to
>>>>> subscribe to every partition in the cluster, which is inefficient. The
>>>>> actual benefit of this approach is also questionable. On the surface,
>> it
>>>>> might seem that we could reduce the number of lines that need to be
>>>> changed
>>>>> for this KIP. However, the changes are quite intrusive to the classic
>>>>> partition's code path and will probably make the code base harder to
>>>>> maintain in the long run. I like the original approach based on the
>> batch
>>>>> coordinator much better than this one. We could probably refactor the
>>>>> producer state code so that it could be reused in the batch
>> coordinator.
>>>> 
>>>> It’s hard to disagree with this. The explicit coordinator is more a side
>>>> thing, while coordinator-less approach is more about extending
>>>> ReplicaManager, UnifiedLog and others substantially.
>>>> 
>>>>> Thanks for addressing the concerns on the number of RPCs in the produce
>>>>> path. I agree that with the metadata crafting mechanism, we could
>>>> mitigate
>>>>> the PRC problem. However, since we now require the metadata to be
>>>>> collocated with the data on the same set of brokers, it's weird that
>> they
>>>>> are now managed by different mechanisms. The data assignment now uses
>> the
>>>>> metadata crafting mechanism, but the metadata is stored in the classic
>>>>> partition using its own assignment strategy. It will be complicated to
>>>> keep
>>>>> them collocated.
>>>> 
>>>> I would like to note that the metadata crafting is needed only to tell
>>>> producers which brokers they should send Produce requests to, but data
>> (as
>>>> in “locally materialized log”) is located on partition replicas, i.e.
>>>> automatically co-located with metadata.
>>>> 
>>>> As a side note, it would probably be better that instead of implicitly
>>>> crafting partition metadata, we extend the metadata protocol so that for
>>>> diskless partitions we return not only the leader and replicas, but also
>>>> some “recommended produce brokers”, selected for optimal performance and
>>>> costs. Producers will pick ones in their racks.
>>>> 
>>>>> I am also concerned about the removal of the object compaction/merging
>>>>> step.
>>>> 
>>>> We should have mentioned this explicitly, but this step, in fact,
>> remains
>>>> in the form of segments offloading to tiered storage. When we assemble a
>>>> segment and hand it over to RemoteLogManager, we’re effectively doing
>>>> metadata compaction: replacing a big number of pieces of metadata about
>>>> individual batches with a single record in __remote_log_metadata.
>>>> 
>>>> We could create a Diskless-specific merging mechanism instead if needed.
>>>> It’s rather easy with the explicit coordinator approach. With the
>>>> coordinator-less approach, this would probably be a bit more tricky
>>>> (rewriting the tail of the log by the leader + replicating this change
>>>> reliably).
>>>> 
>>>>> I see a tendency toward primarily optimizing for the fewest code
>> changes
>>>> in
>>>>> the KIP. Instead, our primary goal should be a clean design that can
>> last
>>>>> for the long term.
>>>> 
>>>> Yes, totally agree.
>>>> 
>>>> 
>>>> 
>>>> Luke,
>>>>> I'm wondering if the complexity of designing txn and queue is because
>> of
>>>>> leaderless cluster, do you think it will be simpler if we only focus on
>>>> the
>>>>> "diskless" design to handle object compaction/merging to/from the
>> remote
>>>>> storage to save the cross-AZ cost?
>>>> 
>>>> After some evolution of the original proposal, leaderless is now
>> limited.
>>>> We only need to be able to accept Produce requests on more than one
>> broker
>>>> to eliminate the cross-AZ costs for producers. Do I get it right that
>> you
>>>> propose to get rid of this? Or do I misunderstand?
>>>> 
>>>> 
>>>> 
>>>> Let’s now look at this problem from a higher level, as Jack proposed. As
>>>> it was said, the big choice we need to make is whether we 1) create an
>>>> explicit batch coordinator; or 2) go for the coordinator-less approach,
>>>> where each diskless partition is managed by its leader as in classic
>> topics.
>>>> 
>>>> If we try to compare the two approaches:
>>>> 
>>>> Pluggability:
>>>> - Explicit coordinator: Possible. For example, some setups may benefit
>>>> from batch metadata being stored in a cloud database (such as AWS
>> DynamoDB
>>>> or GCP Spanner).
>>>> - Coordinator-less: Impossible.
>>>> 
>>>> Scalability and fault tolerance:
>>>> - Explicit coordinator: Depends on the implementation and it’s also
>>>> necessary to actively work for it.
>>>> - Coordinator-less: Closer to classic Kafka topics. Scaling is done by
>>>> partition placement, partitions could fail independently.
>>>> 
>>>> Separation of concerns:
>>>> - Explicit coordinator: Very good. Diskless remains more independent
>> from
>>>> classic topics in terms of code and workflows. For example, the
>>>> above-mentioned non-tiered storage metadata compaction mechanism could
>> be
>>>> relatively simply implemented with it. As a flip side of this, some
>>>> workflows (e.g. transactions) will have to be adapted.
>>>> - Coordinator-less: Less so. It leans to the opposite: bringing diskless
>>>> closer to classic topics. Some code paths and workflows could be more
>>>> straightforwardly reused, but they will inevitably have to be adapted to
>>>> accommodate both topic types as also discussed.
>>>> 
>>>> Cloud-nativeness. This is a vague concept, also related to the previous,
>>>> but let’s try:
>>>> - Explicit coordinator: Storing and processing metadata separately makes
>>>> it easier for brokers to take different roles, be purely stateless if
>>>> needed, etc.
>>>> - Coordinator-less: Less so. Something could be achieved with creative
>>>> partition placement, but not much.
>>>> 
>>>> Both seem to have their pros and cons. However, answering Jack’s
>> question,
>>>> the explicit coordinator approach may indeed lead to a more flexible
>> design.
>>>> 
>>>> 
>>>> The purpose of this deviation in the discussion was to receive a
>>>> preliminary community evaluation of the coordinator-less approach
>> without
>>>> taking on the task of writing a separate KIP and fitting it in the
>> system
>>>> of KIP-1150 and its children. We’re open to stopping it and getting
>> back to
>>>> working out the coordinator design if the community doesn’t favor the
>>>> proposed approach.
>>>> 
>>>> Best,
>>>> Ivan and Diskless team
>>>> 
>>>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
>>>>> Hi Ivan,
>>>>> 
>>>>> As Jun pointed out, the updated design seems to have some shortcomings
>>>>> although it simplifies the implementation.
>>>>> 
>>>>> I'm wondering if the complexity of designing txn and queue is because
>> of
>>>>> leaderless cluster, do you think it will be simpler if we only focus on
>>>> the
>>>>> "diskless" design to handle object compaction/merging to/from the
>> remote
>>>>> storage to save the cross-AZ cost?
>>>>> 
>>>>> 
>>>>> Thank you,
>>>>> Luke
>>>>> 
>>>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid>
>>>> wrote:
>>>>> 
>>>>>> Hi, Ivan,
>>>>>> 
>>>>>> Thanks for the explanation.
>>>>>> 
>>>>>> "we write the reference to the WAL file with the batch data"
>>>>>> 
>>>>>> I understand the approach now, but I think it is a hacky one. There
>> are
>>>>>> multiple short comings with this design. First, it degrades the
>>>> durability.
>>>>>> For each partition, now there are two files being actively written at
>> a
>>>>>> given point of time, one for the data and another for the metadata.
>>>>>> Flushing each file requires a separate IO. If the disk has 1K IOPS and
>>>> we
>>>>>> have 5K partitions in a broker, currently we can afford to flush each
>>>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we double
>>>> the
>>>>>> number of files per partition, we can only flush each partition every
>>>> 10
>>>>>> seconds, which makes RPO twice as bad. Second, if we ever need this
>>>>>> metadata somewhere else, say in the WAL file manager, the consumer
>>>> needs to
>>>>>> subscribe to every partition in the cluster, which is inefficient. The
>>>>>> actual benefit of this approach is also questionable. On the surface,
>>>> it
>>>>>> might seem that we could reduce the number of lines that need to be
>>>> changed
>>>>>> for this KIP. However, the changes are quite intrusive to the classic
>>>>>> partition's code path and will probably make the code base harder to
>>>>>> maintain in the long run. I like the original approach based on the
>>>> batch
>>>>>> coordinator much better than this one. We could probably refactor the
>>>>>> producer state code so that it could be reused in the batch
>>>> coordinator.
>>>>>> 
>>>>>> Thanks for addressing the concerns on the number of RPCs in the
>> produce
>>>>>> path. I agree that with the metadata crafting mechanism, we could
>>>> mitigate
>>>>>> the PRC problem. However, since we now require the metadata to be
>>>>>> collocated with the data on the same set of brokers, it's weird that
>>>> they
>>>>>> are now managed by different mechanisms. The data assignment now uses
>>>> the
>>>>>> metadata crafting mechanism, but the metadata is stored in the classic
>>>>>> partition using its own assignment strategy. It will be complicated to
>>>> keep
>>>>>> them collocated.
>>>>>> 
>>>>>> I am also concerned about the removal of the object compaction/merging
>>>>>> step. My first concern is on the amount of metadata that need to be
>>>> kept.
>>>>>> Without object compcation, the metadata generated in the produce path
>>>> can
>>>>>> only be deleted after remote tiering kicks in. Let's say for every
>>>> 250ms we
>>>>>> produce 100 byte of metadata per partition. Let's say remoting tiering
>>>>>> kicks in after 5 hours. In a cluster with 100K partitions, we need to
>>>> keep
>>>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
>>>>>> signficant. A second concern is on performance. Every time we need to
>>>>>> rebuild the caching data, we need to read a bunch of small objects
>>>> from S3,
>>>>>> slowing down the building process. If a consumer happens to need such
>>>> data,
>>>>>> it could slow down the application.
>>>>>> 
>>>>>> I see a tendency toward primarily optimizing for the fewest code
>>>> changes in
>>>>>> the KIP. Instead, our primary goal should be a clean design that can
>>>> last
>>>>>> for the long term.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me>
>>>> wrote:
>>>>>> 
>>>>>>> Hi Jun,
>>>>>>> 
>>>>>>> Thank you for your message. I’m sorry that I failed to clearly
>>>> explain
>>>>>> the
>>>>>>> idea. Let me try to fix this.
>>>>>>> 
>>>>>>>> Does each partition now have a metadata partition and a separate
>>>> data
>>>>>>>> partition? If so, I am concerned that it essentially doubles the
>>>> number
>>>>>>> of
>>>>>>>> partitions, which impacts the number of open file descriptors and
>>>> the
>>>>>>>> required IOPS, and so on. It also seems wasteful to have a separate
>>>>>>>> partition just to store the metadata. It's as if we are creating an
>>>>>>>> internal topic with an unbounded number of partitions.
>>>>>>> 
>>>>>>> No. There will be only one physical partition per diskless
>>>> partition. Let
>>>>>>> me explain this with an example. Let’s say we have a diskless
>>>> partition
>>>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce
>>>> some
>>>>>>> batches to this partition. The content of the segment file will be
>>>>>>> something like this (for each batch):
>>>>>>> 
>>>>>>> BaseOffset: 00000000000000000000 (like in classic)
>>>>>>> Length: 123456 (like in classic)
>>>>>>> PartitionLeaderEpoch: like in classic
>>>>>>> Magic: like in classic
>>>>>>> CRC: like in classic
>>>>>>> Attributes: like in classic
>>>>>>> LastOffsetDelta: like in classic
>>>>>>> BaseTimestamp: like in classic
>>>>>>> MaxTimestamp: like in classic
>>>>>>> ProducerId: like in classic
>>>>>>> ProducerEpoch: like in classic
>>>>>>> BaseSequence: like in classic
>>>>>>> RecordsCount: like in classic
>>>>>>> Records:
>>>>>>> path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
>>>>>>> 123456
>>>>>>> 
>>>>>>> It looks very much like classic log entries. The only difference is
>>>> that
>>>>>>> instead of writing real Records, we write the reference to the WAL
>>>> file
>>>>>>> with the batch data (I guess we need only the name and the byte
>>>> offset,
>>>>>>> because the byte length is the standard field above). Otherwise,
>>>> it’s a
>>>>>>> normal Kafka log with the leader and replicas.
>>>>>>> 
>>>>>>> So we have as many partitions for diskless as for classic. As of open
>>>>>> file
>>>>>>> descriptors, let’s proceed to the following:
>>>>>>> 
>>>>>>>> Are the metadata and
>>>>>>>> the data for the same partition always collocated on the same
>>>> broker?
>>>>>> If
>>>>>>>> so, how do we enforce that when replicas are reassigned?
>>>>>>> 
>>>>>>> The source of truth for the data is still in WAL files on object
>>>> storage.
>>>>>>> The source of truth for the metadata is in segment files on the
>>>> brokers
>>>>>> in
>>>>>>> the replica set. Two new mechanisms are planned, both independent of
>>>> this
>>>>>>> new proposal, but I want to present them to give the idea that only a
>>>>>>> limited amount of data files will be operated locally:
>>>>>>> - We want to assemble batches into segment files and offload them to
>>>>>>> tiered storage in order to prevent the unbounded growth of batch
>>>>>> metadata.
>>>>>>> For this, we need to open only  a few file descriptors (for the
>>>> segment
>>>>>>> file itself + the necessary indexes) before the segment is fully
>>>> written
>>>>>>> and handed over to RemoteLogManager.
>>>>>>> - We want to assemble local segment files for caching purposes as
>>>> well,
>>>>>>> i.e. to speed up fetching. This will not materialize the full
>>>> content of
>>>>>>> the log, but only the hot set according to some policy (or
>>>> configurable
>>>>>>> policies), i.e. the number of segments and file descriptors will
>>>> also be
>>>>>>> limited.
>>>>>>> 
>>>>>>>> The number of RPCs in the produce path is significantly higher. For
>>>>>>>> example, if a produce request has 100 partitions, in a cluster
>>>> with 100
>>>>>>>> brokers, each produce request could generate 100 more RPC requests.
>>>>>> This
>>>>>>>> will significantly increase the request rate.
>>>>>>> 
>>>>>>> This is a valid concern that we considered, but this issue can be
>>>>>>> mitigated. I’ll try to explain the approach.
>>>>>>> The situation with a single broker is trivial: all the commit
>>>> requests go
>>>>>>> from the broker to itself.
>>>>>>> Let’s scale this to a multi-broker cluster, but located in the single
>>>>>> rack
>>>>>>> (AZ). Any broker can accept Produce requests for diskless
>>>> partitions, but
>>>>>>> we can tell producers (through metadata) to always send Produce
>>>> requests
>>>>>> to
>>>>>>> leaders. For example, broker 0 hosts the leader replicas for diskless
>>>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless Produce
>>>> requests
>>>>>> for
>>>>>>> these partitions in various combinations, but only for them.
>>>>>>> 
>>>>>>>                   Broker 0
>>>>>>>             +-----------------+
>>>>>>>             |    t1-0         |
>>>>>>>             |    t2-1 <--------------------+
>>>>>>>             |    t3-0         |            |
>>>>>>> produce      | +-------------+ |            |
>>>>>>> requests     | |  diskless   | |            |
>>>>>>> --------------->|   produce   +--------------+
>>>>>>> for these    | | WAL buffer  | |    commit requests
>>>>>>> partitions   | +-------------+ |    for these partitions
>>>>>>>             |                 |
>>>>>>>             +-----------------+
>>>>>>> 
>>>>>>> The same applies for other brokers in this cluster. Effectively, each
>>>>>>> broker will commit only to itself, which effectively means 1 commit
>>>>>> request
>>>>>>> per WAL buffer (this may be 0 physical network calls, if we wish,
>>>> just a
>>>>>>> local function call).
>>>>>>> 
>>>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we cannot
>>>> always
>>>>>>> send Produce requests to the designated leaders of diskless
>>>> partitions:
>>>>>>> this would mean inter-AZ network traffic, which we would like to
>>>> avoid.
>>>>>> To
>>>>>>> avoid it, we say that every broker has a “diskless produce
>>>>>> representative”
>>>>>>> in every AZ. If we continue our example: when a Produce request for
>>>> t1-0,
>>>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0
>>>> (in the
>>>>>>> broker’s AZ the representative is the broker itself). However, if it
>>>>>> comes
>>>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
>>>>>>> 
>>>>>>> |produce requests         |produce requests        |produce
>>>> requests
>>>>>>> |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
>>>>>> t3-0
>>>>>>> |from AZ 0                |from AZ 1               |from AZ 2
>>>>>>> v                         v                        v
>>>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
>>>>>>> +---------------+      +---------------+      +---------------+
>>>>>>> |     t1-0      |      |               |      |               |
>>>>>>> |     t2-1      |      |               |      |               |
>>>>>>> |     t3-0      |      |               |      |               |
>>>>>>> +---------------+      +--------+------+      +--------+------+
>>>>>>>    ^     ^                    |                      |
>>>>>>>    |     +--------------------+                      |
>>>>>>>    |     commit requests for these partitions        |
>>>>>>>    |                                                 |
>>>>>>>    +-------------------------------------------------+
>>>>>>>          commit requests for these partitions
>>>>>>> 
>>>>>>> All the partitions that broker 0 is the leader of will be
>>>> “represented”
>>>>>> by
>>>>>>> brokers 1 and 2 in their AZs.
>>>>>>> 
>>>>>>> Of course, this relationship goes both ways between AZs (not
>>>> necessarily
>>>>>>> between the same brokers). It means that provided the cluster is
>>>> balanced
>>>>>>> by the number of brokers per AZ, each broker will represent
>>>>>> (number_of_azs
>>>>>>> - 1) other brokers. This will result in the situation that for the
>>>>>> majority
>>>>>>> of commits, each broker will do up to (number_of_azs - 1) network
>>>> commit
>>>>>>> requests (plus one local). Cloud regions tend to have 3 AZs, very
>>>> rarely
>>>>>>> more. That means, brokers will be doing up to 2 network commit
>>>> requests
>>>>>> per
>>>>>>> WAL file.
>>>>>>> 
>>>>>>> There are the following exceptions:
>>>>>>> 1. Broker count imbalance between AZs. For example, when we have 2
>>>> AZs
>>>>>> and
>>>>>>> one has three brokers and another AZ has one. This one broker will do
>>>>>>> between 1 and 3 commit requests per WAL file. This is not an extreme
>>>>>>> amplification. Such an imbalance is not healthy in most practical
>>>> setups
>>>>>>> and should be avoided anyway.
>>>>>>> 2. Leadership changes and metadata propagation period. When the
>>>> partition
>>>>>>> t3-0 is relocated from broker 0 to some broker 3, the producers will
>>>> not
>>>>>>> know this immediately (unless we want to be strict and respond with
>>>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
>>>> together
>>>>>> in a
>>>>>>> WAL buffer on broker 2, it will have to send two commit requests: to
>>>>>> broker
>>>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
>>>> situation
>>>>>>> is not permanent and as producers update the cluster metadata, it
>>>> will be
>>>>>>> resolved.
>>>>>>> 
>>>>>>> This all could be built with the metadata crafting mechanism only
>>>> (which
>>>>>>> is anyway needed for Diskless in one way or another to direct
>>>> producers
>>>>>> and
>>>>>>> consumers where we need to avoid inter-AZ traffic), just with the
>>>> right
>>>>>>> policy for it (for example, some deterministic hash-based formula).
>>>> I.e.
>>>>>> no
>>>>>>> explicit support for “produce representative” or anything like this
>>>> is
>>>>>>> needed on the cluster level, in KRaft, etc.
>>>>>>> 
>>>>>>>> The same WAL file metadata is now duplicated into two places,
>>>> partition
>>>>>>>> leader and WAL File Manager. Which one is the source of truth, and
>>>> how
>>>>>> do
>>>>>>>> we maintain consistency between the two places?
>>>>>>> 
>>>>>>> We do only two operations on WAL files that span multiple diskless
>>>>>>> partitions: committing and deleting. Commits can be done
>>>> independently as
>>>>>>> described above. But deletes are different, because when a file is
>>>>>> deleted,
>>>>>>> this affects all the partitions that still have alive batches in this
>>>>>> file
>>>>>>> (if any).
>>>>>>> 
>>>>>>> The WAL file manager is a necessary point of coordination to delete
>>>> WAL
>>>>>>> files safely. We can say it is the source of truth about files
>>>>>> themselves,
>>>>>>> while the partition leaders and their logs hold the truth about
>>>> whether a
>>>>>>> particular file contains live batches of this particular partition.
>>>>>>> 
>>>>>>> The file manager will do this important task: be able to say for sure
>>>>>> that
>>>>>>> a file does not contain any live batch of any existing partition. For
>>>>>> this,
>>>>>>> it will have to periodically check against the partition leaders.
>>>>>>> Considering that batch deletion is irreversible, when we declare a
>>>> file
>>>>>>> “empty”, this is guaranteed to be and stay so.
>>>>>>> 
>>>>>>> The file manager has to know about files being committed to start
>>>> track
>>>>>>> them and periodically check if they are empty. We can consider
>>>> various
>>>>>> ways
>>>>>>> to achieve this:
>>>>>>> 1. As was proposed in my previous message: best effort commit by
>>>> brokers
>>>>>> +
>>>>>>> periodic prefix scans of object storage to detect files that went
>>>> below
>>>>>> the
>>>>>>> radar due to network issue or the file manager temporary
>>>> unavailability.
>>>>>>> We’re speaking about listing the file names only and opening only
>>>>>>> previously unknown files in order to find the partitions involved
>>>> with
>>>>>> them.
>>>>>>> 2. Only do scans without explicit commit, i.e. fill the list of files
>>>>>>> fully asynchronously and in the background. This may be not ideal
>>>> due to
>>>>>>> costs and performance of scanning tons of files. However, the number
>>>> of
>>>>>>> live WAL files should be limited due to tiered storage offloading +
>>>> we
>>>>>> can
>>>>>>> optimize this if we give files some global soft order in their names.
>>>>>>> 
>>>>>>>> I am not sure how this design simplifies the implementation. The
>>>>>> existing
>>>>>>>> producer/replication code can't be simply reused. Adjusting both
>>>> the
>>>>>>> write
>>>>>>>> path in the leader and the replication path in the follower to
>>>>>> understand
>>>>>>>> batch-header only data is quite intrusive to the existing logic.
>>>>>>> 
>>>>>>> It is true that we’ll have to change LocalLog and UnifiedLog in
>>>> order to
>>>>>>> support these changes. However, it seems that idempotence,
>>>> transactions,
>>>>>>> queues, tiered storage will have to be changed less than with the
>>>>>> original
>>>>>>> design. This is because the partition leader state would remain in
>>>> the
>>>>>> same
>>>>>>> place (on brokers) and existing workflows that involve it would have
>>>> to
>>>>>> be
>>>>>>> changed less compared to the situation where we globalize the
>>>> partition
>>>>>>> leader state in the batch coordinator. I admit this is hard to make
>>>>>>> convincing without both real implementations to hand :)
>>>>>>> 
>>>>>>>> I am also
>>>>>>>> not sure how this enables seamless switching the topic modes
>>>> between
>>>>>>>> diskless and classic. Could you provide more details on those?
>>>>>>> 
>>>>>>> Let’s consider the scenario of turning a classic topic into
>>>> diskless. The
>>>>>>> user sets diskless.enabled=true, the leader receives this metadata
>>>> update
>>>>>>> and does the following:
>>>>>>> 1. Stop accepting normal append writes.
>>>>>>> 2. Close the current active segment.
>>>>>>> 3. Start a new segment that will be written in the diskless format
>>>> (i.e.
>>>>>>> without data).
>>>>>>> 4. Start accepting diskless commits.
>>>>>>> 
>>>>>>> Since it’s the same log, the followers will know about that switch
>>>>>>> consistently. They will finish replicating the classic segments and
>>>> start
>>>>>>> replicating the diskless ones. They will always know where each
>>>> batch is
>>>>>>> located (either inside a classic segment or referenced by a diskless
>>>>>> one).
>>>>>>> Switching back should be similar.
>>>>>>> 
>>>>>>> Doing this with the coordinator is possible, but has some caveats.
>>>> The
>>>>>>> leader must do the following:
>>>>>>> 1. Stop accepting normal append writes.
>>>>>>> 2. Close the current active segment.
>>>>>>> 3. Write a special control segment to persist and replicate the fact
>>>> that
>>>>>>> from offset N the partition is now in the diskless mode.
>>>>>>> 4. Inform the coordinator about the first offset N of the “diskless
>>>> era”.
>>>>>>> 5. Inform the controller quorum that the transition has finished and
>>>> that
>>>>>>> brokers now can process diskless writes for this partition.
>>>>>>> This could fail at some points, so this will probably require some
>>>>>>> explicit state machine with replication either in the partition log
>>>> or in
>>>>>>> KRaft.
>>>>>>> 
>>>>>>> It seems that the coordinator-less approach makes this simpler
>>>> because
>>>>>> the
>>>>>>> “coordinator” for the partition and the partition leader are the
>>>> same and
>>>>>>> they store the partition metadata in the same log, too. While in the
>>>>>>> coordinator approach we have to perform some kind of a distributed
>>>> commit
>>>>>>> to handover metadata management from the classic partition leader to
>>>> the
>>>>>>> batch coordinator.
>>>>>>> 
>>>>>>> I hope these explanations help to clarify the idea. Please let me
>>>> know if
>>>>>>> I should go deeper anywhere.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Ivan and the Diskless team
>>>>>>> 
>>>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
>>>>>>>> Hi, Ivan,
>>>>>>>> 
>>>>>>>> Thanks for the update.
>>>>>>>> 
>>>>>>>> I am not sure that I fully understand the new design, but it seems
>>>> less
>>>>>>>> clean than before.
>>>>>>>> 
>>>>>>>> Does each partition now have a metadata partition and a separate
>>>> data
>>>>>>>> partition? If so, I am concerned that it essentially doubles the
>>>> number
>>>>>>> of
>>>>>>>> partitions, which impacts the number of open file descriptors and
>>>> the
>>>>>>>> required IOPS, and so on. It also seems wasteful to have a separate
>>>>>>>> partition just to store the metadata. It's as if we are creating an
>>>>>>>> internal topic with an unbounded number of partitions. Are the
>>>> metadata
>>>>>>> and
>>>>>>>> the data for the same partition always collocated on the same
>>>> broker?
>>>>>> If
>>>>>>>> so, how do we enforce that when replicas are reassigned?
>>>>>>>> 
>>>>>>>> The number of RPCs in the produce path is significantly higher. For
>>>>>>>> example, if a produce request has 100 partitions, in a cluster
>>>> with 100
>>>>>>>> brokers, each produce request could generate 100 more RPC requests.
>>>>>> This
>>>>>>>> will significantly increase the request rate.
>>>>>>>> 
>>>>>>>> The same WAL file metadata is now duplicated into two places,
>>>> partition
>>>>>>>> leader and WAL File Manager. Which one is the source of truth, and
>>>> how
>>>>>> do
>>>>>>>> we maintain consistency between the two places?
>>>>>>>> 
>>>>>>>> I am not sure how this design simplifies the implementation. The
>>>>>> existing
>>>>>>>> producer/replication code can't be simply reused. Adjusting both
>>>> the
>>>>>>> write
>>>>>>>> path in the leader and the replication path in the follower to
>>>>>> understand
>>>>>>>> batch-header only data is quite intrusive to the existing logic. I
>>>> am
>>>>>>> also
>>>>>>>> not sure how this enables seamless switching the topic modes
>>>> between
>>>>>>>> diskless and classic. Could you provide more details on those?
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi dear Kafka community,
>>>>>>>>> 
>>>>>>>>> In the initial Diskless proposal, we proposed to have a separate
>>>>>>>>> component, batch/diskless coordinator, whose role would be to
>>>>>> centrally
>>>>>>>>> manage the batch and WAL file metadata for diskless topics. This
>>>>>>> component
>>>>>>>>> drew many reasonable comments from the community about how it
>>>> would
>>>>>>> support
>>>>>>>>> various Kafka features (transactions, queues) and its
>>>> scalability.
>>>>>>> While we
>>>>>>>>> believe we have good answers to all the expressed concerns, we
>>>> took a
>>>>>>> step
>>>>>>>>> back and looked at the problem from a different perspective.
>>>>>>>>> 
>>>>>>>>> We would like to propose an alternative Diskless design *without
>>>> a
>>>>>>>>> centralized coordinator*. We believe this approach has potential
>>>> and
>>>>>>>>> propose to discuss it as it may be more appealing to the
>>>> community.
>>>>>>>>> 
>>>>>>>>> Let us explain the idea. Most of the complications with the
>>>> original
>>>>>>>>> Diskless approach come from one necessary architecture change:
>>>>>>> globalizing
>>>>>>>>> the local state of partition leader in the batch coordinator.
>>>> This
>>>>>>> causes
>>>>>>>>> deviations to the established workflows in various features like
>>>>>>> produce
>>>>>>>>> idempotence and transactions, queues, retention, etc. These
>>>>>> deviations
>>>>>>> need
>>>>>>>>> to be carefully considered, designed, and later implemented and
>>>>>>> tested. In
>>>>>>>>> the new approach we want to avoid this by making partition
>>>> leaders
>>>>>>> again
>>>>>>>>> responsible for managing their partitions, even in diskless
>>>> topics.
>>>>>>>>> 
>>>>>>>>> In classic Kafka topics, batch data and metadata are blended
>>>> together
>>>>>>> in
>>>>>>>>> the one partition log. The crux of the Diskless idea is to
>>>> decouple
>>>>>>> them
>>>>>>>>> and move data to the remote storage, while keeping metadata
>>>> somewhere
>>>>>>> else.
>>>>>>>>> Using the central batch coordinator for managing batch metadata
>>>> is
>>>>>> one
>>>>>>> way,
>>>>>>>>> but not the only.
>>>>>>>>> 
>>>>>>>>> Let’s now think about managing metadata for each user partition
>>>>>>>>> independently. Generally partitions are independent and don’t
>>>> share
>>>>>>>>> anything apart from that their data are mixed in WAL files. If we
>>>>>>> figure
>>>>>>>>> out how to commit and later delete WAL files safely, we will
>>>> achieve
>>>>>>> the
>>>>>>>>> necessary autonomy that allows us to get rid of the central batch
>>>>>>>>> coordinator. Instead, *each diskless user partition will be
>>>> managed
>>>>>> by
>>>>>>> its
>>>>>>>>> leader*, as in classic Kafka topics. Also like in classic
>>>> topics, the
>>>>>>>>> leader uses the partition log as the way to persist batch
>>>> metadata,
>>>>>>> i.e.
>>>>>>>>> the regular batch header + the information about how to find this
>>>>>>> batch on
>>>>>>>>> remote storage. In contrast to classic topics, batch data is in
>>>>>> remote
>>>>>>>>> storage.
>>>>>>>>> 
>>>>>>>>> For clarity, let’s compare the three designs:
>>>>>>>>> • Classic topics:
>>>>>>>>>  • Data and metadata are co-located in the partition log.
>>>>>>>>>  • The partition log content: [Batch header (metadata)|Batch
>>>> data].
>>>>>>>>>  • The partition log is replicated to the followers.
>>>>>>>>>  • The replicas and leader have local state built from
>>>> metadata.
>>>>>>>>> • Original Diskless:
>>>>>>>>>  • Metadata is in the batch coordinator, data is on remote
>>>> storage.
>>>>>>>>>  • The partition state is global in the batch coordinator.
>>>>>>>>> • New Diskless:
>>>>>>>>>  • Metadata is in the partition log, data is on remote storage.
>>>>>>>>>  • Partition log content: [Batch header (metadata)|Batch
>>>>>> coordinates
>>>>>>> on
>>>>>>>>> remote storage].
>>>>>>>>>  • The partition log is replicated to the followers.
>>>>>>>>>  • The replicas and leader have local state built from
>>>> metadata.
>>>>>>>>> 
>>>>>>>>> Let’s consider the produce path. Here’s the reminder of the
>>>> original
>>>>>>>>> Diskless design:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The new approach could be depicted as the following:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> As you can see, the main difference is that now instead of a
>>>> single
>>>>>>> commit
>>>>>>>>> request to the batch coordinator, we send multiple parallel
>>>> commit
>>>>>>> requests
>>>>>>>>> to all the leaders of each partition involved in the WAL file.
>>>> Each
>>>>>> of
>>>>>>> them
>>>>>>>>> will commit its batches independently, without coordinating with
>>>>>> other
>>>>>>>>> leaders and any other components. Batch data is addressed by the
>>>> WAL
>>>>>>> file
>>>>>>>>> name, the byte offset and size, which allows partitions to know
>>>>>> nothing
>>>>>>>>> about other partitions to access their data in shared WAL files.
>>>>>>>>> 
>>>>>>>>> The number of partitions involved in a single WAL file may be
>>>> quite
>>>>>>> large,
>>>>>>>>> e.g. a hundred. A hundred network requests to commit one WAL
>>>> file is
>>>>>>> very
>>>>>>>>> impractical. However, there are ways to reduce this number:
>>>>>>>>> 1. Partition leaders are located on brokers. Requests to
>>>> leaders on
>>>>>>> one
>>>>>>>>> broker could be grouped together into a single physical network
>>>>>> request
>>>>>>>>> (resembling the normal Produce request that may carry batches for
>>>>>> many
>>>>>>>>> partitions inside). This will cap the number of network requests
>>>> to
>>>>>> the
>>>>>>>>> number of brokers in the cluster.
>>>>>>>>> 2. If we craft the cluster metadata to make producers send their
>>>>>>> requests
>>>>>>>>> to the right brokers (with respect to AZs), we may achieve the
>>>> higher
>>>>>>>>> concentration of logical commit requests in physical network
>>>> requests
>>>>>>>>> reducing the number of the latter ones even further, ideally to
>>>> one.
>>>>>>>>> 
>>>>>>>>> Obviously, out of multiple commit requests some may fail or time
>>>> out
>>>>>>> for a
>>>>>>>>> variety of reasons. This is fine. Some producers will receive
>>>> totally
>>>>>>> or
>>>>>>>>> partially failed responses to their Produce requests, similar to
>>>> what
>>>>>>> they
>>>>>>>>> would have received when appending to a classic topic fails or
>>>> times
>>>>>>> out.
>>>>>>>>> If a partition experiences problems, other partitions will not be
>>>>>>> affected
>>>>>>>>> (again, like in classic topics). Of course, the uncommitted data
>>>> will
>>>>>>> be
>>>>>>>>> garbage in WAL files. But WAL files are short-lived (batches are
>>>>>>> constantly
>>>>>>>>> assembled into segments and offloaded to tiered storage), so this
>>>>>>> garbage
>>>>>>>>> will be eventually deleted.
>>>>>>>>> 
>>>>>>>>> For safely deleting WAL files we now need to centrally manage
>>>> them,
>>>>>> as
>>>>>>>>> this is the only state and logic that spans multiple partitions.
>>>> On
>>>>>> the
>>>>>>>>> diagram, you can see another commit request called “Commit file
>>>> (best
>>>>>>>>> effort)” going to the WAL File Manager. This manager will be
>>>>>>> responsible
>>>>>>>>> for the following:
>>>>>>>>> 1. Collecting (by requests from brokers) and persisting
>>>> information
>>>>>>> about
>>>>>>>>> committed WAL files.
>>>>>>>>> 2. To handle potential failures in file information delivery, it
>>>>>> will
>>>>>>> be
>>>>>>>>> doing prefix scan on the remote storage periodically to find and
>>>>>>> register
>>>>>>>>> unknown files. The period of this scan will be configurable and
>>>>>> ideally
>>>>>>>>> should be quite long.
>>>>>>>>> 3. Checking with the relevant partition leaders (after a grace
>>>>>>> period) if
>>>>>>>>> they still have batches in a particular file.
>>>>>>>>> 4. Physically deleting files when they aren’t anymore referred
>>>> to by
>>>>>>> any
>>>>>>>>> partition.
>>>>>>>>> 
>>>>>>>>> This new design offers the following advantages:
>>>>>>>>> 1. It simplifies the implementation of many Kafka features such
>>>> as
>>>>>>>>> idempotence, transactions, queues, tiered storage, retention.
>>>> Now we
>>>>>>> don’t
>>>>>>>>> need to abstract away and reuse the code from partition leaders
>>>> in
>>>>>> the
>>>>>>>>> batch coordinator. Instead, we will literally use the same code
>>>> paths
>>>>>>> in
>>>>>>>>> leaders, with little adaptation. Workflows from classic topics
>>>> mostly
>>>>>>>>> remain unchanged.
>>>>>>>>> For example, it seems that
>>>>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
>>>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for transaction
>>>> support
>>>>>> on
>>>>>>> the
>>>>>>>>> partition leader side could be used for diskless topics with
>>>> little
>>>>>>>>> adaptation. ProducerStateManager, needed for both idempotent
>>>> produce
>>>>>>> and
>>>>>>>>> transactions, would be reused.
>>>>>>>>> Another example is share groups support, where the share
>>>> partition
>>>>>>> leader,
>>>>>>>>> being co-located with the partition leader, would execute the
>>>> same
>>>>>>> logic
>>>>>>>>> for both diskless and classic topics.
>>>>>>>>> 2. It returns to the familiar partition-based scaling model,
>>>> where
>>>>>>>>> partitions are independent.
>>>>>>>>> 3. It makes the operation and failure patterns closer to the
>>>>>> familiar
>>>>>>>>> ones from classic topics.
>>>>>>>>> 4. It opens a straightforward path to seamless switching the
>>>> topics
>>>>>>> modes
>>>>>>>>> between diskless and classic.
>>>>>>>>> 
>>>>>>>>> The rest of the things remain unchanged compared to the previous
>>>>>>> Diskless
>>>>>>>>> design (after all previous discussions). Such things as local
>>>> segment
>>>>>>>>> materialization by replicas, the consume path, tiered storage
>>>>>>> integration,
>>>>>>>>> etc.
>>>>>>>>> 
>>>>>>>>> If the community finds this design more suitable, we will update
>>>> the
>>>>>>>>> KIP(s) accordingly and continue working on it. Please let us know
>>>>>> what
>>>>>>> you
>>>>>>>>> think.
>>>>>>>>> 
>>>>>>>>> Best regards,
>>>>>>>>> Ivan and Diskless team
>>>>>>>>> 
>>>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
>>>>>>>>>> Hi Justine,
>>>>>>>>>> 
>>>>>>>>>> Yes, you're right. We need to track the aborted transactions
>>>> for in
>>>>>>> the
>>>>>>>>> diskless coordinator for as long as the corresponding offsets are
>>>>>>> there.
>>>>>>>>> With the tiered storage unification Greg mentioned earlier, this
>>>> will
>>>>>>> be
>>>>>>>>> finite time even for infinite data retention.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Ivan
>>>>>>>>>> 
>>>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
>>>>>>>>>>> Hey Ivan,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the response. I think most of what you said made
>>>>>> sense,
>>>>>>> but
>>>>>>>>> I
>>>>>>>>>>> did have some questions about this part:
>>>>>>>>>>> 
>>>>>>>>>>>> As we understand this, the partition leader in classic
>>>> topics
>>>>>>> forgets
>>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
>>>> it). The
>>>>>>>>>>> transaction coordinator acts like the main guardian, allowing
>>>>>>> partition
>>>>>>>>>>> leaders to do this safely. Please correct me if this is
>>>> wrong. We
>>>>>>> think
>>>>>>>>>>> about relying on this with the batch coordinator and delete
>>>> the
>>>>>>>>> information
>>>>>>>>>>> about a transaction once it’s finished (as there’s no
>>>> replication
>>>>>>> and
>>>>>>>>> HWM
>>>>>>>>>>> advances immediately).
>>>>>>>>>>> 
>>>>>>>>>>> I didn't quite understand this. In classic topics, we have
>>>> maps
>>>>>> for
>>>>>>>>> ongoing
>>>>>>>>>>> transactions which remove state when the transaction is
>>>> completed
>>>>>>> and
>>>>>>>>> an
>>>>>>>>>>> aborted transactions index which is retained for much longer.
>>>>>> Once
>>>>>>> the
>>>>>>>>>>> transaction is completed, the coordinator is no longer
>>>> involved
>>>>>> in
>>>>>>>>>>> maintaining this partition side state, and it is subject to
>>>>>>> compaction
>>>>>>>>> etc.
>>>>>>>>>>> Looking back at the outline provided above, I didn't see much
>>>>>>> about the
>>>>>>>>>>> fetch path, so maybe that could be expanded a bit further. I
>>>> saw
>>>>>>> the
>>>>>>>>>>> following in a response:
>>>>>>>>>>>> When the broker constructs a fully valid local segment,
>>>> all the
>>>>>>>>> necessary
>>>>>>>>>>> control batches will be inserted and indices, including the
>>>>>>> transaction
>>>>>>>>>>> index will be built to serve FetchRequests exactly as they
>>>> are
>>>>>>> today.
>>>>>>>>>>> 
>>>>>>>>>>> Based on this, it seems like we need to retain the
>>>> information
>>>>>>> about
>>>>>>>>>>> aborted txns for longer.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Justine
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
>>>> ivan@ivanyu.me>
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Justine and all,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thank you for your questions!
>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
>>>> with
>>>>>>>>> producer ID
>>>>>>>>>>>>> and epoch, the positive result of this check could be
>>>> cached
>>>>>>>>> locally
>>>>>>>>>>>>> Are we saying that only new transaction version 2
>>>>>> transactions
>>>>>>> can
>>>>>>>>> be
>>>>>>>>>>>> used
>>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
>>>> with
>>>>>>>>> producer id +
>>>>>>>>>>>>> epoch
>>>>>>>>>>>> 
>>>>>>>>>>>> You’re right that we (probably unintentionally) focused
>>>> only on
>>>>>>>>> version 2.
>>>>>>>>>>>> We can either limit the support to version 2 or consider
>>>> using
>>>>>>> some
>>>>>>>>>>>> surrogates to support version 1.
>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
>>>>>>> checks
>>>>>>>>> of the
>>>>>>>>>>>>> batches. This procedure would output the same errors
>>>> like the
>>>>>>>>> partition
>>>>>>>>>>>>> leader in classic topics would do.
>>>>>>>>>>>>> Can you expand on what these checks are? Would you be
>>>>>> checking
>>>>>>> if
>>>>>>>>> the
>>>>>>>>>>>>> transaction was still ongoing for example?* *
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, the producer epoch, that the transaction is ongoing,
>>>> and
>>>>>> of
>>>>>>>>> course
>>>>>>>>>>>> the normal idempotence checks. What the partition leader
>>>> in the
>>>>>>>>> classic
>>>>>>>>>>>> topics does before appending a batch to the local log
>>>> (e.g. in
>>>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
>>>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In Diskless,
>>>> we
>>>>>>>>> unfortunately
>>>>>>>>>>>> cannot do these checks before appending the data to the WAL
>>>>>>> segment
>>>>>>>>> and
>>>>>>>>>>>> uploading it, but we can “tombstone” these batches in the
>>>> batch
>>>>>>>>> coordinator
>>>>>>>>>>>> during the final commit.
>>>>>>>>>>>> 
>>>>>>>>>>>>> Is there state about ongoing
>>>>>>>>>>>>> transactions in the batch coordinator? I see some other
>>>> state
>>>>>>>>> mentioned
>>>>>>>>>>>> in
>>>>>>>>>>>>> the End transaction section, but it's not super clear
>>>> what
>>>>>>> state is
>>>>>>>>>>>> stored
>>>>>>>>>>>>> and when it is stored.
>>>>>>>>>>>> 
>>>>>>>>>>>> Right, this should have been more explicit. As the
>>>> partition
>>>>>>> leader
>>>>>>>>> tracks
>>>>>>>>>>>> ongoing transactions for classic topics, the batch
>>>> coordinator
>>>>>>> has
>>>>>>>>> to as
>>>>>>>>>>>> well. So when a transaction starts and ends, the
>>>> transaction
>>>>>>>>> coordinator
>>>>>>>>>>>> must inform the batch coordinator about this.
>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
>>>> perhaps
>>>>>>> that
>>>>>>>>> would
>>>>>>>>>>>> be
>>>>>>>>>>>>> stored in the batch coordinator?
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes. This could be deduced from the committed batches and
>>>> other
>>>>>>>>>>>> information, but for the sake of performance we’d better
>>>> store
>>>>>> it
>>>>>>>>>>>> explicitly.
>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
>>>>>>> state is
>>>>>>>>>>>>> maintained in the batch coordinator and how it will be
>>>>>> cleaned
>>>>>>> up?
>>>>>>>>>>>> 
>>>>>>>>>>>> As we understand this, the partition leader in classic
>>>> topics
>>>>>>> forgets
>>>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
>>>> it).
>>>>>> The
>>>>>>>>>>>> transaction coordinator acts like the main guardian,
>>>> allowing
>>>>>>>>> partition
>>>>>>>>>>>> leaders to do this safely. Please correct me if this is
>>>> wrong.
>>>>>> We
>>>>>>>>> think
>>>>>>>>>>>> about relying on this with the batch coordinator and
>>>> delete the
>>>>>>>>> information
>>>>>>>>>>>> about a transaction once it’s finished (as there’s no
>>>>>> replication
>>>>>>>>> and HWM
>>>>>>>>>>>> advances immediately).
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Ivan
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
>>>>>>>>>>>>> Hey folks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Excited to see some updates related to transactions!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I had a few questions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
>>>> with
>>>>>>>>> producer ID
>>>>>>>>>>>>> and epoch, the positive result of this check could be
>>>> cached
>>>>>>>>> locally
>>>>>>>>>>>>> Are we saying that only new transaction version 2
>>>>>> transactions
>>>>>>> can
>>>>>>>>> be
>>>>>>>>>>>> used
>>>>>>>>>>>>> here? If not, we can't uniquely identify transactions
>>>> with
>>>>>>>>> producer id +
>>>>>>>>>>>>> epoch
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
>>>>>>> checks
>>>>>>>>> of the
>>>>>>>>>>>>> batches. This procedure would output the same errors
>>>> like the
>>>>>>>>> partition
>>>>>>>>>>>>> leader in classic topics would do.
>>>>>>>>>>>>> Can you expand on what these checks are? Would you be
>>>>>> checking
>>>>>>> if
>>>>>>>>> the
>>>>>>>>>>>>> transaction was still ongoing for example? Is there state
>>>>>> about
>>>>>>>>> ongoing
>>>>>>>>>>>>> transactions in the batch coordinator? I see some other
>>>> state
>>>>>>>>> mentioned
>>>>>>>>>>>> in
>>>>>>>>>>>>> the End transaction section, but it's not super clear
>>>> what
>>>>>>> state is
>>>>>>>>>>>> stored
>>>>>>>>>>>>> and when it is stored.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
>>>> perhaps
>>>>>>> that
>>>>>>>>> would
>>>>>>>>>>>> be
>>>>>>>>>>>>> stored in the batch coordinator?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
>>>>>>> state is
>>>>>>>>>>>>> maintained in the batch coordinator and how it will be
>>>>>> cleaned
>>>>>>> up?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
>>>>>>> <ju...@confluent.io.invalid>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Greg and Ivan,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the update. A few comments.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
>>>>>> segments,
>>>>>>>>> making
>>>>>>>>>>>> use of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> indexes, page cache, request purgatory, and zero-copy
>>>>>>>>> functionality
>>>>>>>>>>>> already
>>>>>>>>>>>>>> built into classic topics."
>>>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
>>>> each
>>>>>>>>> partition in
>>>>>>>>>>>>>> diskless topics?
>>>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
>>>> need
>>>>>> to
>>>>>>> know
>>>>>>>>>>>> aborted
>>>>>>>>>>>>>> records. How is that achieved?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
>>>> transaction is
>>>>>>>>> finished
>>>>>>>>>>>> and
>>>>>>>>>>>>>> also inserts the control batches in the corresponding
>>>> logs
>>>>>>> of the
>>>>>>>>>>>> involved
>>>>>>>>>>>>>> Diskless topics. This happens only on the metadata
>>>> level,
>>>>>> no
>>>>>>>>> actual
>>>>>>>>>>>> control
>>>>>>>>>>>>>> batches are written to any file. "
>>>>>>>>>>>>>> A fetch response could include multiple transactional
>>>>>>> batches.
>>>>>>>>> How
>>>>>>>>>>>> does the
>>>>>>>>>>>>>> broker obtain the information about the ending control
>>>>>> batch
>>>>>>> for
>>>>>>>>> each
>>>>>>>>>>>>>> batch? Does that mean that a fetch response needs to be
>>>>>>> built by
>>>>>>>>>>>>>> stitching record batches and generated control batches
>>>>>>> together?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR 12. Queues: Is there still a share partition leader
>>>> that
>>>>>>> all
>>>>>>>>>>>> consumers
>>>>>>>>>>>>>> are routed to?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this or
>>>> it's
>>>>>>> too
>>>>>>>>>>>>>> implementation-focused?" It would be useful to include
>>>>>> enough
>>>>>>>>> details
>>>>>>>>>>>> to
>>>>>>>>>>>>>> understand correctness and performance impact.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
>>>>>> producer
>>>>>>>>> contain a
>>>>>>>>>>>>>> sequence number, which is ordered. If a producer sends
>>>>>> every
>>>>>>>>> Produce
>>>>>>>>>>>>>> request to an arbitrary broker, those requests could
>>>> reach
>>>>>>> the
>>>>>>>>> batch
>>>>>>>>>>>>>> coordinator in different order and lead to rejection
>>>> of the
>>>>>>>>> produce
>>>>>>>>>>>>>> requests.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
>>>>>>> ivan@ivanyu.me>
>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We have also thought in a bit more details about
>>>>>>> transactions
>>>>>>>>> and
>>>>>>>>>>>> queues,
>>>>>>>>>>>>>>> here's the plan.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Transactions*
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The support for transactions in *classic topics* is
>>>> based
>>>>>>> on
>>>>>>>>> precise
>>>>>>>>>>>>>>> interactions between three actors: clients (mostly
>>>>>>> producers,
>>>>>>>>> but
>>>>>>>>>>>> also
>>>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
>>>> classes),
>>>>>> and
>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>> coordinators. Brokers also run partition leaders with
>>>>>> their
>>>>>>>>> local
>>>>>>>>>>>> state
>>>>>>>>>>>>>>> (ProducerStateManager and others).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The high level (some details skipped) workflow is the
>>>>>>>>> following.
>>>>>>>>>>>> When a
>>>>>>>>>>>>>>> transactional Produce request is received by the
>>>> broker:
>>>>>>>>>>>>>>> 1. For each partition, the partition leader checks
>>>> if a
>>>>>>>>> non-empty
>>>>>>>>>>>>>>> transaction is running for this partition. This is
>>>> done
>>>>>>> using
>>>>>>>>> its
>>>>>>>>>>>> local
>>>>>>>>>>>>>>> state derived from the log metadata
>>>>>> (ProducerStateManager,
>>>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
>>>>>>>>>>>>>>> 2. The transaction coordinator is informed about all
>>>> the
>>>>>>>>> partitions
>>>>>>>>>>>> that
>>>>>>>>>>>>>>> aren’t part of the transaction to include them.
>>>>>>>>>>>>>>> 3. The partition leaders do additional transactional
>>>>>>> checks.
>>>>>>>>>>>>>>> 4. The partition leaders append the transactional
>>>> data to
>>>>>>>>> their logs
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> update some of their state (for example, log the fact
>>>>>> that
>>>>>>> the
>>>>>>>>>>>>>> transaction
>>>>>>>>>>>>>>> is running for the partition and its first offset).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> When the transaction is committed or aborted:
>>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
>>>>>>> directly
>>>>>>>>> with
>>>>>>>>>>>>>>> EndTxnRequest.
>>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
>>>> or
>>>>>>>>>>>> PREPARE_ABORT to
>>>>>>>>>>>>>>> its log and responds to the producer.
>>>>>>>>>>>>>>> 3. The transaction coordinator sends
>>>>>>> WriteTxnMarkersRequest to
>>>>>>>>> the
>>>>>>>>>>>>>> leaders
>>>>>>>>>>>>>>> of the involved partitions.
>>>>>>>>>>>>>>> 4. The partition leaders write the transaction
>>>> markers to
>>>>>>>>> their logs
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> respond to the coordinator.
>>>>>>>>>>>>>>> 5. The coordinator writes the final transaction state
>>>>>>>>>>>> COMPLETE_COMMIT or
>>>>>>>>>>>>>>> COMPLETE_ABORT.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
>>>> of
>>>>>>>>> important
>>>>>>>>>>>> state
>>>>>>>>>>>>>>> necessary for supporting this workflow is local. The
>>>> main
>>>>>>>>> challenge
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
>>>> are no
>>>>>>>>> partition
>>>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
>>>> to be
>>>>>>>>> globalized
>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> batch coordinator. We are already doing this to
>>>> support
>>>>>>>>> idempotent
>>>>>>>>>>>>>> produce.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The high level workflow for *diskless topics* would
>>>> look
>>>>>>> very
>>>>>>>>>>>> similar:
>>>>>>>>>>>>>>> 1. For each partition, the broker checks if a
>>>> non-empty
>>>>>>>>> transaction
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> running for this partition. In contrast to classic
>>>>>> topics,
>>>>>>>>> this is
>>>>>>>>>>>>>> checked
>>>>>>>>>>>>>>> against the batch coordinator with a single RPC.
>>>> Since a
>>>>>>>>> transaction
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
>>>> the
>>>>>>> positive
>>>>>>>>>>>> result of
>>>>>>>>>>>>>>> this check could be cached locally (for the double
>>>>>>> configured
>>>>>>>>>>>> duration
>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>> transaction, for example).
>>>>>>>>>>>>>>> 2. The same: The transaction coordinator is informed
>>>>>> about
>>>>>>> all
>>>>>>>>> the
>>>>>>>>>>>>>>> partitions that aren’t part of the transaction to
>>>> include
>>>>>>> them.
>>>>>>>>>>>>>>> 3. No transactional checks are done on the broker
>>>> side.
>>>>>>>>>>>>>>> 4. The broker appends the transactional data to the
>>>>>> current
>>>>>>>>> shared
>>>>>>>>>>>> WAL
>>>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
>>>> state
>>>>>>> for
>>>>>>>>> Diskless
>>>>>>>>>>>>>>> topics, because it doesn’t have any.
>>>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
>>>> coordinator
>>>>>>> like
>>>>>>>>> in the
>>>>>>>>>>>>>>> normal produce flow.
>>>>>>>>>>>>>>> 6. The batch coordinator does the final transactional
>>>>>>> checks
>>>>>>>>> of the
>>>>>>>>>>>>>>> batches. This procedure would output the same errors
>>>> like
>>>>>>> the
>>>>>>>>>>>> partition
>>>>>>>>>>>>>>> leader in classic topics would do. I.e. some batches
>>>>>> could
>>>>>>> be
>>>>>>>>>>>> rejected.
>>>>>>>>>>>>>>> This means, there will potentially be garbage in the
>>>> WAL
>>>>>>>>> segment
>>>>>>>>>>>> file in
>>>>>>>>>>>>>>> case of transactional errors. This is preferable to
>>>> doing
>>>>>>> more
>>>>>>>>>>>> network
>>>>>>>>>>>>>>> round trips, especially considering the WAL segments
>>>> will
>>>>>>> be
>>>>>>>>>>>> relatively
>>>>>>>>>>>>>>> short-living (see the Greg's update above).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> When the transaction is committed or aborted:
>>>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
>>>>>>> directly
>>>>>>>>> with
>>>>>>>>>>>>>>> EndTxnRequest.
>>>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
>>>> or
>>>>>>>>>>>> PREPARE_ABORT to
>>>>>>>>>>>>>>> its log and responds to the producer.
>>>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
>>>> batch
>>>>>>>>> coordinator
>>>>>>>>>>>> that
>>>>>>>>>>>>>>> the transaction is finished.
>>>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
>>>>>>> transaction is
>>>>>>>>>>>> finished
>>>>>>>>>>>>>>> and also inserts the control batches in the
>>>> corresponding
>>>>>>> logs
>>>>>>>>> of the
>>>>>>>>>>>>>>> involved Diskless topics. This happens only on the
>>>>>> metadata
>>>>>>>>> level, no
>>>>>>>>>>>>>>> actual control batches are written to any file. They
>>>> will
>>>>>>> be
>>>>>>>>>>>> dynamically
>>>>>>>>>>>>>>> created on Fetch and other read operations. We could
>>>>>>>>> technically
>>>>>>>>>>>> write
>>>>>>>>>>>>>>> these control batches for real, but this would mean
>>>> extra
>>>>>>>>> produce
>>>>>>>>>>>>>> latency,
>>>>>>>>>>>>>>> so it's better just to mark them in the batch
>>>> coordinator
>>>>>>> and
>>>>>>>>> save
>>>>>>>>>>>> these
>>>>>>>>>>>>>>> milliseconds.
>>>>>>>>>>>>>>> 5. The transaction coordinator sends
>>>>>>> WriteTxnMarkersRequest to
>>>>>>>>> the
>>>>>>>>>>>>>> leaders
>>>>>>>>>>>>>>> of the involved partitions. – Now only to classic
>>>> topics
>>>>>>> now.
>>>>>>>>>>>>>>> 6. The partition leaders of classic topics write the
>>>>>>>>> transaction
>>>>>>>>>>>> markers
>>>>>>>>>>>>>>> to their logs and respond to the coordinator.
>>>>>>>>>>>>>>> 7. The coordinator writes the final transaction state
>>>>>>>>>>>> COMPLETE_COMMIT or
>>>>>>>>>>>>>>> COMPLETE_ABORT.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
>>>> get:
>>>>>>>>>>>>>>> 1. An extra network round trip between brokers and
>>>> the
>>>>>>> batch
>>>>>>>>>>>> coordinator
>>>>>>>>>>>>>>> when a new partition appear in the transaction. To
>>>>>>> mitigate the
>>>>>>>>>>>> impact of
>>>>>>>>>>>>>>> them:
>>>>>>>>>>>>>>> - The results will be cached.
>>>>>>>>>>>>>>> - The calls for multiple partitions in one Produce
>>>>>>> request
>>>>>>>>> will be
>>>>>>>>>>>>>>> grouped.
>>>>>>>>>>>>>>> - The batch coordinator should be optimized for
>>>> fast
>>>>>>>>> response to
>>>>>>>>>>>> these
>>>>>>>>>>>>>>> RPCs.
>>>>>>>>>>>>>>> - The fact that a single producer normally will
>>>>>>> communicate
>>>>>>>>> with a
>>>>>>>>>>>>>>> single broker for the duration of the transaction
>>>> further
>>>>>>>>> reduces the
>>>>>>>>>>>>>>> expected number of round trips.
>>>>>>>>>>>>>>> 2. An extra round trip between the transaction
>>>>>> coordinator
>>>>>>> and
>>>>>>>>> batch
>>>>>>>>>>>>>>> coordinator when a transaction is finished.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> With this proposal, transactions will also be able to
>>>>>> span
>>>>>>> both
>>>>>>>>>>>> classic
>>>>>>>>>>>>>>> and Diskless topics.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Queues*
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The share group coordination and management is a
>>>> side job
>>>>>>> that
>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>> interfere with the topic itself (leadership,
>>>> replicas,
>>>>>>> physical
>>>>>>>>>>>> storage
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> records, etc.) and non-queue producers and consumers
>>>>>>> (Fetch and
>>>>>>>>>>>> Produce
>>>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not affected.)
>>>> We
>>>>>>> don't
>>>>>>>>> see any
>>>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
>>>> with
>>>>>>> share
>>>>>>>>>>>> groups the
>>>>>>>>>>>>>>> same way as classic topics are. Even on the code
>>>> level,
>>>>>> we
>>>>>>>>> don't
>>>>>>>>>>>> expect
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>> serious refactoring: the same reading routines are
>>>> used
>>>>>>> that
>>>>>>>>> are
>>>>>>>>>>>> used for
>>>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
>>>> too
>>>>>>>>>>>>>>> implementation-focused?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Ivan
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thank you all for your questions and design input
>>>> on
>>>>>>>>> KIP-1150.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
>>>> new
>>>>>>>>> design. To
>>>>>>>>>>>>>>> summarize
>>>>>>>>>>>>>>>> the changes:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
>>>> existing
>>>>>>>>> KIP-405
>>>>>>>>>>>> Tiered
>>>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
>>>>>>> Diskless
>>>>>>>>> topic
>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>> moved to tiered storage.
>>>>>>>>>>>>>>>> This lowers the scalability requirements for the
>>>> Batch
>>>>>>>>> Coordinator
>>>>>>>>>>>>>>>> component, and allows Diskless to compose with
>>>> Tiered
>>>>>>> Storage
>>>>>>>>>>>> plugin
>>>>>>>>>>>>>>>> features such as encryption and alternative data
>>>>>> formats.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
>>>> segments,
>>>>>>>>> making use
>>>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
>>>> zero-copy
>>>>>>>>> functionality
>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>> built into classic topics.
>>>>>>>>>>>>>>>> However, local segments are now considered cache
>>>>>>> elements,
>>>>>>>>> do not
>>>>>>>>>>>> need
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be durably stored, and can be built without
>>>> contacting
>>>>>>> any
>>>>>>>>> other
>>>>>>>>>>>>>>> replicas.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 3. The design has been simplified substantially, by
>>>>>>> removing
>>>>>>>>> the
>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>> Diskless consume flow, distributed cache
>>>> component, and
>>>>>>>>> "object
>>>>>>>>>>>>>>>> compaction/merging" step.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The design maintains leaderless produces as
>>>> enabled by
>>>>>>> the
>>>>>>>>> Batch
>>>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
>>>>>> earlier
>>>>>>>>> design,
>>>>>>>>>>>> while
>>>>>>>>>>>>>>>> being simpler and integrating better into the
>>>> existing
>>>>>>>>> ecosystem.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
>>>> the
>>>>>> new
>>>>>>>>> design.
>>>>>>>>>>>>>>>> Greg Harris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
>>>>>>>>> <ju...@confluent.io.invalid>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jan,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
>>>> of
>>>>>> all
>>>>>>>>> existing
>>>>>>>>>>>>>> client
>>>>>>>>>>>>>>>>> APIs. Currently, there is no design for
>>>> supporting
>>>>>> APIs
>>>>>>>>> like
>>>>>>>>>>>>>>> transactions
>>>>>>>>>>>>>>>>> and queues.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
>>>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
>>>>>>> status of
>>>>>>>>> this
>>>>>>>>>>>> KIP?
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
>>>>>>> months,
>>>>>>>>> the
>>>>>>>>>>>> vote got
>>>>>>>>>>>>>>>>> vetoed
>>>>>>>>>>>>>>>>>> but I think the pending questions have been
>>>>>> answered
>>>>>>>>> since
>>>>>>>>>>>> then.
>>>>>>>>>>>>>>> KIP-1183
>>>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
>>>> activity
>>>>>>> since
>>>>>>>>> May.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
>>>>>> choices
>>>>>>>>> that can
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
>>>> by
>>>>>> far
>>>>>>> the
>>>>>>>>>>>> dominant
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
>>>>>>>>> standalone
>>>>>>>>>>>>>> products.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
>>>> hate to
>>>>>>> see a
>>>>>>>>>>>> stall. So
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> question is: can we get an update?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
>>>>>> McCabe -
>>>>>>>>> have your
>>>>>>>>>>>>>>> questions
>>>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
>>>> do
>>>>>> to
>>>>>>>>> help? I'm
>>>>>>>>>>>>>> deeply
>>>>>>>>>>>>>>>>>> familiar with both architectures and have
>>>> written
>>>>>>> about
>>>>>>>>> both?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
>>>>>> Kozlovski
>>>>>>> <
>>>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
>>>> main
>>>>>> one
>>>>>>>>> (just a
>>>>>>>>>>>> bunch
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> links
>>>>>>>>>>>>>>>>>>> to everything)
>>>>>>>>>>>>>>>>>>> b) create the email threads
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
>>>> example, I
>>>>>>> was
>>>>>>>>>>>> searching
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
>>>> far
>>>>>> as
>>>>>>> I
>>>>>>>>> can
>>>>>>>>>>>> tell, it
>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>> exist yet.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
>>>> having
>>>>>>> the
>>>>>>>>> root
>>>>>>>>>>>> KIP be
>>>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
>>>> to
>>>>>>>>> sub-KIPs
>>>>>>>>>>>> where
>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>> aimed
>>>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
>>>> would
>>>>>>> be
>>>>>>>>> good to
>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>> DISCUSS
>>>>>>>>>>>>>>>>>>> threads for them all.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Stan
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
>>>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
>>>>>>>>> introducing a
>>>>>>>>>>>> new
>>>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>>>> topics that would make use of Object
>>>> Storage as
>>>>>>> the
>>>>>>>>> primary
>>>>>>>>>>>>>>> source of
>>>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
>>>> decided
>>>>>>> to
>>>>>>>>> split it
>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> multiple
>>>>>>>>>>>>>>>>>>>> related KIPs.
>>>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
>>>> should aim
>>>>>>> to
>>>>>>>>> have
>>>>>>>>>>>> this
>>>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
>>>>>> details
>>>>>>> on
>>>>>>>>> how to
>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>> This follows the same approach used when we
>>>>>>> discussed
>>>>>>>>>>>> KRaft.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
>>>> hard
>>>>>>> to
>>>>>>>>> discuss
>>>>>>>>>>>> on
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> meta
>>>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
>>>> (linked
>>>>>>> in
>>>>>>>>>>>> KIP-1150)
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> offer
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> implementation of this feature.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
>>>>>>> threads
>>>>>>>>> for
>>>>>>>>>>>> each
>>>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
>>>> whether
>>>>>>> Apache
>>>>>>>>> Kafka
>>>>>>>>>>>> wants
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> this feature or not.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
>>>> authors
>>>>>> of
>>>>>>>>> this KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> ------------------
>>>>>>>>>>>>>>>>>>>> Josep Prat
>>>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
>>>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
>>>>>>> aiven.io
>>>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
>>>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
>>>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
>>>>>>> Valtonen,
>>>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
>>>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Andrew,

If we want to focus only on reducing cross-zone replication costs, there is
an alternative design in the KIP-1176 discussion thread that seems simpler
than the proposal here. I am copying the outline of that approach below.

1. A new leader is elected.
2. Leader maintains a first tiered offset, which is initialized to log end
offset.
3. Leader writes produced data from the client to local log.
4. Leader uploads produced data from all local logs as a combined object
5. Leader stores the metadata for the combined object in memory.
6. If a follower fetch request has an offset >= first tiered offset, the
metadata for the corresponding combined object is returned. Otherwise, the
local data is returned.
7. Leader periodically advances first tiered offset.

It's still a bit unnatural, but it could work.

Hi, Ivan,

Are you still committed to proceeding with the original design of KIP-1150?

Thanks,

Jun

On Sun, Nov 2, 2025 at 6:00 AM Andrew Schofield <an...@live.com>
wrote:

> Hi,
> I’ve been following KIP-1150 and friends for a while. I’m going to jump
> into the discussions too.
>
> Looking back at Jack Vanlightly’s message, I am not quite so convinced
> that it’s a kind of fork in the road. The primary aim of the effort is to
> reduce cross-zone replication costs so Apache Kafka is not prohibitively
> expensive to use on cloud storage. I think it would be entirely viable to
> prioritise code reuse for an initial implementation of diskless topics, and
> we could still have a more cloud-native design in the future. It’s hard to
> predict what the community will prioritise in the future.
>
> Of the three major revisions, I’m in the rev3 camp. We can support
> leaderless produce requests, first writing WAL segments into object
> storage, and then using the regular partition leaders to sequence the
> records. The active log segment for a diskless topic will initially contain
> batch coordinates rather than record batches. The batch coordinates can be
> resolved from WAL segments for consumers, and also in order to prepare log
> segments for uploading to tiered storage. Jun is probably correct that we
> need a more frequent object merging process than tiered storage provides.
> This is just the transition from write-optimised WAL segments to
> read-optimised tiered segments, and all of the object storage-based
> implementations of Kafka that I’m aware of do this rearrangement. But
> perhaps this more frequent object merging is a pre-GA improvement, rather
> than a strict requirement for an initial implementation for early access
> use.
>
> For zone-aligned share consumers, the share group assignor is intended to
> be rack-aware. Consumers should be assigned to partitions with leaders in
> their zone. The simple assignor is not rack-aware, but it easily could be
> or we could have a rack-aware assignor.
>
> Thanks,
> Andrew
>
>
> > On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >
> > Hi, Ivan,
> >
> > Thanks for the reply.
> >
> > "As I understand, you’re speaking about locally materialized segments.
> They
> > will indeed consume some IOPS. See them as a cache that could always be
> > restored from the remote storage. While it’s not ideal, it's still OK to
> > lose data in them due to a machine crash, for example. Because of this,
> we
> > can avoid explicit flushing on local materialized segments at all and let
> > the file system and page cache figure out when to flush optimally. This
> > would not eliminate the extra IOPS, but should reduce it dramatically,
> > depending on throughput for each partition. We, of course, continue
> > flushing the metadata segments as before."
> >
> > If we have a mix of classic and diskless topics on the same broker, it's
> > important that the classic topics' data is flushed to disk as quickly as
> > possible. To achieve this, users typically set dirty_expire_centisecs in
> > the kernel based on the number of available disk IOPS. Once you set this
> > number, it applies to all dirty files, including the cached data in
> > diskless topics. So, if there are more files actively accumulating data,
> > the flush frequency and therefore RPO is reduced for classic topics.
> >
> > "We should have mentioned this explicitly, but this step, in fact,
> remains
> > in the form of segments offloading to tiered storage. When we assemble a
> > segment and hand it over to RemoteLogManager, we’re effectively doing
> > metadata compaction: replacing a big number of pieces of metadata about
> > individual batches with a single record in __remote_log_metadata."
> >
> > The object merging in tier storage typically only kicks in after a few
> > hours. The impact is (1) the amount of accumulated metadata is still
> quite
> > large; (2) there are many small objects, leading to poor read
> performance.
> > I think we need a more frequent object merging process than tier storage
> > provides.
> >
> > Jun
> >
> >
> > On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> >> Hello Jack, Jun, Luke, and all!
> >>
> >> Thank you for your messages.
> >>
> >> Let me first address some of Jun’s comments.
> >>
> >>> First, it degrades the durability.
> >>> For each partition, now there are two files being actively written at a
> >>> given point of time, one for the data and another for the metadata.
> >>> Flushing each file requires a separate IO. If the disk has 1K IOPS and
> we
> >>> have 5K partitions in a broker, currently we can afford to flush each
> >>> partition every 5 seconds, achieving an RPO of 5 seconds. If we double
> >> the
> >>> number of files per partition, we can only flush each partition every
> 10
> >>> seconds, which makes RPO twice as bad.
> >>
> >> As I understand, you’re speaking about locally materialized segments.
> They
> >> will indeed consume some IOPS. See them as a cache that could always be
> >> restored from the remote storage. While it’s not ideal, it's still OK to
> >> lose data in them due to a machine crash, for example. Because of this,
> we
> >> can avoid explicit flushing on local materialized segments at all and
> let
> >> the file system and page cache figure out when to flush optimally. This
> >> would not eliminate the extra IOPS, but should reduce it dramatically,
> >> depending on throughput for each partition. We, of course, continue
> >> flushing the metadata segments as before.
> >>
> >> It’s worth making a note on caching. I think nobody will disagree that
> >> doing direct reads from remote storage every time a batch is requested
> by a
> >> consumer will not be practical neither from the performance nor from the
> >> economy point of view. We need a way to keep the number of GET requests
> >> down. There are multiple options, for example:
> >>  1. Rack-aware distributed in-memory caching.
> >>  2. Local in-memory caching. Comes with less network chattiness and
> works
> >> well if we have more or less stable brokers to consume from.
> >>  3. Materialization of diskless logs on local disk. Way lower impact on
> >> RAM and also requires stable brokers for consumption (using just
> assigned
> >> replicas will probably work well).
> >>
> >> Materialization is one of possible options, but we can choose another
> one.
> >> However, we will have this dilemma regardless of whether we have an
> >> explicit coordinator or we go “coordinator-less”.
> >>
> >>> Second, if we ever need this
> >>> metadata somewhere else, say in the WAL file manager, the consumer
> needs
> >> to
> >>> subscribe to every partition in the cluster, which is inefficient. The
> >>> actual benefit of this approach is also questionable. On the surface,
> it
> >>> might seem that we could reduce the number of lines that need to be
> >> changed
> >>> for this KIP. However, the changes are quite intrusive to the classic
> >>> partition's code path and will probably make the code base harder to
> >>> maintain in the long run. I like the original approach based on the
> batch
> >>> coordinator much better than this one. We could probably refactor the
> >>> producer state code so that it could be reused in the batch
> coordinator.
> >>
> >> It’s hard to disagree with this. The explicit coordinator is more a side
> >> thing, while coordinator-less approach is more about extending
> >> ReplicaManager, UnifiedLog and others substantially.
> >>
> >>> Thanks for addressing the concerns on the number of RPCs in the produce
> >>> path. I agree that with the metadata crafting mechanism, we could
> >> mitigate
> >>> the PRC problem. However, since we now require the metadata to be
> >>> collocated with the data on the same set of brokers, it's weird that
> they
> >>> are now managed by different mechanisms. The data assignment now uses
> the
> >>> metadata crafting mechanism, but the metadata is stored in the classic
> >>> partition using its own assignment strategy. It will be complicated to
> >> keep
> >>> them collocated.
> >>
> >> I would like to note that the metadata crafting is needed only to tell
> >> producers which brokers they should send Produce requests to, but data
> (as
> >> in “locally materialized log”) is located on partition replicas, i.e.
> >> automatically co-located with metadata.
> >>
> >> As a side note, it would probably be better that instead of implicitly
> >> crafting partition metadata, we extend the metadata protocol so that for
> >> diskless partitions we return not only the leader and replicas, but also
> >> some “recommended produce brokers”, selected for optimal performance and
> >> costs. Producers will pick ones in their racks.
> >>
> >>> I am also concerned about the removal of the object compaction/merging
> >>> step.
> >>
> >> We should have mentioned this explicitly, but this step, in fact,
> remains
> >> in the form of segments offloading to tiered storage. When we assemble a
> >> segment and hand it over to RemoteLogManager, we’re effectively doing
> >> metadata compaction: replacing a big number of pieces of metadata about
> >> individual batches with a single record in __remote_log_metadata.
> >>
> >> We could create a Diskless-specific merging mechanism instead if needed.
> >> It’s rather easy with the explicit coordinator approach. With the
> >> coordinator-less approach, this would probably be a bit more tricky
> >> (rewriting the tail of the log by the leader + replicating this change
> >> reliably).
> >>
> >>> I see a tendency toward primarily optimizing for the fewest code
> changes
> >> in
> >>> the KIP. Instead, our primary goal should be a clean design that can
> last
> >>> for the long term.
> >>
> >> Yes, totally agree.
> >>
> >>
> >>
> >> Luke,
> >>> I'm wondering if the complexity of designing txn and queue is because
> of
> >>> leaderless cluster, do you think it will be simpler if we only focus on
> >> the
> >>> "diskless" design to handle object compaction/merging to/from the
> remote
> >>> storage to save the cross-AZ cost?
> >>
> >> After some evolution of the original proposal, leaderless is now
> limited.
> >> We only need to be able to accept Produce requests on more than one
> broker
> >> to eliminate the cross-AZ costs for producers. Do I get it right that
> you
> >> propose to get rid of this? Or do I misunderstand?
> >>
> >>
> >>
> >> Let’s now look at this problem from a higher level, as Jack proposed. As
> >> it was said, the big choice we need to make is whether we 1) create an
> >> explicit batch coordinator; or 2) go for the coordinator-less approach,
> >> where each diskless partition is managed by its leader as in classic
> topics.
> >>
> >> If we try to compare the two approaches:
> >>
> >> Pluggability:
> >> - Explicit coordinator: Possible. For example, some setups may benefit
> >> from batch metadata being stored in a cloud database (such as AWS
> DynamoDB
> >> or GCP Spanner).
> >> - Coordinator-less: Impossible.
> >>
> >> Scalability and fault tolerance:
> >> - Explicit coordinator: Depends on the implementation and it’s also
> >> necessary to actively work for it.
> >> - Coordinator-less: Closer to classic Kafka topics. Scaling is done by
> >> partition placement, partitions could fail independently.
> >>
> >> Separation of concerns:
> >> - Explicit coordinator: Very good. Diskless remains more independent
> from
> >> classic topics in terms of code and workflows. For example, the
> >> above-mentioned non-tiered storage metadata compaction mechanism could
> be
> >> relatively simply implemented with it. As a flip side of this, some
> >> workflows (e.g. transactions) will have to be adapted.
> >> - Coordinator-less: Less so. It leans to the opposite: bringing diskless
> >> closer to classic topics. Some code paths and workflows could be more
> >> straightforwardly reused, but they will inevitably have to be adapted to
> >> accommodate both topic types as also discussed.
> >>
> >> Cloud-nativeness. This is a vague concept, also related to the previous,
> >> but let’s try:
> >> - Explicit coordinator: Storing and processing metadata separately makes
> >> it easier for brokers to take different roles, be purely stateless if
> >> needed, etc.
> >> - Coordinator-less: Less so. Something could be achieved with creative
> >> partition placement, but not much.
> >>
> >> Both seem to have their pros and cons. However, answering Jack’s
> question,
> >> the explicit coordinator approach may indeed lead to a more flexible
> design.
> >>
> >>
> >> The purpose of this deviation in the discussion was to receive a
> >> preliminary community evaluation of the coordinator-less approach
> without
> >> taking on the task of writing a separate KIP and fitting it in the
> system
> >> of KIP-1150 and its children. We’re open to stopping it and getting
> back to
> >> working out the coordinator design if the community doesn’t favor the
> >> proposed approach.
> >>
> >> Best,
> >> Ivan and Diskless team
> >>
> >> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> >>> Hi Ivan,
> >>>
> >>> As Jun pointed out, the updated design seems to have some shortcomings
> >>> although it simplifies the implementation.
> >>>
> >>> I'm wondering if the complexity of designing txn and queue is because
> of
> >>> leaderless cluster, do you think it will be simpler if we only focus on
> >> the
> >>> "diskless" design to handle object compaction/merging to/from the
> remote
> >>> storage to save the cross-AZ cost?
> >>>
> >>>
> >>> Thank you,
> >>> Luke
> >>>
> >>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid>
> >> wrote:
> >>>
> >>>> Hi, Ivan,
> >>>>
> >>>> Thanks for the explanation.
> >>>>
> >>>> "we write the reference to the WAL file with the batch data"
> >>>>
> >>>> I understand the approach now, but I think it is a hacky one. There
> are
> >>>> multiple short comings with this design. First, it degrades the
> >> durability.
> >>>> For each partition, now there are two files being actively written at
> a
> >>>> given point of time, one for the data and another for the metadata.
> >>>> Flushing each file requires a separate IO. If the disk has 1K IOPS and
> >> we
> >>>> have 5K partitions in a broker, currently we can afford to flush each
> >>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we double
> >> the
> >>>> number of files per partition, we can only flush each partition every
> >> 10
> >>>> seconds, which makes RPO twice as bad. Second, if we ever need this
> >>>> metadata somewhere else, say in the WAL file manager, the consumer
> >> needs to
> >>>> subscribe to every partition in the cluster, which is inefficient. The
> >>>> actual benefit of this approach is also questionable. On the surface,
> >> it
> >>>> might seem that we could reduce the number of lines that need to be
> >> changed
> >>>> for this KIP. However, the changes are quite intrusive to the classic
> >>>> partition's code path and will probably make the code base harder to
> >>>> maintain in the long run. I like the original approach based on the
> >> batch
> >>>> coordinator much better than this one. We could probably refactor the
> >>>> producer state code so that it could be reused in the batch
> >> coordinator.
> >>>>
> >>>> Thanks for addressing the concerns on the number of RPCs in the
> produce
> >>>> path. I agree that with the metadata crafting mechanism, we could
> >> mitigate
> >>>> the PRC problem. However, since we now require the metadata to be
> >>>> collocated with the data on the same set of brokers, it's weird that
> >> they
> >>>> are now managed by different mechanisms. The data assignment now uses
> >> the
> >>>> metadata crafting mechanism, but the metadata is stored in the classic
> >>>> partition using its own assignment strategy. It will be complicated to
> >> keep
> >>>> them collocated.
> >>>>
> >>>> I am also concerned about the removal of the object compaction/merging
> >>>> step. My first concern is on the amount of metadata that need to be
> >> kept.
> >>>> Without object compcation, the metadata generated in the produce path
> >> can
> >>>> only be deleted after remote tiering kicks in. Let's say for every
> >> 250ms we
> >>>> produce 100 byte of metadata per partition. Let's say remoting tiering
> >>>> kicks in after 5 hours. In a cluster with 100K partitions, we need to
> >> keep
> >>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
> >>>> signficant. A second concern is on performance. Every time we need to
> >>>> rebuild the caching data, we need to read a bunch of small objects
> >> from S3,
> >>>> slowing down the building process. If a consumer happens to need such
> >> data,
> >>>> it could slow down the application.
> >>>>
> >>>> I see a tendency toward primarily optimizing for the fewest code
> >> changes in
> >>>> the KIP. Instead, our primary goal should be a clean design that can
> >> last
> >>>> for the long term.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me>
> >> wrote:
> >>>>
> >>>>> Hi Jun,
> >>>>>
> >>>>> Thank you for your message. I’m sorry that I failed to clearly
> >> explain
> >>>> the
> >>>>> idea. Let me try to fix this.
> >>>>>
> >>>>>> Does each partition now have a metadata partition and a separate
> >> data
> >>>>>> partition? If so, I am concerned that it essentially doubles the
> >> number
> >>>>> of
> >>>>>> partitions, which impacts the number of open file descriptors and
> >> the
> >>>>>> required IOPS, and so on. It also seems wasteful to have a separate
> >>>>>> partition just to store the metadata. It's as if we are creating an
> >>>>>> internal topic with an unbounded number of partitions.
> >>>>>
> >>>>> No. There will be only one physical partition per diskless
> >> partition. Let
> >>>>> me explain this with an example. Let’s say we have a diskless
> >> partition
> >>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce
> >> some
> >>>>> batches to this partition. The content of the segment file will be
> >>>>> something like this (for each batch):
> >>>>>
> >>>>> BaseOffset: 00000000000000000000 (like in classic)
> >>>>> Length: 123456 (like in classic)
> >>>>> PartitionLeaderEpoch: like in classic
> >>>>> Magic: like in classic
> >>>>> CRC: like in classic
> >>>>> Attributes: like in classic
> >>>>> LastOffsetDelta: like in classic
> >>>>> BaseTimestamp: like in classic
> >>>>> MaxTimestamp: like in classic
> >>>>> ProducerId: like in classic
> >>>>> ProducerEpoch: like in classic
> >>>>> BaseSequence: like in classic
> >>>>> RecordsCount: like in classic
> >>>>> Records:
> >>>>>  path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> >>>>> 123456
> >>>>>
> >>>>> It looks very much like classic log entries. The only difference is
> >> that
> >>>>> instead of writing real Records, we write the reference to the WAL
> >> file
> >>>>> with the batch data (I guess we need only the name and the byte
> >> offset,
> >>>>> because the byte length is the standard field above). Otherwise,
> >> it’s a
> >>>>> normal Kafka log with the leader and replicas.
> >>>>>
> >>>>> So we have as many partitions for diskless as for classic. As of open
> >>>> file
> >>>>> descriptors, let’s proceed to the following:
> >>>>>
> >>>>>> Are the metadata and
> >>>>>> the data for the same partition always collocated on the same
> >> broker?
> >>>> If
> >>>>>> so, how do we enforce that when replicas are reassigned?
> >>>>>
> >>>>> The source of truth for the data is still in WAL files on object
> >> storage.
> >>>>> The source of truth for the metadata is in segment files on the
> >> brokers
> >>>> in
> >>>>> the replica set. Two new mechanisms are planned, both independent of
> >> this
> >>>>> new proposal, but I want to present them to give the idea that only a
> >>>>> limited amount of data files will be operated locally:
> >>>>> - We want to assemble batches into segment files and offload them to
> >>>>> tiered storage in order to prevent the unbounded growth of batch
> >>>> metadata.
> >>>>> For this, we need to open only  a few file descriptors (for the
> >> segment
> >>>>> file itself + the necessary indexes) before the segment is fully
> >> written
> >>>>> and handed over to RemoteLogManager.
> >>>>> - We want to assemble local segment files for caching purposes as
> >> well,
> >>>>> i.e. to speed up fetching. This will not materialize the full
> >> content of
> >>>>> the log, but only the hot set according to some policy (or
> >> configurable
> >>>>> policies), i.e. the number of segments and file descriptors will
> >> also be
> >>>>> limited.
> >>>>>
> >>>>>> The number of RPCs in the produce path is significantly higher. For
> >>>>>> example, if a produce request has 100 partitions, in a cluster
> >> with 100
> >>>>>> brokers, each produce request could generate 100 more RPC requests.
> >>>> This
> >>>>>> will significantly increase the request rate.
> >>>>>
> >>>>> This is a valid concern that we considered, but this issue can be
> >>>>> mitigated. I’ll try to explain the approach.
> >>>>> The situation with a single broker is trivial: all the commit
> >> requests go
> >>>>> from the broker to itself.
> >>>>> Let’s scale this to a multi-broker cluster, but located in the single
> >>>> rack
> >>>>> (AZ). Any broker can accept Produce requests for diskless
> >> partitions, but
> >>>>> we can tell producers (through metadata) to always send Produce
> >> requests
> >>>> to
> >>>>> leaders. For example, broker 0 hosts the leader replicas for diskless
> >>>>> partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> >> requests
> >>>> for
> >>>>> these partitions in various combinations, but only for them.
> >>>>>
> >>>>>                    Broker 0
> >>>>>              +-----------------+
> >>>>>              |    t1-0         |
> >>>>>              |    t2-1 <--------------------+
> >>>>>              |    t3-0         |            |
> >>>>> produce      | +-------------+ |            |
> >>>>> requests     | |  diskless   | |            |
> >>>>> --------------->|   produce   +--------------+
> >>>>> for these    | | WAL buffer  | |    commit requests
> >>>>> partitions   | +-------------+ |    for these partitions
> >>>>>              |                 |
> >>>>>              +-----------------+
> >>>>>
> >>>>> The same applies for other brokers in this cluster. Effectively, each
> >>>>> broker will commit only to itself, which effectively means 1 commit
> >>>> request
> >>>>> per WAL buffer (this may be 0 physical network calls, if we wish,
> >> just a
> >>>>> local function call).
> >>>>>
> >>>>> Now let’s scale this to multiple racks (AZs). Obviously, we cannot
> >> always
> >>>>> send Produce requests to the designated leaders of diskless
> >> partitions:
> >>>>> this would mean inter-AZ network traffic, which we would like to
> >> avoid.
> >>>> To
> >>>>> avoid it, we say that every broker has a “diskless produce
> >>>> representative”
> >>>>> in every AZ. If we continue our example: when a Produce request for
> >> t1-0,
> >>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0
> >> (in the
> >>>>> broker’s AZ the representative is the broker itself). However, if it
> >>>> comes
> >>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> >>>>>
> >>>>>  |produce requests         |produce requests        |produce
> >> requests
> >>>>>  |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
> >>>> t3-0
> >>>>>  |from AZ 0                |from AZ 1               |from AZ 2
> >>>>>  v                         v                        v
> >>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> >>>>> +---------------+      +---------------+      +---------------+
> >>>>> |     t1-0      |      |               |      |               |
> >>>>> |     t2-1      |      |               |      |               |
> >>>>> |     t3-0      |      |               |      |               |
> >>>>> +---------------+      +--------+------+      +--------+------+
> >>>>>     ^     ^                    |                      |
> >>>>>     |     +--------------------+                      |
> >>>>>     |     commit requests for these partitions        |
> >>>>>     |                                                 |
> >>>>>     +-------------------------------------------------+
> >>>>>           commit requests for these partitions
> >>>>>
> >>>>> All the partitions that broker 0 is the leader of will be
> >> “represented”
> >>>> by
> >>>>> brokers 1 and 2 in their AZs.
> >>>>>
> >>>>> Of course, this relationship goes both ways between AZs (not
> >> necessarily
> >>>>> between the same brokers). It means that provided the cluster is
> >> balanced
> >>>>> by the number of brokers per AZ, each broker will represent
> >>>> (number_of_azs
> >>>>> - 1) other brokers. This will result in the situation that for the
> >>>> majority
> >>>>> of commits, each broker will do up to (number_of_azs - 1) network
> >> commit
> >>>>> requests (plus one local). Cloud regions tend to have 3 AZs, very
> >> rarely
> >>>>> more. That means, brokers will be doing up to 2 network commit
> >> requests
> >>>> per
> >>>>> WAL file.
> >>>>>
> >>>>> There are the following exceptions:
> >>>>> 1. Broker count imbalance between AZs. For example, when we have 2
> >> AZs
> >>>> and
> >>>>> one has three brokers and another AZ has one. This one broker will do
> >>>>> between 1 and 3 commit requests per WAL file. This is not an extreme
> >>>>> amplification. Such an imbalance is not healthy in most practical
> >> setups
> >>>>> and should be avoided anyway.
> >>>>> 2. Leadership changes and metadata propagation period. When the
> >> partition
> >>>>> t3-0 is relocated from broker 0 to some broker 3, the producers will
> >> not
> >>>>> know this immediately (unless we want to be strict and respond with
> >>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
> >> together
> >>>> in a
> >>>>> WAL buffer on broker 2, it will have to send two commit requests: to
> >>>> broker
> >>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
> >> situation
> >>>>> is not permanent and as producers update the cluster metadata, it
> >> will be
> >>>>> resolved.
> >>>>>
> >>>>> This all could be built with the metadata crafting mechanism only
> >> (which
> >>>>> is anyway needed for Diskless in one way or another to direct
> >> producers
> >>>> and
> >>>>> consumers where we need to avoid inter-AZ traffic), just with the
> >> right
> >>>>> policy for it (for example, some deterministic hash-based formula).
> >> I.e.
> >>>> no
> >>>>> explicit support for “produce representative” or anything like this
> >> is
> >>>>> needed on the cluster level, in KRaft, etc.
> >>>>>
> >>>>>> The same WAL file metadata is now duplicated into two places,
> >> partition
> >>>>>> leader and WAL File Manager. Which one is the source of truth, and
> >> how
> >>>> do
> >>>>>> we maintain consistency between the two places?
> >>>>>
> >>>>> We do only two operations on WAL files that span multiple diskless
> >>>>> partitions: committing and deleting. Commits can be done
> >> independently as
> >>>>> described above. But deletes are different, because when a file is
> >>>> deleted,
> >>>>> this affects all the partitions that still have alive batches in this
> >>>> file
> >>>>> (if any).
> >>>>>
> >>>>> The WAL file manager is a necessary point of coordination to delete
> >> WAL
> >>>>> files safely. We can say it is the source of truth about files
> >>>> themselves,
> >>>>> while the partition leaders and their logs hold the truth about
> >> whether a
> >>>>> particular file contains live batches of this particular partition.
> >>>>>
> >>>>> The file manager will do this important task: be able to say for sure
> >>>> that
> >>>>> a file does not contain any live batch of any existing partition. For
> >>>> this,
> >>>>> it will have to periodically check against the partition leaders.
> >>>>> Considering that batch deletion is irreversible, when we declare a
> >> file
> >>>>> “empty”, this is guaranteed to be and stay so.
> >>>>>
> >>>>> The file manager has to know about files being committed to start
> >> track
> >>>>> them and periodically check if they are empty. We can consider
> >> various
> >>>> ways
> >>>>> to achieve this:
> >>>>> 1. As was proposed in my previous message: best effort commit by
> >> brokers
> >>>> +
> >>>>> periodic prefix scans of object storage to detect files that went
> >> below
> >>>> the
> >>>>> radar due to network issue or the file manager temporary
> >> unavailability.
> >>>>> We’re speaking about listing the file names only and opening only
> >>>>> previously unknown files in order to find the partitions involved
> >> with
> >>>> them.
> >>>>> 2. Only do scans without explicit commit, i.e. fill the list of files
> >>>>> fully asynchronously and in the background. This may be not ideal
> >> due to
> >>>>> costs and performance of scanning tons of files. However, the number
> >> of
> >>>>> live WAL files should be limited due to tiered storage offloading +
> >> we
> >>>> can
> >>>>> optimize this if we give files some global soft order in their names.
> >>>>>
> >>>>>> I am not sure how this design simplifies the implementation. The
> >>>> existing
> >>>>>> producer/replication code can't be simply reused. Adjusting both
> >> the
> >>>>> write
> >>>>>> path in the leader and the replication path in the follower to
> >>>> understand
> >>>>>> batch-header only data is quite intrusive to the existing logic.
> >>>>>
> >>>>> It is true that we’ll have to change LocalLog and UnifiedLog in
> >> order to
> >>>>> support these changes. However, it seems that idempotence,
> >> transactions,
> >>>>> queues, tiered storage will have to be changed less than with the
> >>>> original
> >>>>> design. This is because the partition leader state would remain in
> >> the
> >>>> same
> >>>>> place (on brokers) and existing workflows that involve it would have
> >> to
> >>>> be
> >>>>> changed less compared to the situation where we globalize the
> >> partition
> >>>>> leader state in the batch coordinator. I admit this is hard to make
> >>>>> convincing without both real implementations to hand :)
> >>>>>
> >>>>>> I am also
> >>>>>> not sure how this enables seamless switching the topic modes
> >> between
> >>>>>> diskless and classic. Could you provide more details on those?
> >>>>>
> >>>>> Let’s consider the scenario of turning a classic topic into
> >> diskless. The
> >>>>> user sets diskless.enabled=true, the leader receives this metadata
> >> update
> >>>>> and does the following:
> >>>>> 1. Stop accepting normal append writes.
> >>>>> 2. Close the current active segment.
> >>>>> 3. Start a new segment that will be written in the diskless format
> >> (i.e.
> >>>>> without data).
> >>>>> 4. Start accepting diskless commits.
> >>>>>
> >>>>> Since it’s the same log, the followers will know about that switch
> >>>>> consistently. They will finish replicating the classic segments and
> >> start
> >>>>> replicating the diskless ones. They will always know where each
> >> batch is
> >>>>> located (either inside a classic segment or referenced by a diskless
> >>>> one).
> >>>>> Switching back should be similar.
> >>>>>
> >>>>> Doing this with the coordinator is possible, but has some caveats.
> >> The
> >>>>> leader must do the following:
> >>>>> 1. Stop accepting normal append writes.
> >>>>> 2. Close the current active segment.
> >>>>> 3. Write a special control segment to persist and replicate the fact
> >> that
> >>>>> from offset N the partition is now in the diskless mode.
> >>>>> 4. Inform the coordinator about the first offset N of the “diskless
> >> era”.
> >>>>> 5. Inform the controller quorum that the transition has finished and
> >> that
> >>>>> brokers now can process diskless writes for this partition.
> >>>>> This could fail at some points, so this will probably require some
> >>>>> explicit state machine with replication either in the partition log
> >> or in
> >>>>> KRaft.
> >>>>>
> >>>>> It seems that the coordinator-less approach makes this simpler
> >> because
> >>>> the
> >>>>> “coordinator” for the partition and the partition leader are the
> >> same and
> >>>>> they store the partition metadata in the same log, too. While in the
> >>>>> coordinator approach we have to perform some kind of a distributed
> >> commit
> >>>>> to handover metadata management from the classic partition leader to
> >> the
> >>>>> batch coordinator.
> >>>>>
> >>>>> I hope these explanations help to clarify the idea. Please let me
> >> know if
> >>>>> I should go deeper anywhere.
> >>>>>
> >>>>> Best,
> >>>>> Ivan and the Diskless team
> >>>>>
> >>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> >>>>>> Hi, Ivan,
> >>>>>>
> >>>>>> Thanks for the update.
> >>>>>>
> >>>>>> I am not sure that I fully understand the new design, but it seems
> >> less
> >>>>>> clean than before.
> >>>>>>
> >>>>>> Does each partition now have a metadata partition and a separate
> >> data
> >>>>>> partition? If so, I am concerned that it essentially doubles the
> >> number
> >>>>> of
> >>>>>> partitions, which impacts the number of open file descriptors and
> >> the
> >>>>>> required IOPS, and so on. It also seems wasteful to have a separate
> >>>>>> partition just to store the metadata. It's as if we are creating an
> >>>>>> internal topic with an unbounded number of partitions. Are the
> >> metadata
> >>>>> and
> >>>>>> the data for the same partition always collocated on the same
> >> broker?
> >>>> If
> >>>>>> so, how do we enforce that when replicas are reassigned?
> >>>>>>
> >>>>>> The number of RPCs in the produce path is significantly higher. For
> >>>>>> example, if a produce request has 100 partitions, in a cluster
> >> with 100
> >>>>>> brokers, each produce request could generate 100 more RPC requests.
> >>>> This
> >>>>>> will significantly increase the request rate.
> >>>>>>
> >>>>>> The same WAL file metadata is now duplicated into two places,
> >> partition
> >>>>>> leader and WAL File Manager. Which one is the source of truth, and
> >> how
> >>>> do
> >>>>>> we maintain consistency between the two places?
> >>>>>>
> >>>>>> I am not sure how this design simplifies the implementation. The
> >>>> existing
> >>>>>> producer/replication code can't be simply reused. Adjusting both
> >> the
> >>>>> write
> >>>>>> path in the leader and the replication path in the follower to
> >>>> understand
> >>>>>> batch-header only data is quite intrusive to the existing logic. I
> >> am
> >>>>> also
> >>>>>> not sure how this enables seamless switching the topic modes
> >> between
> >>>>>> diskless and classic. Could you provide more details on those?
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me>
> >> wrote:
> >>>>>>
> >>>>>>> Hi dear Kafka community,
> >>>>>>>
> >>>>>>> In the initial Diskless proposal, we proposed to have a separate
> >>>>>>> component, batch/diskless coordinator, whose role would be to
> >>>> centrally
> >>>>>>> manage the batch and WAL file metadata for diskless topics. This
> >>>>> component
> >>>>>>> drew many reasonable comments from the community about how it
> >> would
> >>>>> support
> >>>>>>> various Kafka features (transactions, queues) and its
> >> scalability.
> >>>>> While we
> >>>>>>> believe we have good answers to all the expressed concerns, we
> >> took a
> >>>>> step
> >>>>>>> back and looked at the problem from a different perspective.
> >>>>>>>
> >>>>>>> We would like to propose an alternative Diskless design *without
> >> a
> >>>>>>> centralized coordinator*. We believe this approach has potential
> >> and
> >>>>>>> propose to discuss it as it may be more appealing to the
> >> community.
> >>>>>>>
> >>>>>>> Let us explain the idea. Most of the complications with the
> >> original
> >>>>>>> Diskless approach come from one necessary architecture change:
> >>>>> globalizing
> >>>>>>> the local state of partition leader in the batch coordinator.
> >> This
> >>>>> causes
> >>>>>>> deviations to the established workflows in various features like
> >>>>> produce
> >>>>>>> idempotence and transactions, queues, retention, etc. These
> >>>> deviations
> >>>>> need
> >>>>>>> to be carefully considered, designed, and later implemented and
> >>>>> tested. In
> >>>>>>> the new approach we want to avoid this by making partition
> >> leaders
> >>>>> again
> >>>>>>> responsible for managing their partitions, even in diskless
> >> topics.
> >>>>>>>
> >>>>>>> In classic Kafka topics, batch data and metadata are blended
> >> together
> >>>>> in
> >>>>>>> the one partition log. The crux of the Diskless idea is to
> >> decouple
> >>>>> them
> >>>>>>> and move data to the remote storage, while keeping metadata
> >> somewhere
> >>>>> else.
> >>>>>>> Using the central batch coordinator for managing batch metadata
> >> is
> >>>> one
> >>>>> way,
> >>>>>>> but not the only.
> >>>>>>>
> >>>>>>> Let’s now think about managing metadata for each user partition
> >>>>>>> independently. Generally partitions are independent and don’t
> >> share
> >>>>>>> anything apart from that their data are mixed in WAL files. If we
> >>>>> figure
> >>>>>>> out how to commit and later delete WAL files safely, we will
> >> achieve
> >>>>> the
> >>>>>>> necessary autonomy that allows us to get rid of the central batch
> >>>>>>> coordinator. Instead, *each diskless user partition will be
> >> managed
> >>>> by
> >>>>> its
> >>>>>>> leader*, as in classic Kafka topics. Also like in classic
> >> topics, the
> >>>>>>> leader uses the partition log as the way to persist batch
> >> metadata,
> >>>>> i.e.
> >>>>>>> the regular batch header + the information about how to find this
> >>>>> batch on
> >>>>>>> remote storage. In contrast to classic topics, batch data is in
> >>>> remote
> >>>>>>> storage.
> >>>>>>>
> >>>>>>> For clarity, let’s compare the three designs:
> >>>>>>> • Classic topics:
> >>>>>>>   • Data and metadata are co-located in the partition log.
> >>>>>>>   • The partition log content: [Batch header (metadata)|Batch
> >> data].
> >>>>>>>   • The partition log is replicated to the followers.
> >>>>>>>   • The replicas and leader have local state built from
> >> metadata.
> >>>>>>> • Original Diskless:
> >>>>>>>   • Metadata is in the batch coordinator, data is on remote
> >> storage.
> >>>>>>>   • The partition state is global in the batch coordinator.
> >>>>>>> • New Diskless:
> >>>>>>>   • Metadata is in the partition log, data is on remote storage.
> >>>>>>>   • Partition log content: [Batch header (metadata)|Batch
> >>>> coordinates
> >>>>> on
> >>>>>>> remote storage].
> >>>>>>>   • The partition log is replicated to the followers.
> >>>>>>>   • The replicas and leader have local state built from
> >> metadata.
> >>>>>>>
> >>>>>>> Let’s consider the produce path. Here’s the reminder of the
> >> original
> >>>>>>> Diskless design:
> >>>>>>>
> >>>>>>>
> >>>>>>> The new approach could be depicted as the following:
> >>>>>>>
> >>>>>>>
> >>>>>>> As you can see, the main difference is that now instead of a
> >> single
> >>>>> commit
> >>>>>>> request to the batch coordinator, we send multiple parallel
> >> commit
> >>>>> requests
> >>>>>>> to all the leaders of each partition involved in the WAL file.
> >> Each
> >>>> of
> >>>>> them
> >>>>>>> will commit its batches independently, without coordinating with
> >>>> other
> >>>>>>> leaders and any other components. Batch data is addressed by the
> >> WAL
> >>>>> file
> >>>>>>> name, the byte offset and size, which allows partitions to know
> >>>> nothing
> >>>>>>> about other partitions to access their data in shared WAL files.
> >>>>>>>
> >>>>>>> The number of partitions involved in a single WAL file may be
> >> quite
> >>>>> large,
> >>>>>>> e.g. a hundred. A hundred network requests to commit one WAL
> >> file is
> >>>>> very
> >>>>>>> impractical. However, there are ways to reduce this number:
> >>>>>>> 1. Partition leaders are located on brokers. Requests to
> >> leaders on
> >>>>> one
> >>>>>>> broker could be grouped together into a single physical network
> >>>> request
> >>>>>>> (resembling the normal Produce request that may carry batches for
> >>>> many
> >>>>>>> partitions inside). This will cap the number of network requests
> >> to
> >>>> the
> >>>>>>> number of brokers in the cluster.
> >>>>>>> 2. If we craft the cluster metadata to make producers send their
> >>>>> requests
> >>>>>>> to the right brokers (with respect to AZs), we may achieve the
> >> higher
> >>>>>>> concentration of logical commit requests in physical network
> >> requests
> >>>>>>> reducing the number of the latter ones even further, ideally to
> >> one.
> >>>>>>>
> >>>>>>> Obviously, out of multiple commit requests some may fail or time
> >> out
> >>>>> for a
> >>>>>>> variety of reasons. This is fine. Some producers will receive
> >> totally
> >>>>> or
> >>>>>>> partially failed responses to their Produce requests, similar to
> >> what
> >>>>> they
> >>>>>>> would have received when appending to a classic topic fails or
> >> times
> >>>>> out.
> >>>>>>> If a partition experiences problems, other partitions will not be
> >>>>> affected
> >>>>>>> (again, like in classic topics). Of course, the uncommitted data
> >> will
> >>>>> be
> >>>>>>> garbage in WAL files. But WAL files are short-lived (batches are
> >>>>> constantly
> >>>>>>> assembled into segments and offloaded to tiered storage), so this
> >>>>> garbage
> >>>>>>> will be eventually deleted.
> >>>>>>>
> >>>>>>> For safely deleting WAL files we now need to centrally manage
> >> them,
> >>>> as
> >>>>>>> this is the only state and logic that spans multiple partitions.
> >> On
> >>>> the
> >>>>>>> diagram, you can see another commit request called “Commit file
> >> (best
> >>>>>>> effort)” going to the WAL File Manager. This manager will be
> >>>>> responsible
> >>>>>>> for the following:
> >>>>>>> 1. Collecting (by requests from brokers) and persisting
> >> information
> >>>>> about
> >>>>>>> committed WAL files.
> >>>>>>> 2. To handle potential failures in file information delivery, it
> >>>> will
> >>>>> be
> >>>>>>> doing prefix scan on the remote storage periodically to find and
> >>>>> register
> >>>>>>> unknown files. The period of this scan will be configurable and
> >>>> ideally
> >>>>>>> should be quite long.
> >>>>>>> 3. Checking with the relevant partition leaders (after a grace
> >>>>> period) if
> >>>>>>> they still have batches in a particular file.
> >>>>>>> 4. Physically deleting files when they aren’t anymore referred
> >> to by
> >>>>> any
> >>>>>>> partition.
> >>>>>>>
> >>>>>>> This new design offers the following advantages:
> >>>>>>> 1. It simplifies the implementation of many Kafka features such
> >> as
> >>>>>>> idempotence, transactions, queues, tiered storage, retention.
> >> Now we
> >>>>> don’t
> >>>>>>> need to abstract away and reuse the code from partition leaders
> >> in
> >>>> the
> >>>>>>> batch coordinator. Instead, we will literally use the same code
> >> paths
> >>>>> in
> >>>>>>> leaders, with little adaptation. Workflows from classic topics
> >> mostly
> >>>>>>> remain unchanged.
> >>>>>>> For example, it seems that
> >>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> >>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for transaction
> >> support
> >>>> on
> >>>>> the
> >>>>>>> partition leader side could be used for diskless topics with
> >> little
> >>>>>>> adaptation. ProducerStateManager, needed for both idempotent
> >> produce
> >>>>> and
> >>>>>>> transactions, would be reused.
> >>>>>>> Another example is share groups support, where the share
> >> partition
> >>>>> leader,
> >>>>>>> being co-located with the partition leader, would execute the
> >> same
> >>>>> logic
> >>>>>>> for both diskless and classic topics.
> >>>>>>> 2. It returns to the familiar partition-based scaling model,
> >> where
> >>>>>>> partitions are independent.
> >>>>>>> 3. It makes the operation and failure patterns closer to the
> >>>> familiar
> >>>>>>> ones from classic topics.
> >>>>>>> 4. It opens a straightforward path to seamless switching the
> >> topics
> >>>>> modes
> >>>>>>> between diskless and classic.
> >>>>>>>
> >>>>>>> The rest of the things remain unchanged compared to the previous
> >>>>> Diskless
> >>>>>>> design (after all previous discussions). Such things as local
> >> segment
> >>>>>>> materialization by replicas, the consume path, tiered storage
> >>>>> integration,
> >>>>>>> etc.
> >>>>>>>
> >>>>>>> If the community finds this design more suitable, we will update
> >> the
> >>>>>>> KIP(s) accordingly and continue working on it. Please let us know
> >>>> what
> >>>>> you
> >>>>>>> think.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Ivan and Diskless team
> >>>>>>>
> >>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> >>>>>>>> Hi Justine,
> >>>>>>>>
> >>>>>>>> Yes, you're right. We need to track the aborted transactions
> >> for in
> >>>>> the
> >>>>>>> diskless coordinator for as long as the corresponding offsets are
> >>>>> there.
> >>>>>>> With the tiered storage unification Greg mentioned earlier, this
> >> will
> >>>>> be
> >>>>>>> finite time even for infinite data retention.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Ivan
> >>>>>>>>
> >>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> >>>>>>>>> Hey Ivan,
> >>>>>>>>>
> >>>>>>>>> Thanks for the response. I think most of what you said made
> >>>> sense,
> >>>>> but
> >>>>>>> I
> >>>>>>>>> did have some questions about this part:
> >>>>>>>>>
> >>>>>>>>>> As we understand this, the partition leader in classic
> >> topics
> >>>>> forgets
> >>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> >> it). The
> >>>>>>>>> transaction coordinator acts like the main guardian, allowing
> >>>>> partition
> >>>>>>>>> leaders to do this safely. Please correct me if this is
> >> wrong. We
> >>>>> think
> >>>>>>>>> about relying on this with the batch coordinator and delete
> >> the
> >>>>>>> information
> >>>>>>>>> about a transaction once it’s finished (as there’s no
> >> replication
> >>>>> and
> >>>>>>> HWM
> >>>>>>>>> advances immediately).
> >>>>>>>>>
> >>>>>>>>> I didn't quite understand this. In classic topics, we have
> >> maps
> >>>> for
> >>>>>>> ongoing
> >>>>>>>>> transactions which remove state when the transaction is
> >> completed
> >>>>> and
> >>>>>>> an
> >>>>>>>>> aborted transactions index which is retained for much longer.
> >>>> Once
> >>>>> the
> >>>>>>>>> transaction is completed, the coordinator is no longer
> >> involved
> >>>> in
> >>>>>>>>> maintaining this partition side state, and it is subject to
> >>>>> compaction
> >>>>>>> etc.
> >>>>>>>>> Looking back at the outline provided above, I didn't see much
> >>>>> about the
> >>>>>>>>> fetch path, so maybe that could be expanded a bit further. I
> >> saw
> >>>>> the
> >>>>>>>>> following in a response:
> >>>>>>>>>> When the broker constructs a fully valid local segment,
> >> all the
> >>>>>>> necessary
> >>>>>>>>> control batches will be inserted and indices, including the
> >>>>> transaction
> >>>>>>>>> index will be built to serve FetchRequests exactly as they
> >> are
> >>>>> today.
> >>>>>>>>>
> >>>>>>>>> Based on this, it seems like we need to retain the
> >> information
> >>>>> about
> >>>>>>>>> aborted txns for longer.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Justine
> >>>>>>>>>
> >>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> >> ivan@ivanyu.me>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Justine and all,
> >>>>>>>>>>
> >>>>>>>>>> Thank you for your questions!
> >>>>>>>>>>
> >>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> >> with
> >>>>>>> producer ID
> >>>>>>>>>>> and epoch, the positive result of this check could be
> >> cached
> >>>>>>> locally
> >>>>>>>>>>> Are we saying that only new transaction version 2
> >>>> transactions
> >>>>> can
> >>>>>>> be
> >>>>>>>>>> used
> >>>>>>>>>>> here? If not, we can't uniquely identify transactions
> >> with
> >>>>>>> producer id +
> >>>>>>>>>>> epoch
> >>>>>>>>>>
> >>>>>>>>>> You’re right that we (probably unintentionally) focused
> >> only on
> >>>>>>> version 2.
> >>>>>>>>>> We can either limit the support to version 2 or consider
> >> using
> >>>>> some
> >>>>>>>>>> surrogates to support version 1.
> >>>>>>>>>>
> >>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> >>>>> checks
> >>>>>>> of the
> >>>>>>>>>>> batches. This procedure would output the same errors
> >> like the
> >>>>>>> partition
> >>>>>>>>>>> leader in classic topics would do.
> >>>>>>>>>>> Can you expand on what these checks are? Would you be
> >>>> checking
> >>>>> if
> >>>>>>> the
> >>>>>>>>>>> transaction was still ongoing for example?* *
> >>>>>>>>>>
> >>>>>>>>>> Yes, the producer epoch, that the transaction is ongoing,
> >> and
> >>>> of
> >>>>>>> course
> >>>>>>>>>> the normal idempotence checks. What the partition leader
> >> in the
> >>>>>>> classic
> >>>>>>>>>> topics does before appending a batch to the local log
> >> (e.g. in
> >>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
> >>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In Diskless,
> >> we
> >>>>>>> unfortunately
> >>>>>>>>>> cannot do these checks before appending the data to the WAL
> >>>>> segment
> >>>>>>> and
> >>>>>>>>>> uploading it, but we can “tombstone” these batches in the
> >> batch
> >>>>>>> coordinator
> >>>>>>>>>> during the final commit.
> >>>>>>>>>>
> >>>>>>>>>>> Is there state about ongoing
> >>>>>>>>>>> transactions in the batch coordinator? I see some other
> >> state
> >>>>>>> mentioned
> >>>>>>>>>> in
> >>>>>>>>>>> the End transaction section, but it's not super clear
> >> what
> >>>>> state is
> >>>>>>>>>> stored
> >>>>>>>>>>> and when it is stored.
> >>>>>>>>>>
> >>>>>>>>>> Right, this should have been more explicit. As the
> >> partition
> >>>>> leader
> >>>>>>> tracks
> >>>>>>>>>> ongoing transactions for classic topics, the batch
> >> coordinator
> >>>>> has
> >>>>>>> to as
> >>>>>>>>>> well. So when a transaction starts and ends, the
> >> transaction
> >>>>>>> coordinator
> >>>>>>>>>> must inform the batch coordinator about this.
> >>>>>>>>>>
> >>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> >> perhaps
> >>>>> that
> >>>>>>> would
> >>>>>>>>>> be
> >>>>>>>>>>> stored in the batch coordinator?
> >>>>>>>>>>
> >>>>>>>>>> Yes. This could be deduced from the committed batches and
> >> other
> >>>>>>>>>> information, but for the sake of performance we’d better
> >> store
> >>>> it
> >>>>>>>>>> explicitly.
> >>>>>>>>>>
> >>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> >>>>> state is
> >>>>>>>>>>> maintained in the batch coordinator and how it will be
> >>>> cleaned
> >>>>> up?
> >>>>>>>>>>
> >>>>>>>>>> As we understand this, the partition leader in classic
> >> topics
> >>>>> forgets
> >>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
> >> it).
> >>>> The
> >>>>>>>>>> transaction coordinator acts like the main guardian,
> >> allowing
> >>>>>>> partition
> >>>>>>>>>> leaders to do this safely. Please correct me if this is
> >> wrong.
> >>>> We
> >>>>>>> think
> >>>>>>>>>> about relying on this with the batch coordinator and
> >> delete the
> >>>>>>> information
> >>>>>>>>>> about a transaction once it’s finished (as there’s no
> >>>> replication
> >>>>>>> and HWM
> >>>>>>>>>> advances immediately).
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Ivan
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> >>>>>>>>>>> Hey folks,
> >>>>>>>>>>>
> >>>>>>>>>>> Excited to see some updates related to transactions!
> >>>>>>>>>>>
> >>>>>>>>>>> I had a few questions.
> >>>>>>>>>>>
> >>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
> >> with
> >>>>>>> producer ID
> >>>>>>>>>>> and epoch, the positive result of this check could be
> >> cached
> >>>>>>> locally
> >>>>>>>>>>> Are we saying that only new transaction version 2
> >>>> transactions
> >>>>> can
> >>>>>>> be
> >>>>>>>>>> used
> >>>>>>>>>>> here? If not, we can't uniquely identify transactions
> >> with
> >>>>>>> producer id +
> >>>>>>>>>>> epoch
> >>>>>>>>>>>
> >>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
> >>>>> checks
> >>>>>>> of the
> >>>>>>>>>>> batches. This procedure would output the same errors
> >> like the
> >>>>>>> partition
> >>>>>>>>>>> leader in classic topics would do.
> >>>>>>>>>>> Can you expand on what these checks are? Would you be
> >>>> checking
> >>>>> if
> >>>>>>> the
> >>>>>>>>>>> transaction was still ongoing for example? Is there state
> >>>> about
> >>>>>>> ongoing
> >>>>>>>>>>> transactions in the batch coordinator? I see some other
> >> state
> >>>>>>> mentioned
> >>>>>>>>>> in
> >>>>>>>>>>> the End transaction section, but it's not super clear
> >> what
> >>>>> state is
> >>>>>>>>>> stored
> >>>>>>>>>>> and when it is stored.
> >>>>>>>>>>>
> >>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
> >> perhaps
> >>>>> that
> >>>>>>> would
> >>>>>>>>>> be
> >>>>>>>>>>> stored in the batch coordinator?
> >>>>>>>>>>>
> >>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
> >>>>> state is
> >>>>>>>>>>> maintained in the batch coordinator and how it will be
> >>>> cleaned
> >>>>> up?
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> >>>>> <ju...@confluent.io.invalid>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi, Greg and Ivan,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the update. A few comments.
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
> >>>> segments,
> >>>>>>> making
> >>>>>>>>>> use of
> >>>>>>>>>>>> the
> >>>>>>>>>>>> indexes, page cache, request purgatory, and zero-copy
> >>>>>>> functionality
> >>>>>>>>>> already
> >>>>>>>>>>>> built into classic topics."
> >>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
> >> each
> >>>>>>> partition in
> >>>>>>>>>>>> diskless topics?
> >>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
> >> need
> >>>> to
> >>>>> know
> >>>>>>>>>> aborted
> >>>>>>>>>>>> records. How is that achieved?
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR 11. "The batch coordinator saves that the
> >> transaction is
> >>>>>>> finished
> >>>>>>>>>> and
> >>>>>>>>>>>> also inserts the control batches in the corresponding
> >> logs
> >>>>> of the
> >>>>>>>>>> involved
> >>>>>>>>>>>> Diskless topics. This happens only on the metadata
> >> level,
> >>>> no
> >>>>>>> actual
> >>>>>>>>>> control
> >>>>>>>>>>>> batches are written to any file. "
> >>>>>>>>>>>> A fetch response could include multiple transactional
> >>>>> batches.
> >>>>>>> How
> >>>>>>>>>> does the
> >>>>>>>>>>>> broker obtain the information about the ending control
> >>>> batch
> >>>>> for
> >>>>>>> each
> >>>>>>>>>>>> batch? Does that mean that a fetch response needs to be
> >>>>> built by
> >>>>>>>>>>>> stitching record batches and generated control batches
> >>>>> together?
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR 12. Queues: Is there still a share partition leader
> >> that
> >>>>> all
> >>>>>>>>>> consumers
> >>>>>>>>>>>> are routed to?
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this or
> >> it's
> >>>>> too
> >>>>>>>>>>>> implementation-focused?" It would be useful to include
> >>>> enough
> >>>>>>> details
> >>>>>>>>>> to
> >>>>>>>>>>>> understand correctness and performance impact.
> >>>>>>>>>>>>
> >>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
> >>>> producer
> >>>>>>> contain a
> >>>>>>>>>>>> sequence number, which is ordered. If a producer sends
> >>>> every
> >>>>>>> Produce
> >>>>>>>>>>>> request to an arbitrary broker, those requests could
> >> reach
> >>>>> the
> >>>>>>> batch
> >>>>>>>>>>>> coordinator in different order and lead to rejection
> >> of the
> >>>>>>> produce
> >>>>>>>>>>>> requests.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> >>>>> ivan@ivanyu.me>
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We have also thought in a bit more details about
> >>>>> transactions
> >>>>>>> and
> >>>>>>>>>> queues,
> >>>>>>>>>>>>> here's the plan.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Transactions*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The support for transactions in *classic topics* is
> >> based
> >>>>> on
> >>>>>>> precise
> >>>>>>>>>>>>> interactions between three actors: clients (mostly
> >>>>> producers,
> >>>>>>> but
> >>>>>>>>>> also
> >>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
> >> classes),
> >>>> and
> >>>>>>>>>> transaction
> >>>>>>>>>>>>> coordinators. Brokers also run partition leaders with
> >>>> their
> >>>>>>> local
> >>>>>>>>>> state
> >>>>>>>>>>>>> (ProducerStateManager and others).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The high level (some details skipped) workflow is the
> >>>>>>> following.
> >>>>>>>>>> When a
> >>>>>>>>>>>>> transactional Produce request is received by the
> >> broker:
> >>>>>>>>>>>>> 1. For each partition, the partition leader checks
> >> if a
> >>>>>>> non-empty
> >>>>>>>>>>>>> transaction is running for this partition. This is
> >> done
> >>>>> using
> >>>>>>> its
> >>>>>>>>>> local
> >>>>>>>>>>>>> state derived from the log metadata
> >>>> (ProducerStateManager,
> >>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
> >>>>>>>>>>>>> 2. The transaction coordinator is informed about all
> >> the
> >>>>>>> partitions
> >>>>>>>>>> that
> >>>>>>>>>>>>> aren’t part of the transaction to include them.
> >>>>>>>>>>>>> 3. The partition leaders do additional transactional
> >>>>> checks.
> >>>>>>>>>>>>> 4. The partition leaders append the transactional
> >> data to
> >>>>>>> their logs
> >>>>>>>>>> and
> >>>>>>>>>>>>> update some of their state (for example, log the fact
> >>>> that
> >>>>> the
> >>>>>>>>>>>> transaction
> >>>>>>>>>>>>> is running for the partition and its first offset).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When the transaction is committed or aborted:
> >>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> >>>>> directly
> >>>>>>> with
> >>>>>>>>>>>>> EndTxnRequest.
> >>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> >> or
> >>>>>>>>>> PREPARE_ABORT to
> >>>>>>>>>>>>> its log and responds to the producer.
> >>>>>>>>>>>>> 3. The transaction coordinator sends
> >>>>> WriteTxnMarkersRequest to
> >>>>>>> the
> >>>>>>>>>>>> leaders
> >>>>>>>>>>>>> of the involved partitions.
> >>>>>>>>>>>>> 4. The partition leaders write the transaction
> >> markers to
> >>>>>>> their logs
> >>>>>>>>>> and
> >>>>>>>>>>>>> respond to the coordinator.
> >>>>>>>>>>>>> 5. The coordinator writes the final transaction state
> >>>>>>>>>> COMPLETE_COMMIT or
> >>>>>>>>>>>>> COMPLETE_ABORT.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In classic topics, partitions have leaders and lots
> >> of
> >>>>>>> important
> >>>>>>>>>> state
> >>>>>>>>>>>>> necessary for supporting this workflow is local. The
> >> main
> >>>>>>> challenge
> >>>>>>>>>> in
> >>>>>>>>>>>>> mapping this to Diskless comes from the fact there
> >> are no
> >>>>>>> partition
> >>>>>>>>>>>>> leaders, so the corresponding pieces of state need
> >> to be
> >>>>>>> globalized
> >>>>>>>>>> in
> >>>>>>>>>>>> the
> >>>>>>>>>>>>> batch coordinator. We are already doing this to
> >> support
> >>>>>>> idempotent
> >>>>>>>>>>>> produce.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The high level workflow for *diskless topics* would
> >> look
> >>>>> very
> >>>>>>>>>> similar:
> >>>>>>>>>>>>> 1. For each partition, the broker checks if a
> >> non-empty
> >>>>>>> transaction
> >>>>>>>>>> is
> >>>>>>>>>>>>> running for this partition. In contrast to classic
> >>>> topics,
> >>>>>>> this is
> >>>>>>>>>>>> checked
> >>>>>>>>>>>>> against the batch coordinator with a single RPC.
> >> Since a
> >>>>>>> transaction
> >>>>>>>>>>>> could
> >>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
> >> the
> >>>>> positive
> >>>>>>>>>> result of
> >>>>>>>>>>>>> this check could be cached locally (for the double
> >>>>> configured
> >>>>>>>>>> duration
> >>>>>>>>>>>> of a
> >>>>>>>>>>>>> transaction, for example).
> >>>>>>>>>>>>> 2. The same: The transaction coordinator is informed
> >>>> about
> >>>>> all
> >>>>>>> the
> >>>>>>>>>>>>> partitions that aren’t part of the transaction to
> >> include
> >>>>> them.
> >>>>>>>>>>>>> 3. No transactional checks are done on the broker
> >> side.
> >>>>>>>>>>>>> 4. The broker appends the transactional data to the
> >>>> current
> >>>>>>> shared
> >>>>>>>>>> WAL
> >>>>>>>>>>>>> segment. It doesn’t update any transaction-related
> >> state
> >>>>> for
> >>>>>>> Diskless
> >>>>>>>>>>>>> topics, because it doesn’t have any.
> >>>>>>>>>>>>> 5. The WAL segment is committed to the batch
> >> coordinator
> >>>>> like
> >>>>>>> in the
> >>>>>>>>>>>>> normal produce flow.
> >>>>>>>>>>>>> 6. The batch coordinator does the final transactional
> >>>>> checks
> >>>>>>> of the
> >>>>>>>>>>>>> batches. This procedure would output the same errors
> >> like
> >>>>> the
> >>>>>>>>>> partition
> >>>>>>>>>>>>> leader in classic topics would do. I.e. some batches
> >>>> could
> >>>>> be
> >>>>>>>>>> rejected.
> >>>>>>>>>>>>> This means, there will potentially be garbage in the
> >> WAL
> >>>>>>> segment
> >>>>>>>>>> file in
> >>>>>>>>>>>>> case of transactional errors. This is preferable to
> >> doing
> >>>>> more
> >>>>>>>>>> network
> >>>>>>>>>>>>> round trips, especially considering the WAL segments
> >> will
> >>>>> be
> >>>>>>>>>> relatively
> >>>>>>>>>>>>> short-living (see the Greg's update above).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When the transaction is committed or aborted:
> >>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
> >>>>> directly
> >>>>>>> with
> >>>>>>>>>>>>> EndTxnRequest.
> >>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
> >> or
> >>>>>>>>>> PREPARE_ABORT to
> >>>>>>>>>>>>> its log and responds to the producer.
> >>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
> >> batch
> >>>>>>> coordinator
> >>>>>>>>>> that
> >>>>>>>>>>>>> the transaction is finished.
> >>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
> >>>>> transaction is
> >>>>>>>>>> finished
> >>>>>>>>>>>>> and also inserts the control batches in the
> >> corresponding
> >>>>> logs
> >>>>>>> of the
> >>>>>>>>>>>>> involved Diskless topics. This happens only on the
> >>>> metadata
> >>>>>>> level, no
> >>>>>>>>>>>>> actual control batches are written to any file. They
> >> will
> >>>>> be
> >>>>>>>>>> dynamically
> >>>>>>>>>>>>> created on Fetch and other read operations. We could
> >>>>>>> technically
> >>>>>>>>>> write
> >>>>>>>>>>>>> these control batches for real, but this would mean
> >> extra
> >>>>>>> produce
> >>>>>>>>>>>> latency,
> >>>>>>>>>>>>> so it's better just to mark them in the batch
> >> coordinator
> >>>>> and
> >>>>>>> save
> >>>>>>>>>> these
> >>>>>>>>>>>>> milliseconds.
> >>>>>>>>>>>>> 5. The transaction coordinator sends
> >>>>> WriteTxnMarkersRequest to
> >>>>>>> the
> >>>>>>>>>>>> leaders
> >>>>>>>>>>>>> of the involved partitions. – Now only to classic
> >> topics
> >>>>> now.
> >>>>>>>>>>>>> 6. The partition leaders of classic topics write the
> >>>>>>> transaction
> >>>>>>>>>> markers
> >>>>>>>>>>>>> to their logs and respond to the coordinator.
> >>>>>>>>>>>>> 7. The coordinator writes the final transaction state
> >>>>>>>>>> COMPLETE_COMMIT or
> >>>>>>>>>>>>> COMPLETE_ABORT.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Compared to the non-transactional produce flow, we
> >> get:
> >>>>>>>>>>>>> 1. An extra network round trip between brokers and
> >> the
> >>>>> batch
> >>>>>>>>>> coordinator
> >>>>>>>>>>>>> when a new partition appear in the transaction. To
> >>>>> mitigate the
> >>>>>>>>>> impact of
> >>>>>>>>>>>>> them:
> >>>>>>>>>>>>>  - The results will be cached.
> >>>>>>>>>>>>>  - The calls for multiple partitions in one Produce
> >>>>> request
> >>>>>>> will be
> >>>>>>>>>>>>> grouped.
> >>>>>>>>>>>>>  - The batch coordinator should be optimized for
> >> fast
> >>>>>>> response to
> >>>>>>>>>> these
> >>>>>>>>>>>>> RPCs.
> >>>>>>>>>>>>>  - The fact that a single producer normally will
> >>>>> communicate
> >>>>>>> with a
> >>>>>>>>>>>>> single broker for the duration of the transaction
> >> further
> >>>>>>> reduces the
> >>>>>>>>>>>>> expected number of round trips.
> >>>>>>>>>>>>> 2. An extra round trip between the transaction
> >>>> coordinator
> >>>>> and
> >>>>>>> batch
> >>>>>>>>>>>>> coordinator when a transaction is finished.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> With this proposal, transactions will also be able to
> >>>> span
> >>>>> both
> >>>>>>>>>> classic
> >>>>>>>>>>>>> and Diskless topics.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Queues*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The share group coordination and management is a
> >> side job
> >>>>> that
> >>>>>>>>>> doesn't
> >>>>>>>>>>>>> interfere with the topic itself (leadership,
> >> replicas,
> >>>>> physical
> >>>>>>>>>> storage
> >>>>>>>>>>>> of
> >>>>>>>>>>>>> records, etc.) and non-queue producers and consumers
> >>>>> (Fetch and
> >>>>>>>>>> Produce
> >>>>>>>>>>>>> RPCs, consumer group-related RPCs are not affected.)
> >> We
> >>>>> don't
> >>>>>>> see any
> >>>>>>>>>>>>> reason why we can't make Diskless topics compatible
> >> with
> >>>>> share
> >>>>>>>>>> groups the
> >>>>>>>>>>>>> same way as classic topics are. Even on the code
> >> level,
> >>>> we
> >>>>>>> don't
> >>>>>>>>>> expect
> >>>>>>>>>>>> any
> >>>>>>>>>>>>> serious refactoring: the same reading routines are
> >> used
> >>>>> that
> >>>>>>> are
> >>>>>>>>>> used for
> >>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Should the KIPs be modified to include this or it's
> >> too
> >>>>>>>>>>>>> implementation-focused?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>> Ivan
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> >>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thank you all for your questions and design input
> >> on
> >>>>>>> KIP-1150.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
> >> new
> >>>>>>> design. To
> >>>>>>>>>>>>> summarize
> >>>>>>>>>>>>>> the changes:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1. The design prioritizes integrating with the
> >> existing
> >>>>>>> KIP-405
> >>>>>>>>>> Tiered
> >>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
> >>>>> Diskless
> >>>>>>> topic
> >>>>>>>>>> to be
> >>>>>>>>>>>>>> moved to tiered storage.
> >>>>>>>>>>>>>> This lowers the scalability requirements for the
> >> Batch
> >>>>>>> Coordinator
> >>>>>>>>>>>>>> component, and allows Diskless to compose with
> >> Tiered
> >>>>> Storage
> >>>>>>>>>> plugin
> >>>>>>>>>>>>>> features such as encryption and alternative data
> >>>> formats.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2. Consumer fetches are now served from local
> >> segments,
> >>>>>>> making use
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> indexes, page cache, request purgatory, and
> >> zero-copy
> >>>>>>> functionality
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>> built into classic topics.
> >>>>>>>>>>>>>> However, local segments are now considered cache
> >>>>> elements,
> >>>>>>> do not
> >>>>>>>>>> need
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> be durably stored, and can be built without
> >> contacting
> >>>>> any
> >>>>>>> other
> >>>>>>>>>>>>> replicas.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3. The design has been simplified substantially, by
> >>>>> removing
> >>>>>>> the
> >>>>>>>>>>>> previous
> >>>>>>>>>>>>>> Diskless consume flow, distributed cache
> >> component, and
> >>>>>>> "object
> >>>>>>>>>>>>>> compaction/merging" step.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The design maintains leaderless produces as
> >> enabled by
> >>>>> the
> >>>>>>> Batch
> >>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
> >>>> earlier
> >>>>>>> design,
> >>>>>>>>>> while
> >>>>>>>>>>>>>> being simpler and integrating better into the
> >> existing
> >>>>>>> ecosystem.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
> >> the
> >>>> new
> >>>>>>> design.
> >>>>>>>>>>>>>> Greg Harris
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> >>>>>>> <ju...@confluent.io.invalid>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Jan,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
> >> of
> >>>> all
> >>>>>>> existing
> >>>>>>>>>>>> client
> >>>>>>>>>>>>>>> APIs. Currently, there is no design for
> >> supporting
> >>>> APIs
> >>>>>>> like
> >>>>>>>>>>>>> transactions
> >>>>>>>>>>>>>>> and queues.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> >>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Would it be a good time to ask for the current
> >>>>> status of
> >>>>>>> this
> >>>>>>>>>> KIP?
> >>>>>>>>>>>> I
> >>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
> >>>>> months,
> >>>>>>> the
> >>>>>>>>>> vote got
> >>>>>>>>>>>>>>> vetoed
> >>>>>>>>>>>>>>>> but I think the pending questions have been
> >>>> answered
> >>>>>>> since
> >>>>>>>>>> then.
> >>>>>>>>>>>>> KIP-1183
> >>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
> >> activity
> >>>>> since
> >>>>>>> May.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
> >>>> choices
> >>>>>>> that can
> >>>>>>>>>> be
> >>>>>>>>>>>>>>>> made, with a coordinator-based approach being
> >> by
> >>>> far
> >>>>> the
> >>>>>>>>>> dominant
> >>>>>>>>>>>> one
> >>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>> it comes to market adoption - but all these are
> >>>>>>> standalone
> >>>>>>>>>>>> products.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
> >> hate to
> >>>>> see a
> >>>>>>>>>> stall. So
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> question is: can we get an update?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
> >>>> McCabe -
> >>>>>>> have your
> >>>>>>>>>>>>> questions
> >>>>>>>>>>>>>>>> been answered? If not, is there anything I can
> >> do
> >>>> to
> >>>>>>> help? I'm
> >>>>>>>>>>>> deeply
> >>>>>>>>>>>>>>>> familiar with both architectures and have
> >> written
> >>>>> about
> >>>>>>> both?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>> Jan
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> >>>> Kozlovski
> >>>>> <
> >>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I have some nits - it may be useful to
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
> >> main
> >>>> one
> >>>>>>> (just a
> >>>>>>>>>> bunch
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>> links
> >>>>>>>>>>>>>>>>> to everything)
> >>>>>>>>>>>>>>>>> b) create the email threads
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
> >> example, I
> >>>>> was
> >>>>>>>>>> searching
> >>>>>>>>>>>> for
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
> >> far
> >>>> as
> >>>>> I
> >>>>>>> can
> >>>>>>>>>> tell, it
> >>>>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>> exist yet.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
> >> having
> >>>>> the
> >>>>>>> root
> >>>>>>>>>> KIP be
> >>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
> >> to
> >>>>>>> sub-KIPs
> >>>>>>>>>> where
> >>>>>>>>>>>> were
> >>>>>>>>>>>>>>> aimed
> >>>>>>>>>>>>>>>>> to move the discussion towards), I think it
> >> would
> >>>>> be
> >>>>>>> good to
> >>>>>>>>>>>> create
> >>>>>>>>>>>>>>>> DISCUSS
> >>>>>>>>>>>>>>>>> threads for them all.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>> Stan
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
> >>>>>>>>>>>>>>>>>> Hi Kafka Devs!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
> >>>>>>> introducing a
> >>>>>>>>>> new
> >>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>> topics that would make use of Object
> >> Storage as
> >>>>> the
> >>>>>>> primary
> >>>>>>>>>>>>> source of
> >>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
> >> decided
> >>>>> to
> >>>>>>> split it
> >>>>>>>>>>>> into
> >>>>>>>>>>>>>>>> multiple
> >>>>>>>>>>>>>>>>>> related KIPs.
> >>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> >>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
> >> should aim
> >>>>> to
> >>>>>>> have
> >>>>>>>>>> this
> >>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
> >>>> details
> >>>>> on
> >>>>>>> how to
> >>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>> This follows the same approach used when we
> >>>>> discussed
> >>>>>>>>>> KRaft.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
> >> hard
> >>>>> to
> >>>>>>> discuss
> >>>>>>>>>> on
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> meta
> >>>>>>>>>>>>>>>>>> level, we also created several sub-kips
> >> (linked
> >>>>> in
> >>>>>>>>>> KIP-1150)
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>> offer
> >>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>> implementation of this feature.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
> >>>>> threads
> >>>>>>> for
> >>>>>>>>>> each
> >>>>>>>>>>>>> type of
> >>>>>>>>>>>>>>>>>> concern and keep this one to discuss
> >> whether
> >>>>> Apache
> >>>>>>> Kafka
> >>>>>>>>>> wants
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>> this feature or not.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
> >> authors
> >>>> of
> >>>>>>> this KIP.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ------------------
> >>>>>>>>>>>>>>>>>> Josep Prat
> >>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
> >>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
> >>>>> aiven.io
> >>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
> >>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
> >>>>> Valtonen,
> >>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
> >>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Andrew Schofield <an...@live.com>]

Hi,
I’ve been following KIP-1150 and friends for a while. I’m going to jump into the discussions too.

Looking back at Jack Vanlightly’s message, I am not quite so convinced that it’s a kind of fork in the road. The primary aim of the effort is to reduce cross-zone replication costs so Apache Kafka is not prohibitively expensive to use on cloud storage. I think it would be entirely viable to prioritise code reuse for an initial implementation of diskless topics, and we could still have a more cloud-native design in the future. It’s hard to predict what the community will prioritise in the future.

Of the three major revisions, I’m in the rev3 camp. We can support leaderless produce requests, first writing WAL segments into object storage, and then using the regular partition leaders to sequence the records. The active log segment for a diskless topic will initially contain batch coordinates rather than record batches. The batch coordinates can be resolved from WAL segments for consumers, and also in order to prepare log segments for uploading to tiered storage. Jun is probably correct that we need a more frequent object merging process than tiered storage provides. This is just the transition from write-optimised WAL segments to read-optimised tiered segments, and all of the object storage-based implementations of Kafka that I’m aware of do this rearrangement. But perhaps this more frequent object merging is a pre-GA improvement, rather than a strict requirement for an initial implementation for early access use.

For zone-aligned share consumers, the share group assignor is intended to be rack-aware. Consumers should be assigned to partitions with leaders in their zone. The simple assignor is not rack-aware, but it easily could be or we could have a rack-aware assignor.

Thanks,
Andrew


> On 24 Oct 2025, at 23:14, Jun Rao <ju...@confluent.io.INVALID> wrote:
>
> Hi, Ivan,
>
> Thanks for the reply.
>
> "As I understand, you’re speaking about locally materialized segments. They
> will indeed consume some IOPS. See them as a cache that could always be
> restored from the remote storage. While it’s not ideal, it's still OK to
> lose data in them due to a machine crash, for example. Because of this, we
> can avoid explicit flushing on local materialized segments at all and let
> the file system and page cache figure out when to flush optimally. This
> would not eliminate the extra IOPS, but should reduce it dramatically,
> depending on throughput for each partition. We, of course, continue
> flushing the metadata segments as before."
>
> If we have a mix of classic and diskless topics on the same broker, it's
> important that the classic topics' data is flushed to disk as quickly as
> possible. To achieve this, users typically set dirty_expire_centisecs in
> the kernel based on the number of available disk IOPS. Once you set this
> number, it applies to all dirty files, including the cached data in
> diskless topics. So, if there are more files actively accumulating data,
> the flush frequency and therefore RPO is reduced for classic topics.
>
> "We should have mentioned this explicitly, but this step, in fact, remains
> in the form of segments offloading to tiered storage. When we assemble a
> segment and hand it over to RemoteLogManager, we’re effectively doing
> metadata compaction: replacing a big number of pieces of metadata about
> individual batches with a single record in __remote_log_metadata."
>
> The object merging in tier storage typically only kicks in after a few
> hours. The impact is (1) the amount of accumulated metadata is still quite
> large; (2) there are many small objects, leading to poor read performance.
> I think we need a more frequent object merging process than tier storage
> provides.
>
> Jun
>
>
> On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>
>> Hello Jack, Jun, Luke, and all!
>>
>> Thank you for your messages.
>>
>> Let me first address some of Jun’s comments.
>>
>>> First, it degrades the durability.
>>> For each partition, now there are two files being actively written at a
>>> given point of time, one for the data and another for the metadata.
>>> Flushing each file requires a separate IO. If the disk has 1K IOPS and we
>>> have 5K partitions in a broker, currently we can afford to flush each
>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we double
>> the
>>> number of files per partition, we can only flush each partition every 10
>>> seconds, which makes RPO twice as bad.
>>
>> As I understand, you’re speaking about locally materialized segments. They
>> will indeed consume some IOPS. See them as a cache that could always be
>> restored from the remote storage. While it’s not ideal, it's still OK to
>> lose data in them due to a machine crash, for example. Because of this, we
>> can avoid explicit flushing on local materialized segments at all and let
>> the file system and page cache figure out when to flush optimally. This
>> would not eliminate the extra IOPS, but should reduce it dramatically,
>> depending on throughput for each partition. We, of course, continue
>> flushing the metadata segments as before.
>>
>> It’s worth making a note on caching. I think nobody will disagree that
>> doing direct reads from remote storage every time a batch is requested by a
>> consumer will not be practical neither from the performance nor from the
>> economy point of view. We need a way to keep the number of GET requests
>> down. There are multiple options, for example:
>>  1. Rack-aware distributed in-memory caching.
>>  2. Local in-memory caching. Comes with less network chattiness and works
>> well if we have more or less stable brokers to consume from.
>>  3. Materialization of diskless logs on local disk. Way lower impact on
>> RAM and also requires stable brokers for consumption (using just assigned
>> replicas will probably work well).
>>
>> Materialization is one of possible options, but we can choose another one.
>> However, we will have this dilemma regardless of whether we have an
>> explicit coordinator or we go “coordinator-less”.
>>
>>> Second, if we ever need this
>>> metadata somewhere else, say in the WAL file manager, the consumer needs
>> to
>>> subscribe to every partition in the cluster, which is inefficient. The
>>> actual benefit of this approach is also questionable. On the surface, it
>>> might seem that we could reduce the number of lines that need to be
>> changed
>>> for this KIP. However, the changes are quite intrusive to the classic
>>> partition's code path and will probably make the code base harder to
>>> maintain in the long run. I like the original approach based on the batch
>>> coordinator much better than this one. We could probably refactor the
>>> producer state code so that it could be reused in the batch coordinator.
>>
>> It’s hard to disagree with this. The explicit coordinator is more a side
>> thing, while coordinator-less approach is more about extending
>> ReplicaManager, UnifiedLog and others substantially.
>>
>>> Thanks for addressing the concerns on the number of RPCs in the produce
>>> path. I agree that with the metadata crafting mechanism, we could
>> mitigate
>>> the PRC problem. However, since we now require the metadata to be
>>> collocated with the data on the same set of brokers, it's weird that they
>>> are now managed by different mechanisms. The data assignment now uses the
>>> metadata crafting mechanism, but the metadata is stored in the classic
>>> partition using its own assignment strategy. It will be complicated to
>> keep
>>> them collocated.
>>
>> I would like to note that the metadata crafting is needed only to tell
>> producers which brokers they should send Produce requests to, but data (as
>> in “locally materialized log”) is located on partition replicas, i.e.
>> automatically co-located with metadata.
>>
>> As a side note, it would probably be better that instead of implicitly
>> crafting partition metadata, we extend the metadata protocol so that for
>> diskless partitions we return not only the leader and replicas, but also
>> some “recommended produce brokers”, selected for optimal performance and
>> costs. Producers will pick ones in their racks.
>>
>>> I am also concerned about the removal of the object compaction/merging
>>> step.
>>
>> We should have mentioned this explicitly, but this step, in fact, remains
>> in the form of segments offloading to tiered storage. When we assemble a
>> segment and hand it over to RemoteLogManager, we’re effectively doing
>> metadata compaction: replacing a big number of pieces of metadata about
>> individual batches with a single record in __remote_log_metadata.
>>
>> We could create a Diskless-specific merging mechanism instead if needed.
>> It’s rather easy with the explicit coordinator approach. With the
>> coordinator-less approach, this would probably be a bit more tricky
>> (rewriting the tail of the log by the leader + replicating this change
>> reliably).
>>
>>> I see a tendency toward primarily optimizing for the fewest code changes
>> in
>>> the KIP. Instead, our primary goal should be a clean design that can last
>>> for the long term.
>>
>> Yes, totally agree.
>>
>>
>>
>> Luke,
>>> I'm wondering if the complexity of designing txn and queue is because of
>>> leaderless cluster, do you think it will be simpler if we only focus on
>> the
>>> "diskless" design to handle object compaction/merging to/from the remote
>>> storage to save the cross-AZ cost?
>>
>> After some evolution of the original proposal, leaderless is now limited.
>> We only need to be able to accept Produce requests on more than one broker
>> to eliminate the cross-AZ costs for producers. Do I get it right that you
>> propose to get rid of this? Or do I misunderstand?
>>
>>
>>
>> Let’s now look at this problem from a higher level, as Jack proposed. As
>> it was said, the big choice we need to make is whether we 1) create an
>> explicit batch coordinator; or 2) go for the coordinator-less approach,
>> where each diskless partition is managed by its leader as in classic topics.
>>
>> If we try to compare the two approaches:
>>
>> Pluggability:
>> - Explicit coordinator: Possible. For example, some setups may benefit
>> from batch metadata being stored in a cloud database (such as AWS DynamoDB
>> or GCP Spanner).
>> - Coordinator-less: Impossible.
>>
>> Scalability and fault tolerance:
>> - Explicit coordinator: Depends on the implementation and it’s also
>> necessary to actively work for it.
>> - Coordinator-less: Closer to classic Kafka topics. Scaling is done by
>> partition placement, partitions could fail independently.
>>
>> Separation of concerns:
>> - Explicit coordinator: Very good. Diskless remains more independent from
>> classic topics in terms of code and workflows. For example, the
>> above-mentioned non-tiered storage metadata compaction mechanism could be
>> relatively simply implemented with it. As a flip side of this, some
>> workflows (e.g. transactions) will have to be adapted.
>> - Coordinator-less: Less so. It leans to the opposite: bringing diskless
>> closer to classic topics. Some code paths and workflows could be more
>> straightforwardly reused, but they will inevitably have to be adapted to
>> accommodate both topic types as also discussed.
>>
>> Cloud-nativeness. This is a vague concept, also related to the previous,
>> but let’s try:
>> - Explicit coordinator: Storing and processing metadata separately makes
>> it easier for brokers to take different roles, be purely stateless if
>> needed, etc.
>> - Coordinator-less: Less so. Something could be achieved with creative
>> partition placement, but not much.
>>
>> Both seem to have their pros and cons. However, answering Jack’s question,
>> the explicit coordinator approach may indeed lead to a more flexible design.
>>
>>
>> The purpose of this deviation in the discussion was to receive a
>> preliminary community evaluation of the coordinator-less approach without
>> taking on the task of writing a separate KIP and fitting it in the system
>> of KIP-1150 and its children. We’re open to stopping it and getting back to
>> working out the coordinator design if the community doesn’t favor the
>> proposed approach.
>>
>> Best,
>> Ivan and Diskless team
>>
>> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
>>> Hi Ivan,
>>>
>>> As Jun pointed out, the updated design seems to have some shortcomings
>>> although it simplifies the implementation.
>>>
>>> I'm wondering if the complexity of designing txn and queue is because of
>>> leaderless cluster, do you think it will be simpler if we only focus on
>> the
>>> "diskless" design to handle object compaction/merging to/from the remote
>>> storage to save the cross-AZ cost?
>>>
>>>
>>> Thank you,
>>> Luke
>>>
>>> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid>
>> wrote:
>>>
>>>> Hi, Ivan,
>>>>
>>>> Thanks for the explanation.
>>>>
>>>> "we write the reference to the WAL file with the batch data"
>>>>
>>>> I understand the approach now, but I think it is a hacky one. There are
>>>> multiple short comings with this design. First, it degrades the
>> durability.
>>>> For each partition, now there are two files being actively written at a
>>>> given point of time, one for the data and another for the metadata.
>>>> Flushing each file requires a separate IO. If the disk has 1K IOPS and
>> we
>>>> have 5K partitions in a broker, currently we can afford to flush each
>>>> partition every 5 seconds, achieving an RPO of 5 seconds. If we double
>> the
>>>> number of files per partition, we can only flush each partition every
>> 10
>>>> seconds, which makes RPO twice as bad. Second, if we ever need this
>>>> metadata somewhere else, say in the WAL file manager, the consumer
>> needs to
>>>> subscribe to every partition in the cluster, which is inefficient. The
>>>> actual benefit of this approach is also questionable. On the surface,
>> it
>>>> might seem that we could reduce the number of lines that need to be
>> changed
>>>> for this KIP. However, the changes are quite intrusive to the classic
>>>> partition's code path and will probably make the code base harder to
>>>> maintain in the long run. I like the original approach based on the
>> batch
>>>> coordinator much better than this one. We could probably refactor the
>>>> producer state code so that it could be reused in the batch
>> coordinator.
>>>>
>>>> Thanks for addressing the concerns on the number of RPCs in the produce
>>>> path. I agree that with the metadata crafting mechanism, we could
>> mitigate
>>>> the PRC problem. However, since we now require the metadata to be
>>>> collocated with the data on the same set of brokers, it's weird that
>> they
>>>> are now managed by different mechanisms. The data assignment now uses
>> the
>>>> metadata crafting mechanism, but the metadata is stored in the classic
>>>> partition using its own assignment strategy. It will be complicated to
>> keep
>>>> them collocated.
>>>>
>>>> I am also concerned about the removal of the object compaction/merging
>>>> step. My first concern is on the amount of metadata that need to be
>> kept.
>>>> Without object compcation, the metadata generated in the produce path
>> can
>>>> only be deleted after remote tiering kicks in. Let's say for every
>> 250ms we
>>>> produce 100 byte of metadata per partition. Let's say remoting tiering
>>>> kicks in after 5 hours. In a cluster with 100K partitions, we need to
>> keep
>>>> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
>>>> signficant. A second concern is on performance. Every time we need to
>>>> rebuild the caching data, we need to read a bunch of small objects
>> from S3,
>>>> slowing down the building process. If a consumer happens to need such
>> data,
>>>> it could slow down the application.
>>>>
>>>> I see a tendency toward primarily optimizing for the fewest code
>> changes in
>>>> the KIP. Instead, our primary goal should be a clean design that can
>> last
>>>> for the long term.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me>
>> wrote:
>>>>
>>>>> Hi Jun,
>>>>>
>>>>> Thank you for your message. I’m sorry that I failed to clearly
>> explain
>>>> the
>>>>> idea. Let me try to fix this.
>>>>>
>>>>>> Does each partition now have a metadata partition and a separate
>> data
>>>>>> partition? If so, I am concerned that it essentially doubles the
>> number
>>>>> of
>>>>>> partitions, which impacts the number of open file descriptors and
>> the
>>>>>> required IOPS, and so on. It also seems wasteful to have a separate
>>>>>> partition just to store the metadata. It's as if we are creating an
>>>>>> internal topic with an unbounded number of partitions.
>>>>>
>>>>> No. There will be only one physical partition per diskless
>> partition. Let
>>>>> me explain this with an example. Let’s say we have a diskless
>> partition
>>>>> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce
>> some
>>>>> batches to this partition. The content of the segment file will be
>>>>> something like this (for each batch):
>>>>>
>>>>> BaseOffset: 00000000000000000000 (like in classic)
>>>>> Length: 123456 (like in classic)
>>>>> PartitionLeaderEpoch: like in classic
>>>>> Magic: like in classic
>>>>> CRC: like in classic
>>>>> Attributes: like in classic
>>>>> LastOffsetDelta: like in classic
>>>>> BaseTimestamp: like in classic
>>>>> MaxTimestamp: like in classic
>>>>> ProducerId: like in classic
>>>>> ProducerEpoch: like in classic
>>>>> BaseSequence: like in classic
>>>>> RecordsCount: like in classic
>>>>> Records:
>>>>>  path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
>>>>> 123456
>>>>>
>>>>> It looks very much like classic log entries. The only difference is
>> that
>>>>> instead of writing real Records, we write the reference to the WAL
>> file
>>>>> with the batch data (I guess we need only the name and the byte
>> offset,
>>>>> because the byte length is the standard field above). Otherwise,
>> it’s a
>>>>> normal Kafka log with the leader and replicas.
>>>>>
>>>>> So we have as many partitions for diskless as for classic. As of open
>>>> file
>>>>> descriptors, let’s proceed to the following:
>>>>>
>>>>>> Are the metadata and
>>>>>> the data for the same partition always collocated on the same
>> broker?
>>>> If
>>>>>> so, how do we enforce that when replicas are reassigned?
>>>>>
>>>>> The source of truth for the data is still in WAL files on object
>> storage.
>>>>> The source of truth for the metadata is in segment files on the
>> brokers
>>>> in
>>>>> the replica set. Two new mechanisms are planned, both independent of
>> this
>>>>> new proposal, but I want to present them to give the idea that only a
>>>>> limited amount of data files will be operated locally:
>>>>> - We want to assemble batches into segment files and offload them to
>>>>> tiered storage in order to prevent the unbounded growth of batch
>>>> metadata.
>>>>> For this, we need to open only  a few file descriptors (for the
>> segment
>>>>> file itself + the necessary indexes) before the segment is fully
>> written
>>>>> and handed over to RemoteLogManager.
>>>>> - We want to assemble local segment files for caching purposes as
>> well,
>>>>> i.e. to speed up fetching. This will not materialize the full
>> content of
>>>>> the log, but only the hot set according to some policy (or
>> configurable
>>>>> policies), i.e. the number of segments and file descriptors will
>> also be
>>>>> limited.
>>>>>
>>>>>> The number of RPCs in the produce path is significantly higher. For
>>>>>> example, if a produce request has 100 partitions, in a cluster
>> with 100
>>>>>> brokers, each produce request could generate 100 more RPC requests.
>>>> This
>>>>>> will significantly increase the request rate.
>>>>>
>>>>> This is a valid concern that we considered, but this issue can be
>>>>> mitigated. I’ll try to explain the approach.
>>>>> The situation with a single broker is trivial: all the commit
>> requests go
>>>>> from the broker to itself.
>>>>> Let’s scale this to a multi-broker cluster, but located in the single
>>>> rack
>>>>> (AZ). Any broker can accept Produce requests for diskless
>> partitions, but
>>>>> we can tell producers (through metadata) to always send Produce
>> requests
>>>> to
>>>>> leaders. For example, broker 0 hosts the leader replicas for diskless
>>>>> partitions t1-0, t2-1, t3-0. It will receive diskless Produce
>> requests
>>>> for
>>>>> these partitions in various combinations, but only for them.
>>>>>
>>>>>                    Broker 0
>>>>>              +-----------------+
>>>>>              |    t1-0         |
>>>>>              |    t2-1 <--------------------+
>>>>>              |    t3-0         |            |
>>>>> produce      | +-------------+ |            |
>>>>> requests     | |  diskless   | |            |
>>>>> --------------->|   produce   +--------------+
>>>>> for these    | | WAL buffer  | |    commit requests
>>>>> partitions   | +-------------+ |    for these partitions
>>>>>              |                 |
>>>>>              +-----------------+
>>>>>
>>>>> The same applies for other brokers in this cluster. Effectively, each
>>>>> broker will commit only to itself, which effectively means 1 commit
>>>> request
>>>>> per WAL buffer (this may be 0 physical network calls, if we wish,
>> just a
>>>>> local function call).
>>>>>
>>>>> Now let’s scale this to multiple racks (AZs). Obviously, we cannot
>> always
>>>>> send Produce requests to the designated leaders of diskless
>> partitions:
>>>>> this would mean inter-AZ network traffic, which we would like to
>> avoid.
>>>> To
>>>>> avoid it, we say that every broker has a “diskless produce
>>>> representative”
>>>>> in every AZ. If we continue our example: when a Produce request for
>> t1-0,
>>>>> t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0
>> (in the
>>>>> broker’s AZ the representative is the broker itself). However, if it
>>>> comes
>>>>> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
>>>>>
>>>>>  |produce requests         |produce requests        |produce
>> requests
>>>>>  |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
>>>> t3-0
>>>>>  |from AZ 0                |from AZ 1               |from AZ 2
>>>>>  v                         v                        v
>>>>> Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
>>>>> +---------------+      +---------------+      +---------------+
>>>>> |     t1-0      |      |               |      |               |
>>>>> |     t2-1      |      |               |      |               |
>>>>> |     t3-0      |      |               |      |               |
>>>>> +---------------+      +--------+------+      +--------+------+
>>>>>     ^     ^                    |                      |
>>>>>     |     +--------------------+                      |
>>>>>     |     commit requests for these partitions        |
>>>>>     |                                                 |
>>>>>     +-------------------------------------------------+
>>>>>           commit requests for these partitions
>>>>>
>>>>> All the partitions that broker 0 is the leader of will be
>> “represented”
>>>> by
>>>>> brokers 1 and 2 in their AZs.
>>>>>
>>>>> Of course, this relationship goes both ways between AZs (not
>> necessarily
>>>>> between the same brokers). It means that provided the cluster is
>> balanced
>>>>> by the number of brokers per AZ, each broker will represent
>>>> (number_of_azs
>>>>> - 1) other brokers. This will result in the situation that for the
>>>> majority
>>>>> of commits, each broker will do up to (number_of_azs - 1) network
>> commit
>>>>> requests (plus one local). Cloud regions tend to have 3 AZs, very
>> rarely
>>>>> more. That means, brokers will be doing up to 2 network commit
>> requests
>>>> per
>>>>> WAL file.
>>>>>
>>>>> There are the following exceptions:
>>>>> 1. Broker count imbalance between AZs. For example, when we have 2
>> AZs
>>>> and
>>>>> one has three brokers and another AZ has one. This one broker will do
>>>>> between 1 and 3 commit requests per WAL file. This is not an extreme
>>>>> amplification. Such an imbalance is not healthy in most practical
>> setups
>>>>> and should be avoided anyway.
>>>>> 2. Leadership changes and metadata propagation period. When the
>> partition
>>>>> t3-0 is relocated from broker 0 to some broker 3, the producers will
>> not
>>>>> know this immediately (unless we want to be strict and respond with
>>>>> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
>> together
>>>> in a
>>>>> WAL buffer on broker 2, it will have to send two commit requests: to
>>>> broker
>>>>> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
>> situation
>>>>> is not permanent and as producers update the cluster metadata, it
>> will be
>>>>> resolved.
>>>>>
>>>>> This all could be built with the metadata crafting mechanism only
>> (which
>>>>> is anyway needed for Diskless in one way or another to direct
>> producers
>>>> and
>>>>> consumers where we need to avoid inter-AZ traffic), just with the
>> right
>>>>> policy for it (for example, some deterministic hash-based formula).
>> I.e.
>>>> no
>>>>> explicit support for “produce representative” or anything like this
>> is
>>>>> needed on the cluster level, in KRaft, etc.
>>>>>
>>>>>> The same WAL file metadata is now duplicated into two places,
>> partition
>>>>>> leader and WAL File Manager. Which one is the source of truth, and
>> how
>>>> do
>>>>>> we maintain consistency between the two places?
>>>>>
>>>>> We do only two operations on WAL files that span multiple diskless
>>>>> partitions: committing and deleting. Commits can be done
>> independently as
>>>>> described above. But deletes are different, because when a file is
>>>> deleted,
>>>>> this affects all the partitions that still have alive batches in this
>>>> file
>>>>> (if any).
>>>>>
>>>>> The WAL file manager is a necessary point of coordination to delete
>> WAL
>>>>> files safely. We can say it is the source of truth about files
>>>> themselves,
>>>>> while the partition leaders and their logs hold the truth about
>> whether a
>>>>> particular file contains live batches of this particular partition.
>>>>>
>>>>> The file manager will do this important task: be able to say for sure
>>>> that
>>>>> a file does not contain any live batch of any existing partition. For
>>>> this,
>>>>> it will have to periodically check against the partition leaders.
>>>>> Considering that batch deletion is irreversible, when we declare a
>> file
>>>>> “empty”, this is guaranteed to be and stay so.
>>>>>
>>>>> The file manager has to know about files being committed to start
>> track
>>>>> them and periodically check if they are empty. We can consider
>> various
>>>> ways
>>>>> to achieve this:
>>>>> 1. As was proposed in my previous message: best effort commit by
>> brokers
>>>> +
>>>>> periodic prefix scans of object storage to detect files that went
>> below
>>>> the
>>>>> radar due to network issue or the file manager temporary
>> unavailability.
>>>>> We’re speaking about listing the file names only and opening only
>>>>> previously unknown files in order to find the partitions involved
>> with
>>>> them.
>>>>> 2. Only do scans without explicit commit, i.e. fill the list of files
>>>>> fully asynchronously and in the background. This may be not ideal
>> due to
>>>>> costs and performance of scanning tons of files. However, the number
>> of
>>>>> live WAL files should be limited due to tiered storage offloading +
>> we
>>>> can
>>>>> optimize this if we give files some global soft order in their names.
>>>>>
>>>>>> I am not sure how this design simplifies the implementation. The
>>>> existing
>>>>>> producer/replication code can't be simply reused. Adjusting both
>> the
>>>>> write
>>>>>> path in the leader and the replication path in the follower to
>>>> understand
>>>>>> batch-header only data is quite intrusive to the existing logic.
>>>>>
>>>>> It is true that we’ll have to change LocalLog and UnifiedLog in
>> order to
>>>>> support these changes. However, it seems that idempotence,
>> transactions,
>>>>> queues, tiered storage will have to be changed less than with the
>>>> original
>>>>> design. This is because the partition leader state would remain in
>> the
>>>> same
>>>>> place (on brokers) and existing workflows that involve it would have
>> to
>>>> be
>>>>> changed less compared to the situation where we globalize the
>> partition
>>>>> leader state in the batch coordinator. I admit this is hard to make
>>>>> convincing without both real implementations to hand :)
>>>>>
>>>>>> I am also
>>>>>> not sure how this enables seamless switching the topic modes
>> between
>>>>>> diskless and classic. Could you provide more details on those?
>>>>>
>>>>> Let’s consider the scenario of turning a classic topic into
>> diskless. The
>>>>> user sets diskless.enabled=true, the leader receives this metadata
>> update
>>>>> and does the following:
>>>>> 1. Stop accepting normal append writes.
>>>>> 2. Close the current active segment.
>>>>> 3. Start a new segment that will be written in the diskless format
>> (i.e.
>>>>> without data).
>>>>> 4. Start accepting diskless commits.
>>>>>
>>>>> Since it’s the same log, the followers will know about that switch
>>>>> consistently. They will finish replicating the classic segments and
>> start
>>>>> replicating the diskless ones. They will always know where each
>> batch is
>>>>> located (either inside a classic segment or referenced by a diskless
>>>> one).
>>>>> Switching back should be similar.
>>>>>
>>>>> Doing this with the coordinator is possible, but has some caveats.
>> The
>>>>> leader must do the following:
>>>>> 1. Stop accepting normal append writes.
>>>>> 2. Close the current active segment.
>>>>> 3. Write a special control segment to persist and replicate the fact
>> that
>>>>> from offset N the partition is now in the diskless mode.
>>>>> 4. Inform the coordinator about the first offset N of the “diskless
>> era”.
>>>>> 5. Inform the controller quorum that the transition has finished and
>> that
>>>>> brokers now can process diskless writes for this partition.
>>>>> This could fail at some points, so this will probably require some
>>>>> explicit state machine with replication either in the partition log
>> or in
>>>>> KRaft.
>>>>>
>>>>> It seems that the coordinator-less approach makes this simpler
>> because
>>>> the
>>>>> “coordinator” for the partition and the partition leader are the
>> same and
>>>>> they store the partition metadata in the same log, too. While in the
>>>>> coordinator approach we have to perform some kind of a distributed
>> commit
>>>>> to handover metadata management from the classic partition leader to
>> the
>>>>> batch coordinator.
>>>>>
>>>>> I hope these explanations help to clarify the idea. Please let me
>> know if
>>>>> I should go deeper anywhere.
>>>>>
>>>>> Best,
>>>>> Ivan and the Diskless team
>>>>>
>>>>> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
>>>>>> Hi, Ivan,
>>>>>>
>>>>>> Thanks for the update.
>>>>>>
>>>>>> I am not sure that I fully understand the new design, but it seems
>> less
>>>>>> clean than before.
>>>>>>
>>>>>> Does each partition now have a metadata partition and a separate
>> data
>>>>>> partition? If so, I am concerned that it essentially doubles the
>> number
>>>>> of
>>>>>> partitions, which impacts the number of open file descriptors and
>> the
>>>>>> required IOPS, and so on. It also seems wasteful to have a separate
>>>>>> partition just to store the metadata. It's as if we are creating an
>>>>>> internal topic with an unbounded number of partitions. Are the
>> metadata
>>>>> and
>>>>>> the data for the same partition always collocated on the same
>> broker?
>>>> If
>>>>>> so, how do we enforce that when replicas are reassigned?
>>>>>>
>>>>>> The number of RPCs in the produce path is significantly higher. For
>>>>>> example, if a produce request has 100 partitions, in a cluster
>> with 100
>>>>>> brokers, each produce request could generate 100 more RPC requests.
>>>> This
>>>>>> will significantly increase the request rate.
>>>>>>
>>>>>> The same WAL file metadata is now duplicated into two places,
>> partition
>>>>>> leader and WAL File Manager. Which one is the source of truth, and
>> how
>>>> do
>>>>>> we maintain consistency between the two places?
>>>>>>
>>>>>> I am not sure how this design simplifies the implementation. The
>>>> existing
>>>>>> producer/replication code can't be simply reused. Adjusting both
>> the
>>>>> write
>>>>>> path in the leader and the replication path in the follower to
>>>> understand
>>>>>> batch-header only data is quite intrusive to the existing logic. I
>> am
>>>>> also
>>>>>> not sure how this enables seamless switching the topic modes
>> between
>>>>>> diskless and classic. Could you provide more details on those?
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me>
>> wrote:
>>>>>>
>>>>>>> Hi dear Kafka community,
>>>>>>>
>>>>>>> In the initial Diskless proposal, we proposed to have a separate
>>>>>>> component, batch/diskless coordinator, whose role would be to
>>>> centrally
>>>>>>> manage the batch and WAL file metadata for diskless topics. This
>>>>> component
>>>>>>> drew many reasonable comments from the community about how it
>> would
>>>>> support
>>>>>>> various Kafka features (transactions, queues) and its
>> scalability.
>>>>> While we
>>>>>>> believe we have good answers to all the expressed concerns, we
>> took a
>>>>> step
>>>>>>> back and looked at the problem from a different perspective.
>>>>>>>
>>>>>>> We would like to propose an alternative Diskless design *without
>> a
>>>>>>> centralized coordinator*. We believe this approach has potential
>> and
>>>>>>> propose to discuss it as it may be more appealing to the
>> community.
>>>>>>>
>>>>>>> Let us explain the idea. Most of the complications with the
>> original
>>>>>>> Diskless approach come from one necessary architecture change:
>>>>> globalizing
>>>>>>> the local state of partition leader in the batch coordinator.
>> This
>>>>> causes
>>>>>>> deviations to the established workflows in various features like
>>>>> produce
>>>>>>> idempotence and transactions, queues, retention, etc. These
>>>> deviations
>>>>> need
>>>>>>> to be carefully considered, designed, and later implemented and
>>>>> tested. In
>>>>>>> the new approach we want to avoid this by making partition
>> leaders
>>>>> again
>>>>>>> responsible for managing their partitions, even in diskless
>> topics.
>>>>>>>
>>>>>>> In classic Kafka topics, batch data and metadata are blended
>> together
>>>>> in
>>>>>>> the one partition log. The crux of the Diskless idea is to
>> decouple
>>>>> them
>>>>>>> and move data to the remote storage, while keeping metadata
>> somewhere
>>>>> else.
>>>>>>> Using the central batch coordinator for managing batch metadata
>> is
>>>> one
>>>>> way,
>>>>>>> but not the only.
>>>>>>>
>>>>>>> Let’s now think about managing metadata for each user partition
>>>>>>> independently. Generally partitions are independent and don’t
>> share
>>>>>>> anything apart from that their data are mixed in WAL files. If we
>>>>> figure
>>>>>>> out how to commit and later delete WAL files safely, we will
>> achieve
>>>>> the
>>>>>>> necessary autonomy that allows us to get rid of the central batch
>>>>>>> coordinator. Instead, *each diskless user partition will be
>> managed
>>>> by
>>>>> its
>>>>>>> leader*, as in classic Kafka topics. Also like in classic
>> topics, the
>>>>>>> leader uses the partition log as the way to persist batch
>> metadata,
>>>>> i.e.
>>>>>>> the regular batch header + the information about how to find this
>>>>> batch on
>>>>>>> remote storage. In contrast to classic topics, batch data is in
>>>> remote
>>>>>>> storage.
>>>>>>>
>>>>>>> For clarity, let’s compare the three designs:
>>>>>>> • Classic topics:
>>>>>>>   • Data and metadata are co-located in the partition log.
>>>>>>>   • The partition log content: [Batch header (metadata)|Batch
>> data].
>>>>>>>   • The partition log is replicated to the followers.
>>>>>>>   • The replicas and leader have local state built from
>> metadata.
>>>>>>> • Original Diskless:
>>>>>>>   • Metadata is in the batch coordinator, data is on remote
>> storage.
>>>>>>>   • The partition state is global in the batch coordinator.
>>>>>>> • New Diskless:
>>>>>>>   • Metadata is in the partition log, data is on remote storage.
>>>>>>>   • Partition log content: [Batch header (metadata)|Batch
>>>> coordinates
>>>>> on
>>>>>>> remote storage].
>>>>>>>   • The partition log is replicated to the followers.
>>>>>>>   • The replicas and leader have local state built from
>> metadata.
>>>>>>>
>>>>>>> Let’s consider the produce path. Here’s the reminder of the
>> original
>>>>>>> Diskless design:
>>>>>>>
>>>>>>>
>>>>>>> The new approach could be depicted as the following:
>>>>>>>
>>>>>>>
>>>>>>> As you can see, the main difference is that now instead of a
>> single
>>>>> commit
>>>>>>> request to the batch coordinator, we send multiple parallel
>> commit
>>>>> requests
>>>>>>> to all the leaders of each partition involved in the WAL file.
>> Each
>>>> of
>>>>> them
>>>>>>> will commit its batches independently, without coordinating with
>>>> other
>>>>>>> leaders and any other components. Batch data is addressed by the
>> WAL
>>>>> file
>>>>>>> name, the byte offset and size, which allows partitions to know
>>>> nothing
>>>>>>> about other partitions to access their data in shared WAL files.
>>>>>>>
>>>>>>> The number of partitions involved in a single WAL file may be
>> quite
>>>>> large,
>>>>>>> e.g. a hundred. A hundred network requests to commit one WAL
>> file is
>>>>> very
>>>>>>> impractical. However, there are ways to reduce this number:
>>>>>>> 1. Partition leaders are located on brokers. Requests to
>> leaders on
>>>>> one
>>>>>>> broker could be grouped together into a single physical network
>>>> request
>>>>>>> (resembling the normal Produce request that may carry batches for
>>>> many
>>>>>>> partitions inside). This will cap the number of network requests
>> to
>>>> the
>>>>>>> number of brokers in the cluster.
>>>>>>> 2. If we craft the cluster metadata to make producers send their
>>>>> requests
>>>>>>> to the right brokers (with respect to AZs), we may achieve the
>> higher
>>>>>>> concentration of logical commit requests in physical network
>> requests
>>>>>>> reducing the number of the latter ones even further, ideally to
>> one.
>>>>>>>
>>>>>>> Obviously, out of multiple commit requests some may fail or time
>> out
>>>>> for a
>>>>>>> variety of reasons. This is fine. Some producers will receive
>> totally
>>>>> or
>>>>>>> partially failed responses to their Produce requests, similar to
>> what
>>>>> they
>>>>>>> would have received when appending to a classic topic fails or
>> times
>>>>> out.
>>>>>>> If a partition experiences problems, other partitions will not be
>>>>> affected
>>>>>>> (again, like in classic topics). Of course, the uncommitted data
>> will
>>>>> be
>>>>>>> garbage in WAL files. But WAL files are short-lived (batches are
>>>>> constantly
>>>>>>> assembled into segments and offloaded to tiered storage), so this
>>>>> garbage
>>>>>>> will be eventually deleted.
>>>>>>>
>>>>>>> For safely deleting WAL files we now need to centrally manage
>> them,
>>>> as
>>>>>>> this is the only state and logic that spans multiple partitions.
>> On
>>>> the
>>>>>>> diagram, you can see another commit request called “Commit file
>> (best
>>>>>>> effort)” going to the WAL File Manager. This manager will be
>>>>> responsible
>>>>>>> for the following:
>>>>>>> 1. Collecting (by requests from brokers) and persisting
>> information
>>>>> about
>>>>>>> committed WAL files.
>>>>>>> 2. To handle potential failures in file information delivery, it
>>>> will
>>>>> be
>>>>>>> doing prefix scan on the remote storage periodically to find and
>>>>> register
>>>>>>> unknown files. The period of this scan will be configurable and
>>>> ideally
>>>>>>> should be quite long.
>>>>>>> 3. Checking with the relevant partition leaders (after a grace
>>>>> period) if
>>>>>>> they still have batches in a particular file.
>>>>>>> 4. Physically deleting files when they aren’t anymore referred
>> to by
>>>>> any
>>>>>>> partition.
>>>>>>>
>>>>>>> This new design offers the following advantages:
>>>>>>> 1. It simplifies the implementation of many Kafka features such
>> as
>>>>>>> idempotence, transactions, queues, tiered storage, retention.
>> Now we
>>>>> don’t
>>>>>>> need to abstract away and reuse the code from partition leaders
>> in
>>>> the
>>>>>>> batch coordinator. Instead, we will literally use the same code
>> paths
>>>>> in
>>>>>>> leaders, with little adaptation. Workflows from classic topics
>> mostly
>>>>>>> remain unchanged.
>>>>>>> For example, it seems that
>>>>>>> ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
>>>>>>> KafkaApis.handleWriteTxnMarkersRequest used for transaction
>> support
>>>> on
>>>>> the
>>>>>>> partition leader side could be used for diskless topics with
>> little
>>>>>>> adaptation. ProducerStateManager, needed for both idempotent
>> produce
>>>>> and
>>>>>>> transactions, would be reused.
>>>>>>> Another example is share groups support, where the share
>> partition
>>>>> leader,
>>>>>>> being co-located with the partition leader, would execute the
>> same
>>>>> logic
>>>>>>> for both diskless and classic topics.
>>>>>>> 2. It returns to the familiar partition-based scaling model,
>> where
>>>>>>> partitions are independent.
>>>>>>> 3. It makes the operation and failure patterns closer to the
>>>> familiar
>>>>>>> ones from classic topics.
>>>>>>> 4. It opens a straightforward path to seamless switching the
>> topics
>>>>> modes
>>>>>>> between diskless and classic.
>>>>>>>
>>>>>>> The rest of the things remain unchanged compared to the previous
>>>>> Diskless
>>>>>>> design (after all previous discussions). Such things as local
>> segment
>>>>>>> materialization by replicas, the consume path, tiered storage
>>>>> integration,
>>>>>>> etc.
>>>>>>>
>>>>>>> If the community finds this design more suitable, we will update
>> the
>>>>>>> KIP(s) accordingly and continue working on it. Please let us know
>>>> what
>>>>> you
>>>>>>> think.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Ivan and Diskless team
>>>>>>>
>>>>>>> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
>>>>>>>> Hi Justine,
>>>>>>>>
>>>>>>>> Yes, you're right. We need to track the aborted transactions
>> for in
>>>>> the
>>>>>>> diskless coordinator for as long as the corresponding offsets are
>>>>> there.
>>>>>>> With the tiered storage unification Greg mentioned earlier, this
>> will
>>>>> be
>>>>>>> finite time even for infinite data retention.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Ivan
>>>>>>>>
>>>>>>>> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
>>>>>>>>> Hey Ivan,
>>>>>>>>>
>>>>>>>>> Thanks for the response. I think most of what you said made
>>>> sense,
>>>>> but
>>>>>>> I
>>>>>>>>> did have some questions about this part:
>>>>>>>>>
>>>>>>>>>> As we understand this, the partition leader in classic
>> topics
>>>>> forgets
>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
>> it). The
>>>>>>>>> transaction coordinator acts like the main guardian, allowing
>>>>> partition
>>>>>>>>> leaders to do this safely. Please correct me if this is
>> wrong. We
>>>>> think
>>>>>>>>> about relying on this with the batch coordinator and delete
>> the
>>>>>>> information
>>>>>>>>> about a transaction once it’s finished (as there’s no
>> replication
>>>>> and
>>>>>>> HWM
>>>>>>>>> advances immediately).
>>>>>>>>>
>>>>>>>>> I didn't quite understand this. In classic topics, we have
>> maps
>>>> for
>>>>>>> ongoing
>>>>>>>>> transactions which remove state when the transaction is
>> completed
>>>>> and
>>>>>>> an
>>>>>>>>> aborted transactions index which is retained for much longer.
>>>> Once
>>>>> the
>>>>>>>>> transaction is completed, the coordinator is no longer
>> involved
>>>> in
>>>>>>>>> maintaining this partition side state, and it is subject to
>>>>> compaction
>>>>>>> etc.
>>>>>>>>> Looking back at the outline provided above, I didn't see much
>>>>> about the
>>>>>>>>> fetch path, so maybe that could be expanded a bit further. I
>> saw
>>>>> the
>>>>>>>>> following in a response:
>>>>>>>>>> When the broker constructs a fully valid local segment,
>> all the
>>>>>>> necessary
>>>>>>>>> control batches will be inserted and indices, including the
>>>>> transaction
>>>>>>>>> index will be built to serve FetchRequests exactly as they
>> are
>>>>> today.
>>>>>>>>>
>>>>>>>>> Based on this, it seems like we need to retain the
>> information
>>>>> about
>>>>>>>>> aborted txns for longer.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Justine
>>>>>>>>>
>>>>>>>>> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
>> ivan@ivanyu.me>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Justine and all,
>>>>>>>>>>
>>>>>>>>>> Thank you for your questions!
>>>>>>>>>>
>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
>> with
>>>>>>> producer ID
>>>>>>>>>>> and epoch, the positive result of this check could be
>> cached
>>>>>>> locally
>>>>>>>>>>> Are we saying that only new transaction version 2
>>>> transactions
>>>>> can
>>>>>>> be
>>>>>>>>>> used
>>>>>>>>>>> here? If not, we can't uniquely identify transactions
>> with
>>>>>>> producer id +
>>>>>>>>>>> epoch
>>>>>>>>>>
>>>>>>>>>> You’re right that we (probably unintentionally) focused
>> only on
>>>>>>> version 2.
>>>>>>>>>> We can either limit the support to version 2 or consider
>> using
>>>>> some
>>>>>>>>>> surrogates to support version 1.
>>>>>>>>>>
>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
>>>>> checks
>>>>>>> of the
>>>>>>>>>>> batches. This procedure would output the same errors
>> like the
>>>>>>> partition
>>>>>>>>>>> leader in classic topics would do.
>>>>>>>>>>> Can you expand on what these checks are? Would you be
>>>> checking
>>>>> if
>>>>>>> the
>>>>>>>>>>> transaction was still ongoing for example?* *
>>>>>>>>>>
>>>>>>>>>> Yes, the producer epoch, that the transaction is ongoing,
>> and
>>>> of
>>>>>>> course
>>>>>>>>>> the normal idempotence checks. What the partition leader
>> in the
>>>>>>> classic
>>>>>>>>>> topics does before appending a batch to the local log
>> (e.g. in
>>>>>>>>>> UnifiedLog.maybeStartTransactionVerification and
>>>>>>>>>> UnifiedLog.analyzeAndValidateProducerState). In Diskless,
>> we
>>>>>>> unfortunately
>>>>>>>>>> cannot do these checks before appending the data to the WAL
>>>>> segment
>>>>>>> and
>>>>>>>>>> uploading it, but we can “tombstone” these batches in the
>> batch
>>>>>>> coordinator
>>>>>>>>>> during the final commit.
>>>>>>>>>>
>>>>>>>>>>> Is there state about ongoing
>>>>>>>>>>> transactions in the batch coordinator? I see some other
>> state
>>>>>>> mentioned
>>>>>>>>>> in
>>>>>>>>>>> the End transaction section, but it's not super clear
>> what
>>>>> state is
>>>>>>>>>> stored
>>>>>>>>>>> and when it is stored.
>>>>>>>>>>
>>>>>>>>>> Right, this should have been more explicit. As the
>> partition
>>>>> leader
>>>>>>> tracks
>>>>>>>>>> ongoing transactions for classic topics, the batch
>> coordinator
>>>>> has
>>>>>>> to as
>>>>>>>>>> well. So when a transaction starts and ends, the
>> transaction
>>>>>>> coordinator
>>>>>>>>>> must inform the batch coordinator about this.
>>>>>>>>>>
>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
>> perhaps
>>>>> that
>>>>>>> would
>>>>>>>>>> be
>>>>>>>>>>> stored in the batch coordinator?
>>>>>>>>>>
>>>>>>>>>> Yes. This could be deduced from the committed batches and
>> other
>>>>>>>>>> information, but for the sake of performance we’d better
>> store
>>>> it
>>>>>>>>>> explicitly.
>>>>>>>>>>
>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
>>>>> state is
>>>>>>>>>>> maintained in the batch coordinator and how it will be
>>>> cleaned
>>>>> up?
>>>>>>>>>>
>>>>>>>>>> As we understand this, the partition leader in classic
>> topics
>>>>> forgets
>>>>>>>>>> about a transaction once it’s replicated (HWM overpasses
>> it).
>>>> The
>>>>>>>>>> transaction coordinator acts like the main guardian,
>> allowing
>>>>>>> partition
>>>>>>>>>> leaders to do this safely. Please correct me if this is
>> wrong.
>>>> We
>>>>>>> think
>>>>>>>>>> about relying on this with the batch coordinator and
>> delete the
>>>>>>> information
>>>>>>>>>> about a transaction once it’s finished (as there’s no
>>>> replication
>>>>>>> and HWM
>>>>>>>>>> advances immediately).
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Ivan
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
>>>>>>>>>>> Hey folks,
>>>>>>>>>>>
>>>>>>>>>>> Excited to see some updates related to transactions!
>>>>>>>>>>>
>>>>>>>>>>> I had a few questions.
>>>>>>>>>>>
>>>>>>>>>>> JO 1. >Since a transaction could be uniquely identified
>> with
>>>>>>> producer ID
>>>>>>>>>>> and epoch, the positive result of this check could be
>> cached
>>>>>>> locally
>>>>>>>>>>> Are we saying that only new transaction version 2
>>>> transactions
>>>>> can
>>>>>>> be
>>>>>>>>>> used
>>>>>>>>>>> here? If not, we can't uniquely identify transactions
>> with
>>>>>>> producer id +
>>>>>>>>>>> epoch
>>>>>>>>>>>
>>>>>>>>>>> JO 2. >The batch coordinator does the final transactional
>>>>> checks
>>>>>>> of the
>>>>>>>>>>> batches. This procedure would output the same errors
>> like the
>>>>>>> partition
>>>>>>>>>>> leader in classic topics would do.
>>>>>>>>>>> Can you expand on what these checks are? Would you be
>>>> checking
>>>>> if
>>>>>>> the
>>>>>>>>>>> transaction was still ongoing for example? Is there state
>>>> about
>>>>>>> ongoing
>>>>>>>>>>> transactions in the batch coordinator? I see some other
>> state
>>>>>>> mentioned
>>>>>>>>>> in
>>>>>>>>>>> the End transaction section, but it's not super clear
>> what
>>>>> state is
>>>>>>>>>> stored
>>>>>>>>>>> and when it is stored.
>>>>>>>>>>>
>>>>>>>>>>> JO 3. I didn't see anything about maintaining LSO --
>> perhaps
>>>>> that
>>>>>>> would
>>>>>>>>>> be
>>>>>>>>>>> stored in the batch coordinator?
>>>>>>>>>>>
>>>>>>>>>>> JO 4. Are there any thoughts about how long transactional
>>>>> state is
>>>>>>>>>>> maintained in the batch coordinator and how it will be
>>>> cleaned
>>>>> up?
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
>>>>> <ju...@confluent.io.invalid>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, Greg and Ivan,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the update. A few comments.
>>>>>>>>>>>>
>>>>>>>>>>>> JR 10. "Consumer fetches are now served from local
>>>> segments,
>>>>>>> making
>>>>>>>>>> use of
>>>>>>>>>>>> the
>>>>>>>>>>>> indexes, page cache, request purgatory, and zero-copy
>>>>>>> functionality
>>>>>>>>>> already
>>>>>>>>>>>> built into classic topics."
>>>>>>>>>>>> JR 10.1 Does the broker build the producer state for
>> each
>>>>>>> partition in
>>>>>>>>>>>> diskless topics?
>>>>>>>>>>>> JR 10.2 For transactional data, the consumer fetches
>> need
>>>> to
>>>>> know
>>>>>>>>>> aborted
>>>>>>>>>>>> records. How is that achieved?
>>>>>>>>>>>>
>>>>>>>>>>>> JR 11. "The batch coordinator saves that the
>> transaction is
>>>>>>> finished
>>>>>>>>>> and
>>>>>>>>>>>> also inserts the control batches in the corresponding
>> logs
>>>>> of the
>>>>>>>>>> involved
>>>>>>>>>>>> Diskless topics. This happens only on the metadata
>> level,
>>>> no
>>>>>>> actual
>>>>>>>>>> control
>>>>>>>>>>>> batches are written to any file. "
>>>>>>>>>>>> A fetch response could include multiple transactional
>>>>> batches.
>>>>>>> How
>>>>>>>>>> does the
>>>>>>>>>>>> broker obtain the information about the ending control
>>>> batch
>>>>> for
>>>>>>> each
>>>>>>>>>>>> batch? Does that mean that a fetch response needs to be
>>>>> built by
>>>>>>>>>>>> stitching record batches and generated control batches
>>>>> together?
>>>>>>>>>>>>
>>>>>>>>>>>> JR 12. Queues: Is there still a share partition leader
>> that
>>>>> all
>>>>>>>>>> consumers
>>>>>>>>>>>> are routed to?
>>>>>>>>>>>>
>>>>>>>>>>>> JR 13. "Should the KIPs be modified to include this or
>> it's
>>>>> too
>>>>>>>>>>>> implementation-focused?" It would be useful to include
>>>> enough
>>>>>>> details
>>>>>>>>>> to
>>>>>>>>>>>> understand correctness and performance impact.
>>>>>>>>>>>>
>>>>>>>>>>>> HC5. Henry has a valid point. Requests from a given
>>>> producer
>>>>>>> contain a
>>>>>>>>>>>> sequence number, which is ordered. If a producer sends
>>>> every
>>>>>>> Produce
>>>>>>>>>>>> request to an arbitrary broker, those requests could
>> reach
>>>>> the
>>>>>>> batch
>>>>>>>>>>>> coordinator in different order and lead to rejection
>> of the
>>>>>>> produce
>>>>>>>>>>>> requests.
>>>>>>>>>>>>
>>>>>>>>>>>> Jun
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
>>>>> ivan@ivanyu.me>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have also thought in a bit more details about
>>>>> transactions
>>>>>>> and
>>>>>>>>>> queues,
>>>>>>>>>>>>> here's the plan.
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Transactions*
>>>>>>>>>>>>>
>>>>>>>>>>>>> The support for transactions in *classic topics* is
>> based
>>>>> on
>>>>>>> precise
>>>>>>>>>>>>> interactions between three actors: clients (mostly
>>>>> producers,
>>>>>>> but
>>>>>>>>>> also
>>>>>>>>>>>>> consumers), brokers (ReplicaManager and other
>> classes),
>>>> and
>>>>>>>>>> transaction
>>>>>>>>>>>>> coordinators. Brokers also run partition leaders with
>>>> their
>>>>>>> local
>>>>>>>>>> state
>>>>>>>>>>>>> (ProducerStateManager and others).
>>>>>>>>>>>>>
>>>>>>>>>>>>> The high level (some details skipped) workflow is the
>>>>>>> following.
>>>>>>>>>> When a
>>>>>>>>>>>>> transactional Produce request is received by the
>> broker:
>>>>>>>>>>>>> 1. For each partition, the partition leader checks
>> if a
>>>>>>> non-empty
>>>>>>>>>>>>> transaction is running for this partition. This is
>> done
>>>>> using
>>>>>>> its
>>>>>>>>>> local
>>>>>>>>>>>>> state derived from the log metadata
>>>> (ProducerStateManager,
>>>>>>>>>>>>> VerificationStateEntry, VerificationGuard).
>>>>>>>>>>>>> 2. The transaction coordinator is informed about all
>> the
>>>>>>> partitions
>>>>>>>>>> that
>>>>>>>>>>>>> aren’t part of the transaction to include them.
>>>>>>>>>>>>> 3. The partition leaders do additional transactional
>>>>> checks.
>>>>>>>>>>>>> 4. The partition leaders append the transactional
>> data to
>>>>>>> their logs
>>>>>>>>>> and
>>>>>>>>>>>>> update some of their state (for example, log the fact
>>>> that
>>>>> the
>>>>>>>>>>>> transaction
>>>>>>>>>>>>> is running for the partition and its first offset).
>>>>>>>>>>>>>
>>>>>>>>>>>>> When the transaction is committed or aborted:
>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
>>>>> directly
>>>>>>> with
>>>>>>>>>>>>> EndTxnRequest.
>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
>> or
>>>>>>>>>> PREPARE_ABORT to
>>>>>>>>>>>>> its log and responds to the producer.
>>>>>>>>>>>>> 3. The transaction coordinator sends
>>>>> WriteTxnMarkersRequest to
>>>>>>> the
>>>>>>>>>>>> leaders
>>>>>>>>>>>>> of the involved partitions.
>>>>>>>>>>>>> 4. The partition leaders write the transaction
>> markers to
>>>>>>> their logs
>>>>>>>>>> and
>>>>>>>>>>>>> respond to the coordinator.
>>>>>>>>>>>>> 5. The coordinator writes the final transaction state
>>>>>>>>>> COMPLETE_COMMIT or
>>>>>>>>>>>>> COMPLETE_ABORT.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In classic topics, partitions have leaders and lots
>> of
>>>>>>> important
>>>>>>>>>> state
>>>>>>>>>>>>> necessary for supporting this workflow is local. The
>> main
>>>>>>> challenge
>>>>>>>>>> in
>>>>>>>>>>>>> mapping this to Diskless comes from the fact there
>> are no
>>>>>>> partition
>>>>>>>>>>>>> leaders, so the corresponding pieces of state need
>> to be
>>>>>>> globalized
>>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>> batch coordinator. We are already doing this to
>> support
>>>>>>> idempotent
>>>>>>>>>>>> produce.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The high level workflow for *diskless topics* would
>> look
>>>>> very
>>>>>>>>>> similar:
>>>>>>>>>>>>> 1. For each partition, the broker checks if a
>> non-empty
>>>>>>> transaction
>>>>>>>>>> is
>>>>>>>>>>>>> running for this partition. In contrast to classic
>>>> topics,
>>>>>>> this is
>>>>>>>>>>>> checked
>>>>>>>>>>>>> against the batch coordinator with a single RPC.
>> Since a
>>>>>>> transaction
>>>>>>>>>>>> could
>>>>>>>>>>>>> be uniquely identified with producer ID and epoch,
>> the
>>>>> positive
>>>>>>>>>> result of
>>>>>>>>>>>>> this check could be cached locally (for the double
>>>>> configured
>>>>>>>>>> duration
>>>>>>>>>>>> of a
>>>>>>>>>>>>> transaction, for example).
>>>>>>>>>>>>> 2. The same: The transaction coordinator is informed
>>>> about
>>>>> all
>>>>>>> the
>>>>>>>>>>>>> partitions that aren’t part of the transaction to
>> include
>>>>> them.
>>>>>>>>>>>>> 3. No transactional checks are done on the broker
>> side.
>>>>>>>>>>>>> 4. The broker appends the transactional data to the
>>>> current
>>>>>>> shared
>>>>>>>>>> WAL
>>>>>>>>>>>>> segment. It doesn’t update any transaction-related
>> state
>>>>> for
>>>>>>> Diskless
>>>>>>>>>>>>> topics, because it doesn’t have any.
>>>>>>>>>>>>> 5. The WAL segment is committed to the batch
>> coordinator
>>>>> like
>>>>>>> in the
>>>>>>>>>>>>> normal produce flow.
>>>>>>>>>>>>> 6. The batch coordinator does the final transactional
>>>>> checks
>>>>>>> of the
>>>>>>>>>>>>> batches. This procedure would output the same errors
>> like
>>>>> the
>>>>>>>>>> partition
>>>>>>>>>>>>> leader in classic topics would do. I.e. some batches
>>>> could
>>>>> be
>>>>>>>>>> rejected.
>>>>>>>>>>>>> This means, there will potentially be garbage in the
>> WAL
>>>>>>> segment
>>>>>>>>>> file in
>>>>>>>>>>>>> case of transactional errors. This is preferable to
>> doing
>>>>> more
>>>>>>>>>> network
>>>>>>>>>>>>> round trips, especially considering the WAL segments
>> will
>>>>> be
>>>>>>>>>> relatively
>>>>>>>>>>>>> short-living (see the Greg's update above).
>>>>>>>>>>>>>
>>>>>>>>>>>>> When the transaction is committed or aborted:
>>>>>>>>>>>>> 1. The producer contacts the transaction coordinator
>>>>> directly
>>>>>>> with
>>>>>>>>>>>>> EndTxnRequest.
>>>>>>>>>>>>> 2. The transaction coordinator writes PREPARE_COMMIT
>> or
>>>>>>>>>> PREPARE_ABORT to
>>>>>>>>>>>>> its log and responds to the producer.
>>>>>>>>>>>>> 3. *[NEW]* The transaction coordinator informs the
>> batch
>>>>>>> coordinator
>>>>>>>>>> that
>>>>>>>>>>>>> the transaction is finished.
>>>>>>>>>>>>> 4. *[NEW]* The batch coordinator saves that the
>>>>> transaction is
>>>>>>>>>> finished
>>>>>>>>>>>>> and also inserts the control batches in the
>> corresponding
>>>>> logs
>>>>>>> of the
>>>>>>>>>>>>> involved Diskless topics. This happens only on the
>>>> metadata
>>>>>>> level, no
>>>>>>>>>>>>> actual control batches are written to any file. They
>> will
>>>>> be
>>>>>>>>>> dynamically
>>>>>>>>>>>>> created on Fetch and other read operations. We could
>>>>>>> technically
>>>>>>>>>> write
>>>>>>>>>>>>> these control batches for real, but this would mean
>> extra
>>>>>>> produce
>>>>>>>>>>>> latency,
>>>>>>>>>>>>> so it's better just to mark them in the batch
>> coordinator
>>>>> and
>>>>>>> save
>>>>>>>>>> these
>>>>>>>>>>>>> milliseconds.
>>>>>>>>>>>>> 5. The transaction coordinator sends
>>>>> WriteTxnMarkersRequest to
>>>>>>> the
>>>>>>>>>>>> leaders
>>>>>>>>>>>>> of the involved partitions. – Now only to classic
>> topics
>>>>> now.
>>>>>>>>>>>>> 6. The partition leaders of classic topics write the
>>>>>>> transaction
>>>>>>>>>> markers
>>>>>>>>>>>>> to their logs and respond to the coordinator.
>>>>>>>>>>>>> 7. The coordinator writes the final transaction state
>>>>>>>>>> COMPLETE_COMMIT or
>>>>>>>>>>>>> COMPLETE_ABORT.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Compared to the non-transactional produce flow, we
>> get:
>>>>>>>>>>>>> 1. An extra network round trip between brokers and
>> the
>>>>> batch
>>>>>>>>>> coordinator
>>>>>>>>>>>>> when a new partition appear in the transaction. To
>>>>> mitigate the
>>>>>>>>>> impact of
>>>>>>>>>>>>> them:
>>>>>>>>>>>>>  - The results will be cached.
>>>>>>>>>>>>>  - The calls for multiple partitions in one Produce
>>>>> request
>>>>>>> will be
>>>>>>>>>>>>> grouped.
>>>>>>>>>>>>>  - The batch coordinator should be optimized for
>> fast
>>>>>>> response to
>>>>>>>>>> these
>>>>>>>>>>>>> RPCs.
>>>>>>>>>>>>>  - The fact that a single producer normally will
>>>>> communicate
>>>>>>> with a
>>>>>>>>>>>>> single broker for the duration of the transaction
>> further
>>>>>>> reduces the
>>>>>>>>>>>>> expected number of round trips.
>>>>>>>>>>>>> 2. An extra round trip between the transaction
>>>> coordinator
>>>>> and
>>>>>>> batch
>>>>>>>>>>>>> coordinator when a transaction is finished.
>>>>>>>>>>>>>
>>>>>>>>>>>>> With this proposal, transactions will also be able to
>>>> span
>>>>> both
>>>>>>>>>> classic
>>>>>>>>>>>>> and Diskless topics.
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Queues*
>>>>>>>>>>>>>
>>>>>>>>>>>>> The share group coordination and management is a
>> side job
>>>>> that
>>>>>>>>>> doesn't
>>>>>>>>>>>>> interfere with the topic itself (leadership,
>> replicas,
>>>>> physical
>>>>>>>>>> storage
>>>>>>>>>>>> of
>>>>>>>>>>>>> records, etc.) and non-queue producers and consumers
>>>>> (Fetch and
>>>>>>>>>> Produce
>>>>>>>>>>>>> RPCs, consumer group-related RPCs are not affected.)
>> We
>>>>> don't
>>>>>>> see any
>>>>>>>>>>>>> reason why we can't make Diskless topics compatible
>> with
>>>>> share
>>>>>>>>>> groups the
>>>>>>>>>>>>> same way as classic topics are. Even on the code
>> level,
>>>> we
>>>>>>> don't
>>>>>>>>>> expect
>>>>>>>>>>>> any
>>>>>>>>>>>>> serious refactoring: the same reading routines are
>> used
>>>>> that
>>>>>>> are
>>>>>>>>>> used for
>>>>>>>>>>>>> fetching (e.g. ReplicaManager.readFromLog).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should the KIPs be modified to include this or it's
>> too
>>>>>>>>>>>>> implementation-focused?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Ivan
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you all for your questions and design input
>> on
>>>>>>> KIP-1150.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have just updated KIP-1150 and KIP-1163 with a
>> new
>>>>>>> design. To
>>>>>>>>>>>>> summarize
>>>>>>>>>>>>>> the changes:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. The design prioritizes integrating with the
>> existing
>>>>>>> KIP-405
>>>>>>>>>> Tiered
>>>>>>>>>>>>>> Storage interfaces, permitting data produced to a
>>>>> Diskless
>>>>>>> topic
>>>>>>>>>> to be
>>>>>>>>>>>>>> moved to tiered storage.
>>>>>>>>>>>>>> This lowers the scalability requirements for the
>> Batch
>>>>>>> Coordinator
>>>>>>>>>>>>>> component, and allows Diskless to compose with
>> Tiered
>>>>> Storage
>>>>>>>>>> plugin
>>>>>>>>>>>>>> features such as encryption and alternative data
>>>> formats.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. Consumer fetches are now served from local
>> segments,
>>>>>>> making use
>>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>> indexes, page cache, request purgatory, and
>> zero-copy
>>>>>>> functionality
>>>>>>>>>>>>> already
>>>>>>>>>>>>>> built into classic topics.
>>>>>>>>>>>>>> However, local segments are now considered cache
>>>>> elements,
>>>>>>> do not
>>>>>>>>>> need
>>>>>>>>>>>> to
>>>>>>>>>>>>>> be durably stored, and can be built without
>> contacting
>>>>> any
>>>>>>> other
>>>>>>>>>>>>> replicas.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. The design has been simplified substantially, by
>>>>> removing
>>>>>>> the
>>>>>>>>>>>> previous
>>>>>>>>>>>>>> Diskless consume flow, distributed cache
>> component, and
>>>>>>> "object
>>>>>>>>>>>>>> compaction/merging" step.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The design maintains leaderless produces as
>> enabled by
>>>>> the
>>>>>>> Batch
>>>>>>>>>>>>>> Coordinator, and the same latency profiles as the
>>>> earlier
>>>>>>> design,
>>>>>>>>>> while
>>>>>>>>>>>>>> being simpler and integrating better into the
>> existing
>>>>>>> ecosystem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, and we are eager to hear your feedback on
>> the
>>>> new
>>>>>>> design.
>>>>>>>>>>>>>> Greg Harris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
>>>>>>> <ju...@confluent.io.invalid>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, Jan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For me, the main gap of KIP-1150 is the support
>> of
>>>> all
>>>>>>> existing
>>>>>>>>>>>> client
>>>>>>>>>>>>>>> APIs. Currently, there is no design for
>> supporting
>>>> APIs
>>>>>>> like
>>>>>>>>>>>>> transactions
>>>>>>>>>>>>>>> and queues.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
>>>>>>>>>>>>>>> <ja...@kentra.io.invalid> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Would it be a good time to ask for the current
>>>>> status of
>>>>>>> this
>>>>>>>>>> KIP?
>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> haven't seen much activity here for the past 2
>>>>> months,
>>>>>>> the
>>>>>>>>>> vote got
>>>>>>>>>>>>>>> vetoed
>>>>>>>>>>>>>>>> but I think the pending questions have been
>>>> answered
>>>>>>> since
>>>>>>>>>> then.
>>>>>>>>>>>>> KIP-1183
>>>>>>>>>>>>>>>> (AutoMQ's proposal) also didn't have any
>> activity
>>>>> since
>>>>>>> May.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In my eyes KIP-1150 and KIP-1183 are two real
>>>> choices
>>>>>>> that can
>>>>>>>>>> be
>>>>>>>>>>>>>>>> made, with a coordinator-based approach being
>> by
>>>> far
>>>>> the
>>>>>>>>>> dominant
>>>>>>>>>>>> one
>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> it comes to market adoption - but all these are
>>>>>>> standalone
>>>>>>>>>>>> products.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm a big fan of both approaches, but would
>> hate to
>>>>> see a
>>>>>>>>>> stall. So
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> question is: can we get an update?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Maybe it's time to start another vote? Colin
>>>> McCabe -
>>>>>>> have your
>>>>>>>>>>>>> questions
>>>>>>>>>>>>>>>> been answered? If not, is there anything I can
>> do
>>>> to
>>>>>>> help? I'm
>>>>>>>>>>>> deeply
>>>>>>>>>>>>>>>> familiar with both architectures and have
>> written
>>>>> about
>>>>>>> both?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Jun 24, 2025 at 10:42 AM Stanislav
>>>> Kozlovski
>>>>> <
>>>>>>>>>>>>>>>> stanislavkozlovski@apache.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have some nits - it may be useful to
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> a) group all the KIP email threads in the
>> main
>>>> one
>>>>>>> (just a
>>>>>>>>>> bunch
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> links
>>>>>>>>>>>>>>>>> to everything)
>>>>>>>>>>>>>>>>> b) create the email threads
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's a bit hard to track it all - for
>> example, I
>>>>> was
>>>>>>>>>> searching
>>>>>>>>>>>> for
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> discuss thread for KIP-1165 for a while; As
>> far
>>>> as
>>>>> I
>>>>>>> can
>>>>>>>>>> tell, it
>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>> exist yet.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Since the KIPs are published (by virtue of
>> having
>>>>> the
>>>>>>> root
>>>>>>>>>> KIP be
>>>>>>>>>>>>>>>>> published, having a DISCUSS thread and links
>> to
>>>>>>> sub-KIPs
>>>>>>>>>> where
>>>>>>>>>>>> were
>>>>>>>>>>>>>>> aimed
>>>>>>>>>>>>>>>>> to move the discussion towards), I think it
>> would
>>>>> be
>>>>>>> good to
>>>>>>>>>>>> create
>>>>>>>>>>>>>>>> DISCUSS
>>>>>>>>>>>>>>>>> threads for them all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Stan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 2025/04/16 11:58:22 Josep Prat wrote:
>>>>>>>>>>>>>>>>>> Hi Kafka Devs!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We want to start a new KIP discussion about
>>>>>>> introducing a
>>>>>>>>>> new
>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>> topics that would make use of Object
>> Storage as
>>>>> the
>>>>>>> primary
>>>>>>>>>>>>> source of
>>>>>>>>>>>>>>>>>> storage. However, as this KIP is big we
>> decided
>>>>> to
>>>>>>> split it
>>>>>>>>>>>> into
>>>>>>>>>>>>>>>> multiple
>>>>>>>>>>>>>>>>>> related KIPs.
>>>>>>>>>>>>>>>>>> We have the motivational KIP-1150 (
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>> that aims to discuss if Apache Kafka
>> should aim
>>>>> to
>>>>>>> have
>>>>>>>>>> this
>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>> feature at all. This KIP doesn't go onto
>>>> details
>>>>> on
>>>>>>> how to
>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>> This follows the same approach used when we
>>>>> discussed
>>>>>>>>>> KRaft.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But as we know that it is sometimes really
>> hard
>>>>> to
>>>>>>> discuss
>>>>>>>>>> on
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> meta
>>>>>>>>>>>>>>>>>> level, we also created several sub-kips
>> (linked
>>>>> in
>>>>>>>>>> KIP-1150)
>>>>>>>>>>>> that
>>>>>>>>>>>>>>> offer
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> implementation of this feature.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We kindly ask you to use the proper DISCUSS
>>>>> threads
>>>>>>> for
>>>>>>>>>> each
>>>>>>>>>>>>> type of
>>>>>>>>>>>>>>>>>> concern and keep this one to discuss
>> whether
>>>>> Apache
>>>>>>> Kafka
>>>>>>>>>> wants
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> this feature or not.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks in advance on behalf of all the
>> authors
>>>> of
>>>>>>> this KIP.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ------------------
>>>>>>>>>>>>>>>>>> Josep Prat
>>>>>>>>>>>>>>>>>> Open Source Engineering Director, Aiven
>>>>>>>>>>>>>>>>>> josep.prat@aiven.io   |   +491715557497 |
>>>>> aiven.io
>>>>>>>>>>>>>>>>>> Aiven Deutschland GmbH
>>>>>>>>>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
>>>>>>>>>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu
>>>>> Valtonen,
>>>>>>>>>>>>>>>>>> Anna Richardson, Kenneth Chen
>>>>>>>>>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Ivan,

Thanks for the reply.

"As I understand, you’re speaking about locally materialized segments. They
will indeed consume some IOPS. See them as a cache that could always be
restored from the remote storage. While it’s not ideal, it's still OK to
lose data in them due to a machine crash, for example. Because of this, we
can avoid explicit flushing on local materialized segments at all and let
the file system and page cache figure out when to flush optimally. This
would not eliminate the extra IOPS, but should reduce it dramatically,
depending on throughput for each partition. We, of course, continue
flushing the metadata segments as before."

If we have a mix of classic and diskless topics on the same broker, it's
important that the classic topics' data is flushed to disk as quickly as
possible. To achieve this, users typically set dirty_expire_centisecs in
the kernel based on the number of available disk IOPS. Once you set this
number, it applies to all dirty files, including the cached data in
diskless topics. So, if there are more files actively accumulating data,
the flush frequency and therefore RPO is reduced for classic topics.

"We should have mentioned this explicitly, but this step, in fact, remains
in the form of segments offloading to tiered storage. When we assemble a
segment and hand it over to RemoteLogManager, we’re effectively doing
metadata compaction: replacing a big number of pieces of metadata about
individual batches with a single record in __remote_log_metadata."

The object merging in tier storage typically only kicks in after a few
hours. The impact is (1) the amount of accumulated metadata is still quite
large; (2) there are many small objects, leading to poor read performance.
I think we need a more frequent object merging process than tier storage
provides.

Jun


On Thu, Oct 23, 2025 at 10:12 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hello Jack, Jun, Luke, and all!
>
> Thank you for your messages.
>
> Let me first address some of Jun’s comments.
>
> > First, it degrades the durability.
> > For each partition, now there are two files being actively written at a
> > given point of time, one for the data and another for the metadata.
> > Flushing each file requires a separate IO. If the disk has 1K IOPS and we
> > have 5K partitions in a broker, currently we can afford to flush each
> > partition every 5 seconds, achieving an RPO of 5 seconds. If we double
> the
> > number of files per partition, we can only flush each partition every 10
> > seconds, which makes RPO twice as bad.
>
> As I understand, you’re speaking about locally materialized segments. They
> will indeed consume some IOPS. See them as a cache that could always be
> restored from the remote storage. While it’s not ideal, it's still OK to
> lose data in them due to a machine crash, for example. Because of this, we
> can avoid explicit flushing on local materialized segments at all and let
> the file system and page cache figure out when to flush optimally. This
> would not eliminate the extra IOPS, but should reduce it dramatically,
> depending on throughput for each partition. We, of course, continue
> flushing the metadata segments as before.
>
> It’s worth making a note on caching. I think nobody will disagree that
> doing direct reads from remote storage every time a batch is requested by a
> consumer will not be practical neither from the performance nor from the
> economy point of view. We need a way to keep the number of GET requests
> down. There are multiple options, for example:
>   1. Rack-aware distributed in-memory caching.
>   2. Local in-memory caching. Comes with less network chattiness and works
> well if we have more or less stable brokers to consume from.
>   3. Materialization of diskless logs on local disk. Way lower impact on
> RAM and also requires stable brokers for consumption (using just assigned
> replicas will probably work well).
>
> Materialization is one of possible options, but we can choose another one.
> However, we will have this dilemma regardless of whether we have an
> explicit coordinator or we go “coordinator-less”.
>
> > Second, if we ever need this
> > metadata somewhere else, say in the WAL file manager, the consumer needs
> to
> > subscribe to every partition in the cluster, which is inefficient. The
> > actual benefit of this approach is also questionable. On the surface, it
> > might seem that we could reduce the number of lines that need to be
> changed
> > for this KIP. However, the changes are quite intrusive to the classic
> > partition's code path and will probably make the code base harder to
> > maintain in the long run. I like the original approach based on the batch
> > coordinator much better than this one. We could probably refactor the
> > producer state code so that it could be reused in the batch coordinator.
>
> It’s hard to disagree with this. The explicit coordinator is more a side
> thing, while coordinator-less approach is more about extending
> ReplicaManager, UnifiedLog and others substantially.
>
> > Thanks for addressing the concerns on the number of RPCs in the produce
> > path. I agree that with the metadata crafting mechanism, we could
> mitigate
> > the PRC problem. However, since we now require the metadata to be
> > collocated with the data on the same set of brokers, it's weird that they
> > are now managed by different mechanisms. The data assignment now uses the
> > metadata crafting mechanism, but the metadata is stored in the classic
> > partition using its own assignment strategy. It will be complicated to
> keep
> > them collocated.
>
> I would like to note that the metadata crafting is needed only to tell
> producers which brokers they should send Produce requests to, but data (as
> in “locally materialized log”) is located on partition replicas, i.e.
> automatically co-located with metadata.
>
> As a side note, it would probably be better that instead of implicitly
> crafting partition metadata, we extend the metadata protocol so that for
> diskless partitions we return not only the leader and replicas, but also
> some “recommended produce brokers”, selected for optimal performance and
> costs. Producers will pick ones in their racks.
>
> > I am also concerned about the removal of the object compaction/merging
> > step.
>
> We should have mentioned this explicitly, but this step, in fact, remains
> in the form of segments offloading to tiered storage. When we assemble a
> segment and hand it over to RemoteLogManager, we’re effectively doing
> metadata compaction: replacing a big number of pieces of metadata about
> individual batches with a single record in __remote_log_metadata.
>
> We could create a Diskless-specific merging mechanism instead if needed.
> It’s rather easy with the explicit coordinator approach. With the
> coordinator-less approach, this would probably be a bit more tricky
> (rewriting the tail of the log by the leader + replicating this change
> reliably).
>
> > I see a tendency toward primarily optimizing for the fewest code changes
> in
> > the KIP. Instead, our primary goal should be a clean design that can last
> > for the long term.
>
> Yes, totally agree.
>
>
>
> Luke,
> > I'm wondering if the complexity of designing txn and queue is because of
> > leaderless cluster, do you think it will be simpler if we only focus on
> the
> > "diskless" design to handle object compaction/merging to/from the remote
> > storage to save the cross-AZ cost?
>
> After some evolution of the original proposal, leaderless is now limited.
> We only need to be able to accept Produce requests on more than one broker
> to eliminate the cross-AZ costs for producers. Do I get it right that you
> propose to get rid of this? Or do I misunderstand?
>
>
>
> Let’s now look at this problem from a higher level, as Jack proposed. As
> it was said, the big choice we need to make is whether we 1) create an
> explicit batch coordinator; or 2) go for the coordinator-less approach,
> where each diskless partition is managed by its leader as in classic topics.
>
> If we try to compare the two approaches:
>
> Pluggability:
> - Explicit coordinator: Possible. For example, some setups may benefit
> from batch metadata being stored in a cloud database (such as AWS DynamoDB
> or GCP Spanner).
> - Coordinator-less: Impossible.
>
> Scalability and fault tolerance:
> - Explicit coordinator: Depends on the implementation and it’s also
> necessary to actively work for it.
> - Coordinator-less: Closer to classic Kafka topics. Scaling is done by
> partition placement, partitions could fail independently.
>
> Separation of concerns:
> - Explicit coordinator: Very good. Diskless remains more independent from
> classic topics in terms of code and workflows. For example, the
> above-mentioned non-tiered storage metadata compaction mechanism could be
> relatively simply implemented with it. As a flip side of this, some
> workflows (e.g. transactions) will have to be adapted.
> - Coordinator-less: Less so. It leans to the opposite: bringing diskless
> closer to classic topics. Some code paths and workflows could be more
> straightforwardly reused, but they will inevitably have to be adapted to
> accommodate both topic types as also discussed.
>
> Cloud-nativeness. This is a vague concept, also related to the previous,
> but let’s try:
> - Explicit coordinator: Storing and processing metadata separately makes
> it easier for brokers to take different roles, be purely stateless if
> needed, etc.
> - Coordinator-less: Less so. Something could be achieved with creative
> partition placement, but not much.
>
> Both seem to have their pros and cons. However, answering Jack’s question,
> the explicit coordinator approach may indeed lead to a more flexible design.
>
>
> The purpose of this deviation in the discussion was to receive a
> preliminary community evaluation of the coordinator-less approach without
> taking on the task of writing a separate KIP and fitting it in the system
> of KIP-1150 and its children. We’re open to stopping it and getting back to
> working out the coordinator design if the community doesn’t favor the
> proposed approach.
>
> Best,
> Ivan and Diskless team
>
> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > Hi Ivan,
> >
> > As Jun pointed out, the updated design seems to have some shortcomings
> > although it simplifies the implementation.
> >
> > I'm wondering if the complexity of designing txn and queue is because of
> > leaderless cluster, do you think it will be simpler if we only focus on
> the
> > "diskless" design to handle object compaction/merging to/from the remote
> > storage to save the cross-AZ cost?
> >
> >
> > Thank you,
> > Luke
> >
> > On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Ivan,
> > >
> > > Thanks for the explanation.
> > >
> > > "we write the reference to the WAL file with the batch data"
> > >
> > > I understand the approach now, but I think it is a hacky one. There are
> > > multiple short comings with this design. First, it degrades the
> durability.
> > > For each partition, now there are two files being actively written at a
> > > given point of time, one for the data and another for the metadata.
> > > Flushing each file requires a separate IO. If the disk has 1K IOPS and
> we
> > > have 5K partitions in a broker, currently we can afford to flush each
> > > partition every 5 seconds, achieving an RPO of 5 seconds. If we double
> the
> > > number of files per partition, we can only flush each partition every
> 10
> > > seconds, which makes RPO twice as bad. Second, if we ever need this
> > > metadata somewhere else, say in the WAL file manager, the consumer
> needs to
> > > subscribe to every partition in the cluster, which is inefficient. The
> > > actual benefit of this approach is also questionable. On the surface,
> it
> > > might seem that we could reduce the number of lines that need to be
> changed
> > > for this KIP. However, the changes are quite intrusive to the classic
> > > partition's code path and will probably make the code base harder to
> > > maintain in the long run. I like the original approach based on the
> batch
> > > coordinator much better than this one. We could probably refactor the
> > > producer state code so that it could be reused in the batch
> coordinator.
> > >
> > > Thanks for addressing the concerns on the number of RPCs in the produce
> > > path. I agree that with the metadata crafting mechanism, we could
> mitigate
> > > the PRC problem. However, since we now require the metadata to be
> > > collocated with the data on the same set of brokers, it's weird that
> they
> > > are now managed by different mechanisms. The data assignment now uses
> the
> > > metadata crafting mechanism, but the metadata is stored in the classic
> > > partition using its own assignment strategy. It will be complicated to
> keep
> > > them collocated.
> > >
> > > I am also concerned about the removal of the object compaction/merging
> > > step. My first concern is on the amount of metadata that need to be
> kept.
> > > Without object compcation, the metadata generated in the produce path
> can
> > > only be deleted after remote tiering kicks in. Let's say for every
> 250ms we
> > > produce 100 byte of metadata per partition. Let's say remoting tiering
> > > kicks in after 5 hours. In a cluster with 100K partitions, we need to
> keep
> > > about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
> > > signficant. A second concern is on performance. Every time we need to
> > > rebuild the caching data, we need to read a bunch of small objects
> from S3,
> > > slowing down the building process. If a consumer happens to need such
> data,
> > > it could slow down the application.
> > >
> > > I see a tendency toward primarily optimizing for the fewest code
> changes in
> > > the KIP. Instead, our primary goal should be a clean design that can
> last
> > > for the long term.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thank you for your message. I’m sorry that I failed to clearly
> explain
> > > the
> > > > idea. Let me try to fix this.
> > > >
> > > > > Does each partition now have a metadata partition and a separate
> data
> > > > > partition? If so, I am concerned that it essentially doubles the
> number
> > > > of
> > > > > partitions, which impacts the number of open file descriptors and
> the
> > > > > required IOPS, and so on. It also seems wasteful to have a separate
> > > > > partition just to store the metadata. It's as if we are creating an
> > > > > internal topic with an unbounded number of partitions.
> > > >
> > > > No. There will be only one physical partition per diskless
> partition. Let
> > > > me explain this with an example. Let’s say we have a diskless
> partition
> > > > topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce
> some
> > > > batches to this partition. The content of the segment file will be
> > > > something like this (for each batch):
> > > >
> > > > BaseOffset: 00000000000000000000 (like in classic)
> > > > Length: 123456 (like in classic)
> > > > PartitionLeaderEpoch: like in classic
> > > > Magic: like in classic
> > > > CRC: like in classic
> > > > Attributes: like in classic
> > > > LastOffsetDelta: like in classic
> > > > BaseTimestamp: like in classic
> > > > MaxTimestamp: like in classic
> > > > ProducerId: like in classic
> > > > ProducerEpoch: like in classic
> > > > BaseSequence: like in classic
> > > > RecordsCount: like in classic
> > > > Records:
> > > >   path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> > > > 123456
> > > >
> > > > It looks very much like classic log entries. The only difference is
> that
> > > > instead of writing real Records, we write the reference to the WAL
> file
> > > > with the batch data (I guess we need only the name and the byte
> offset,
> > > > because the byte length is the standard field above). Otherwise,
> it’s a
> > > > normal Kafka log with the leader and replicas.
> > > >
> > > > So we have as many partitions for diskless as for classic. As of open
> > > file
> > > > descriptors, let’s proceed to the following:
> > > >
> > > > > Are the metadata and
> > > > > the data for the same partition always collocated on the same
> broker?
> > > If
> > > > > so, how do we enforce that when replicas are reassigned?
> > > >
> > > > The source of truth for the data is still in WAL files on object
> storage.
> > > > The source of truth for the metadata is in segment files on the
> brokers
> > > in
> > > > the replica set. Two new mechanisms are planned, both independent of
> this
> > > > new proposal, but I want to present them to give the idea that only a
> > > > limited amount of data files will be operated locally:
> > > > - We want to assemble batches into segment files and offload them to
> > > > tiered storage in order to prevent the unbounded growth of batch
> > > metadata.
> > > > For this, we need to open only  a few file descriptors (for the
> segment
> > > > file itself + the necessary indexes) before the segment is fully
> written
> > > > and handed over to RemoteLogManager.
> > > > - We want to assemble local segment files for caching purposes as
> well,
> > > > i.e. to speed up fetching. This will not materialize the full
> content of
> > > > the log, but only the hot set according to some policy (or
> configurable
> > > > policies), i.e. the number of segments and file descriptors will
> also be
> > > > limited.
> > > >
> > > > > The number of RPCs in the produce path is significantly higher. For
> > > > > example, if a produce request has 100 partitions, in a cluster
> with 100
> > > > > brokers, each produce request could generate 100 more RPC requests.
> > > This
> > > > > will significantly increase the request rate.
> > > >
> > > > This is a valid concern that we considered, but this issue can be
> > > > mitigated. I’ll try to explain the approach.
> > > > The situation with a single broker is trivial: all the commit
> requests go
> > > > from the broker to itself.
> > > > Let’s scale this to a multi-broker cluster, but located in the single
> > > rack
> > > > (AZ). Any broker can accept Produce requests for diskless
> partitions, but
> > > > we can tell producers (through metadata) to always send Produce
> requests
> > > to
> > > > leaders. For example, broker 0 hosts the leader replicas for diskless
> > > > partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> requests
> > > for
> > > > these partitions in various combinations, but only for them.
> > > >
> > > >                     Broker 0
> > > >               +-----------------+
> > > >               |    t1-0         |
> > > >               |    t2-1 <--------------------+
> > > >               |    t3-0         |            |
> > > >  produce      | +-------------+ |            |
> > > >  requests     | |  diskless   | |            |
> > > > --------------->|   produce   +--------------+
> > > >  for these    | | WAL buffer  | |    commit requests
> > > >  partitions   | +-------------+ |    for these partitions
> > > >               |                 |
> > > >               +-----------------+
> > > >
> > > > The same applies for other brokers in this cluster. Effectively, each
> > > > broker will commit only to itself, which effectively means 1 commit
> > > request
> > > > per WAL buffer (this may be 0 physical network calls, if we wish,
> just a
> > > > local function call).
> > > >
> > > > Now let’s scale this to multiple racks (AZs). Obviously, we cannot
> always
> > > > send Produce requests to the designated leaders of diskless
> partitions:
> > > > this would mean inter-AZ network traffic, which we would like to
> avoid.
> > > To
> > > > avoid it, we say that every broker has a “diskless produce
> > > representative”
> > > > in every AZ. If we continue our example: when a Produce request for
> t1-0,
> > > > t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0
> (in the
> > > > broker’s AZ the representative is the broker itself). However, if it
> > > comes
> > > > from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > > >
> > > >   |produce requests         |produce requests        |produce
> requests
> > > >   |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
> > > t3-0
> > > >   |from AZ 0                |from AZ 1               |from AZ 2
> > > >   v                         v                        v
> > > >  Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > > > +---------------+      +---------------+      +---------------+
> > > > |     t1-0      |      |               |      |               |
> > > > |     t2-1      |      |               |      |               |
> > > > |     t3-0      |      |               |      |               |
> > > > +---------------+      +--------+------+      +--------+------+
> > > >      ^     ^                    |                      |
> > > >      |     +--------------------+                      |
> > > >      |     commit requests for these partitions        |
> > > >      |                                                 |
> > > >      +-------------------------------------------------+
> > > >            commit requests for these partitions
> > > >
> > > > All the partitions that broker 0 is the leader of will be
> “represented”
> > > by
> > > > brokers 1 and 2 in their AZs.
> > > >
> > > > Of course, this relationship goes both ways between AZs (not
> necessarily
> > > > between the same brokers). It means that provided the cluster is
> balanced
> > > > by the number of brokers per AZ, each broker will represent
> > > (number_of_azs
> > > > - 1) other brokers. This will result in the situation that for the
> > > majority
> > > > of commits, each broker will do up to (number_of_azs - 1) network
> commit
> > > > requests (plus one local). Cloud regions tend to have 3 AZs, very
> rarely
> > > > more. That means, brokers will be doing up to 2 network commit
> requests
> > > per
> > > > WAL file.
> > > >
> > > > There are the following exceptions:
> > > > 1. Broker count imbalance between AZs. For example, when we have 2
> AZs
> > > and
> > > > one has three brokers and another AZ has one. This one broker will do
> > > > between 1 and 3 commit requests per WAL file. This is not an extreme
> > > > amplification. Such an imbalance is not healthy in most practical
> setups
> > > > and should be avoided anyway.
> > > > 2. Leadership changes and metadata propagation period. When the
> partition
> > > > t3-0 is relocated from broker 0 to some broker 3, the producers will
> not
> > > > know this immediately (unless we want to be strict and respond with
> > > > NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
> together
> > > in a
> > > > WAL buffer on broker 2, it will have to send two commit requests: to
> > > broker
> > > > 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
> situation
> > > > is not permanent and as producers update the cluster metadata, it
> will be
> > > > resolved.
> > > >
> > > > This all could be built with the metadata crafting mechanism only
> (which
> > > > is anyway needed for Diskless in one way or another to direct
> producers
> > > and
> > > > consumers where we need to avoid inter-AZ traffic), just with the
> right
> > > > policy for it (for example, some deterministic hash-based formula).
> I.e.
> > > no
> > > > explicit support for “produce representative” or anything like this
> is
> > > > needed on the cluster level, in KRaft, etc.
> > > >
> > > > > The same WAL file metadata is now duplicated into two places,
> partition
> > > > > leader and WAL File Manager. Which one is the source of truth, and
> how
> > > do
> > > > > we maintain consistency between the two places?
> > > >
> > > > We do only two operations on WAL files that span multiple diskless
> > > > partitions: committing and deleting. Commits can be done
> independently as
> > > > described above. But deletes are different, because when a file is
> > > deleted,
> > > > this affects all the partitions that still have alive batches in this
> > > file
> > > > (if any).
> > > >
> > > > The WAL file manager is a necessary point of coordination to delete
> WAL
> > > > files safely. We can say it is the source of truth about files
> > > themselves,
> > > > while the partition leaders and their logs hold the truth about
> whether a
> > > > particular file contains live batches of this particular partition.
> > > >
> > > > The file manager will do this important task: be able to say for sure
> > > that
> > > > a file does not contain any live batch of any existing partition. For
> > > this,
> > > > it will have to periodically check against the partition leaders.
> > > > Considering that batch deletion is irreversible, when we declare a
> file
> > > > “empty”, this is guaranteed to be and stay so.
> > > >
> > > > The file manager has to know about files being committed to start
> track
> > > > them and periodically check if they are empty. We can consider
> various
> > > ways
> > > > to achieve this:
> > > > 1. As was proposed in my previous message: best effort commit by
> brokers
> > > +
> > > > periodic prefix scans of object storage to detect files that went
> below
> > > the
> > > > radar due to network issue or the file manager temporary
> unavailability.
> > > > We’re speaking about listing the file names only and opening only
> > > > previously unknown files in order to find the partitions involved
> with
> > > them.
> > > > 2. Only do scans without explicit commit, i.e. fill the list of files
> > > > fully asynchronously and in the background. This may be not ideal
> due to
> > > > costs and performance of scanning tons of files. However, the number
> of
> > > > live WAL files should be limited due to tiered storage offloading +
> we
> > > can
> > > > optimize this if we give files some global soft order in their names.
> > > >
> > > > > I am not sure how this design simplifies the implementation. The
> > > existing
> > > > > producer/replication code can't be simply reused. Adjusting both
> the
> > > > write
> > > > > path in the leader and the replication path in the follower to
> > > understand
> > > > > batch-header only data is quite intrusive to the existing logic.
> > > >
> > > > It is true that we’ll have to change LocalLog and UnifiedLog in
> order to
> > > > support these changes. However, it seems that idempotence,
> transactions,
> > > > queues, tiered storage will have to be changed less than with the
> > > original
> > > > design. This is because the partition leader state would remain in
> the
> > > same
> > > > place (on brokers) and existing workflows that involve it would have
> to
> > > be
> > > > changed less compared to the situation where we globalize the
> partition
> > > > leader state in the batch coordinator. I admit this is hard to make
> > > > convincing without both real implementations to hand :)
> > > >
> > > > > I am also
> > > > > not sure how this enables seamless switching the topic modes
> between
> > > > > diskless and classic. Could you provide more details on those?
> > > >
> > > > Let’s consider the scenario of turning a classic topic into
> diskless. The
> > > > user sets diskless.enabled=true, the leader receives this metadata
> update
> > > > and does the following:
> > > > 1. Stop accepting normal append writes.
> > > > 2. Close the current active segment.
> > > > 3. Start a new segment that will be written in the diskless format
> (i.e.
> > > > without data).
> > > > 4. Start accepting diskless commits.
> > > >
> > > > Since it’s the same log, the followers will know about that switch
> > > > consistently. They will finish replicating the classic segments and
> start
> > > > replicating the diskless ones. They will always know where each
> batch is
> > > > located (either inside a classic segment or referenced by a diskless
> > > one).
> > > > Switching back should be similar.
> > > >
> > > > Doing this with the coordinator is possible, but has some caveats.
> The
> > > > leader must do the following:
> > > > 1. Stop accepting normal append writes.
> > > > 2. Close the current active segment.
> > > > 3. Write a special control segment to persist and replicate the fact
> that
> > > > from offset N the partition is now in the diskless mode.
> > > > 4. Inform the coordinator about the first offset N of the “diskless
> era”.
> > > > 5. Inform the controller quorum that the transition has finished and
> that
> > > > brokers now can process diskless writes for this partition.
> > > > This could fail at some points, so this will probably require some
> > > > explicit state machine with replication either in the partition log
> or in
> > > > KRaft.
> > > >
> > > > It seems that the coordinator-less approach makes this simpler
> because
> > > the
> > > > “coordinator” for the partition and the partition leader are the
> same and
> > > > they store the partition metadata in the same log, too. While in the
> > > > coordinator approach we have to perform some kind of a distributed
> commit
> > > > to handover metadata management from the classic partition leader to
> the
> > > > batch coordinator.
> > > >
> > > > I hope these explanations help to clarify the idea. Please let me
> know if
> > > > I should go deeper anywhere.
> > > >
> > > > Best,
> > > > Ivan and the Diskless team
> > > >
> > > > On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > Hi, Ivan,
> > > > >
> > > > > Thanks for the update.
> > > > >
> > > > > I am not sure that I fully understand the new design, but it seems
> less
> > > > > clean than before.
> > > > >
> > > > > Does each partition now have a metadata partition and a separate
> data
> > > > > partition? If so, I am concerned that it essentially doubles the
> number
> > > > of
> > > > > partitions, which impacts the number of open file descriptors and
> the
> > > > > required IOPS, and so on. It also seems wasteful to have a separate
> > > > > partition just to store the metadata. It's as if we are creating an
> > > > > internal topic with an unbounded number of partitions. Are the
> metadata
> > > > and
> > > > > the data for the same partition always collocated on the same
> broker?
> > > If
> > > > > so, how do we enforce that when replicas are reassigned?
> > > > >
> > > > > The number of RPCs in the produce path is significantly higher. For
> > > > > example, if a produce request has 100 partitions, in a cluster
> with 100
> > > > > brokers, each produce request could generate 100 more RPC requests.
> > > This
> > > > > will significantly increase the request rate.
> > > > >
> > > > > The same WAL file metadata is now duplicated into two places,
> partition
> > > > > leader and WAL File Manager. Which one is the source of truth, and
> how
> > > do
> > > > > we maintain consistency between the two places?
> > > > >
> > > > > I am not sure how this design simplifies the implementation. The
> > > existing
> > > > > producer/replication code can't be simply reused. Adjusting both
> the
> > > > write
> > > > > path in the leader and the replication path in the follower to
> > > understand
> > > > > batch-header only data is quite intrusive to the existing logic. I
> am
> > > > also
> > > > > not sure how this enables seamless switching the topic modes
> between
> > > > > diskless and classic. Could you provide more details on those?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> > > > >
> > > > > > Hi dear Kafka community,
> > > > > >
> > > > > > In the initial Diskless proposal, we proposed to have a separate
> > > > > > component, batch/diskless coordinator, whose role would be to
> > > centrally
> > > > > > manage the batch and WAL file metadata for diskless topics. This
> > > > component
> > > > > > drew many reasonable comments from the community about how it
> would
> > > > support
> > > > > > various Kafka features (transactions, queues) and its
> scalability.
> > > > While we
> > > > > > believe we have good answers to all the expressed concerns, we
> took a
> > > > step
> > > > > > back and looked at the problem from a different perspective.
> > > > > >
> > > > > > We would like to propose an alternative Diskless design *without
> a
> > > > > > centralized coordinator*. We believe this approach has potential
> and
> > > > > > propose to discuss it as it may be more appealing to the
> community.
> > > > > >
> > > > > > Let us explain the idea. Most of the complications with the
> original
> > > > > > Diskless approach come from one necessary architecture change:
> > > > globalizing
> > > > > > the local state of partition leader in the batch coordinator.
> This
> > > > causes
> > > > > > deviations to the established workflows in various features like
> > > > produce
> > > > > > idempotence and transactions, queues, retention, etc. These
> > > deviations
> > > > need
> > > > > > to be carefully considered, designed, and later implemented and
> > > > tested. In
> > > > > > the new approach we want to avoid this by making partition
> leaders
> > > > again
> > > > > > responsible for managing their partitions, even in diskless
> topics.
> > > > > >
> > > > > > In classic Kafka topics, batch data and metadata are blended
> together
> > > > in
> > > > > > the one partition log. The crux of the Diskless idea is to
> decouple
> > > > them
> > > > > > and move data to the remote storage, while keeping metadata
> somewhere
> > > > else.
> > > > > > Using the central batch coordinator for managing batch metadata
> is
> > > one
> > > > way,
> > > > > > but not the only.
> > > > > >
> > > > > > Let’s now think about managing metadata for each user partition
> > > > > > independently. Generally partitions are independent and don’t
> share
> > > > > > anything apart from that their data are mixed in WAL files. If we
> > > > figure
> > > > > > out how to commit and later delete WAL files safely, we will
> achieve
> > > > the
> > > > > > necessary autonomy that allows us to get rid of the central batch
> > > > > > coordinator. Instead, *each diskless user partition will be
> managed
> > > by
> > > > its
> > > > > > leader*, as in classic Kafka topics. Also like in classic
> topics, the
> > > > > > leader uses the partition log as the way to persist batch
> metadata,
> > > > i.e.
> > > > > > the regular batch header + the information about how to find this
> > > > batch on
> > > > > > remote storage. In contrast to classic topics, batch data is in
> > > remote
> > > > > > storage.
> > > > > >
> > > > > > For clarity, let’s compare the three designs:
> > > > > >  • Classic topics:
> > > > > >    • Data and metadata are co-located in the partition log.
> > > > > >    • The partition log content: [Batch header (metadata)|Batch
> data].
> > > > > >    • The partition log is replicated to the followers.
> > > > > >    • The replicas and leader have local state built from
> metadata.
> > > > > >  • Original Diskless:
> > > > > >    • Metadata is in the batch coordinator, data is on remote
> storage.
> > > > > >    • The partition state is global in the batch coordinator.
> > > > > >  • New Diskless:
> > > > > >    • Metadata is in the partition log, data is on remote storage.
> > > > > >    • Partition log content: [Batch header (metadata)|Batch
> > > coordinates
> > > > on
> > > > > > remote storage].
> > > > > >    • The partition log is replicated to the followers.
> > > > > >    • The replicas and leader have local state built from
> metadata.
> > > > > >
> > > > > > Let’s consider the produce path. Here’s the reminder of the
> original
> > > > > > Diskless design:
> > > > > >
> > > > > >
> > > > > > The new approach could be depicted as the following:
> > > > > >
> > > > > >
> > > > > > As you can see, the main difference is that now instead of a
> single
> > > > commit
> > > > > > request to the batch coordinator, we send multiple parallel
> commit
> > > > requests
> > > > > > to all the leaders of each partition involved in the WAL file.
> Each
> > > of
> > > > them
> > > > > > will commit its batches independently, without coordinating with
> > > other
> > > > > > leaders and any other components. Batch data is addressed by the
> WAL
> > > > file
> > > > > > name, the byte offset and size, which allows partitions to know
> > > nothing
> > > > > > about other partitions to access their data in shared WAL files.
> > > > > >
> > > > > > The number of partitions involved in a single WAL file may be
> quite
> > > > large,
> > > > > > e.g. a hundred. A hundred network requests to commit one WAL
> file is
> > > > very
> > > > > > impractical. However, there are ways to reduce this number:
> > > > > >  1. Partition leaders are located on brokers. Requests to
> leaders on
> > > > one
> > > > > > broker could be grouped together into a single physical network
> > > request
> > > > > > (resembling the normal Produce request that may carry batches for
> > > many
> > > > > > partitions inside). This will cap the number of network requests
> to
> > > the
> > > > > > number of brokers in the cluster.
> > > > > >  2. If we craft the cluster metadata to make producers send their
> > > > requests
> > > > > > to the right brokers (with respect to AZs), we may achieve the
> higher
> > > > > > concentration of logical commit requests in physical network
> requests
> > > > > > reducing the number of the latter ones even further, ideally to
> one.
> > > > > >
> > > > > > Obviously, out of multiple commit requests some may fail or time
> out
> > > > for a
> > > > > > variety of reasons. This is fine. Some producers will receive
> totally
> > > > or
> > > > > > partially failed responses to their Produce requests, similar to
> what
> > > > they
> > > > > > would have received when appending to a classic topic fails or
> times
> > > > out.
> > > > > > If a partition experiences problems, other partitions will not be
> > > > affected
> > > > > > (again, like in classic topics). Of course, the uncommitted data
> will
> > > > be
> > > > > > garbage in WAL files. But WAL files are short-lived (batches are
> > > > constantly
> > > > > > assembled into segments and offloaded to tiered storage), so this
> > > > garbage
> > > > > > will be eventually deleted.
> > > > > >
> > > > > > For safely deleting WAL files we now need to centrally manage
> them,
> > > as
> > > > > > this is the only state and logic that spans multiple partitions.
> On
> > > the
> > > > > > diagram, you can see another commit request called “Commit file
> (best
> > > > > > effort)” going to the WAL File Manager. This manager will be
> > > > responsible
> > > > > > for the following:
> > > > > >  1. Collecting (by requests from brokers) and persisting
> information
> > > > about
> > > > > > committed WAL files.
> > > > > >  2. To handle potential failures in file information delivery, it
> > > will
> > > > be
> > > > > > doing prefix scan on the remote storage periodically to find and
> > > > register
> > > > > > unknown files. The period of this scan will be configurable and
> > > ideally
> > > > > > should be quite long.
> > > > > >  3. Checking with the relevant partition leaders (after a grace
> > > > period) if
> > > > > > they still have batches in a particular file.
> > > > > >  4. Physically deleting files when they aren’t anymore referred
> to by
> > > > any
> > > > > > partition.
> > > > > >
> > > > > > This new design offers the following advantages:
> > > > > >  1. It simplifies the implementation of many Kafka features such
> as
> > > > > > idempotence, transactions, queues, tiered storage, retention.
> Now we
> > > > don’t
> > > > > > need to abstract away and reuse the code from partition leaders
> in
> > > the
> > > > > > batch coordinator. Instead, we will literally use the same code
> paths
> > > > in
> > > > > > leaders, with little adaptation. Workflows from classic topics
> mostly
> > > > > > remain unchanged.
> > > > > > For example, it seems that
> > > > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > > > > > KafkaApis.handleWriteTxnMarkersRequest used for transaction
> support
> > > on
> > > > the
> > > > > > partition leader side could be used for diskless topics with
> little
> > > > > > adaptation. ProducerStateManager, needed for both idempotent
> produce
> > > > and
> > > > > > transactions, would be reused.
> > > > > > Another example is share groups support, where the share
> partition
> > > > leader,
> > > > > > being co-located with the partition leader, would execute the
> same
> > > > logic
> > > > > > for both diskless and classic topics.
> > > > > >  2. It returns to the familiar partition-based scaling model,
> where
> > > > > > partitions are independent.
> > > > > >  3. It makes the operation and failure patterns closer to the
> > > familiar
> > > > > > ones from classic topics.
> > > > > >  4. It opens a straightforward path to seamless switching the
> topics
> > > > modes
> > > > > > between diskless and classic.
> > > > > >
> > > > > > The rest of the things remain unchanged compared to the previous
> > > > Diskless
> > > > > > design (after all previous discussions). Such things as local
> segment
> > > > > > materialization by replicas, the consume path, tiered storage
> > > > integration,
> > > > > > etc.
> > > > > >
> > > > > > If the community finds this design more suitable, we will update
> the
> > > > > > KIP(s) accordingly and continue working on it. Please let us know
> > > what
> > > > you
> > > > > > think.
> > > > > >
> > > > > > Best regards,
> > > > > > Ivan and Diskless team
> > > > > >
> > > > > > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Yes, you're right. We need to track the aborted transactions
> for in
> > > > the
> > > > > > diskless coordinator for as long as the corresponding offsets are
> > > > there.
> > > > > > With the tiered storage unification Greg mentioned earlier, this
> will
> > > > be
> > > > > > finite time even for infinite data retention.
> > > > > > >
> > > > > > > Best,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > > > > Hey Ivan,
> > > > > > > >
> > > > > > > > Thanks for the response. I think most of what you said made
> > > sense,
> > > > but
> > > > > > I
> > > > > > > > did have some questions about this part:
> > > > > > > >
> > > > > > > > > As we understand this, the partition leader in classic
> topics
> > > > forgets
> > > > > > > > about a transaction once it’s replicated (HWM overpasses
> it). The
> > > > > > > > transaction coordinator acts like the main guardian, allowing
> > > > partition
> > > > > > > > leaders to do this safely. Please correct me if this is
> wrong. We
> > > > think
> > > > > > > > about relying on this with the batch coordinator and delete
> the
> > > > > > information
> > > > > > > > about a transaction once it’s finished (as there’s no
> replication
> > > > and
> > > > > > HWM
> > > > > > > > advances immediately).
> > > > > > > >
> > > > > > > > I didn't quite understand this. In classic topics, we have
> maps
> > > for
> > > > > > ongoing
> > > > > > > > transactions which remove state when the transaction is
> completed
> > > > and
> > > > > > an
> > > > > > > > aborted transactions index which is retained for much longer.
> > > Once
> > > > the
> > > > > > > > transaction is completed, the coordinator is no longer
> involved
> > > in
> > > > > > > > maintaining this partition side state, and it is subject to
> > > > compaction
> > > > > > etc.
> > > > > > > > Looking back at the outline provided above, I didn't see much
> > > > about the
> > > > > > > > fetch path, so maybe that could be expanded a bit further. I
> saw
> > > > the
> > > > > > > > following in a response:
> > > > > > > > > When the broker constructs a fully valid local segment,
> all the
> > > > > > necessary
> > > > > > > > control batches will be inserted and indices, including the
> > > > transaction
> > > > > > > > index will be built to serve FetchRequests exactly as they
> are
> > > > today.
> > > > > > > >
> > > > > > > > Based on this, it seems like we need to retain the
> information
> > > > about
> > > > > > > > aborted txns for longer.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> ivan@ivanyu.me>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Justine and all,
> > > > > > > > >
> > > > > > > > > Thank you for your questions!
> > > > > > > > >
> > > > > > > > > > JO 1. >Since a transaction could be uniquely identified
> with
> > > > > > producer ID
> > > > > > > > > > and epoch, the positive result of this check could be
> cached
> > > > > > locally
> > > > > > > > > > Are we saying that only new transaction version 2
> > > transactions
> > > > can
> > > > > > be
> > > > > > > > > used
> > > > > > > > > > here? If not, we can't uniquely identify transactions
> with
> > > > > > producer id +
> > > > > > > > > > epoch
> > > > > > > > >
> > > > > > > > > You’re right that we (probably unintentionally) focused
> only on
> > > > > > version 2.
> > > > > > > > > We can either limit the support to version 2 or consider
> using
> > > > some
> > > > > > > > > surrogates to support version 1.
> > > > > > > > >
> > > > > > > > > > JO 2. >The batch coordinator does the final transactional
> > > > checks
> > > > > > of the
> > > > > > > > > > batches. This procedure would output the same errors
> like the
> > > > > > partition
> > > > > > > > > > leader in classic topics would do.
> > > > > > > > > > Can you expand on what these checks are? Would you be
> > > checking
> > > > if
> > > > > > the
> > > > > > > > > > transaction was still ongoing for example?* *
> > > > > > > > >
> > > > > > > > > Yes, the producer epoch, that the transaction is ongoing,
> and
> > > of
> > > > > > course
> > > > > > > > > the normal idempotence checks. What the partition leader
> in the
> > > > > > classic
> > > > > > > > > topics does before appending a batch to the local log
> (e.g. in
> > > > > > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless,
> we
> > > > > > unfortunately
> > > > > > > > > cannot do these checks before appending the data to the WAL
> > > > segment
> > > > > > and
> > > > > > > > > uploading it, but we can “tombstone” these batches in the
> batch
> > > > > > coordinator
> > > > > > > > > during the final commit.
> > > > > > > > >
> > > > > > > > > > Is there state about ongoing
> > > > > > > > > > transactions in the batch coordinator? I see some other
> state
> > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > the End transaction section, but it's not super clear
> what
> > > > state is
> > > > > > > > > stored
> > > > > > > > > > and when it is stored.
> > > > > > > > >
> > > > > > > > > Right, this should have been more explicit. As the
> partition
> > > > leader
> > > > > > tracks
> > > > > > > > > ongoing transactions for classic topics, the batch
> coordinator
> > > > has
> > > > > > to as
> > > > > > > > > well. So when a transaction starts and ends, the
> transaction
> > > > > > coordinator
> > > > > > > > > must inform the batch coordinator about this.
> > > > > > > > >
> > > > > > > > > > JO 3. I didn't see anything about maintaining LSO --
> perhaps
> > > > that
> > > > > > would
> > > > > > > > > be
> > > > > > > > > > stored in the batch coordinator?
> > > > > > > > >
> > > > > > > > > Yes. This could be deduced from the committed batches and
> other
> > > > > > > > > information, but for the sake of performance we’d better
> store
> > > it
> > > > > > > > > explicitly.
> > > > > > > > >
> > > > > > > > > > JO 4. Are there any thoughts about how long transactional
> > > > state is
> > > > > > > > > > maintained in the batch coordinator and how it will be
> > > cleaned
> > > > up?
> > > > > > > > >
> > > > > > > > > As we understand this, the partition leader in classic
> topics
> > > > forgets
> > > > > > > > > about a transaction once it’s replicated (HWM overpasses
> it).
> > > The
> > > > > > > > > transaction coordinator acts like the main guardian,
> allowing
> > > > > > partition
> > > > > > > > > leaders to do this safely. Please correct me if this is
> wrong.
> > > We
> > > > > > think
> > > > > > > > > about relying on this with the batch coordinator and
> delete the
> > > > > > information
> > > > > > > > > about a transaction once it’s finished (as there’s no
> > > replication
> > > > > > and HWM
> > > > > > > > > advances immediately).
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ivan
> > > > > > > > >
> > > > > > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > > > > > Hey folks,
> > > > > > > > > >
> > > > > > > > > > Excited to see some updates related to transactions!
> > > > > > > > > >
> > > > > > > > > > I had a few questions.
> > > > > > > > > >
> > > > > > > > > > JO 1. >Since a transaction could be uniquely identified
> with
> > > > > > producer ID
> > > > > > > > > > and epoch, the positive result of this check could be
> cached
> > > > > > locally
> > > > > > > > > > Are we saying that only new transaction version 2
> > > transactions
> > > > can
> > > > > > be
> > > > > > > > > used
> > > > > > > > > > here? If not, we can't uniquely identify transactions
> with
> > > > > > producer id +
> > > > > > > > > > epoch
> > > > > > > > > >
> > > > > > > > > > JO 2. >The batch coordinator does the final transactional
> > > > checks
> > > > > > of the
> > > > > > > > > > batches. This procedure would output the same errors
> like the
> > > > > > partition
> > > > > > > > > > leader in classic topics would do.
> > > > > > > > > > Can you expand on what these checks are? Would you be
> > > checking
> > > > if
> > > > > > the
> > > > > > > > > > transaction was still ongoing for example? Is there state
> > > about
> > > > > > ongoing
> > > > > > > > > > transactions in the batch coordinator? I see some other
> state
> > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > the End transaction section, but it's not super clear
> what
> > > > state is
> > > > > > > > > stored
> > > > > > > > > > and when it is stored.
> > > > > > > > > >
> > > > > > > > > > JO 3. I didn't see anything about maintaining LSO --
> perhaps
> > > > that
> > > > > > would
> > > > > > > > > be
> > > > > > > > > > stored in the batch coordinator?
> > > > > > > > > >
> > > > > > > > > > JO 4. Are there any thoughts about how long transactional
> > > > state is
> > > > > > > > > > maintained in the batch coordinator and how it will be
> > > cleaned
> > > > up?
> > > > > > > > > >
> > > > > > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > <ju...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Greg and Ivan,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the update. A few comments.
> > > > > > > > > > >
> > > > > > > > > > > JR 10. "Consumer fetches are now served from local
> > > segments,
> > > > > > making
> > > > > > > > > use of
> > > > > > > > > > > the
> > > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > > > functionality
> > > > > > > > > already
> > > > > > > > > > > built into classic topics."
> > > > > > > > > > > JR 10.1 Does the broker build the producer state for
> each
> > > > > > partition in
> > > > > > > > > > > diskless topics?
> > > > > > > > > > > JR 10.2 For transactional data, the consumer fetches
> need
> > > to
> > > > know
> > > > > > > > > aborted
> > > > > > > > > > > records. How is that achieved?
> > > > > > > > > > >
> > > > > > > > > > > JR 11. "The batch coordinator saves that the
> transaction is
> > > > > > finished
> > > > > > > > > and
> > > > > > > > > > > also inserts the control batches in the corresponding
> logs
> > > > of the
> > > > > > > > > involved
> > > > > > > > > > > Diskless topics. This happens only on the metadata
> level,
> > > no
> > > > > > actual
> > > > > > > > > control
> > > > > > > > > > > batches are written to any file. "
> > > > > > > > > > > A fetch response could include multiple transactional
> > > > batches.
> > > > > > How
> > > > > > > > > does the
> > > > > > > > > > > broker obtain the information about the ending control
> > > batch
> > > > for
> > > > > > each
> > > > > > > > > > > batch? Does that mean that a fetch response needs to be
> > > > built by
> > > > > > > > > > > stitching record batches and generated control batches
> > > > together?
> > > > > > > > > > >
> > > > > > > > > > > JR 12. Queues: Is there still a share partition leader
> that
> > > > all
> > > > > > > > > consumers
> > > > > > > > > > > are routed to?
> > > > > > > > > > >
> > > > > > > > > > > JR 13. "Should the KIPs be modified to include this or
> it's
> > > > too
> > > > > > > > > > > implementation-focused?" It would be useful to include
> > > enough
> > > > > > details
> > > > > > > > > to
> > > > > > > > > > > understand correctness and performance impact.
> > > > > > > > > > >
> > > > > > > > > > > HC5. Henry has a valid point. Requests from a given
> > > producer
> > > > > > contain a
> > > > > > > > > > > sequence number, which is ordered. If a producer sends
> > > every
> > > > > > Produce
> > > > > > > > > > > request to an arbitrary broker, those requests could
> reach
> > > > the
> > > > > > batch
> > > > > > > > > > > coordinator in different order and lead to rejection
> of the
> > > > > > produce
> > > > > > > > > > > requests.
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > > ivan@ivanyu.me>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > We have also thought in a bit more details about
> > > > transactions
> > > > > > and
> > > > > > > > > queues,
> > > > > > > > > > > > here's the plan.
> > > > > > > > > > > >
> > > > > > > > > > > > *Transactions*
> > > > > > > > > > > >
> > > > > > > > > > > > The support for transactions in *classic topics* is
> based
> > > > on
> > > > > > precise
> > > > > > > > > > > > interactions between three actors: clients (mostly
> > > > producers,
> > > > > > but
> > > > > > > > > also
> > > > > > > > > > > > consumers), brokers (ReplicaManager and other
> classes),
> > > and
> > > > > > > > > transaction
> > > > > > > > > > > > coordinators. Brokers also run partition leaders with
> > > their
> > > > > > local
> > > > > > > > > state
> > > > > > > > > > > > (ProducerStateManager and others).
> > > > > > > > > > > >
> > > > > > > > > > > > The high level (some details skipped) workflow is the
> > > > > > following.
> > > > > > > > > When a
> > > > > > > > > > > > transactional Produce request is received by the
> broker:
> > > > > > > > > > > > 1. For each partition, the partition leader checks
> if a
> > > > > > non-empty
> > > > > > > > > > > > transaction is running for this partition. This is
> done
> > > > using
> > > > > > its
> > > > > > > > > local
> > > > > > > > > > > > state derived from the log metadata
> > > (ProducerStateManager,
> > > > > > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > > > > > 2. The transaction coordinator is informed about all
> the
> > > > > > partitions
> > > > > > > > > that
> > > > > > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > > > > > 3. The partition leaders do additional transactional
> > > > checks.
> > > > > > > > > > > > 4. The partition leaders append the transactional
> data to
> > > > > > their logs
> > > > > > > > > and
> > > > > > > > > > > > update some of their state (for example, log the fact
> > > that
> > > > the
> > > > > > > > > > > transaction
> > > > > > > > > > > > is running for the partition and its first offset).
> > > > > > > > > > > >
> > > > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > > > 1. The producer contacts the transaction coordinator
> > > > directly
> > > > > > with
> > > > > > > > > > > > EndTxnRequest.
> > > > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT
> or
> > > > > > > > > PREPARE_ABORT to
> > > > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > > > 3. The transaction coordinator sends
> > > > WriteTxnMarkersRequest to
> > > > > > the
> > > > > > > > > > > leaders
> > > > > > > > > > > > of the involved partitions.
> > > > > > > > > > > > 4. The partition leaders write the transaction
> markers to
> > > > > > their logs
> > > > > > > > > and
> > > > > > > > > > > > respond to the coordinator.
> > > > > > > > > > > > 5. The coordinator writes the final transaction state
> > > > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > > > >
> > > > > > > > > > > > In classic topics, partitions have leaders and lots
> of
> > > > > > important
> > > > > > > > > state
> > > > > > > > > > > > necessary for supporting this workflow is local. The
> main
> > > > > > challenge
> > > > > > > > > in
> > > > > > > > > > > > mapping this to Diskless comes from the fact there
> are no
> > > > > > partition
> > > > > > > > > > > > leaders, so the corresponding pieces of state need
> to be
> > > > > > globalized
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > batch coordinator. We are already doing this to
> support
> > > > > > idempotent
> > > > > > > > > > > produce.
> > > > > > > > > > > >
> > > > > > > > > > > > The high level workflow for *diskless topics* would
> look
> > > > very
> > > > > > > > > similar:
> > > > > > > > > > > > 1. For each partition, the broker checks if a
> non-empty
> > > > > > transaction
> > > > > > > > > is
> > > > > > > > > > > > running for this partition. In contrast to classic
> > > topics,
> > > > > > this is
> > > > > > > > > > > checked
> > > > > > > > > > > > against the batch coordinator with a single RPC.
> Since a
> > > > > > transaction
> > > > > > > > > > > could
> > > > > > > > > > > > be uniquely identified with producer ID and epoch,
> the
> > > > positive
> > > > > > > > > result of
> > > > > > > > > > > > this check could be cached locally (for the double
> > > > configured
> > > > > > > > > duration
> > > > > > > > > > > of a
> > > > > > > > > > > > transaction, for example).
> > > > > > > > > > > > 2. The same: The transaction coordinator is informed
> > > about
> > > > all
> > > > > > the
> > > > > > > > > > > > partitions that aren’t part of the transaction to
> include
> > > > them.
> > > > > > > > > > > > 3. No transactional checks are done on the broker
> side.
> > > > > > > > > > > > 4. The broker appends the transactional data to the
> > > current
> > > > > > shared
> > > > > > > > > WAL
> > > > > > > > > > > > segment. It doesn’t update any transaction-related
> state
> > > > for
> > > > > > Diskless
> > > > > > > > > > > > topics, because it doesn’t have any.
> > > > > > > > > > > > 5. The WAL segment is committed to the batch
> coordinator
> > > > like
> > > > > > in the
> > > > > > > > > > > > normal produce flow.
> > > > > > > > > > > > 6. The batch coordinator does the final transactional
> > > > checks
> > > > > > of the
> > > > > > > > > > > > batches. This procedure would output the same errors
> like
> > > > the
> > > > > > > > > partition
> > > > > > > > > > > > leader in classic topics would do. I.e. some batches
> > > could
> > > > be
> > > > > > > > > rejected.
> > > > > > > > > > > > This means, there will potentially be garbage in the
> WAL
> > > > > > segment
> > > > > > > > > file in
> > > > > > > > > > > > case of transactional errors. This is preferable to
> doing
> > > > more
> > > > > > > > > network
> > > > > > > > > > > > round trips, especially considering the WAL segments
> will
> > > > be
> > > > > > > > > relatively
> > > > > > > > > > > > short-living (see the Greg's update above).
> > > > > > > > > > > >
> > > > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > > > 1. The producer contacts the transaction coordinator
> > > > directly
> > > > > > with
> > > > > > > > > > > > EndTxnRequest.
> > > > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT
> or
> > > > > > > > > PREPARE_ABORT to
> > > > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > > > 3. *[NEW]* The transaction coordinator informs the
> batch
> > > > > > coordinator
> > > > > > > > > that
> > > > > > > > > > > > the transaction is finished.
> > > > > > > > > > > > 4. *[NEW]* The batch coordinator saves that the
> > > > transaction is
> > > > > > > > > finished
> > > > > > > > > > > > and also inserts the control batches in the
> corresponding
> > > > logs
> > > > > > of the
> > > > > > > > > > > > involved Diskless topics. This happens only on the
> > > metadata
> > > > > > level, no
> > > > > > > > > > > > actual control batches are written to any file. They
> will
> > > > be
> > > > > > > > > dynamically
> > > > > > > > > > > > created on Fetch and other read operations. We could
> > > > > > technically
> > > > > > > > > write
> > > > > > > > > > > > these control batches for real, but this would mean
> extra
> > > > > > produce
> > > > > > > > > > > latency,
> > > > > > > > > > > > so it's better just to mark them in the batch
> coordinator
> > > > and
> > > > > > save
> > > > > > > > > these
> > > > > > > > > > > > milliseconds.
> > > > > > > > > > > > 5. The transaction coordinator sends
> > > > WriteTxnMarkersRequest to
> > > > > > the
> > > > > > > > > > > leaders
> > > > > > > > > > > > of the involved partitions. – Now only to classic
> topics
> > > > now.
> > > > > > > > > > > > 6. The partition leaders of classic topics write the
> > > > > > transaction
> > > > > > > > > markers
> > > > > > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > > > > > 7. The coordinator writes the final transaction state
> > > > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > > > >
> > > > > > > > > > > > Compared to the non-transactional produce flow, we
> get:
> > > > > > > > > > > > 1. An extra network round trip between brokers and
> the
> > > > batch
> > > > > > > > > coordinator
> > > > > > > > > > > > when a new partition appear in the transaction. To
> > > > mitigate the
> > > > > > > > > impact of
> > > > > > > > > > > > them:
> > > > > > > > > > > >   - The results will be cached.
> > > > > > > > > > > >   - The calls for multiple partitions in one Produce
> > > > request
> > > > > > will be
> > > > > > > > > > > > grouped.
> > > > > > > > > > > >   - The batch coordinator should be optimized for
> fast
> > > > > > response to
> > > > > > > > > these
> > > > > > > > > > > > RPCs.
> > > > > > > > > > > >   - The fact that a single producer normally will
> > > > communicate
> > > > > > with a
> > > > > > > > > > > > single broker for the duration of the transaction
> further
> > > > > > reduces the
> > > > > > > > > > > > expected number of round trips.
> > > > > > > > > > > > 2. An extra round trip between the transaction
> > > coordinator
> > > > and
> > > > > > batch
> > > > > > > > > > > > coordinator when a transaction is finished.
> > > > > > > > > > > >
> > > > > > > > > > > > With this proposal, transactions will also be able to
> > > span
> > > > both
> > > > > > > > > classic
> > > > > > > > > > > > and Diskless topics.
> > > > > > > > > > > >
> > > > > > > > > > > > *Queues*
> > > > > > > > > > > >
> > > > > > > > > > > > The share group coordination and management is a
> side job
> > > > that
> > > > > > > > > doesn't
> > > > > > > > > > > > interfere with the topic itself (leadership,
> replicas,
> > > > physical
> > > > > > > > > storage
> > > > > > > > > > > of
> > > > > > > > > > > > records, etc.) and non-queue producers and consumers
> > > > (Fetch and
> > > > > > > > > Produce
> > > > > > > > > > > > RPCs, consumer group-related RPCs are not affected.)
> We
> > > > don't
> > > > > > see any
> > > > > > > > > > > > reason why we can't make Diskless topics compatible
> with
> > > > share
> > > > > > > > > groups the
> > > > > > > > > > > > same way as classic topics are. Even on the code
> level,
> > > we
> > > > > > don't
> > > > > > > > > expect
> > > > > > > > > > > any
> > > > > > > > > > > > serious refactoring: the same reading routines are
> used
> > > > that
> > > > > > are
> > > > > > > > > used for
> > > > > > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Should the KIPs be modified to include this or it's
> too
> > > > > > > > > > > > implementation-focused?
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Ivan
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you all for your questions and design input
> on
> > > > > > KIP-1150.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a
> new
> > > > > > design. To
> > > > > > > > > > > > summarize
> > > > > > > > > > > > > the changes:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. The design prioritizes integrating with the
> existing
> > > > > > KIP-405
> > > > > > > > > Tiered
> > > > > > > > > > > > > Storage interfaces, permitting data produced to a
> > > > Diskless
> > > > > > topic
> > > > > > > > > to be
> > > > > > > > > > > > > moved to tiered storage.
> > > > > > > > > > > > > This lowers the scalability requirements for the
> Batch
> > > > > > Coordinator
> > > > > > > > > > > > > component, and allows Diskless to compose with
> Tiered
> > > > Storage
> > > > > > > > > plugin
> > > > > > > > > > > > > features such as encryption and alternative data
> > > formats.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. Consumer fetches are now served from local
> segments,
> > > > > > making use
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > indexes, page cache, request purgatory, and
> zero-copy
> > > > > > functionality
> > > > > > > > > > > > already
> > > > > > > > > > > > > built into classic topics.
> > > > > > > > > > > > > However, local segments are now considered cache
> > > > elements,
> > > > > > do not
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > be durably stored, and can be built without
> contacting
> > > > any
> > > > > > other
> > > > > > > > > > > > replicas.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. The design has been simplified substantially, by
> > > > removing
> > > > > > the
> > > > > > > > > > > previous
> > > > > > > > > > > > > Diskless consume flow, distributed cache
> component, and
> > > > > > "object
> > > > > > > > > > > > > compaction/merging" step.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The design maintains leaderless produces as
> enabled by
> > > > the
> > > > > > Batch
> > > > > > > > > > > > > Coordinator, and the same latency profiles as the
> > > earlier
> > > > > > design,
> > > > > > > > > while
> > > > > > > > > > > > > being simpler and integrating better into the
> existing
> > > > > > ecosystem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks, and we are eager to hear your feedback on
> the
> > > new
> > > > > > design.
> > > > > > > > > > > > > Greg Harris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > <ju...@confluent.io.invalid>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi, Jan,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For me, the main gap of KIP-1150 is the support
> of
> > > all
> > > > > > existing
> > > > > > > > > > > client
> > > > > > > > > > > > > > APIs. Currently, there is no design for
> supporting
> > > APIs
> > > > > > like
> > > > > > > > > > > > transactions
> > > > > > > > > > > > > > and queues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Would it be a good time to ask for the current
> > > > status of
> > > > > > this
> > > > > > > > > KIP?
> > > > > > > > > > > I
> > > > > > > > > > > > > > > haven't seen much activity here for the past 2
> > > > months,
> > > > > > the
> > > > > > > > > vote got
> > > > > > > > > > > > > > vetoed
> > > > > > > > > > > > > > > but I think the pending questions have been
> > > answered
> > > > > > since
> > > > > > > > > then.
> > > > > > > > > > > > KIP-1183
> > > > > > > > > > > > > > > (AutoMQ's proposal) also didn't have any
> activity
> > > > since
> > > > > > May.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real
> > > choices
> > > > > > that can
> > > > > > > > > be
> > > > > > > > > > > > > > > made, with a coordinator-based approach being
> by
> > > far
> > > > the
> > > > > > > > > dominant
> > > > > > > > > > > one
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > it comes to market adoption - but all these are
> > > > > > standalone
> > > > > > > > > > > products.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm a big fan of both approaches, but would
> hate to
> > > > see a
> > > > > > > > > stall. So
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Maybe it's time to start another vote? Colin
> > > McCabe -
> > > > > > have your
> > > > > > > > > > > > questions
> > > > > > > > > > > > > > > been answered? If not, is there anything I can
> do
> > > to
> > > > > > help? I'm
> > > > > > > > > > > deeply
> > > > > > > > > > > > > > > familiar with both architectures and have
> written
> > > > about
> > > > > > both?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > > Jan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > Kozlovski
> > > > <
> > > > > > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > a) group all the KIP email threads in the
> main
> > > one
> > > > > > (just a
> > > > > > > > > bunch
> > > > > > > > > > > of
> > > > > > > > > > > > > > links
> > > > > > > > > > > > > > > > to everything)
> > > > > > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It's a bit hard to track it all - for
> example, I
> > > > was
> > > > > > > > > searching
> > > > > > > > > > > for
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > discuss thread for KIP-1165 for a while; As
> far
> > > as
> > > > I
> > > > > > can
> > > > > > > > > tell, it
> > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > exist yet.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Since the KIPs are published (by virtue of
> having
> > > > the
> > > > > > root
> > > > > > > > > KIP be
> > > > > > > > > > > > > > > > published, having a DISCUSS thread and links
> to
> > > > > > sub-KIPs
> > > > > > > > > where
> > > > > > > > > > > were
> > > > > > > > > > > > > > aimed
> > > > > > > > > > > > > > > > to move the discussion towards), I think it
> would
> > > > be
> > > > > > good to
> > > > > > > > > > > create
> > > > > > > > > > > > > > > DISCUSS
> > > > > > > > > > > > > > > > threads for them all.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Stan
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We want to start a new KIP discussion about
> > > > > > introducing a
> > > > > > > > > new
> > > > > > > > > > > > type of
> > > > > > > > > > > > > > > > > topics that would make use of Object
> Storage as
> > > > the
> > > > > > primary
> > > > > > > > > > > > source of
> > > > > > > > > > > > > > > > > storage. However, as this KIP is big we
> decided
> > > > to
> > > > > > split it
> > > > > > > > > > > into
> > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > that aims to discuss if Apache Kafka
> should aim
> > > > to
> > > > > > have
> > > > > > > > > this
> > > > > > > > > > > > type of
> > > > > > > > > > > > > > > > > feature at all. This KIP doesn't go onto
> > > details
> > > > on
> > > > > > how to
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > This follows the same approach used when we
> > > > discussed
> > > > > > > > > KRaft.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > But as we know that it is sometimes really
> hard
> > > > to
> > > > > > discuss
> > > > > > > > > on
> > > > > > > > > > > > that
> > > > > > > > > > > > > > meta
> > > > > > > > > > > > > > > > > level, we also created several sub-kips
> (linked
> > > > in
> > > > > > > > > KIP-1150)
> > > > > > > > > > > that
> > > > > > > > > > > > > > offer
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS
> > > > threads
> > > > > > for
> > > > > > > > > each
> > > > > > > > > > > > type of
> > > > > > > > > > > > > > > > > concern and keep this one to discuss
> whether
> > > > Apache
> > > > > > Kafka
> > > > > > > > > wants
> > > > > > > > > > > > to
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks in advance on behalf of all the
> authors
> > > of
> > > > > > this KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 |
> > > > aiven.io
> > > > > > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > Valtonen,
> > > > > > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Luke Chen <sh...@gmail.com>]

Hi Ivan,

> After some evolution of the original proposal, leaderless is now limited.
We only need to be able to accept Produce requests on more than one broker
to eliminate the cross-AZ costs for producers. Do I get it right that you
propose to get rid of this? Or do I misunderstand?

Yes, that's what I meant.

About the "batch coordinator approach" VS "coordinator-less approach", I'd
like to clarify one thing: If I remember correctly, the reason we proposed
the coordinator-less approach is because we encountered some bottleneck
when designing the batch coordinator approach to support kafka queue
feature. If that's the case, I'm wondering if the "batch coordinator
approach" is still a valid option here? Or do you already have solutions to
support the kafka queue feature in batch coordinator approach?

Thank you,
Luke

On Fri, Oct 24, 2025 at 2:12 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hello Jack, Jun, Luke, and all!
>
> Thank you for your messages.
>
> Let me first address some of Jun’s comments.
>
> > First, it degrades the durability.
> > For each partition, now there are two files being actively written at a
> > given point of time, one for the data and another for the metadata.
> > Flushing each file requires a separate IO. If the disk has 1K IOPS and we
> > have 5K partitions in a broker, currently we can afford to flush each
> > partition every 5 seconds, achieving an RPO of 5 seconds. If we double
> the
> > number of files per partition, we can only flush each partition every 10
> > seconds, which makes RPO twice as bad.
>
> As I understand, you’re speaking about locally materialized segments. They
> will indeed consume some IOPS. See them as a cache that could always be
> restored from the remote storage. While it’s not ideal, it's still OK to
> lose data in them due to a machine crash, for example. Because of this, we
> can avoid explicit flushing on local materialized segments at all and let
> the file system and page cache figure out when to flush optimally. This
> would not eliminate the extra IOPS, but should reduce it dramatically,
> depending on throughput for each partition. We, of course, continue
> flushing the metadata segments as before.
>
> It’s worth making a note on caching. I think nobody will disagree that
> doing direct reads from remote storage every time a batch is requested by a
> consumer will not be practical neither from the performance nor from the
> economy point of view. We need a way to keep the number of GET requests
> down. There are multiple options, for example:
>   1. Rack-aware distributed in-memory caching.
>   2. Local in-memory caching. Comes with less network chattiness and works
> well if we have more or less stable brokers to consume from.
>   3. Materialization of diskless logs on local disk. Way lower impact on
> RAM and also requires stable brokers for consumption (using just assigned
> replicas will probably work well).
>
> Materialization is one of possible options, but we can choose another one.
> However, we will have this dilemma regardless of whether we have an
> explicit coordinator or we go “coordinator-less”.
>
> > Second, if we ever need this
> > metadata somewhere else, say in the WAL file manager, the consumer needs
> to
> > subscribe to every partition in the cluster, which is inefficient. The
> > actual benefit of this approach is also questionable. On the surface, it
> > might seem that we could reduce the number of lines that need to be
> changed
> > for this KIP. However, the changes are quite intrusive to the classic
> > partition's code path and will probably make the code base harder to
> > maintain in the long run. I like the original approach based on the batch
> > coordinator much better than this one. We could probably refactor the
> > producer state code so that it could be reused in the batch coordinator.
>
> It’s hard to disagree with this. The explicit coordinator is more a side
> thing, while coordinator-less approach is more about extending
> ReplicaManager, UnifiedLog and others substantially.
>
> > Thanks for addressing the concerns on the number of RPCs in the produce
> > path. I agree that with the metadata crafting mechanism, we could
> mitigate
> > the PRC problem. However, since we now require the metadata to be
> > collocated with the data on the same set of brokers, it's weird that they
> > are now managed by different mechanisms. The data assignment now uses the
> > metadata crafting mechanism, but the metadata is stored in the classic
> > partition using its own assignment strategy. It will be complicated to
> keep
> > them collocated.
>
> I would like to note that the metadata crafting is needed only to tell
> producers which brokers they should send Produce requests to, but data (as
> in “locally materialized log”) is located on partition replicas, i.e.
> automatically co-located with metadata.
>
> As a side note, it would probably be better that instead of implicitly
> crafting partition metadata, we extend the metadata protocol so that for
> diskless partitions we return not only the leader and replicas, but also
> some “recommended produce brokers”, selected for optimal performance and
> costs. Producers will pick ones in their racks.
>
> > I am also concerned about the removal of the object compaction/merging
> > step.
>
> We should have mentioned this explicitly, but this step, in fact, remains
> in the form of segments offloading to tiered storage. When we assemble a
> segment and hand it over to RemoteLogManager, we’re effectively doing
> metadata compaction: replacing a big number of pieces of metadata about
> individual batches with a single record in __remote_log_metadata.
>
> We could create a Diskless-specific merging mechanism instead if needed.
> It’s rather easy with the explicit coordinator approach. With the
> coordinator-less approach, this would probably be a bit more tricky
> (rewriting the tail of the log by the leader + replicating this change
> reliably).
>
> > I see a tendency toward primarily optimizing for the fewest code changes
> in
> > the KIP. Instead, our primary goal should be a clean design that can last
> > for the long term.
>
> Yes, totally agree.
>
>
>
> Luke,
> > I'm wondering if the complexity of designing txn and queue is because of
> > leaderless cluster, do you think it will be simpler if we only focus on
> the
> > "diskless" design to handle object compaction/merging to/from the remote
> > storage to save the cross-AZ cost?
>
> After some evolution of the original proposal, leaderless is now limited.
> We only need to be able to accept Produce requests on more than one broker
> to eliminate the cross-AZ costs for producers. Do I get it right that you
> propose to get rid of this? Or do I misunderstand?
>
>
>
> Let’s now look at this problem from a higher level, as Jack proposed. As
> it was said, the big choice we need to make is whether we 1) create an
> explicit batch coordinator; or 2) go for the coordinator-less approach,
> where each diskless partition is managed by its leader as in classic topics.
>
> If we try to compare the two approaches:
>
> Pluggability:
> - Explicit coordinator: Possible. For example, some setups may benefit
> from batch metadata being stored in a cloud database (such as AWS DynamoDB
> or GCP Spanner).
> - Coordinator-less: Impossible.
>
> Scalability and fault tolerance:
> - Explicit coordinator: Depends on the implementation and it’s also
> necessary to actively work for it.
> - Coordinator-less: Closer to classic Kafka topics. Scaling is done by
> partition placement, partitions could fail independently.
>
> Separation of concerns:
> - Explicit coordinator: Very good. Diskless remains more independent from
> classic topics in terms of code and workflows. For example, the
> above-mentioned non-tiered storage metadata compaction mechanism could be
> relatively simply implemented with it. As a flip side of this, some
> workflows (e.g. transactions) will have to be adapted.
> - Coordinator-less: Less so. It leans to the opposite: bringing diskless
> closer to classic topics. Some code paths and workflows could be more
> straightforwardly reused, but they will inevitably have to be adapted to
> accommodate both topic types as also discussed.
>
> Cloud-nativeness. This is a vague concept, also related to the previous,
> but let’s try:
> - Explicit coordinator: Storing and processing metadata separately makes
> it easier for brokers to take different roles, be purely stateless if
> needed, etc.
> - Coordinator-less: Less so. Something could be achieved with creative
> partition placement, but not much.
>
> Both seem to have their pros and cons. However, answering Jack’s question,
> the explicit coordinator approach may indeed lead to a more flexible design.
>
>
> The purpose of this deviation in the discussion was to receive a
> preliminary community evaluation of the coordinator-less approach without
> taking on the task of writing a separate KIP and fitting it in the system
> of KIP-1150 and its children. We’re open to stopping it and getting back to
> working out the coordinator design if the community doesn’t favor the
> proposed approach.
>
> Best,
> Ivan and Diskless team
>
> On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> > Hi Ivan,
> >
> > As Jun pointed out, the updated design seems to have some shortcomings
> > although it simplifies the implementation.
> >
> > I'm wondering if the complexity of designing txn and queue is because of
> > leaderless cluster, do you think it will be simpler if we only focus on
> the
> > "diskless" design to handle object compaction/merging to/from the remote
> > storage to save the cross-AZ cost?
> >
> >
> > Thank you,
> > Luke
> >
> > On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Ivan,
> > >
> > > Thanks for the explanation.
> > >
> > > "we write the reference to the WAL file with the batch data"
> > >
> > > I understand the approach now, but I think it is a hacky one. There are
> > > multiple short comings with this design. First, it degrades the
> durability.
> > > For each partition, now there are two files being actively written at a
> > > given point of time, one for the data and another for the metadata.
> > > Flushing each file requires a separate IO. If the disk has 1K IOPS and
> we
> > > have 5K partitions in a broker, currently we can afford to flush each
> > > partition every 5 seconds, achieving an RPO of 5 seconds. If we double
> the
> > > number of files per partition, we can only flush each partition every
> 10
> > > seconds, which makes RPO twice as bad. Second, if we ever need this
> > > metadata somewhere else, say in the WAL file manager, the consumer
> needs to
> > > subscribe to every partition in the cluster, which is inefficient. The
> > > actual benefit of this approach is also questionable. On the surface,
> it
> > > might seem that we could reduce the number of lines that need to be
> changed
> > > for this KIP. However, the changes are quite intrusive to the classic
> > > partition's code path and will probably make the code base harder to
> > > maintain in the long run. I like the original approach based on the
> batch
> > > coordinator much better than this one. We could probably refactor the
> > > producer state code so that it could be reused in the batch
> coordinator.
> > >
> > > Thanks for addressing the concerns on the number of RPCs in the produce
> > > path. I agree that with the metadata crafting mechanism, we could
> mitigate
> > > the PRC problem. However, since we now require the metadata to be
> > > collocated with the data on the same set of brokers, it's weird that
> they
> > > are now managed by different mechanisms. The data assignment now uses
> the
> > > metadata crafting mechanism, but the metadata is stored in the classic
> > > partition using its own assignment strategy. It will be complicated to
> keep
> > > them collocated.
> > >
> > > I am also concerned about the removal of the object compaction/merging
> > > step. My first concern is on the amount of metadata that need to be
> kept.
> > > Without object compcation, the metadata generated in the produce path
> can
> > > only be deleted after remote tiering kicks in. Let's say for every
> 250ms we
> > > produce 100 byte of metadata per partition. Let's say remoting tiering
> > > kicks in after 5 hours. In a cluster with 100K partitions, we need to
> keep
> > > about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
> > > signficant. A second concern is on performance. Every time we need to
> > > rebuild the caching data, we need to read a bunch of small objects
> from S3,
> > > slowing down the building process. If a consumer happens to need such
> data,
> > > it could slow down the application.
> > >
> > > I see a tendency toward primarily optimizing for the fewest code
> changes in
> > > the KIP. Instead, our primary goal should be a clean design that can
> last
> > > for the long term.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thank you for your message. I’m sorry that I failed to clearly
> explain
> > > the
> > > > idea. Let me try to fix this.
> > > >
> > > > > Does each partition now have a metadata partition and a separate
> data
> > > > > partition? If so, I am concerned that it essentially doubles the
> number
> > > > of
> > > > > partitions, which impacts the number of open file descriptors and
> the
> > > > > required IOPS, and so on. It also seems wasteful to have a separate
> > > > > partition just to store the metadata. It's as if we are creating an
> > > > > internal topic with an unbounded number of partitions.
> > > >
> > > > No. There will be only one physical partition per diskless
> partition. Let
> > > > me explain this with an example. Let’s say we have a diskless
> partition
> > > > topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce
> some
> > > > batches to this partition. The content of the segment file will be
> > > > something like this (for each batch):
> > > >
> > > > BaseOffset: 00000000000000000000 (like in classic)
> > > > Length: 123456 (like in classic)
> > > > PartitionLeaderEpoch: like in classic
> > > > Magic: like in classic
> > > > CRC: like in classic
> > > > Attributes: like in classic
> > > > LastOffsetDelta: like in classic
> > > > BaseTimestamp: like in classic
> > > > MaxTimestamp: like in classic
> > > > ProducerId: like in classic
> > > > ProducerEpoch: like in classic
> > > > BaseSequence: like in classic
> > > > RecordsCount: like in classic
> > > > Records:
> > > >   path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> > > > 123456
> > > >
> > > > It looks very much like classic log entries. The only difference is
> that
> > > > instead of writing real Records, we write the reference to the WAL
> file
> > > > with the batch data (I guess we need only the name and the byte
> offset,
> > > > because the byte length is the standard field above). Otherwise,
> it’s a
> > > > normal Kafka log with the leader and replicas.
> > > >
> > > > So we have as many partitions for diskless as for classic. As of open
> > > file
> > > > descriptors, let’s proceed to the following:
> > > >
> > > > > Are the metadata and
> > > > > the data for the same partition always collocated on the same
> broker?
> > > If
> > > > > so, how do we enforce that when replicas are reassigned?
> > > >
> > > > The source of truth for the data is still in WAL files on object
> storage.
> > > > The source of truth for the metadata is in segment files on the
> brokers
> > > in
> > > > the replica set. Two new mechanisms are planned, both independent of
> this
> > > > new proposal, but I want to present them to give the idea that only a
> > > > limited amount of data files will be operated locally:
> > > > - We want to assemble batches into segment files and offload them to
> > > > tiered storage in order to prevent the unbounded growth of batch
> > > metadata.
> > > > For this, we need to open only  a few file descriptors (for the
> segment
> > > > file itself + the necessary indexes) before the segment is fully
> written
> > > > and handed over to RemoteLogManager.
> > > > - We want to assemble local segment files for caching purposes as
> well,
> > > > i.e. to speed up fetching. This will not materialize the full
> content of
> > > > the log, but only the hot set according to some policy (or
> configurable
> > > > policies), i.e. the number of segments and file descriptors will
> also be
> > > > limited.
> > > >
> > > > > The number of RPCs in the produce path is significantly higher. For
> > > > > example, if a produce request has 100 partitions, in a cluster
> with 100
> > > > > brokers, each produce request could generate 100 more RPC requests.
> > > This
> > > > > will significantly increase the request rate.
> > > >
> > > > This is a valid concern that we considered, but this issue can be
> > > > mitigated. I’ll try to explain the approach.
> > > > The situation with a single broker is trivial: all the commit
> requests go
> > > > from the broker to itself.
> > > > Let’s scale this to a multi-broker cluster, but located in the single
> > > rack
> > > > (AZ). Any broker can accept Produce requests for diskless
> partitions, but
> > > > we can tell producers (through metadata) to always send Produce
> requests
> > > to
> > > > leaders. For example, broker 0 hosts the leader replicas for diskless
> > > > partitions t1-0, t2-1, t3-0. It will receive diskless Produce
> requests
> > > for
> > > > these partitions in various combinations, but only for them.
> > > >
> > > >                     Broker 0
> > > >               +-----------------+
> > > >               |    t1-0         |
> > > >               |    t2-1 <--------------------+
> > > >               |    t3-0         |            |
> > > >  produce      | +-------------+ |            |
> > > >  requests     | |  diskless   | |            |
> > > > --------------->|   produce   +--------------+
> > > >  for these    | | WAL buffer  | |    commit requests
> > > >  partitions   | +-------------+ |    for these partitions
> > > >               |                 |
> > > >               +-----------------+
> > > >
> > > > The same applies for other brokers in this cluster. Effectively, each
> > > > broker will commit only to itself, which effectively means 1 commit
> > > request
> > > > per WAL buffer (this may be 0 physical network calls, if we wish,
> just a
> > > > local function call).
> > > >
> > > > Now let’s scale this to multiple racks (AZs). Obviously, we cannot
> always
> > > > send Produce requests to the designated leaders of diskless
> partitions:
> > > > this would mean inter-AZ network traffic, which we would like to
> avoid.
> > > To
> > > > avoid it, we say that every broker has a “diskless produce
> > > representative”
> > > > in every AZ. If we continue our example: when a Produce request for
> t1-0,
> > > > t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0
> (in the
> > > > broker’s AZ the representative is the broker itself). However, if it
> > > comes
> > > > from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > > >
> > > >   |produce requests         |produce requests        |produce
> requests
> > > >   |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
> > > t3-0
> > > >   |from AZ 0                |from AZ 1               |from AZ 2
> > > >   v                         v                        v
> > > >  Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > > > +---------------+      +---------------+      +---------------+
> > > > |     t1-0      |      |               |      |               |
> > > > |     t2-1      |      |               |      |               |
> > > > |     t3-0      |      |               |      |               |
> > > > +---------------+      +--------+------+      +--------+------+
> > > >      ^     ^                    |                      |
> > > >      |     +--------------------+                      |
> > > >      |     commit requests for these partitions        |
> > > >      |                                                 |
> > > >      +-------------------------------------------------+
> > > >            commit requests for these partitions
> > > >
> > > > All the partitions that broker 0 is the leader of will be
> “represented”
> > > by
> > > > brokers 1 and 2 in their AZs.
> > > >
> > > > Of course, this relationship goes both ways between AZs (not
> necessarily
> > > > between the same brokers). It means that provided the cluster is
> balanced
> > > > by the number of brokers per AZ, each broker will represent
> > > (number_of_azs
> > > > - 1) other brokers. This will result in the situation that for the
> > > majority
> > > > of commits, each broker will do up to (number_of_azs - 1) network
> commit
> > > > requests (plus one local). Cloud regions tend to have 3 AZs, very
> rarely
> > > > more. That means, brokers will be doing up to 2 network commit
> requests
> > > per
> > > > WAL file.
> > > >
> > > > There are the following exceptions:
> > > > 1. Broker count imbalance between AZs. For example, when we have 2
> AZs
> > > and
> > > > one has three brokers and another AZ has one. This one broker will do
> > > > between 1 and 3 commit requests per WAL file. This is not an extreme
> > > > amplification. Such an imbalance is not healthy in most practical
> setups
> > > > and should be avoided anyway.
> > > > 2. Leadership changes and metadata propagation period. When the
> partition
> > > > t3-0 is relocated from broker 0 to some broker 3, the producers will
> not
> > > > know this immediately (unless we want to be strict and respond with
> > > > NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come
> together
> > > in a
> > > > WAL buffer on broker 2, it will have to send two commit requests: to
> > > broker
> > > > 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This
> situation
> > > > is not permanent and as producers update the cluster metadata, it
> will be
> > > > resolved.
> > > >
> > > > This all could be built with the metadata crafting mechanism only
> (which
> > > > is anyway needed for Diskless in one way or another to direct
> producers
> > > and
> > > > consumers where we need to avoid inter-AZ traffic), just with the
> right
> > > > policy for it (for example, some deterministic hash-based formula).
> I.e.
> > > no
> > > > explicit support for “produce representative” or anything like this
> is
> > > > needed on the cluster level, in KRaft, etc.
> > > >
> > > > > The same WAL file metadata is now duplicated into two places,
> partition
> > > > > leader and WAL File Manager. Which one is the source of truth, and
> how
> > > do
> > > > > we maintain consistency between the two places?
> > > >
> > > > We do only two operations on WAL files that span multiple diskless
> > > > partitions: committing and deleting. Commits can be done
> independently as
> > > > described above. But deletes are different, because when a file is
> > > deleted,
> > > > this affects all the partitions that still have alive batches in this
> > > file
> > > > (if any).
> > > >
> > > > The WAL file manager is a necessary point of coordination to delete
> WAL
> > > > files safely. We can say it is the source of truth about files
> > > themselves,
> > > > while the partition leaders and their logs hold the truth about
> whether a
> > > > particular file contains live batches of this particular partition.
> > > >
> > > > The file manager will do this important task: be able to say for sure
> > > that
> > > > a file does not contain any live batch of any existing partition. For
> > > this,
> > > > it will have to periodically check against the partition leaders.
> > > > Considering that batch deletion is irreversible, when we declare a
> file
> > > > “empty”, this is guaranteed to be and stay so.
> > > >
> > > > The file manager has to know about files being committed to start
> track
> > > > them and periodically check if they are empty. We can consider
> various
> > > ways
> > > > to achieve this:
> > > > 1. As was proposed in my previous message: best effort commit by
> brokers
> > > +
> > > > periodic prefix scans of object storage to detect files that went
> below
> > > the
> > > > radar due to network issue or the file manager temporary
> unavailability.
> > > > We’re speaking about listing the file names only and opening only
> > > > previously unknown files in order to find the partitions involved
> with
> > > them.
> > > > 2. Only do scans without explicit commit, i.e. fill the list of files
> > > > fully asynchronously and in the background. This may be not ideal
> due to
> > > > costs and performance of scanning tons of files. However, the number
> of
> > > > live WAL files should be limited due to tiered storage offloading +
> we
> > > can
> > > > optimize this if we give files some global soft order in their names.
> > > >
> > > > > I am not sure how this design simplifies the implementation. The
> > > existing
> > > > > producer/replication code can't be simply reused. Adjusting both
> the
> > > > write
> > > > > path in the leader and the replication path in the follower to
> > > understand
> > > > > batch-header only data is quite intrusive to the existing logic.
> > > >
> > > > It is true that we’ll have to change LocalLog and UnifiedLog in
> order to
> > > > support these changes. However, it seems that idempotence,
> transactions,
> > > > queues, tiered storage will have to be changed less than with the
> > > original
> > > > design. This is because the partition leader state would remain in
> the
> > > same
> > > > place (on brokers) and existing workflows that involve it would have
> to
> > > be
> > > > changed less compared to the situation where we globalize the
> partition
> > > > leader state in the batch coordinator. I admit this is hard to make
> > > > convincing without both real implementations to hand :)
> > > >
> > > > > I am also
> > > > > not sure how this enables seamless switching the topic modes
> between
> > > > > diskless and classic. Could you provide more details on those?
> > > >
> > > > Let’s consider the scenario of turning a classic topic into
> diskless. The
> > > > user sets diskless.enabled=true, the leader receives this metadata
> update
> > > > and does the following:
> > > > 1. Stop accepting normal append writes.
> > > > 2. Close the current active segment.
> > > > 3. Start a new segment that will be written in the diskless format
> (i.e.
> > > > without data).
> > > > 4. Start accepting diskless commits.
> > > >
> > > > Since it’s the same log, the followers will know about that switch
> > > > consistently. They will finish replicating the classic segments and
> start
> > > > replicating the diskless ones. They will always know where each
> batch is
> > > > located (either inside a classic segment or referenced by a diskless
> > > one).
> > > > Switching back should be similar.
> > > >
> > > > Doing this with the coordinator is possible, but has some caveats.
> The
> > > > leader must do the following:
> > > > 1. Stop accepting normal append writes.
> > > > 2. Close the current active segment.
> > > > 3. Write a special control segment to persist and replicate the fact
> that
> > > > from offset N the partition is now in the diskless mode.
> > > > 4. Inform the coordinator about the first offset N of the “diskless
> era”.
> > > > 5. Inform the controller quorum that the transition has finished and
> that
> > > > brokers now can process diskless writes for this partition.
> > > > This could fail at some points, so this will probably require some
> > > > explicit state machine with replication either in the partition log
> or in
> > > > KRaft.
> > > >
> > > > It seems that the coordinator-less approach makes this simpler
> because
> > > the
> > > > “coordinator” for the partition and the partition leader are the
> same and
> > > > they store the partition metadata in the same log, too. While in the
> > > > coordinator approach we have to perform some kind of a distributed
> commit
> > > > to handover metadata management from the classic partition leader to
> the
> > > > batch coordinator.
> > > >
> > > > I hope these explanations help to clarify the idea. Please let me
> know if
> > > > I should go deeper anywhere.
> > > >
> > > > Best,
> > > > Ivan and the Diskless team
> > > >
> > > > On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > > Hi, Ivan,
> > > > >
> > > > > Thanks for the update.
> > > > >
> > > > > I am not sure that I fully understand the new design, but it seems
> less
> > > > > clean than before.
> > > > >
> > > > > Does each partition now have a metadata partition and a separate
> data
> > > > > partition? If so, I am concerned that it essentially doubles the
> number
> > > > of
> > > > > partitions, which impacts the number of open file descriptors and
> the
> > > > > required IOPS, and so on. It also seems wasteful to have a separate
> > > > > partition just to store the metadata. It's as if we are creating an
> > > > > internal topic with an unbounded number of partitions. Are the
> metadata
> > > > and
> > > > > the data for the same partition always collocated on the same
> broker?
> > > If
> > > > > so, how do we enforce that when replicas are reassigned?
> > > > >
> > > > > The number of RPCs in the produce path is significantly higher. For
> > > > > example, if a produce request has 100 partitions, in a cluster
> with 100
> > > > > brokers, each produce request could generate 100 more RPC requests.
> > > This
> > > > > will significantly increase the request rate.
> > > > >
> > > > > The same WAL file metadata is now duplicated into two places,
> partition
> > > > > leader and WAL File Manager. Which one is the source of truth, and
> how
> > > do
> > > > > we maintain consistency between the two places?
> > > > >
> > > > > I am not sure how this design simplifies the implementation. The
> > > existing
> > > > > producer/replication code can't be simply reused. Adjusting both
> the
> > > > write
> > > > > path in the leader and the replication path in the follower to
> > > understand
> > > > > batch-header only data is quite intrusive to the existing logic. I
> am
> > > > also
> > > > > not sure how this enables seamless switching the topic modes
> between
> > > > > diskless and classic. Could you provide more details on those?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> > > > >
> > > > > > Hi dear Kafka community,
> > > > > >
> > > > > > In the initial Diskless proposal, we proposed to have a separate
> > > > > > component, batch/diskless coordinator, whose role would be to
> > > centrally
> > > > > > manage the batch and WAL file metadata for diskless topics. This
> > > > component
> > > > > > drew many reasonable comments from the community about how it
> would
> > > > support
> > > > > > various Kafka features (transactions, queues) and its
> scalability.
> > > > While we
> > > > > > believe we have good answers to all the expressed concerns, we
> took a
> > > > step
> > > > > > back and looked at the problem from a different perspective.
> > > > > >
> > > > > > We would like to propose an alternative Diskless design *without
> a
> > > > > > centralized coordinator*. We believe this approach has potential
> and
> > > > > > propose to discuss it as it may be more appealing to the
> community.
> > > > > >
> > > > > > Let us explain the idea. Most of the complications with the
> original
> > > > > > Diskless approach come from one necessary architecture change:
> > > > globalizing
> > > > > > the local state of partition leader in the batch coordinator.
> This
> > > > causes
> > > > > > deviations to the established workflows in various features like
> > > > produce
> > > > > > idempotence and transactions, queues, retention, etc. These
> > > deviations
> > > > need
> > > > > > to be carefully considered, designed, and later implemented and
> > > > tested. In
> > > > > > the new approach we want to avoid this by making partition
> leaders
> > > > again
> > > > > > responsible for managing their partitions, even in diskless
> topics.
> > > > > >
> > > > > > In classic Kafka topics, batch data and metadata are blended
> together
> > > > in
> > > > > > the one partition log. The crux of the Diskless idea is to
> decouple
> > > > them
> > > > > > and move data to the remote storage, while keeping metadata
> somewhere
> > > > else.
> > > > > > Using the central batch coordinator for managing batch metadata
> is
> > > one
> > > > way,
> > > > > > but not the only.
> > > > > >
> > > > > > Let’s now think about managing metadata for each user partition
> > > > > > independently. Generally partitions are independent and don’t
> share
> > > > > > anything apart from that their data are mixed in WAL files. If we
> > > > figure
> > > > > > out how to commit and later delete WAL files safely, we will
> achieve
> > > > the
> > > > > > necessary autonomy that allows us to get rid of the central batch
> > > > > > coordinator. Instead, *each diskless user partition will be
> managed
> > > by
> > > > its
> > > > > > leader*, as in classic Kafka topics. Also like in classic
> topics, the
> > > > > > leader uses the partition log as the way to persist batch
> metadata,
> > > > i.e.
> > > > > > the regular batch header + the information about how to find this
> > > > batch on
> > > > > > remote storage. In contrast to classic topics, batch data is in
> > > remote
> > > > > > storage.
> > > > > >
> > > > > > For clarity, let’s compare the three designs:
> > > > > >  • Classic topics:
> > > > > >    • Data and metadata are co-located in the partition log.
> > > > > >    • The partition log content: [Batch header (metadata)|Batch
> data].
> > > > > >    • The partition log is replicated to the followers.
> > > > > >    • The replicas and leader have local state built from
> metadata.
> > > > > >  • Original Diskless:
> > > > > >    • Metadata is in the batch coordinator, data is on remote
> storage.
> > > > > >    • The partition state is global in the batch coordinator.
> > > > > >  • New Diskless:
> > > > > >    • Metadata is in the partition log, data is on remote storage.
> > > > > >    • Partition log content: [Batch header (metadata)|Batch
> > > coordinates
> > > > on
> > > > > > remote storage].
> > > > > >    • The partition log is replicated to the followers.
> > > > > >    • The replicas and leader have local state built from
> metadata.
> > > > > >
> > > > > > Let’s consider the produce path. Here’s the reminder of the
> original
> > > > > > Diskless design:
> > > > > >
> > > > > >
> > > > > > The new approach could be depicted as the following:
> > > > > >
> > > > > >
> > > > > > As you can see, the main difference is that now instead of a
> single
> > > > commit
> > > > > > request to the batch coordinator, we send multiple parallel
> commit
> > > > requests
> > > > > > to all the leaders of each partition involved in the WAL file.
> Each
> > > of
> > > > them
> > > > > > will commit its batches independently, without coordinating with
> > > other
> > > > > > leaders and any other components. Batch data is addressed by the
> WAL
> > > > file
> > > > > > name, the byte offset and size, which allows partitions to know
> > > nothing
> > > > > > about other partitions to access their data in shared WAL files.
> > > > > >
> > > > > > The number of partitions involved in a single WAL file may be
> quite
> > > > large,
> > > > > > e.g. a hundred. A hundred network requests to commit one WAL
> file is
> > > > very
> > > > > > impractical. However, there are ways to reduce this number:
> > > > > >  1. Partition leaders are located on brokers. Requests to
> leaders on
> > > > one
> > > > > > broker could be grouped together into a single physical network
> > > request
> > > > > > (resembling the normal Produce request that may carry batches for
> > > many
> > > > > > partitions inside). This will cap the number of network requests
> to
> > > the
> > > > > > number of brokers in the cluster.
> > > > > >  2. If we craft the cluster metadata to make producers send their
> > > > requests
> > > > > > to the right brokers (with respect to AZs), we may achieve the
> higher
> > > > > > concentration of logical commit requests in physical network
> requests
> > > > > > reducing the number of the latter ones even further, ideally to
> one.
> > > > > >
> > > > > > Obviously, out of multiple commit requests some may fail or time
> out
> > > > for a
> > > > > > variety of reasons. This is fine. Some producers will receive
> totally
> > > > or
> > > > > > partially failed responses to their Produce requests, similar to
> what
> > > > they
> > > > > > would have received when appending to a classic topic fails or
> times
> > > > out.
> > > > > > If a partition experiences problems, other partitions will not be
> > > > affected
> > > > > > (again, like in classic topics). Of course, the uncommitted data
> will
> > > > be
> > > > > > garbage in WAL files. But WAL files are short-lived (batches are
> > > > constantly
> > > > > > assembled into segments and offloaded to tiered storage), so this
> > > > garbage
> > > > > > will be eventually deleted.
> > > > > >
> > > > > > For safely deleting WAL files we now need to centrally manage
> them,
> > > as
> > > > > > this is the only state and logic that spans multiple partitions.
> On
> > > the
> > > > > > diagram, you can see another commit request called “Commit file
> (best
> > > > > > effort)” going to the WAL File Manager. This manager will be
> > > > responsible
> > > > > > for the following:
> > > > > >  1. Collecting (by requests from brokers) and persisting
> information
> > > > about
> > > > > > committed WAL files.
> > > > > >  2. To handle potential failures in file information delivery, it
> > > will
> > > > be
> > > > > > doing prefix scan on the remote storage periodically to find and
> > > > register
> > > > > > unknown files. The period of this scan will be configurable and
> > > ideally
> > > > > > should be quite long.
> > > > > >  3. Checking with the relevant partition leaders (after a grace
> > > > period) if
> > > > > > they still have batches in a particular file.
> > > > > >  4. Physically deleting files when they aren’t anymore referred
> to by
> > > > any
> > > > > > partition.
> > > > > >
> > > > > > This new design offers the following advantages:
> > > > > >  1. It simplifies the implementation of many Kafka features such
> as
> > > > > > idempotence, transactions, queues, tiered storage, retention.
> Now we
> > > > don’t
> > > > > > need to abstract away and reuse the code from partition leaders
> in
> > > the
> > > > > > batch coordinator. Instead, we will literally use the same code
> paths
> > > > in
> > > > > > leaders, with little adaptation. Workflows from classic topics
> mostly
> > > > > > remain unchanged.
> > > > > > For example, it seems that
> > > > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > > > > > KafkaApis.handleWriteTxnMarkersRequest used for transaction
> support
> > > on
> > > > the
> > > > > > partition leader side could be used for diskless topics with
> little
> > > > > > adaptation. ProducerStateManager, needed for both idempotent
> produce
> > > > and
> > > > > > transactions, would be reused.
> > > > > > Another example is share groups support, where the share
> partition
> > > > leader,
> > > > > > being co-located with the partition leader, would execute the
> same
> > > > logic
> > > > > > for both diskless and classic topics.
> > > > > >  2. It returns to the familiar partition-based scaling model,
> where
> > > > > > partitions are independent.
> > > > > >  3. It makes the operation and failure patterns closer to the
> > > familiar
> > > > > > ones from classic topics.
> > > > > >  4. It opens a straightforward path to seamless switching the
> topics
> > > > modes
> > > > > > between diskless and classic.
> > > > > >
> > > > > > The rest of the things remain unchanged compared to the previous
> > > > Diskless
> > > > > > design (after all previous discussions). Such things as local
> segment
> > > > > > materialization by replicas, the consume path, tiered storage
> > > > integration,
> > > > > > etc.
> > > > > >
> > > > > > If the community finds this design more suitable, we will update
> the
> > > > > > KIP(s) accordingly and continue working on it. Please let us know
> > > what
> > > > you
> > > > > > think.
> > > > > >
> > > > > > Best regards,
> > > > > > Ivan and Diskless team
> > > > > >
> > > > > > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Yes, you're right. We need to track the aborted transactions
> for in
> > > > the
> > > > > > diskless coordinator for as long as the corresponding offsets are
> > > > there.
> > > > > > With the tiered storage unification Greg mentioned earlier, this
> will
> > > > be
> > > > > > finite time even for infinite data retention.
> > > > > > >
> > > > > > > Best,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > > > > Hey Ivan,
> > > > > > > >
> > > > > > > > Thanks for the response. I think most of what you said made
> > > sense,
> > > > but
> > > > > > I
> > > > > > > > did have some questions about this part:
> > > > > > > >
> > > > > > > > > As we understand this, the partition leader in classic
> topics
> > > > forgets
> > > > > > > > about a transaction once it’s replicated (HWM overpasses
> it). The
> > > > > > > > transaction coordinator acts like the main guardian, allowing
> > > > partition
> > > > > > > > leaders to do this safely. Please correct me if this is
> wrong. We
> > > > think
> > > > > > > > about relying on this with the batch coordinator and delete
> the
> > > > > > information
> > > > > > > > about a transaction once it’s finished (as there’s no
> replication
> > > > and
> > > > > > HWM
> > > > > > > > advances immediately).
> > > > > > > >
> > > > > > > > I didn't quite understand this. In classic topics, we have
> maps
> > > for
> > > > > > ongoing
> > > > > > > > transactions which remove state when the transaction is
> completed
> > > > and
> > > > > > an
> > > > > > > > aborted transactions index which is retained for much longer.
> > > Once
> > > > the
> > > > > > > > transaction is completed, the coordinator is no longer
> involved
> > > in
> > > > > > > > maintaining this partition side state, and it is subject to
> > > > compaction
> > > > > > etc.
> > > > > > > > Looking back at the outline provided above, I didn't see much
> > > > about the
> > > > > > > > fetch path, so maybe that could be expanded a bit further. I
> saw
> > > > the
> > > > > > > > following in a response:
> > > > > > > > > When the broker constructs a fully valid local segment,
> all the
> > > > > > necessary
> > > > > > > > control batches will be inserted and indices, including the
> > > > transaction
> > > > > > > > index will be built to serve FetchRequests exactly as they
> are
> > > > today.
> > > > > > > >
> > > > > > > > Based on this, it seems like we need to retain the
> information
> > > > about
> > > > > > > > aborted txns for longer.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <
> ivan@ivanyu.me>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Justine and all,
> > > > > > > > >
> > > > > > > > > Thank you for your questions!
> > > > > > > > >
> > > > > > > > > > JO 1. >Since a transaction could be uniquely identified
> with
> > > > > > producer ID
> > > > > > > > > > and epoch, the positive result of this check could be
> cached
> > > > > > locally
> > > > > > > > > > Are we saying that only new transaction version 2
> > > transactions
> > > > can
> > > > > > be
> > > > > > > > > used
> > > > > > > > > > here? If not, we can't uniquely identify transactions
> with
> > > > > > producer id +
> > > > > > > > > > epoch
> > > > > > > > >
> > > > > > > > > You’re right that we (probably unintentionally) focused
> only on
> > > > > > version 2.
> > > > > > > > > We can either limit the support to version 2 or consider
> using
> > > > some
> > > > > > > > > surrogates to support version 1.
> > > > > > > > >
> > > > > > > > > > JO 2. >The batch coordinator does the final transactional
> > > > checks
> > > > > > of the
> > > > > > > > > > batches. This procedure would output the same errors
> like the
> > > > > > partition
> > > > > > > > > > leader in classic topics would do.
> > > > > > > > > > Can you expand on what these checks are? Would you be
> > > checking
> > > > if
> > > > > > the
> > > > > > > > > > transaction was still ongoing for example?* *
> > > > > > > > >
> > > > > > > > > Yes, the producer epoch, that the transaction is ongoing,
> and
> > > of
> > > > > > course
> > > > > > > > > the normal idempotence checks. What the partition leader
> in the
> > > > > > classic
> > > > > > > > > topics does before appending a batch to the local log
> (e.g. in
> > > > > > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless,
> we
> > > > > > unfortunately
> > > > > > > > > cannot do these checks before appending the data to the WAL
> > > > segment
> > > > > > and
> > > > > > > > > uploading it, but we can “tombstone” these batches in the
> batch
> > > > > > coordinator
> > > > > > > > > during the final commit.
> > > > > > > > >
> > > > > > > > > > Is there state about ongoing
> > > > > > > > > > transactions in the batch coordinator? I see some other
> state
> > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > the End transaction section, but it's not super clear
> what
> > > > state is
> > > > > > > > > stored
> > > > > > > > > > and when it is stored.
> > > > > > > > >
> > > > > > > > > Right, this should have been more explicit. As the
> partition
> > > > leader
> > > > > > tracks
> > > > > > > > > ongoing transactions for classic topics, the batch
> coordinator
> > > > has
> > > > > > to as
> > > > > > > > > well. So when a transaction starts and ends, the
> transaction
> > > > > > coordinator
> > > > > > > > > must inform the batch coordinator about this.
> > > > > > > > >
> > > > > > > > > > JO 3. I didn't see anything about maintaining LSO --
> perhaps
> > > > that
> > > > > > would
> > > > > > > > > be
> > > > > > > > > > stored in the batch coordinator?
> > > > > > > > >
> > > > > > > > > Yes. This could be deduced from the committed batches and
> other
> > > > > > > > > information, but for the sake of performance we’d better
> store
> > > it
> > > > > > > > > explicitly.
> > > > > > > > >
> > > > > > > > > > JO 4. Are there any thoughts about how long transactional
> > > > state is
> > > > > > > > > > maintained in the batch coordinator and how it will be
> > > cleaned
> > > > up?
> > > > > > > > >
> > > > > > > > > As we understand this, the partition leader in classic
> topics
> > > > forgets
> > > > > > > > > about a transaction once it’s replicated (HWM overpasses
> it).
> > > The
> > > > > > > > > transaction coordinator acts like the main guardian,
> allowing
> > > > > > partition
> > > > > > > > > leaders to do this safely. Please correct me if this is
> wrong.
> > > We
> > > > > > think
> > > > > > > > > about relying on this with the batch coordinator and
> delete the
> > > > > > information
> > > > > > > > > about a transaction once it’s finished (as there’s no
> > > replication
> > > > > > and HWM
> > > > > > > > > advances immediately).
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ivan
> > > > > > > > >
> > > > > > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > > > > > Hey folks,
> > > > > > > > > >
> > > > > > > > > > Excited to see some updates related to transactions!
> > > > > > > > > >
> > > > > > > > > > I had a few questions.
> > > > > > > > > >
> > > > > > > > > > JO 1. >Since a transaction could be uniquely identified
> with
> > > > > > producer ID
> > > > > > > > > > and epoch, the positive result of this check could be
> cached
> > > > > > locally
> > > > > > > > > > Are we saying that only new transaction version 2
> > > transactions
> > > > can
> > > > > > be
> > > > > > > > > used
> > > > > > > > > > here? If not, we can't uniquely identify transactions
> with
> > > > > > producer id +
> > > > > > > > > > epoch
> > > > > > > > > >
> > > > > > > > > > JO 2. >The batch coordinator does the final transactional
> > > > checks
> > > > > > of the
> > > > > > > > > > batches. This procedure would output the same errors
> like the
> > > > > > partition
> > > > > > > > > > leader in classic topics would do.
> > > > > > > > > > Can you expand on what these checks are? Would you be
> > > checking
> > > > if
> > > > > > the
> > > > > > > > > > transaction was still ongoing for example? Is there state
> > > about
> > > > > > ongoing
> > > > > > > > > > transactions in the batch coordinator? I see some other
> state
> > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > the End transaction section, but it's not super clear
> what
> > > > state is
> > > > > > > > > stored
> > > > > > > > > > and when it is stored.
> > > > > > > > > >
> > > > > > > > > > JO 3. I didn't see anything about maintaining LSO --
> perhaps
> > > > that
> > > > > > would
> > > > > > > > > be
> > > > > > > > > > stored in the batch coordinator?
> > > > > > > > > >
> > > > > > > > > > JO 4. Are there any thoughts about how long transactional
> > > > state is
> > > > > > > > > > maintained in the batch coordinator and how it will be
> > > cleaned
> > > > up?
> > > > > > > > > >
> > > > > > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > > <ju...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Greg and Ivan,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the update. A few comments.
> > > > > > > > > > >
> > > > > > > > > > > JR 10. "Consumer fetches are now served from local
> > > segments,
> > > > > > making
> > > > > > > > > use of
> > > > > > > > > > > the
> > > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > > > functionality
> > > > > > > > > already
> > > > > > > > > > > built into classic topics."
> > > > > > > > > > > JR 10.1 Does the broker build the producer state for
> each
> > > > > > partition in
> > > > > > > > > > > diskless topics?
> > > > > > > > > > > JR 10.2 For transactional data, the consumer fetches
> need
> > > to
> > > > know
> > > > > > > > > aborted
> > > > > > > > > > > records. How is that achieved?
> > > > > > > > > > >
> > > > > > > > > > > JR 11. "The batch coordinator saves that the
> transaction is
> > > > > > finished
> > > > > > > > > and
> > > > > > > > > > > also inserts the control batches in the corresponding
> logs
> > > > of the
> > > > > > > > > involved
> > > > > > > > > > > Diskless topics. This happens only on the metadata
> level,
> > > no
> > > > > > actual
> > > > > > > > > control
> > > > > > > > > > > batches are written to any file. "
> > > > > > > > > > > A fetch response could include multiple transactional
> > > > batches.
> > > > > > How
> > > > > > > > > does the
> > > > > > > > > > > broker obtain the information about the ending control
> > > batch
> > > > for
> > > > > > each
> > > > > > > > > > > batch? Does that mean that a fetch response needs to be
> > > > built by
> > > > > > > > > > > stitching record batches and generated control batches
> > > > together?
> > > > > > > > > > >
> > > > > > > > > > > JR 12. Queues: Is there still a share partition leader
> that
> > > > all
> > > > > > > > > consumers
> > > > > > > > > > > are routed to?
> > > > > > > > > > >
> > > > > > > > > > > JR 13. "Should the KIPs be modified to include this or
> it's
> > > > too
> > > > > > > > > > > implementation-focused?" It would be useful to include
> > > enough
> > > > > > details
> > > > > > > > > to
> > > > > > > > > > > understand correctness and performance impact.
> > > > > > > > > > >
> > > > > > > > > > > HC5. Henry has a valid point. Requests from a given
> > > producer
> > > > > > contain a
> > > > > > > > > > > sequence number, which is ordered. If a producer sends
> > > every
> > > > > > Produce
> > > > > > > > > > > request to an arbitrary broker, those requests could
> reach
> > > > the
> > > > > > batch
> > > > > > > > > > > coordinator in different order and lead to rejection
> of the
> > > > > > produce
> > > > > > > > > > > requests.
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > > ivan@ivanyu.me>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > We have also thought in a bit more details about
> > > > transactions
> > > > > > and
> > > > > > > > > queues,
> > > > > > > > > > > > here's the plan.
> > > > > > > > > > > >
> > > > > > > > > > > > *Transactions*
> > > > > > > > > > > >
> > > > > > > > > > > > The support for transactions in *classic topics* is
> based
> > > > on
> > > > > > precise
> > > > > > > > > > > > interactions between three actors: clients (mostly
> > > > producers,
> > > > > > but
> > > > > > > > > also
> > > > > > > > > > > > consumers), brokers (ReplicaManager and other
> classes),
> > > and
> > > > > > > > > transaction
> > > > > > > > > > > > coordinators. Brokers also run partition leaders with
> > > their
> > > > > > local
> > > > > > > > > state
> > > > > > > > > > > > (ProducerStateManager and others).
> > > > > > > > > > > >
> > > > > > > > > > > > The high level (some details skipped) workflow is the
> > > > > > following.
> > > > > > > > > When a
> > > > > > > > > > > > transactional Produce request is received by the
> broker:
> > > > > > > > > > > > 1. For each partition, the partition leader checks
> if a
> > > > > > non-empty
> > > > > > > > > > > > transaction is running for this partition. This is
> done
> > > > using
> > > > > > its
> > > > > > > > > local
> > > > > > > > > > > > state derived from the log metadata
> > > (ProducerStateManager,
> > > > > > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > > > > > 2. The transaction coordinator is informed about all
> the
> > > > > > partitions
> > > > > > > > > that
> > > > > > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > > > > > 3. The partition leaders do additional transactional
> > > > checks.
> > > > > > > > > > > > 4. The partition leaders append the transactional
> data to
> > > > > > their logs
> > > > > > > > > and
> > > > > > > > > > > > update some of their state (for example, log the fact
> > > that
> > > > the
> > > > > > > > > > > transaction
> > > > > > > > > > > > is running for the partition and its first offset).
> > > > > > > > > > > >
> > > > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > > > 1. The producer contacts the transaction coordinator
> > > > directly
> > > > > > with
> > > > > > > > > > > > EndTxnRequest.
> > > > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT
> or
> > > > > > > > > PREPARE_ABORT to
> > > > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > > > 3. The transaction coordinator sends
> > > > WriteTxnMarkersRequest to
> > > > > > the
> > > > > > > > > > > leaders
> > > > > > > > > > > > of the involved partitions.
> > > > > > > > > > > > 4. The partition leaders write the transaction
> markers to
> > > > > > their logs
> > > > > > > > > and
> > > > > > > > > > > > respond to the coordinator.
> > > > > > > > > > > > 5. The coordinator writes the final transaction state
> > > > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > > > >
> > > > > > > > > > > > In classic topics, partitions have leaders and lots
> of
> > > > > > important
> > > > > > > > > state
> > > > > > > > > > > > necessary for supporting this workflow is local. The
> main
> > > > > > challenge
> > > > > > > > > in
> > > > > > > > > > > > mapping this to Diskless comes from the fact there
> are no
> > > > > > partition
> > > > > > > > > > > > leaders, so the corresponding pieces of state need
> to be
> > > > > > globalized
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > batch coordinator. We are already doing this to
> support
> > > > > > idempotent
> > > > > > > > > > > produce.
> > > > > > > > > > > >
> > > > > > > > > > > > The high level workflow for *diskless topics* would
> look
> > > > very
> > > > > > > > > similar:
> > > > > > > > > > > > 1. For each partition, the broker checks if a
> non-empty
> > > > > > transaction
> > > > > > > > > is
> > > > > > > > > > > > running for this partition. In contrast to classic
> > > topics,
> > > > > > this is
> > > > > > > > > > > checked
> > > > > > > > > > > > against the batch coordinator with a single RPC.
> Since a
> > > > > > transaction
> > > > > > > > > > > could
> > > > > > > > > > > > be uniquely identified with producer ID and epoch,
> the
> > > > positive
> > > > > > > > > result of
> > > > > > > > > > > > this check could be cached locally (for the double
> > > > configured
> > > > > > > > > duration
> > > > > > > > > > > of a
> > > > > > > > > > > > transaction, for example).
> > > > > > > > > > > > 2. The same: The transaction coordinator is informed
> > > about
> > > > all
> > > > > > the
> > > > > > > > > > > > partitions that aren’t part of the transaction to
> include
> > > > them.
> > > > > > > > > > > > 3. No transactional checks are done on the broker
> side.
> > > > > > > > > > > > 4. The broker appends the transactional data to the
> > > current
> > > > > > shared
> > > > > > > > > WAL
> > > > > > > > > > > > segment. It doesn’t update any transaction-related
> state
> > > > for
> > > > > > Diskless
> > > > > > > > > > > > topics, because it doesn’t have any.
> > > > > > > > > > > > 5. The WAL segment is committed to the batch
> coordinator
> > > > like
> > > > > > in the
> > > > > > > > > > > > normal produce flow.
> > > > > > > > > > > > 6. The batch coordinator does the final transactional
> > > > checks
> > > > > > of the
> > > > > > > > > > > > batches. This procedure would output the same errors
> like
> > > > the
> > > > > > > > > partition
> > > > > > > > > > > > leader in classic topics would do. I.e. some batches
> > > could
> > > > be
> > > > > > > > > rejected.
> > > > > > > > > > > > This means, there will potentially be garbage in the
> WAL
> > > > > > segment
> > > > > > > > > file in
> > > > > > > > > > > > case of transactional errors. This is preferable to
> doing
> > > > more
> > > > > > > > > network
> > > > > > > > > > > > round trips, especially considering the WAL segments
> will
> > > > be
> > > > > > > > > relatively
> > > > > > > > > > > > short-living (see the Greg's update above).
> > > > > > > > > > > >
> > > > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > > > 1. The producer contacts the transaction coordinator
> > > > directly
> > > > > > with
> > > > > > > > > > > > EndTxnRequest.
> > > > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT
> or
> > > > > > > > > PREPARE_ABORT to
> > > > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > > > 3. *[NEW]* The transaction coordinator informs the
> batch
> > > > > > coordinator
> > > > > > > > > that
> > > > > > > > > > > > the transaction is finished.
> > > > > > > > > > > > 4. *[NEW]* The batch coordinator saves that the
> > > > transaction is
> > > > > > > > > finished
> > > > > > > > > > > > and also inserts the control batches in the
> corresponding
> > > > logs
> > > > > > of the
> > > > > > > > > > > > involved Diskless topics. This happens only on the
> > > metadata
> > > > > > level, no
> > > > > > > > > > > > actual control batches are written to any file. They
> will
> > > > be
> > > > > > > > > dynamically
> > > > > > > > > > > > created on Fetch and other read operations. We could
> > > > > > technically
> > > > > > > > > write
> > > > > > > > > > > > these control batches for real, but this would mean
> extra
> > > > > > produce
> > > > > > > > > > > latency,
> > > > > > > > > > > > so it's better just to mark them in the batch
> coordinator
> > > > and
> > > > > > save
> > > > > > > > > these
> > > > > > > > > > > > milliseconds.
> > > > > > > > > > > > 5. The transaction coordinator sends
> > > > WriteTxnMarkersRequest to
> > > > > > the
> > > > > > > > > > > leaders
> > > > > > > > > > > > of the involved partitions. – Now only to classic
> topics
> > > > now.
> > > > > > > > > > > > 6. The partition leaders of classic topics write the
> > > > > > transaction
> > > > > > > > > markers
> > > > > > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > > > > > 7. The coordinator writes the final transaction state
> > > > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > > > >
> > > > > > > > > > > > Compared to the non-transactional produce flow, we
> get:
> > > > > > > > > > > > 1. An extra network round trip between brokers and
> the
> > > > batch
> > > > > > > > > coordinator
> > > > > > > > > > > > when a new partition appear in the transaction. To
> > > > mitigate the
> > > > > > > > > impact of
> > > > > > > > > > > > them:
> > > > > > > > > > > >   - The results will be cached.
> > > > > > > > > > > >   - The calls for multiple partitions in one Produce
> > > > request
> > > > > > will be
> > > > > > > > > > > > grouped.
> > > > > > > > > > > >   - The batch coordinator should be optimized for
> fast
> > > > > > response to
> > > > > > > > > these
> > > > > > > > > > > > RPCs.
> > > > > > > > > > > >   - The fact that a single producer normally will
> > > > communicate
> > > > > > with a
> > > > > > > > > > > > single broker for the duration of the transaction
> further
> > > > > > reduces the
> > > > > > > > > > > > expected number of round trips.
> > > > > > > > > > > > 2. An extra round trip between the transaction
> > > coordinator
> > > > and
> > > > > > batch
> > > > > > > > > > > > coordinator when a transaction is finished.
> > > > > > > > > > > >
> > > > > > > > > > > > With this proposal, transactions will also be able to
> > > span
> > > > both
> > > > > > > > > classic
> > > > > > > > > > > > and Diskless topics.
> > > > > > > > > > > >
> > > > > > > > > > > > *Queues*
> > > > > > > > > > > >
> > > > > > > > > > > > The share group coordination and management is a
> side job
> > > > that
> > > > > > > > > doesn't
> > > > > > > > > > > > interfere with the topic itself (leadership,
> replicas,
> > > > physical
> > > > > > > > > storage
> > > > > > > > > > > of
> > > > > > > > > > > > records, etc.) and non-queue producers and consumers
> > > > (Fetch and
> > > > > > > > > Produce
> > > > > > > > > > > > RPCs, consumer group-related RPCs are not affected.)
> We
> > > > don't
> > > > > > see any
> > > > > > > > > > > > reason why we can't make Diskless topics compatible
> with
> > > > share
> > > > > > > > > groups the
> > > > > > > > > > > > same way as classic topics are. Even on the code
> level,
> > > we
> > > > > > don't
> > > > > > > > > expect
> > > > > > > > > > > any
> > > > > > > > > > > > serious refactoring: the same reading routines are
> used
> > > > that
> > > > > > are
> > > > > > > > > used for
> > > > > > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Should the KIPs be modified to include this or it's
> too
> > > > > > > > > > > > implementation-focused?
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Ivan
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you all for your questions and design input
> on
> > > > > > KIP-1150.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a
> new
> > > > > > design. To
> > > > > > > > > > > > summarize
> > > > > > > > > > > > > the changes:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. The design prioritizes integrating with the
> existing
> > > > > > KIP-405
> > > > > > > > > Tiered
> > > > > > > > > > > > > Storage interfaces, permitting data produced to a
> > > > Diskless
> > > > > > topic
> > > > > > > > > to be
> > > > > > > > > > > > > moved to tiered storage.
> > > > > > > > > > > > > This lowers the scalability requirements for the
> Batch
> > > > > > Coordinator
> > > > > > > > > > > > > component, and allows Diskless to compose with
> Tiered
> > > > Storage
> > > > > > > > > plugin
> > > > > > > > > > > > > features such as encryption and alternative data
> > > formats.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. Consumer fetches are now served from local
> segments,
> > > > > > making use
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > indexes, page cache, request purgatory, and
> zero-copy
> > > > > > functionality
> > > > > > > > > > > > already
> > > > > > > > > > > > > built into classic topics.
> > > > > > > > > > > > > However, local segments are now considered cache
> > > > elements,
> > > > > > do not
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > be durably stored, and can be built without
> contacting
> > > > any
> > > > > > other
> > > > > > > > > > > > replicas.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. The design has been simplified substantially, by
> > > > removing
> > > > > > the
> > > > > > > > > > > previous
> > > > > > > > > > > > > Diskless consume flow, distributed cache
> component, and
> > > > > > "object
> > > > > > > > > > > > > compaction/merging" step.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The design maintains leaderless produces as
> enabled by
> > > > the
> > > > > > Batch
> > > > > > > > > > > > > Coordinator, and the same latency profiles as the
> > > earlier
> > > > > > design,
> > > > > > > > > while
> > > > > > > > > > > > > being simpler and integrating better into the
> existing
> > > > > > ecosystem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks, and we are eager to hear your feedback on
> the
> > > new
> > > > > > design.
> > > > > > > > > > > > > Greg Harris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > > <ju...@confluent.io.invalid>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi, Jan,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For me, the main gap of KIP-1150 is the support
> of
> > > all
> > > > > > existing
> > > > > > > > > > > client
> > > > > > > > > > > > > > APIs. Currently, there is no design for
> supporting
> > > APIs
> > > > > > like
> > > > > > > > > > > > transactions
> > > > > > > > > > > > > > and queues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Would it be a good time to ask for the current
> > > > status of
> > > > > > this
> > > > > > > > > KIP?
> > > > > > > > > > > I
> > > > > > > > > > > > > > > haven't seen much activity here for the past 2
> > > > months,
> > > > > > the
> > > > > > > > > vote got
> > > > > > > > > > > > > > vetoed
> > > > > > > > > > > > > > > but I think the pending questions have been
> > > answered
> > > > > > since
> > > > > > > > > then.
> > > > > > > > > > > > KIP-1183
> > > > > > > > > > > > > > > (AutoMQ's proposal) also didn't have any
> activity
> > > > since
> > > > > > May.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real
> > > choices
> > > > > > that can
> > > > > > > > > be
> > > > > > > > > > > > > > > made, with a coordinator-based approach being
> by
> > > far
> > > > the
> > > > > > > > > dominant
> > > > > > > > > > > one
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > it comes to market adoption - but all these are
> > > > > > standalone
> > > > > > > > > > > products.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm a big fan of both approaches, but would
> hate to
> > > > see a
> > > > > > > > > stall. So
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Maybe it's time to start another vote? Colin
> > > McCabe -
> > > > > > have your
> > > > > > > > > > > > questions
> > > > > > > > > > > > > > > been answered? If not, is there anything I can
> do
> > > to
> > > > > > help? I'm
> > > > > > > > > > > deeply
> > > > > > > > > > > > > > > familiar with both architectures and have
> written
> > > > about
> > > > > > both?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > > Jan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > > Kozlovski
> > > > <
> > > > > > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > a) group all the KIP email threads in the
> main
> > > one
> > > > > > (just a
> > > > > > > > > bunch
> > > > > > > > > > > of
> > > > > > > > > > > > > > links
> > > > > > > > > > > > > > > > to everything)
> > > > > > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It's a bit hard to track it all - for
> example, I
> > > > was
> > > > > > > > > searching
> > > > > > > > > > > for
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > discuss thread for KIP-1165 for a while; As
> far
> > > as
> > > > I
> > > > > > can
> > > > > > > > > tell, it
> > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > exist yet.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Since the KIPs are published (by virtue of
> having
> > > > the
> > > > > > root
> > > > > > > > > KIP be
> > > > > > > > > > > > > > > > published, having a DISCUSS thread and links
> to
> > > > > > sub-KIPs
> > > > > > > > > where
> > > > > > > > > > > were
> > > > > > > > > > > > > > aimed
> > > > > > > > > > > > > > > > to move the discussion towards), I think it
> would
> > > > be
> > > > > > good to
> > > > > > > > > > > create
> > > > > > > > > > > > > > > DISCUSS
> > > > > > > > > > > > > > > > threads for them all.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Stan
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We want to start a new KIP discussion about
> > > > > > introducing a
> > > > > > > > > new
> > > > > > > > > > > > type of
> > > > > > > > > > > > > > > > > topics that would make use of Object
> Storage as
> > > > the
> > > > > > primary
> > > > > > > > > > > > source of
> > > > > > > > > > > > > > > > > storage. However, as this KIP is big we
> decided
> > > > to
> > > > > > split it
> > > > > > > > > > > into
> > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > that aims to discuss if Apache Kafka
> should aim
> > > > to
> > > > > > have
> > > > > > > > > this
> > > > > > > > > > > > type of
> > > > > > > > > > > > > > > > > feature at all. This KIP doesn't go onto
> > > details
> > > > on
> > > > > > how to
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > This follows the same approach used when we
> > > > discussed
> > > > > > > > > KRaft.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > But as we know that it is sometimes really
> hard
> > > > to
> > > > > > discuss
> > > > > > > > > on
> > > > > > > > > > > > that
> > > > > > > > > > > > > > meta
> > > > > > > > > > > > > > > > > level, we also created several sub-kips
> (linked
> > > > in
> > > > > > > > > KIP-1150)
> > > > > > > > > > > that
> > > > > > > > > > > > > > offer
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS
> > > > threads
> > > > > > for
> > > > > > > > > each
> > > > > > > > > > > > type of
> > > > > > > > > > > > > > > > > concern and keep this one to discuss
> whether
> > > > Apache
> > > > > > Kafka
> > > > > > > > > wants
> > > > > > > > > > > > to
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks in advance on behalf of all the
> authors
> > > of
> > > > > > this KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 |
> > > > aiven.io
> > > > > > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu
> > > > Valtonen,
> > > > > > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hello Jack, Jun, Luke, and all!

Thank you for your messages.

Let me first address some of Jun’s comments.

> First, it degrades the durability.
> For each partition, now there are two files being actively written at a
> given point of time, one for the data and another for the metadata.
> Flushing each file requires a separate IO. If the disk has 1K IOPS and we
> have 5K partitions in a broker, currently we can afford to flush each
> partition every 5 seconds, achieving an RPO of 5 seconds. If we double the
> number of files per partition, we can only flush each partition every 10
> seconds, which makes RPO twice as bad.

As I understand, you’re speaking about locally materialized segments. They will indeed consume some IOPS. See them as a cache that could always be restored from the remote storage. While it’s not ideal, it's still OK to lose data in them due to a machine crash, for example. Because of this, we can avoid explicit flushing on local materialized segments at all and let the file system and page cache figure out when to flush optimally. This would not eliminate the extra IOPS, but should reduce it dramatically, depending on throughput for each partition. We, of course, continue flushing the metadata segments as before.

It’s worth making a note on caching. I think nobody will disagree that doing direct reads from remote storage every time a batch is requested by a consumer will not be practical neither from the performance nor from the economy point of view. We need a way to keep the number of GET requests down. There are multiple options, for example:
  1. Rack-aware distributed in-memory caching.
  2. Local in-memory caching. Comes with less network chattiness and works well if we have more or less stable brokers to consume from.
  3. Materialization of diskless logs on local disk. Way lower impact on RAM and also requires stable brokers for consumption (using just assigned replicas will probably work well).

Materialization is one of possible options, but we can choose another one. However, we will have this dilemma regardless of whether we have an explicit coordinator or we go “coordinator-less”. 

> Second, if we ever need this
> metadata somewhere else, say in the WAL file manager, the consumer needs to
> subscribe to every partition in the cluster, which is inefficient. The
> actual benefit of this approach is also questionable. On the surface, it
> might seem that we could reduce the number of lines that need to be changed
> for this KIP. However, the changes are quite intrusive to the classic
> partition's code path and will probably make the code base harder to
> maintain in the long run. I like the original approach based on the batch
> coordinator much better than this one. We could probably refactor the
> producer state code so that it could be reused in the batch coordinator.

It’s hard to disagree with this. The explicit coordinator is more a side thing, while coordinator-less approach is more about extending ReplicaManager, UnifiedLog and others substantially.

> Thanks for addressing the concerns on the number of RPCs in the produce
> path. I agree that with the metadata crafting mechanism, we could mitigate
> the PRC problem. However, since we now require the metadata to be
> collocated with the data on the same set of brokers, it's weird that they
> are now managed by different mechanisms. The data assignment now uses the
> metadata crafting mechanism, but the metadata is stored in the classic
> partition using its own assignment strategy. It will be complicated to keep
> them collocated.

I would like to note that the metadata crafting is needed only to tell producers which brokers they should send Produce requests to, but data (as in “locally materialized log”) is located on partition replicas, i.e. automatically co-located with metadata. 

As a side note, it would probably be better that instead of implicitly crafting partition metadata, we extend the metadata protocol so that for diskless partitions we return not only the leader and replicas, but also some “recommended produce brokers”, selected for optimal performance and costs. Producers will pick ones in their racks.

> I am also concerned about the removal of the object compaction/merging
> step.

We should have mentioned this explicitly, but this step, in fact, remains in the form of segments offloading to tiered storage. When we assemble a segment and hand it over to RemoteLogManager, we’re effectively doing metadata compaction: replacing a big number of pieces of metadata about individual batches with a single record in __remote_log_metadata.

We could create a Diskless-specific merging mechanism instead if needed. It’s rather easy with the explicit coordinator approach. With the coordinator-less approach, this would probably be a bit more tricky (rewriting the tail of the log by the leader + replicating this change reliably).

> I see a tendency toward primarily optimizing for the fewest code changes in
> the KIP. Instead, our primary goal should be a clean design that can last
> for the long term.

Yes, totally agree.



Luke,
> I'm wondering if the complexity of designing txn and queue is because of
> leaderless cluster, do you think it will be simpler if we only focus on the
> "diskless" design to handle object compaction/merging to/from the remote
> storage to save the cross-AZ cost?

After some evolution of the original proposal, leaderless is now limited. We only need to be able to accept Produce requests on more than one broker to eliminate the cross-AZ costs for producers. Do I get it right that you propose to get rid of this? Or do I misunderstand?



Let’s now look at this problem from a higher level, as Jack proposed. As it was said, the big choice we need to make is whether we 1) create an explicit batch coordinator; or 2) go for the coordinator-less approach, where each diskless partition is managed by its leader as in classic topics.

If we try to compare the two approaches:

Pluggability:
- Explicit coordinator: Possible. For example, some setups may benefit from batch metadata being stored in a cloud database (such as AWS DynamoDB or GCP Spanner).
- Coordinator-less: Impossible.

Scalability and fault tolerance:
- Explicit coordinator: Depends on the implementation and it’s also necessary to actively work for it.
- Coordinator-less: Closer to classic Kafka topics. Scaling is done by partition placement, partitions could fail independently.

Separation of concerns:
- Explicit coordinator: Very good. Diskless remains more independent from classic topics in terms of code and workflows. For example, the above-mentioned non-tiered storage metadata compaction mechanism could be relatively simply implemented with it. As a flip side of this, some workflows (e.g. transactions) will have to be adapted.
- Coordinator-less: Less so. It leans to the opposite: bringing diskless closer to classic topics. Some code paths and workflows could be more straightforwardly reused, but they will inevitably have to be adapted to accommodate both topic types as also discussed.

Cloud-nativeness. This is a vague concept, also related to the previous, but let’s try:
- Explicit coordinator: Storing and processing metadata separately makes it easier for brokers to take different roles, be purely stateless if needed, etc.
- Coordinator-less: Less so. Something could be achieved with creative partition placement, but not much.

Both seem to have their pros and cons. However, answering Jack’s question, the explicit coordinator approach may indeed lead to a more flexible design.


The purpose of this deviation in the discussion was to receive a preliminary community evaluation of the coordinator-less approach without taking on the task of writing a separate KIP and fitting it in the system of KIP-1150 and its children. We’re open to stopping it and getting back to working out the coordinator design if the community doesn’t favor the proposed approach.

Best,
Ivan and Diskless team

On Mon, Oct 20, 2025, at 05:58, Luke Chen wrote:
> Hi Ivan,
> 
> As Jun pointed out, the updated design seems to have some shortcomings
> although it simplifies the implementation.
> 
> I'm wondering if the complexity of designing txn and queue is because of
> leaderless cluster, do you think it will be simpler if we only focus on the
> "diskless" design to handle object compaction/merging to/from the remote
> storage to save the cross-AZ cost?
> 
> 
> Thank you,
> Luke
> 
> On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid> wrote:
> 
> > Hi, Ivan,
> >
> > Thanks for the explanation.
> >
> > "we write the reference to the WAL file with the batch data"
> >
> > I understand the approach now, but I think it is a hacky one. There are
> > multiple short comings with this design. First, it degrades the durability.
> > For each partition, now there are two files being actively written at a
> > given point of time, one for the data and another for the metadata.
> > Flushing each file requires a separate IO. If the disk has 1K IOPS and we
> > have 5K partitions in a broker, currently we can afford to flush each
> > partition every 5 seconds, achieving an RPO of 5 seconds. If we double the
> > number of files per partition, we can only flush each partition every 10
> > seconds, which makes RPO twice as bad. Second, if we ever need this
> > metadata somewhere else, say in the WAL file manager, the consumer needs to
> > subscribe to every partition in the cluster, which is inefficient. The
> > actual benefit of this approach is also questionable. On the surface, it
> > might seem that we could reduce the number of lines that need to be changed
> > for this KIP. However, the changes are quite intrusive to the classic
> > partition's code path and will probably make the code base harder to
> > maintain in the long run. I like the original approach based on the batch
> > coordinator much better than this one. We could probably refactor the
> > producer state code so that it could be reused in the batch coordinator.
> >
> > Thanks for addressing the concerns on the number of RPCs in the produce
> > path. I agree that with the metadata crafting mechanism, we could mitigate
> > the PRC problem. However, since we now require the metadata to be
> > collocated with the data on the same set of brokers, it's weird that they
> > are now managed by different mechanisms. The data assignment now uses the
> > metadata crafting mechanism, but the metadata is stored in the classic
> > partition using its own assignment strategy. It will be complicated to keep
> > them collocated.
> >
> > I am also concerned about the removal of the object compaction/merging
> > step. My first concern is on the amount of metadata that need to be kept.
> > Without object compcation, the metadata generated in the produce path can
> > only be deleted after remote tiering kicks in. Let's say for every 250ms we
> > produce 100 byte of metadata per partition. Let's say remoting tiering
> > kicks in after 5 hours. In a cluster with 100K partitions, we need to keep
> > about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
> > signficant. A second concern is on performance. Every time we need to
> > rebuild the caching data, we need to read a bunch of small objects from S3,
> > slowing down the building process. If a consumer happens to need such data,
> > it could slow down the application.
> >
> > I see a tendency toward primarily optimizing for the fewest code changes in
> > the KIP. Instead, our primary goal should be a clean design that can last
> > for the long term.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi Jun,
> > >
> > > Thank you for your message. I’m sorry that I failed to clearly explain
> > the
> > > idea. Let me try to fix this.
> > >
> > > > Does each partition now have a metadata partition and a separate data
> > > > partition? If so, I am concerned that it essentially doubles the number
> > > of
> > > > partitions, which impacts the number of open file descriptors and the
> > > > required IOPS, and so on. It also seems wasteful to have a separate
> > > > partition just to store the metadata. It's as if we are creating an
> > > > internal topic with an unbounded number of partitions.
> > >
> > > No. There will be only one physical partition per diskless partition. Let
> > > me explain this with an example. Let’s say we have a diskless partition
> > > topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce some
> > > batches to this partition. The content of the segment file will be
> > > something like this (for each batch):
> > >
> > > BaseOffset: 00000000000000000000 (like in classic)
> > > Length: 123456 (like in classic)
> > > PartitionLeaderEpoch: like in classic
> > > Magic: like in classic
> > > CRC: like in classic
> > > Attributes: like in classic
> > > LastOffsetDelta: like in classic
> > > BaseTimestamp: like in classic
> > > MaxTimestamp: like in classic
> > > ProducerId: like in classic
> > > ProducerEpoch: like in classic
> > > BaseSequence: like in classic
> > > RecordsCount: like in classic
> > > Records:
> > >   path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> > > 123456
> > >
> > > It looks very much like classic log entries. The only difference is that
> > > instead of writing real Records, we write the reference to the WAL file
> > > with the batch data (I guess we need only the name and the byte offset,
> > > because the byte length is the standard field above). Otherwise, it’s a
> > > normal Kafka log with the leader and replicas.
> > >
> > > So we have as many partitions for diskless as for classic. As of open
> > file
> > > descriptors, let’s proceed to the following:
> > >
> > > > Are the metadata and
> > > > the data for the same partition always collocated on the same broker?
> > If
> > > > so, how do we enforce that when replicas are reassigned?
> > >
> > > The source of truth for the data is still in WAL files on object storage.
> > > The source of truth for the metadata is in segment files on the brokers
> > in
> > > the replica set. Two new mechanisms are planned, both independent of this
> > > new proposal, but I want to present them to give the idea that only a
> > > limited amount of data files will be operated locally:
> > > - We want to assemble batches into segment files and offload them to
> > > tiered storage in order to prevent the unbounded growth of batch
> > metadata.
> > > For this, we need to open only  a few file descriptors (for the segment
> > > file itself + the necessary indexes) before the segment is fully written
> > > and handed over to RemoteLogManager.
> > > - We want to assemble local segment files for caching purposes as well,
> > > i.e. to speed up fetching. This will not materialize the full content of
> > > the log, but only the hot set according to some policy (or configurable
> > > policies), i.e. the number of segments and file descriptors will also be
> > > limited.
> > >
> > > > The number of RPCs in the produce path is significantly higher. For
> > > > example, if a produce request has 100 partitions, in a cluster with 100
> > > > brokers, each produce request could generate 100 more RPC requests.
> > This
> > > > will significantly increase the request rate.
> > >
> > > This is a valid concern that we considered, but this issue can be
> > > mitigated. I’ll try to explain the approach.
> > > The situation with a single broker is trivial: all the commit requests go
> > > from the broker to itself.
> > > Let’s scale this to a multi-broker cluster, but located in the single
> > rack
> > > (AZ). Any broker can accept Produce requests for diskless partitions, but
> > > we can tell producers (through metadata) to always send Produce requests
> > to
> > > leaders. For example, broker 0 hosts the leader replicas for diskless
> > > partitions t1-0, t2-1, t3-0. It will receive diskless Produce requests
> > for
> > > these partitions in various combinations, but only for them.
> > >
> > >                     Broker 0
> > >               +-----------------+
> > >               |    t1-0         |
> > >               |    t2-1 <--------------------+
> > >               |    t3-0         |            |
> > >  produce      | +-------------+ |            |
> > >  requests     | |  diskless   | |            |
> > > --------------->|   produce   +--------------+
> > >  for these    | | WAL buffer  | |    commit requests
> > >  partitions   | +-------------+ |    for these partitions
> > >               |                 |
> > >               +-----------------+
> > >
> > > The same applies for other brokers in this cluster. Effectively, each
> > > broker will commit only to itself, which effectively means 1 commit
> > request
> > > per WAL buffer (this may be 0 physical network calls, if we wish, just a
> > > local function call).
> > >
> > > Now let’s scale this to multiple racks (AZs). Obviously, we cannot always
> > > send Produce requests to the designated leaders of diskless partitions:
> > > this would mean inter-AZ network traffic, which we would like to avoid.
> > To
> > > avoid it, we say that every broker has a “diskless produce
> > representative”
> > > in every AZ. If we continue our example: when a Produce request for t1-0,
> > > t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0 (in the
> > > broker’s AZ the representative is the broker itself). However, if it
> > comes
> > > from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> > >
> > >   |produce requests         |produce requests        |produce requests
> > >   |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
> > t3-0
> > >   |from AZ 0                |from AZ 1               |from AZ 2
> > >   v                         v                        v
> > >  Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > > +---------------+      +---------------+      +---------------+
> > > |     t1-0      |      |               |      |               |
> > > |     t2-1      |      |               |      |               |
> > > |     t3-0      |      |               |      |               |
> > > +---------------+      +--------+------+      +--------+------+
> > >      ^     ^                    |                      |
> > >      |     +--------------------+                      |
> > >      |     commit requests for these partitions        |
> > >      |                                                 |
> > >      +-------------------------------------------------+
> > >            commit requests for these partitions
> > >
> > > All the partitions that broker 0 is the leader of will be “represented”
> > by
> > > brokers 1 and 2 in their AZs.
> > >
> > > Of course, this relationship goes both ways between AZs (not necessarily
> > > between the same brokers). It means that provided the cluster is balanced
> > > by the number of brokers per AZ, each broker will represent
> > (number_of_azs
> > > - 1) other brokers. This will result in the situation that for the
> > majority
> > > of commits, each broker will do up to (number_of_azs - 1) network commit
> > > requests (plus one local). Cloud regions tend to have 3 AZs, very rarely
> > > more. That means, brokers will be doing up to 2 network commit requests
> > per
> > > WAL file.
> > >
> > > There are the following exceptions:
> > > 1. Broker count imbalance between AZs. For example, when we have 2 AZs
> > and
> > > one has three brokers and another AZ has one. This one broker will do
> > > between 1 and 3 commit requests per WAL file. This is not an extreme
> > > amplification. Such an imbalance is not healthy in most practical setups
> > > and should be avoided anyway.
> > > 2. Leadership changes and metadata propagation period. When the partition
> > > t3-0 is relocated from broker 0 to some broker 3, the producers will not
> > > know this immediately (unless we want to be strict and respond with
> > > NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come together
> > in a
> > > WAL buffer on broker 2, it will have to send two commit requests: to
> > broker
> > > 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This situation
> > > is not permanent and as producers update the cluster metadata, it will be
> > > resolved.
> > >
> > > This all could be built with the metadata crafting mechanism only (which
> > > is anyway needed for Diskless in one way or another to direct producers
> > and
> > > consumers where we need to avoid inter-AZ traffic), just with the right
> > > policy for it (for example, some deterministic hash-based formula). I.e.
> > no
> > > explicit support for “produce representative” or anything like this is
> > > needed on the cluster level, in KRaft, etc.
> > >
> > > > The same WAL file metadata is now duplicated into two places, partition
> > > > leader and WAL File Manager. Which one is the source of truth, and how
> > do
> > > > we maintain consistency between the two places?
> > >
> > > We do only two operations on WAL files that span multiple diskless
> > > partitions: committing and deleting. Commits can be done independently as
> > > described above. But deletes are different, because when a file is
> > deleted,
> > > this affects all the partitions that still have alive batches in this
> > file
> > > (if any).
> > >
> > > The WAL file manager is a necessary point of coordination to delete WAL
> > > files safely. We can say it is the source of truth about files
> > themselves,
> > > while the partition leaders and their logs hold the truth about whether a
> > > particular file contains live batches of this particular partition.
> > >
> > > The file manager will do this important task: be able to say for sure
> > that
> > > a file does not contain any live batch of any existing partition. For
> > this,
> > > it will have to periodically check against the partition leaders.
> > > Considering that batch deletion is irreversible, when we declare a file
> > > “empty”, this is guaranteed to be and stay so.
> > >
> > > The file manager has to know about files being committed to start track
> > > them and periodically check if they are empty. We can consider various
> > ways
> > > to achieve this:
> > > 1. As was proposed in my previous message: best effort commit by brokers
> > +
> > > periodic prefix scans of object storage to detect files that went below
> > the
> > > radar due to network issue or the file manager temporary unavailability.
> > > We’re speaking about listing the file names only and opening only
> > > previously unknown files in order to find the partitions involved with
> > them.
> > > 2. Only do scans without explicit commit, i.e. fill the list of files
> > > fully asynchronously and in the background. This may be not ideal due to
> > > costs and performance of scanning tons of files. However, the number of
> > > live WAL files should be limited due to tiered storage offloading + we
> > can
> > > optimize this if we give files some global soft order in their names.
> > >
> > > > I am not sure how this design simplifies the implementation. The
> > existing
> > > > producer/replication code can't be simply reused. Adjusting both the
> > > write
> > > > path in the leader and the replication path in the follower to
> > understand
> > > > batch-header only data is quite intrusive to the existing logic.
> > >
> > > It is true that we’ll have to change LocalLog and UnifiedLog in order to
> > > support these changes. However, it seems that idempotence, transactions,
> > > queues, tiered storage will have to be changed less than with the
> > original
> > > design. This is because the partition leader state would remain in the
> > same
> > > place (on brokers) and existing workflows that involve it would have to
> > be
> > > changed less compared to the situation where we globalize the partition
> > > leader state in the batch coordinator. I admit this is hard to make
> > > convincing without both real implementations to hand :)
> > >
> > > > I am also
> > > > not sure how this enables seamless switching the topic modes between
> > > > diskless and classic. Could you provide more details on those?
> > >
> > > Let’s consider the scenario of turning a classic topic into diskless. The
> > > user sets diskless.enabled=true, the leader receives this metadata update
> > > and does the following:
> > > 1. Stop accepting normal append writes.
> > > 2. Close the current active segment.
> > > 3. Start a new segment that will be written in the diskless format (i.e.
> > > without data).
> > > 4. Start accepting diskless commits.
> > >
> > > Since it’s the same log, the followers will know about that switch
> > > consistently. They will finish replicating the classic segments and start
> > > replicating the diskless ones. They will always know where each batch is
> > > located (either inside a classic segment or referenced by a diskless
> > one).
> > > Switching back should be similar.
> > >
> > > Doing this with the coordinator is possible, but has some caveats. The
> > > leader must do the following:
> > > 1. Stop accepting normal append writes.
> > > 2. Close the current active segment.
> > > 3. Write a special control segment to persist and replicate the fact that
> > > from offset N the partition is now in the diskless mode.
> > > 4. Inform the coordinator about the first offset N of the “diskless era”.
> > > 5. Inform the controller quorum that the transition has finished and that
> > > brokers now can process diskless writes for this partition.
> > > This could fail at some points, so this will probably require some
> > > explicit state machine with replication either in the partition log or in
> > > KRaft.
> > >
> > > It seems that the coordinator-less approach makes this simpler because
> > the
> > > “coordinator” for the partition and the partition leader are the same and
> > > they store the partition metadata in the same log, too. While in the
> > > coordinator approach we have to perform some kind of a distributed commit
> > > to handover metadata management from the classic partition leader to the
> > > batch coordinator.
> > >
> > > I hope these explanations help to clarify the idea. Please let me know if
> > > I should go deeper anywhere.
> > >
> > > Best,
> > > Ivan and the Diskless team
> > >
> > > On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > > Hi, Ivan,
> > > >
> > > > Thanks for the update.
> > > >
> > > > I am not sure that I fully understand the new design, but it seems less
> > > > clean than before.
> > > >
> > > > Does each partition now have a metadata partition and a separate data
> > > > partition? If so, I am concerned that it essentially doubles the number
> > > of
> > > > partitions, which impacts the number of open file descriptors and the
> > > > required IOPS, and so on. It also seems wasteful to have a separate
> > > > partition just to store the metadata. It's as if we are creating an
> > > > internal topic with an unbounded number of partitions. Are the metadata
> > > and
> > > > the data for the same partition always collocated on the same broker?
> > If
> > > > so, how do we enforce that when replicas are reassigned?
> > > >
> > > > The number of RPCs in the produce path is significantly higher. For
> > > > example, if a produce request has 100 partitions, in a cluster with 100
> > > > brokers, each produce request could generate 100 more RPC requests.
> > This
> > > > will significantly increase the request rate.
> > > >
> > > > The same WAL file metadata is now duplicated into two places, partition
> > > > leader and WAL File Manager. Which one is the source of truth, and how
> > do
> > > > we maintain consistency between the two places?
> > > >
> > > > I am not sure how this design simplifies the implementation. The
> > existing
> > > > producer/replication code can't be simply reused. Adjusting both the
> > > write
> > > > path in the leader and the replication path in the follower to
> > understand
> > > > batch-header only data is quite intrusive to the existing logic. I am
> > > also
> > > > not sure how this enables seamless switching the topic modes between
> > > > diskless and classic. Could you provide more details on those?
> > > >
> > > > Jun
> > > >
> > > > On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > >
> > > > > Hi dear Kafka community,
> > > > >
> > > > > In the initial Diskless proposal, we proposed to have a separate
> > > > > component, batch/diskless coordinator, whose role would be to
> > centrally
> > > > > manage the batch and WAL file metadata for diskless topics. This
> > > component
> > > > > drew many reasonable comments from the community about how it would
> > > support
> > > > > various Kafka features (transactions, queues) and its scalability.
> > > While we
> > > > > believe we have good answers to all the expressed concerns, we took a
> > > step
> > > > > back and looked at the problem from a different perspective.
> > > > >
> > > > > We would like to propose an alternative Diskless design *without a
> > > > > centralized coordinator*. We believe this approach has potential and
> > > > > propose to discuss it as it may be more appealing to the community.
> > > > >
> > > > > Let us explain the idea. Most of the complications with the original
> > > > > Diskless approach come from one necessary architecture change:
> > > globalizing
> > > > > the local state of partition leader in the batch coordinator. This
> > > causes
> > > > > deviations to the established workflows in various features like
> > > produce
> > > > > idempotence and transactions, queues, retention, etc. These
> > deviations
> > > need
> > > > > to be carefully considered, designed, and later implemented and
> > > tested. In
> > > > > the new approach we want to avoid this by making partition leaders
> > > again
> > > > > responsible for managing their partitions, even in diskless topics.
> > > > >
> > > > > In classic Kafka topics, batch data and metadata are blended together
> > > in
> > > > > the one partition log. The crux of the Diskless idea is to decouple
> > > them
> > > > > and move data to the remote storage, while keeping metadata somewhere
> > > else.
> > > > > Using the central batch coordinator for managing batch metadata is
> > one
> > > way,
> > > > > but not the only.
> > > > >
> > > > > Let’s now think about managing metadata for each user partition
> > > > > independently. Generally partitions are independent and don’t share
> > > > > anything apart from that their data are mixed in WAL files. If we
> > > figure
> > > > > out how to commit and later delete WAL files safely, we will achieve
> > > the
> > > > > necessary autonomy that allows us to get rid of the central batch
> > > > > coordinator. Instead, *each diskless user partition will be managed
> > by
> > > its
> > > > > leader*, as in classic Kafka topics. Also like in classic topics, the
> > > > > leader uses the partition log as the way to persist batch metadata,
> > > i.e.
> > > > > the regular batch header + the information about how to find this
> > > batch on
> > > > > remote storage. In contrast to classic topics, batch data is in
> > remote
> > > > > storage.
> > > > >
> > > > > For clarity, let’s compare the three designs:
> > > > >  • Classic topics:
> > > > >    • Data and metadata are co-located in the partition log.
> > > > >    • The partition log content: [Batch header (metadata)|Batch data].
> > > > >    • The partition log is replicated to the followers.
> > > > >    • The replicas and leader have local state built from metadata.
> > > > >  • Original Diskless:
> > > > >    • Metadata is in the batch coordinator, data is on remote storage.
> > > > >    • The partition state is global in the batch coordinator.
> > > > >  • New Diskless:
> > > > >    • Metadata is in the partition log, data is on remote storage.
> > > > >    • Partition log content: [Batch header (metadata)|Batch
> > coordinates
> > > on
> > > > > remote storage].
> > > > >    • The partition log is replicated to the followers.
> > > > >    • The replicas and leader have local state built from metadata.
> > > > >
> > > > > Let’s consider the produce path. Here’s the reminder of the original
> > > > > Diskless design:
> > > > >
> > > > >
> > > > > The new approach could be depicted as the following:
> > > > >
> > > > >
> > > > > As you can see, the main difference is that now instead of a single
> > > commit
> > > > > request to the batch coordinator, we send multiple parallel commit
> > > requests
> > > > > to all the leaders of each partition involved in the WAL file. Each
> > of
> > > them
> > > > > will commit its batches independently, without coordinating with
> > other
> > > > > leaders and any other components. Batch data is addressed by the WAL
> > > file
> > > > > name, the byte offset and size, which allows partitions to know
> > nothing
> > > > > about other partitions to access their data in shared WAL files.
> > > > >
> > > > > The number of partitions involved in a single WAL file may be quite
> > > large,
> > > > > e.g. a hundred. A hundred network requests to commit one WAL file is
> > > very
> > > > > impractical. However, there are ways to reduce this number:
> > > > >  1. Partition leaders are located on brokers. Requests to leaders on
> > > one
> > > > > broker could be grouped together into a single physical network
> > request
> > > > > (resembling the normal Produce request that may carry batches for
> > many
> > > > > partitions inside). This will cap the number of network requests to
> > the
> > > > > number of brokers in the cluster.
> > > > >  2. If we craft the cluster metadata to make producers send their
> > > requests
> > > > > to the right brokers (with respect to AZs), we may achieve the higher
> > > > > concentration of logical commit requests in physical network requests
> > > > > reducing the number of the latter ones even further, ideally to one.
> > > > >
> > > > > Obviously, out of multiple commit requests some may fail or time out
> > > for a
> > > > > variety of reasons. This is fine. Some producers will receive totally
> > > or
> > > > > partially failed responses to their Produce requests, similar to what
> > > they
> > > > > would have received when appending to a classic topic fails or times
> > > out.
> > > > > If a partition experiences problems, other partitions will not be
> > > affected
> > > > > (again, like in classic topics). Of course, the uncommitted data will
> > > be
> > > > > garbage in WAL files. But WAL files are short-lived (batches are
> > > constantly
> > > > > assembled into segments and offloaded to tiered storage), so this
> > > garbage
> > > > > will be eventually deleted.
> > > > >
> > > > > For safely deleting WAL files we now need to centrally manage them,
> > as
> > > > > this is the only state and logic that spans multiple partitions. On
> > the
> > > > > diagram, you can see another commit request called “Commit file (best
> > > > > effort)” going to the WAL File Manager. This manager will be
> > > responsible
> > > > > for the following:
> > > > >  1. Collecting (by requests from brokers) and persisting information
> > > about
> > > > > committed WAL files.
> > > > >  2. To handle potential failures in file information delivery, it
> > will
> > > be
> > > > > doing prefix scan on the remote storage periodically to find and
> > > register
> > > > > unknown files. The period of this scan will be configurable and
> > ideally
> > > > > should be quite long.
> > > > >  3. Checking with the relevant partition leaders (after a grace
> > > period) if
> > > > > they still have batches in a particular file.
> > > > >  4. Physically deleting files when they aren’t anymore referred to by
> > > any
> > > > > partition.
> > > > >
> > > > > This new design offers the following advantages:
> > > > >  1. It simplifies the implementation of many Kafka features such as
> > > > > idempotence, transactions, queues, tiered storage, retention. Now we
> > > don’t
> > > > > need to abstract away and reuse the code from partition leaders in
> > the
> > > > > batch coordinator. Instead, we will literally use the same code paths
> > > in
> > > > > leaders, with little adaptation. Workflows from classic topics mostly
> > > > > remain unchanged.
> > > > > For example, it seems that
> > > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > > > > KafkaApis.handleWriteTxnMarkersRequest used for transaction support
> > on
> > > the
> > > > > partition leader side could be used for diskless topics with little
> > > > > adaptation. ProducerStateManager, needed for both idempotent produce
> > > and
> > > > > transactions, would be reused.
> > > > > Another example is share groups support, where the share partition
> > > leader,
> > > > > being co-located with the partition leader, would execute the same
> > > logic
> > > > > for both diskless and classic topics.
> > > > >  2. It returns to the familiar partition-based scaling model, where
> > > > > partitions are independent.
> > > > >  3. It makes the operation and failure patterns closer to the
> > familiar
> > > > > ones from classic topics.
> > > > >  4. It opens a straightforward path to seamless switching the topics
> > > modes
> > > > > between diskless and classic.
> > > > >
> > > > > The rest of the things remain unchanged compared to the previous
> > > Diskless
> > > > > design (after all previous discussions). Such things as local segment
> > > > > materialization by replicas, the consume path, tiered storage
> > > integration,
> > > > > etc.
> > > > >
> > > > > If the community finds this design more suitable, we will update the
> > > > > KIP(s) accordingly and continue working on it. Please let us know
> > what
> > > you
> > > > > think.
> > > > >
> > > > > Best regards,
> > > > > Ivan and Diskless team
> > > > >
> > > > > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > > Hi Justine,
> > > > > >
> > > > > > Yes, you're right. We need to track the aborted transactions for in
> > > the
> > > > > diskless coordinator for as long as the corresponding offsets are
> > > there.
> > > > > With the tiered storage unification Greg mentioned earlier, this will
> > > be
> > > > > finite time even for infinite data retention.
> > > > > >
> > > > > > Best,
> > > > > > Ivan
> > > > > >
> > > > > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > > > Hey Ivan,
> > > > > > >
> > > > > > > Thanks for the response. I think most of what you said made
> > sense,
> > > but
> > > > > I
> > > > > > > did have some questions about this part:
> > > > > > >
> > > > > > > > As we understand this, the partition leader in classic topics
> > > forgets
> > > > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > > > transaction coordinator acts like the main guardian, allowing
> > > partition
> > > > > > > leaders to do this safely. Please correct me if this is wrong. We
> > > think
> > > > > > > about relying on this with the batch coordinator and delete the
> > > > > information
> > > > > > > about a transaction once it’s finished (as there’s no replication
> > > and
> > > > > HWM
> > > > > > > advances immediately).
> > > > > > >
> > > > > > > I didn't quite understand this. In classic topics, we have maps
> > for
> > > > > ongoing
> > > > > > > transactions which remove state when the transaction is completed
> > > and
> > > > > an
> > > > > > > aborted transactions index which is retained for much longer.
> > Once
> > > the
> > > > > > > transaction is completed, the coordinator is no longer involved
> > in
> > > > > > > maintaining this partition side state, and it is subject to
> > > compaction
> > > > > etc.
> > > > > > > Looking back at the outline provided above, I didn't see much
> > > about the
> > > > > > > fetch path, so maybe that could be expanded a bit further. I saw
> > > the
> > > > > > > following in a response:
> > > > > > > > When the broker constructs a fully valid local segment, all the
> > > > > necessary
> > > > > > > control batches will be inserted and indices, including the
> > > transaction
> > > > > > > index will be built to serve FetchRequests exactly as they are
> > > today.
> > > > > > >
> > > > > > > Based on this, it seems like we need to retain the information
> > > about
> > > > > > > aborted txns for longer.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Justine and all,
> > > > > > > >
> > > > > > > > Thank you for your questions!
> > > > > > > >
> > > > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > > > producer ID
> > > > > > > > > and epoch, the positive result of this check could be cached
> > > > > locally
> > > > > > > > > Are we saying that only new transaction version 2
> > transactions
> > > can
> > > > > be
> > > > > > > > used
> > > > > > > > > here? If not, we can't uniquely identify transactions with
> > > > > producer id +
> > > > > > > > > epoch
> > > > > > > >
> > > > > > > > You’re right that we (probably unintentionally) focused only on
> > > > > version 2.
> > > > > > > > We can either limit the support to version 2 or consider using
> > > some
> > > > > > > > surrogates to support version 1.
> > > > > > > >
> > > > > > > > > JO 2. >The batch coordinator does the final transactional
> > > checks
> > > > > of the
> > > > > > > > > batches. This procedure would output the same errors like the
> > > > > partition
> > > > > > > > > leader in classic topics would do.
> > > > > > > > > Can you expand on what these checks are? Would you be
> > checking
> > > if
> > > > > the
> > > > > > > > > transaction was still ongoing for example?* *
> > > > > > > >
> > > > > > > > Yes, the producer epoch, that the transaction is ongoing, and
> > of
> > > > > course
> > > > > > > > the normal idempotence checks. What the partition leader in the
> > > > > classic
> > > > > > > > topics does before appending a batch to the local log (e.g. in
> > > > > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> > > > > unfortunately
> > > > > > > > cannot do these checks before appending the data to the WAL
> > > segment
> > > > > and
> > > > > > > > uploading it, but we can “tombstone” these batches in the batch
> > > > > coordinator
> > > > > > > > during the final commit.
> > > > > > > >
> > > > > > > > > Is there state about ongoing
> > > > > > > > > transactions in the batch coordinator? I see some other state
> > > > > mentioned
> > > > > > > > in
> > > > > > > > > the End transaction section, but it's not super clear what
> > > state is
> > > > > > > > stored
> > > > > > > > > and when it is stored.
> > > > > > > >
> > > > > > > > Right, this should have been more explicit. As the partition
> > > leader
> > > > > tracks
> > > > > > > > ongoing transactions for classic topics, the batch coordinator
> > > has
> > > > > to as
> > > > > > > > well. So when a transaction starts and ends, the transaction
> > > > > coordinator
> > > > > > > > must inform the batch coordinator about this.
> > > > > > > >
> > > > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> > > that
> > > > > would
> > > > > > > > be
> > > > > > > > > stored in the batch coordinator?
> > > > > > > >
> > > > > > > > Yes. This could be deduced from the committed batches and other
> > > > > > > > information, but for the sake of performance we’d better store
> > it
> > > > > > > > explicitly.
> > > > > > > >
> > > > > > > > > JO 4. Are there any thoughts about how long transactional
> > > state is
> > > > > > > > > maintained in the batch coordinator and how it will be
> > cleaned
> > > up?
> > > > > > > >
> > > > > > > > As we understand this, the partition leader in classic topics
> > > forgets
> > > > > > > > about a transaction once it’s replicated (HWM overpasses it).
> > The
> > > > > > > > transaction coordinator acts like the main guardian, allowing
> > > > > partition
> > > > > > > > leaders to do this safely. Please correct me if this is wrong.
> > We
> > > > > think
> > > > > > > > about relying on this with the batch coordinator and delete the
> > > > > information
> > > > > > > > about a transaction once it’s finished (as there’s no
> > replication
> > > > > and HWM
> > > > > > > > advances immediately).
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ivan
> > > > > > > >
> > > > > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > > > > Hey folks,
> > > > > > > > >
> > > > > > > > > Excited to see some updates related to transactions!
> > > > > > > > >
> > > > > > > > > I had a few questions.
> > > > > > > > >
> > > > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > > > producer ID
> > > > > > > > > and epoch, the positive result of this check could be cached
> > > > > locally
> > > > > > > > > Are we saying that only new transaction version 2
> > transactions
> > > can
> > > > > be
> > > > > > > > used
> > > > > > > > > here? If not, we can't uniquely identify transactions with
> > > > > producer id +
> > > > > > > > > epoch
> > > > > > > > >
> > > > > > > > > JO 2. >The batch coordinator does the final transactional
> > > checks
> > > > > of the
> > > > > > > > > batches. This procedure would output the same errors like the
> > > > > partition
> > > > > > > > > leader in classic topics would do.
> > > > > > > > > Can you expand on what these checks are? Would you be
> > checking
> > > if
> > > > > the
> > > > > > > > > transaction was still ongoing for example? Is there state
> > about
> > > > > ongoing
> > > > > > > > > transactions in the batch coordinator? I see some other state
> > > > > mentioned
> > > > > > > > in
> > > > > > > > > the End transaction section, but it's not super clear what
> > > state is
> > > > > > > > stored
> > > > > > > > > and when it is stored.
> > > > > > > > >
> > > > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> > > that
> > > > > would
> > > > > > > > be
> > > > > > > > > stored in the batch coordinator?
> > > > > > > > >
> > > > > > > > > JO 4. Are there any thoughts about how long transactional
> > > state is
> > > > > > > > > maintained in the batch coordinator and how it will be
> > cleaned
> > > up?
> > > > > > > > >
> > > > > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > > <ju...@confluent.io.invalid>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Greg and Ivan,
> > > > > > > > > >
> > > > > > > > > > Thanks for the update. A few comments.
> > > > > > > > > >
> > > > > > > > > > JR 10. "Consumer fetches are now served from local
> > segments,
> > > > > making
> > > > > > > > use of
> > > > > > > > > > the
> > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > > functionality
> > > > > > > > already
> > > > > > > > > > built into classic topics."
> > > > > > > > > > JR 10.1 Does the broker build the producer state for each
> > > > > partition in
> > > > > > > > > > diskless topics?
> > > > > > > > > > JR 10.2 For transactional data, the consumer fetches need
> > to
> > > know
> > > > > > > > aborted
> > > > > > > > > > records. How is that achieved?
> > > > > > > > > >
> > > > > > > > > > JR 11. "The batch coordinator saves that the transaction is
> > > > > finished
> > > > > > > > and
> > > > > > > > > > also inserts the control batches in the corresponding logs
> > > of the
> > > > > > > > involved
> > > > > > > > > > Diskless topics. This happens only on the metadata level,
> > no
> > > > > actual
> > > > > > > > control
> > > > > > > > > > batches are written to any file. "
> > > > > > > > > > A fetch response could include multiple transactional
> > > batches.
> > > > > How
> > > > > > > > does the
> > > > > > > > > > broker obtain the information about the ending control
> > batch
> > > for
> > > > > each
> > > > > > > > > > batch? Does that mean that a fetch response needs to be
> > > built by
> > > > > > > > > > stitching record batches and generated control batches
> > > together?
> > > > > > > > > >
> > > > > > > > > > JR 12. Queues: Is there still a share partition leader that
> > > all
> > > > > > > > consumers
> > > > > > > > > > are routed to?
> > > > > > > > > >
> > > > > > > > > > JR 13. "Should the KIPs be modified to include this or it's
> > > too
> > > > > > > > > > implementation-focused?" It would be useful to include
> > enough
> > > > > details
> > > > > > > > to
> > > > > > > > > > understand correctness and performance impact.
> > > > > > > > > >
> > > > > > > > > > HC5. Henry has a valid point. Requests from a given
> > producer
> > > > > contain a
> > > > > > > > > > sequence number, which is ordered. If a producer sends
> > every
> > > > > Produce
> > > > > > > > > > request to an arbitrary broker, those requests could reach
> > > the
> > > > > batch
> > > > > > > > > > coordinator in different order and lead to rejection of the
> > > > > produce
> > > > > > > > > > requests.
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > > ivan@ivanyu.me>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > We have also thought in a bit more details about
> > > transactions
> > > > > and
> > > > > > > > queues,
> > > > > > > > > > > here's the plan.
> > > > > > > > > > >
> > > > > > > > > > > *Transactions*
> > > > > > > > > > >
> > > > > > > > > > > The support for transactions in *classic topics* is based
> > > on
> > > > > precise
> > > > > > > > > > > interactions between three actors: clients (mostly
> > > producers,
> > > > > but
> > > > > > > > also
> > > > > > > > > > > consumers), brokers (ReplicaManager and other classes),
> > and
> > > > > > > > transaction
> > > > > > > > > > > coordinators. Brokers also run partition leaders with
> > their
> > > > > local
> > > > > > > > state
> > > > > > > > > > > (ProducerStateManager and others).
> > > > > > > > > > >
> > > > > > > > > > > The high level (some details skipped) workflow is the
> > > > > following.
> > > > > > > > When a
> > > > > > > > > > > transactional Produce request is received by the broker:
> > > > > > > > > > > 1. For each partition, the partition leader checks if a
> > > > > non-empty
> > > > > > > > > > > transaction is running for this partition. This is done
> > > using
> > > > > its
> > > > > > > > local
> > > > > > > > > > > state derived from the log metadata
> > (ProducerStateManager,
> > > > > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > > > > 2. The transaction coordinator is informed about all the
> > > > > partitions
> > > > > > > > that
> > > > > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > > > > 3. The partition leaders do additional transactional
> > > checks.
> > > > > > > > > > > 4. The partition leaders append the transactional data to
> > > > > their logs
> > > > > > > > and
> > > > > > > > > > > update some of their state (for example, log the fact
> > that
> > > the
> > > > > > > > > > transaction
> > > > > > > > > > > is running for the partition and its first offset).
> > > > > > > > > > >
> > > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > > 1. The producer contacts the transaction coordinator
> > > directly
> > > > > with
> > > > > > > > > > > EndTxnRequest.
> > > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > > > PREPARE_ABORT to
> > > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > > 3. The transaction coordinator sends
> > > WriteTxnMarkersRequest to
> > > > > the
> > > > > > > > > > leaders
> > > > > > > > > > > of the involved partitions.
> > > > > > > > > > > 4. The partition leaders write the transaction markers to
> > > > > their logs
> > > > > > > > and
> > > > > > > > > > > respond to the coordinator.
> > > > > > > > > > > 5. The coordinator writes the final transaction state
> > > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > > >
> > > > > > > > > > > In classic topics, partitions have leaders and lots of
> > > > > important
> > > > > > > > state
> > > > > > > > > > > necessary for supporting this workflow is local. The main
> > > > > challenge
> > > > > > > > in
> > > > > > > > > > > mapping this to Diskless comes from the fact there are no
> > > > > partition
> > > > > > > > > > > leaders, so the corresponding pieces of state need to be
> > > > > globalized
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > batch coordinator. We are already doing this to support
> > > > > idempotent
> > > > > > > > > > produce.
> > > > > > > > > > >
> > > > > > > > > > > The high level workflow for *diskless topics* would look
> > > very
> > > > > > > > similar:
> > > > > > > > > > > 1. For each partition, the broker checks if a non-empty
> > > > > transaction
> > > > > > > > is
> > > > > > > > > > > running for this partition. In contrast to classic
> > topics,
> > > > > this is
> > > > > > > > > > checked
> > > > > > > > > > > against the batch coordinator with a single RPC. Since a
> > > > > transaction
> > > > > > > > > > could
> > > > > > > > > > > be uniquely identified with producer ID and epoch, the
> > > positive
> > > > > > > > result of
> > > > > > > > > > > this check could be cached locally (for the double
> > > configured
> > > > > > > > duration
> > > > > > > > > > of a
> > > > > > > > > > > transaction, for example).
> > > > > > > > > > > 2. The same: The transaction coordinator is informed
> > about
> > > all
> > > > > the
> > > > > > > > > > > partitions that aren’t part of the transaction to include
> > > them.
> > > > > > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > > > > > 4. The broker appends the transactional data to the
> > current
> > > > > shared
> > > > > > > > WAL
> > > > > > > > > > > segment. It doesn’t update any transaction-related state
> > > for
> > > > > Diskless
> > > > > > > > > > > topics, because it doesn’t have any.
> > > > > > > > > > > 5. The WAL segment is committed to the batch coordinator
> > > like
> > > > > in the
> > > > > > > > > > > normal produce flow.
> > > > > > > > > > > 6. The batch coordinator does the final transactional
> > > checks
> > > > > of the
> > > > > > > > > > > batches. This procedure would output the same errors like
> > > the
> > > > > > > > partition
> > > > > > > > > > > leader in classic topics would do. I.e. some batches
> > could
> > > be
> > > > > > > > rejected.
> > > > > > > > > > > This means, there will potentially be garbage in the WAL
> > > > > segment
> > > > > > > > file in
> > > > > > > > > > > case of transactional errors. This is preferable to doing
> > > more
> > > > > > > > network
> > > > > > > > > > > round trips, especially considering the WAL segments will
> > > be
> > > > > > > > relatively
> > > > > > > > > > > short-living (see the Greg's update above).
> > > > > > > > > > >
> > > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > > 1. The producer contacts the transaction coordinator
> > > directly
> > > > > with
> > > > > > > > > > > EndTxnRequest.
> > > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > > > PREPARE_ABORT to
> > > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> > > > > coordinator
> > > > > > > > that
> > > > > > > > > > > the transaction is finished.
> > > > > > > > > > > 4. *[NEW]* The batch coordinator saves that the
> > > transaction is
> > > > > > > > finished
> > > > > > > > > > > and also inserts the control batches in the corresponding
> > > logs
> > > > > of the
> > > > > > > > > > > involved Diskless topics. This happens only on the
> > metadata
> > > > > level, no
> > > > > > > > > > > actual control batches are written to any file. They will
> > > be
> > > > > > > > dynamically
> > > > > > > > > > > created on Fetch and other read operations. We could
> > > > > technically
> > > > > > > > write
> > > > > > > > > > > these control batches for real, but this would mean extra
> > > > > produce
> > > > > > > > > > latency,
> > > > > > > > > > > so it's better just to mark them in the batch coordinator
> > > and
> > > > > save
> > > > > > > > these
> > > > > > > > > > > milliseconds.
> > > > > > > > > > > 5. The transaction coordinator sends
> > > WriteTxnMarkersRequest to
> > > > > the
> > > > > > > > > > leaders
> > > > > > > > > > > of the involved partitions. – Now only to classic topics
> > > now.
> > > > > > > > > > > 6. The partition leaders of classic topics write the
> > > > > transaction
> > > > > > > > markers
> > > > > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > > > > 7. The coordinator writes the final transaction state
> > > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > > >
> > > > > > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > > > > > 1. An extra network round trip between brokers and the
> > > batch
> > > > > > > > coordinator
> > > > > > > > > > > when a new partition appear in the transaction. To
> > > mitigate the
> > > > > > > > impact of
> > > > > > > > > > > them:
> > > > > > > > > > >   - The results will be cached.
> > > > > > > > > > >   - The calls for multiple partitions in one Produce
> > > request
> > > > > will be
> > > > > > > > > > > grouped.
> > > > > > > > > > >   - The batch coordinator should be optimized for fast
> > > > > response to
> > > > > > > > these
> > > > > > > > > > > RPCs.
> > > > > > > > > > >   - The fact that a single producer normally will
> > > communicate
> > > > > with a
> > > > > > > > > > > single broker for the duration of the transaction further
> > > > > reduces the
> > > > > > > > > > > expected number of round trips.
> > > > > > > > > > > 2. An extra round trip between the transaction
> > coordinator
> > > and
> > > > > batch
> > > > > > > > > > > coordinator when a transaction is finished.
> > > > > > > > > > >
> > > > > > > > > > > With this proposal, transactions will also be able to
> > span
> > > both
> > > > > > > > classic
> > > > > > > > > > > and Diskless topics.
> > > > > > > > > > >
> > > > > > > > > > > *Queues*
> > > > > > > > > > >
> > > > > > > > > > > The share group coordination and management is a side job
> > > that
> > > > > > > > doesn't
> > > > > > > > > > > interfere with the topic itself (leadership, replicas,
> > > physical
> > > > > > > > storage
> > > > > > > > > > of
> > > > > > > > > > > records, etc.) and non-queue producers and consumers
> > > (Fetch and
> > > > > > > > Produce
> > > > > > > > > > > RPCs, consumer group-related RPCs are not affected.) We
> > > don't
> > > > > see any
> > > > > > > > > > > reason why we can't make Diskless topics compatible with
> > > share
> > > > > > > > groups the
> > > > > > > > > > > same way as classic topics are. Even on the code level,
> > we
> > > > > don't
> > > > > > > > expect
> > > > > > > > > > any
> > > > > > > > > > > serious refactoring: the same reading routines are used
> > > that
> > > > > are
> > > > > > > > used for
> > > > > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > > > > > implementation-focused?
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Ivan
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you all for your questions and design input on
> > > > > KIP-1150.
> > > > > > > > > > > >
> > > > > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> > > > > design. To
> > > > > > > > > > > summarize
> > > > > > > > > > > > the changes:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. The design prioritizes integrating with the existing
> > > > > KIP-405
> > > > > > > > Tiered
> > > > > > > > > > > > Storage interfaces, permitting data produced to a
> > > Diskless
> > > > > topic
> > > > > > > > to be
> > > > > > > > > > > > moved to tiered storage.
> > > > > > > > > > > > This lowers the scalability requirements for the Batch
> > > > > Coordinator
> > > > > > > > > > > > component, and allows Diskless to compose with Tiered
> > > Storage
> > > > > > > > plugin
> > > > > > > > > > > > features such as encryption and alternative data
> > formats.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. Consumer fetches are now served from local segments,
> > > > > making use
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > > functionality
> > > > > > > > > > > already
> > > > > > > > > > > > built into classic topics.
> > > > > > > > > > > > However, local segments are now considered cache
> > > elements,
> > > > > do not
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > be durably stored, and can be built without contacting
> > > any
> > > > > other
> > > > > > > > > > > replicas.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. The design has been simplified substantially, by
> > > removing
> > > > > the
> > > > > > > > > > previous
> > > > > > > > > > > > Diskless consume flow, distributed cache component, and
> > > > > "object
> > > > > > > > > > > > compaction/merging" step.
> > > > > > > > > > > >
> > > > > > > > > > > > The design maintains leaderless produces as enabled by
> > > the
> > > > > Batch
> > > > > > > > > > > > Coordinator, and the same latency profiles as the
> > earlier
> > > > > design,
> > > > > > > > while
> > > > > > > > > > > > being simpler and integrating better into the existing
> > > > > ecosystem.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks, and we are eager to hear your feedback on the
> > new
> > > > > design.
> > > > > > > > > > > > Greg Harris
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > > <ju...@confluent.io.invalid>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi, Jan,
> > > > > > > > > > > > >
> > > > > > > > > > > > > For me, the main gap of KIP-1150 is the support of
> > all
> > > > > existing
> > > > > > > > > > client
> > > > > > > > > > > > > APIs. Currently, there is no design for supporting
> > APIs
> > > > > like
> > > > > > > > > > > transactions
> > > > > > > > > > > > > and queues.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Would it be a good time to ask for the current
> > > status of
> > > > > this
> > > > > > > > KIP?
> > > > > > > > > > I
> > > > > > > > > > > > > > haven't seen much activity here for the past 2
> > > months,
> > > > > the
> > > > > > > > vote got
> > > > > > > > > > > > > vetoed
> > > > > > > > > > > > > > but I think the pending questions have been
> > answered
> > > > > since
> > > > > > > > then.
> > > > > > > > > > > KIP-1183
> > > > > > > > > > > > > > (AutoMQ's proposal) also didn't have any activity
> > > since
> > > > > May.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real
> > choices
> > > > > that can
> > > > > > > > be
> > > > > > > > > > > > > > made, with a coordinator-based approach being by
> > far
> > > the
> > > > > > > > dominant
> > > > > > > > > > one
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > it comes to market adoption - but all these are
> > > > > standalone
> > > > > > > > > > products.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm a big fan of both approaches, but would hate to
> > > see a
> > > > > > > > stall. So
> > > > > > > > > > > the
> > > > > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Maybe it's time to start another vote? Colin
> > McCabe -
> > > > > have your
> > > > > > > > > > > questions
> > > > > > > > > > > > > > been answered? If not, is there anything I can do
> > to
> > > > > help? I'm
> > > > > > > > > > deeply
> > > > > > > > > > > > > > familiar with both architectures and have written
> > > about
> > > > > both?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > Jan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> > Kozlovski
> > > <
> > > > > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > a) group all the KIP email threads in the main
> > one
> > > > > (just a
> > > > > > > > bunch
> > > > > > > > > > of
> > > > > > > > > > > > > links
> > > > > > > > > > > > > > > to everything)
> > > > > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It's a bit hard to track it all - for example, I
> > > was
> > > > > > > > searching
> > > > > > > > > > for
> > > > > > > > > > > a
> > > > > > > > > > > > > > > discuss thread for KIP-1165 for a while; As far
> > as
> > > I
> > > > > can
> > > > > > > > tell, it
> > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > exist yet.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Since the KIPs are published (by virtue of having
> > > the
> > > > > root
> > > > > > > > KIP be
> > > > > > > > > > > > > > > published, having a DISCUSS thread and links to
> > > > > sub-KIPs
> > > > > > > > where
> > > > > > > > > > were
> > > > > > > > > > > > > aimed
> > > > > > > > > > > > > > > to move the discussion towards), I think it would
> > > be
> > > > > good to
> > > > > > > > > > create
> > > > > > > > > > > > > > DISCUSS
> > > > > > > > > > > > > > > threads for them all.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Stan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We want to start a new KIP discussion about
> > > > > introducing a
> > > > > > > > new
> > > > > > > > > > > type of
> > > > > > > > > > > > > > > > topics that would make use of Object Storage as
> > > the
> > > > > primary
> > > > > > > > > > > source of
> > > > > > > > > > > > > > > > storage. However, as this KIP is big we decided
> > > to
> > > > > split it
> > > > > > > > > > into
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > that aims to discuss if Apache Kafka should aim
> > > to
> > > > > have
> > > > > > > > this
> > > > > > > > > > > type of
> > > > > > > > > > > > > > > > feature at all. This KIP doesn't go onto
> > details
> > > on
> > > > > how to
> > > > > > > > > > > implement
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > This follows the same approach used when we
> > > discussed
> > > > > > > > KRaft.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > But as we know that it is sometimes really hard
> > > to
> > > > > discuss
> > > > > > > > on
> > > > > > > > > > > that
> > > > > > > > > > > > > meta
> > > > > > > > > > > > > > > > level, we also created several sub-kips (linked
> > > in
> > > > > > > > KIP-1150)
> > > > > > > > > > that
> > > > > > > > > > > > > offer
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS
> > > threads
> > > > > for
> > > > > > > > each
> > > > > > > > > > > type of
> > > > > > > > > > > > > > > > concern and keep this one to discuss whether
> > > Apache
> > > > > Kafka
> > > > > > > > wants
> > > > > > > > > > > to
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks in advance on behalf of all the authors
> > of
> > > > > this KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 |
> > > aiven.io
> > > > > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu
> > > Valtonen,
> > > > > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Luke Chen <sh...@gmail.com>]

Hi Ivan,

As Jun pointed out, the updated design seems to have some shortcomings
although it simplifies the implementation.

I'm wondering if the complexity of designing txn and queue is because of
leaderless cluster, do you think it will be simpler if we only focus on the
"diskless" design to handle object compaction/merging to/from the remote
storage to save the cross-AZ cost?


Thank you,
Luke

On Sat, Oct 18, 2025 at 5:22 AM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Ivan,
>
> Thanks for the explanation.
>
> "we write the reference to the WAL file with the batch data"
>
> I understand the approach now, but I think it is a hacky one. There are
> multiple short comings with this design. First, it degrades the durability.
> For each partition, now there are two files being actively written at a
> given point of time, one for the data and another for the metadata.
> Flushing each file requires a separate IO. If the disk has 1K IOPS and we
> have 5K partitions in a broker, currently we can afford to flush each
> partition every 5 seconds, achieving an RPO of 5 seconds. If we double the
> number of files per partition, we can only flush each partition every 10
> seconds, which makes RPO twice as bad. Second, if we ever need this
> metadata somewhere else, say in the WAL file manager, the consumer needs to
> subscribe to every partition in the cluster, which is inefficient. The
> actual benefit of this approach is also questionable. On the surface, it
> might seem that we could reduce the number of lines that need to be changed
> for this KIP. However, the changes are quite intrusive to the classic
> partition's code path and will probably make the code base harder to
> maintain in the long run. I like the original approach based on the batch
> coordinator much better than this one. We could probably refactor the
> producer state code so that it could be reused in the batch coordinator.
>
> Thanks for addressing the concerns on the number of RPCs in the produce
> path. I agree that with the metadata crafting mechanism, we could mitigate
> the PRC problem. However, since we now require the metadata to be
> collocated with the data on the same set of brokers, it's weird that they
> are now managed by different mechanisms. The data assignment now uses the
> metadata crafting mechanism, but the metadata is stored in the classic
> partition using its own assignment strategy. It will be complicated to keep
> them collocated.
>
> I am also concerned about the removal of the object compaction/merging
> step. My first concern is on the amount of metadata that need to be kept.
> Without object compcation, the metadata generated in the produce path can
> only be deleted after remote tiering kicks in. Let's say for every 250ms we
> produce 100 byte of metadata per partition. Let's say remoting tiering
> kicks in after 5 hours. In a cluster with 100K partitions, we need to keep
> about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
> signficant. A second concern is on performance. Every time we need to
> rebuild the caching data, we need to read a bunch of small objects from S3,
> slowing down the building process. If a consumer happens to need such data,
> it could slow down the application.
>
> I see a tendency toward primarily optimizing for the fewest code changes in
> the KIP. Instead, our primary goal should be a clean design that can last
> for the long term.
>
> Thanks,
>
> Jun
>
> On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>
> > Hi Jun,
> >
> > Thank you for your message. I’m sorry that I failed to clearly explain
> the
> > idea. Let me try to fix this.
> >
> > > Does each partition now have a metadata partition and a separate data
> > > partition? If so, I am concerned that it essentially doubles the number
> > of
> > > partitions, which impacts the number of open file descriptors and the
> > > required IOPS, and so on. It also seems wasteful to have a separate
> > > partition just to store the metadata. It's as if we are creating an
> > > internal topic with an unbounded number of partitions.
> >
> > No. There will be only one physical partition per diskless partition. Let
> > me explain this with an example. Let’s say we have a diskless partition
> > topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce some
> > batches to this partition. The content of the segment file will be
> > something like this (for each batch):
> >
> > BaseOffset: 00000000000000000000 (like in classic)
> > Length: 123456 (like in classic)
> > PartitionLeaderEpoch: like in classic
> > Magic: like in classic
> > CRC: like in classic
> > Attributes: like in classic
> > LastOffsetDelta: like in classic
> > BaseTimestamp: like in classic
> > MaxTimestamp: like in classic
> > ProducerId: like in classic
> > ProducerEpoch: like in classic
> > BaseSequence: like in classic
> > RecordsCount: like in classic
> > Records:
> >   path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> > 123456
> >
> > It looks very much like classic log entries. The only difference is that
> > instead of writing real Records, we write the reference to the WAL file
> > with the batch data (I guess we need only the name and the byte offset,
> > because the byte length is the standard field above). Otherwise, it’s a
> > normal Kafka log with the leader and replicas.
> >
> > So we have as many partitions for diskless as for classic. As of open
> file
> > descriptors, let’s proceed to the following:
> >
> > > Are the metadata and
> > > the data for the same partition always collocated on the same broker?
> If
> > > so, how do we enforce that when replicas are reassigned?
> >
> > The source of truth for the data is still in WAL files on object storage.
> > The source of truth for the metadata is in segment files on the brokers
> in
> > the replica set. Two new mechanisms are planned, both independent of this
> > new proposal, but I want to present them to give the idea that only a
> > limited amount of data files will be operated locally:
> > - We want to assemble batches into segment files and offload them to
> > tiered storage in order to prevent the unbounded growth of batch
> metadata.
> > For this, we need to open only  a few file descriptors (for the segment
> > file itself + the necessary indexes) before the segment is fully written
> > and handed over to RemoteLogManager.
> > - We want to assemble local segment files for caching purposes as well,
> > i.e. to speed up fetching. This will not materialize the full content of
> > the log, but only the hot set according to some policy (or configurable
> > policies), i.e. the number of segments and file descriptors will also be
> > limited.
> >
> > > The number of RPCs in the produce path is significantly higher. For
> > > example, if a produce request has 100 partitions, in a cluster with 100
> > > brokers, each produce request could generate 100 more RPC requests.
> This
> > > will significantly increase the request rate.
> >
> > This is a valid concern that we considered, but this issue can be
> > mitigated. I’ll try to explain the approach.
> > The situation with a single broker is trivial: all the commit requests go
> > from the broker to itself.
> > Let’s scale this to a multi-broker cluster, but located in the single
> rack
> > (AZ). Any broker can accept Produce requests for diskless partitions, but
> > we can tell producers (through metadata) to always send Produce requests
> to
> > leaders. For example, broker 0 hosts the leader replicas for diskless
> > partitions t1-0, t2-1, t3-0. It will receive diskless Produce requests
> for
> > these partitions in various combinations, but only for them.
> >
> >                     Broker 0
> >               +-----------------+
> >               |    t1-0         |
> >               |    t2-1 <--------------------+
> >               |    t3-0         |            |
> >  produce      | +-------------+ |            |
> >  requests     | |  diskless   | |            |
> > --------------->|   produce   +--------------+
> >  for these    | | WAL buffer  | |    commit requests
> >  partitions   | +-------------+ |    for these partitions
> >               |                 |
> >               +-----------------+
> >
> > The same applies for other brokers in this cluster. Effectively, each
> > broker will commit only to itself, which effectively means 1 commit
> request
> > per WAL buffer (this may be 0 physical network calls, if we wish, just a
> > local function call).
> >
> > Now let’s scale this to multiple racks (AZs). Obviously, we cannot always
> > send Produce requests to the designated leaders of diskless partitions:
> > this would mean inter-AZ network traffic, which we would like to avoid.
> To
> > avoid it, we say that every broker has a “diskless produce
> representative”
> > in every AZ. If we continue our example: when a Produce request for t1-0,
> > t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0 (in the
> > broker’s AZ the representative is the broker itself). However, if it
> comes
> > from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
> >
> >   |produce requests         |produce requests        |produce requests
> >   |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1,
> t3-0
> >   |from AZ 0                |from AZ 1               |from AZ 2
> >   v                         v                        v
> >  Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> > +---------------+      +---------------+      +---------------+
> > |     t1-0      |      |               |      |               |
> > |     t2-1      |      |               |      |               |
> > |     t3-0      |      |               |      |               |
> > +---------------+      +--------+------+      +--------+------+
> >      ^     ^                    |                      |
> >      |     +--------------------+                      |
> >      |     commit requests for these partitions        |
> >      |                                                 |
> >      +-------------------------------------------------+
> >            commit requests for these partitions
> >
> > All the partitions that broker 0 is the leader of will be “represented”
> by
> > brokers 1 and 2 in their AZs.
> >
> > Of course, this relationship goes both ways between AZs (not necessarily
> > between the same brokers). It means that provided the cluster is balanced
> > by the number of brokers per AZ, each broker will represent
> (number_of_azs
> > - 1) other brokers. This will result in the situation that for the
> majority
> > of commits, each broker will do up to (number_of_azs - 1) network commit
> > requests (plus one local). Cloud regions tend to have 3 AZs, very rarely
> > more. That means, brokers will be doing up to 2 network commit requests
> per
> > WAL file.
> >
> > There are the following exceptions:
> > 1. Broker count imbalance between AZs. For example, when we have 2 AZs
> and
> > one has three brokers and another AZ has one. This one broker will do
> > between 1 and 3 commit requests per WAL file. This is not an extreme
> > amplification. Such an imbalance is not healthy in most practical setups
> > and should be avoided anyway.
> > 2. Leadership changes and metadata propagation period. When the partition
> > t3-0 is relocated from broker 0 to some broker 3, the producers will not
> > know this immediately (unless we want to be strict and respond with
> > NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come together
> in a
> > WAL buffer on broker 2, it will have to send two commit requests: to
> broker
> > 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This situation
> > is not permanent and as producers update the cluster metadata, it will be
> > resolved.
> >
> > This all could be built with the metadata crafting mechanism only (which
> > is anyway needed for Diskless in one way or another to direct producers
> and
> > consumers where we need to avoid inter-AZ traffic), just with the right
> > policy for it (for example, some deterministic hash-based formula). I.e.
> no
> > explicit support for “produce representative” or anything like this is
> > needed on the cluster level, in KRaft, etc.
> >
> > > The same WAL file metadata is now duplicated into two places, partition
> > > leader and WAL File Manager. Which one is the source of truth, and how
> do
> > > we maintain consistency between the two places?
> >
> > We do only two operations on WAL files that span multiple diskless
> > partitions: committing and deleting. Commits can be done independently as
> > described above. But deletes are different, because when a file is
> deleted,
> > this affects all the partitions that still have alive batches in this
> file
> > (if any).
> >
> > The WAL file manager is a necessary point of coordination to delete WAL
> > files safely. We can say it is the source of truth about files
> themselves,
> > while the partition leaders and their logs hold the truth about whether a
> > particular file contains live batches of this particular partition.
> >
> > The file manager will do this important task: be able to say for sure
> that
> > a file does not contain any live batch of any existing partition. For
> this,
> > it will have to periodically check against the partition leaders.
> > Considering that batch deletion is irreversible, when we declare a file
> > “empty”, this is guaranteed to be and stay so.
> >
> > The file manager has to know about files being committed to start track
> > them and periodically check if they are empty. We can consider various
> ways
> > to achieve this:
> > 1. As was proposed in my previous message: best effort commit by brokers
> +
> > periodic prefix scans of object storage to detect files that went below
> the
> > radar due to network issue or the file manager temporary unavailability.
> > We’re speaking about listing the file names only and opening only
> > previously unknown files in order to find the partitions involved with
> them.
> > 2. Only do scans without explicit commit, i.e. fill the list of files
> > fully asynchronously and in the background. This may be not ideal due to
> > costs and performance of scanning tons of files. However, the number of
> > live WAL files should be limited due to tiered storage offloading + we
> can
> > optimize this if we give files some global soft order in their names.
> >
> > > I am not sure how this design simplifies the implementation. The
> existing
> > > producer/replication code can't be simply reused. Adjusting both the
> > write
> > > path in the leader and the replication path in the follower to
> understand
> > > batch-header only data is quite intrusive to the existing logic.
> >
> > It is true that we’ll have to change LocalLog and UnifiedLog in order to
> > support these changes. However, it seems that idempotence, transactions,
> > queues, tiered storage will have to be changed less than with the
> original
> > design. This is because the partition leader state would remain in the
> same
> > place (on brokers) and existing workflows that involve it would have to
> be
> > changed less compared to the situation where we globalize the partition
> > leader state in the batch coordinator. I admit this is hard to make
> > convincing without both real implementations to hand :)
> >
> > > I am also
> > > not sure how this enables seamless switching the topic modes between
> > > diskless and classic. Could you provide more details on those?
> >
> > Let’s consider the scenario of turning a classic topic into diskless. The
> > user sets diskless.enabled=true, the leader receives this metadata update
> > and does the following:
> > 1. Stop accepting normal append writes.
> > 2. Close the current active segment.
> > 3. Start a new segment that will be written in the diskless format (i.e.
> > without data).
> > 4. Start accepting diskless commits.
> >
> > Since it’s the same log, the followers will know about that switch
> > consistently. They will finish replicating the classic segments and start
> > replicating the diskless ones. They will always know where each batch is
> > located (either inside a classic segment or referenced by a diskless
> one).
> > Switching back should be similar.
> >
> > Doing this with the coordinator is possible, but has some caveats. The
> > leader must do the following:
> > 1. Stop accepting normal append writes.
> > 2. Close the current active segment.
> > 3. Write a special control segment to persist and replicate the fact that
> > from offset N the partition is now in the diskless mode.
> > 4. Inform the coordinator about the first offset N of the “diskless era”.
> > 5. Inform the controller quorum that the transition has finished and that
> > brokers now can process diskless writes for this partition.
> > This could fail at some points, so this will probably require some
> > explicit state machine with replication either in the partition log or in
> > KRaft.
> >
> > It seems that the coordinator-less approach makes this simpler because
> the
> > “coordinator” for the partition and the partition leader are the same and
> > they store the partition metadata in the same log, too. While in the
> > coordinator approach we have to perform some kind of a distributed commit
> > to handover metadata management from the classic partition leader to the
> > batch coordinator.
> >
> > I hope these explanations help to clarify the idea. Please let me know if
> > I should go deeper anywhere.
> >
> > Best,
> > Ivan and the Diskless team
> >
> > On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > > Hi, Ivan,
> > >
> > > Thanks for the update.
> > >
> > > I am not sure that I fully understand the new design, but it seems less
> > > clean than before.
> > >
> > > Does each partition now have a metadata partition and a separate data
> > > partition? If so, I am concerned that it essentially doubles the number
> > of
> > > partitions, which impacts the number of open file descriptors and the
> > > required IOPS, and so on. It also seems wasteful to have a separate
> > > partition just to store the metadata. It's as if we are creating an
> > > internal topic with an unbounded number of partitions. Are the metadata
> > and
> > > the data for the same partition always collocated on the same broker?
> If
> > > so, how do we enforce that when replicas are reassigned?
> > >
> > > The number of RPCs in the produce path is significantly higher. For
> > > example, if a produce request has 100 partitions, in a cluster with 100
> > > brokers, each produce request could generate 100 more RPC requests.
> This
> > > will significantly increase the request rate.
> > >
> > > The same WAL file metadata is now duplicated into two places, partition
> > > leader and WAL File Manager. Which one is the source of truth, and how
> do
> > > we maintain consistency between the two places?
> > >
> > > I am not sure how this design simplifies the implementation. The
> existing
> > > producer/replication code can't be simply reused. Adjusting both the
> > write
> > > path in the leader and the replication path in the follower to
> understand
> > > batch-header only data is quite intrusive to the existing logic. I am
> > also
> > > not sure how this enables seamless switching the topic modes between
> > > diskless and classic. Could you provide more details on those?
> > >
> > > Jun
> > >
> > > On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > >
> > > > Hi dear Kafka community,
> > > >
> > > > In the initial Diskless proposal, we proposed to have a separate
> > > > component, batch/diskless coordinator, whose role would be to
> centrally
> > > > manage the batch and WAL file metadata for diskless topics. This
> > component
> > > > drew many reasonable comments from the community about how it would
> > support
> > > > various Kafka features (transactions, queues) and its scalability.
> > While we
> > > > believe we have good answers to all the expressed concerns, we took a
> > step
> > > > back and looked at the problem from a different perspective.
> > > >
> > > > We would like to propose an alternative Diskless design *without a
> > > > centralized coordinator*. We believe this approach has potential and
> > > > propose to discuss it as it may be more appealing to the community.
> > > >
> > > > Let us explain the idea. Most of the complications with the original
> > > > Diskless approach come from one necessary architecture change:
> > globalizing
> > > > the local state of partition leader in the batch coordinator. This
> > causes
> > > > deviations to the established workflows in various features like
> > produce
> > > > idempotence and transactions, queues, retention, etc. These
> deviations
> > need
> > > > to be carefully considered, designed, and later implemented and
> > tested. In
> > > > the new approach we want to avoid this by making partition leaders
> > again
> > > > responsible for managing their partitions, even in diskless topics.
> > > >
> > > > In classic Kafka topics, batch data and metadata are blended together
> > in
> > > > the one partition log. The crux of the Diskless idea is to decouple
> > them
> > > > and move data to the remote storage, while keeping metadata somewhere
> > else.
> > > > Using the central batch coordinator for managing batch metadata is
> one
> > way,
> > > > but not the only.
> > > >
> > > > Let’s now think about managing metadata for each user partition
> > > > independently. Generally partitions are independent and don’t share
> > > > anything apart from that their data are mixed in WAL files. If we
> > figure
> > > > out how to commit and later delete WAL files safely, we will achieve
> > the
> > > > necessary autonomy that allows us to get rid of the central batch
> > > > coordinator. Instead, *each diskless user partition will be managed
> by
> > its
> > > > leader*, as in classic Kafka topics. Also like in classic topics, the
> > > > leader uses the partition log as the way to persist batch metadata,
> > i.e.
> > > > the regular batch header + the information about how to find this
> > batch on
> > > > remote storage. In contrast to classic topics, batch data is in
> remote
> > > > storage.
> > > >
> > > > For clarity, let’s compare the three designs:
> > > >  • Classic topics:
> > > >    • Data and metadata are co-located in the partition log.
> > > >    • The partition log content: [Batch header (metadata)|Batch data].
> > > >    • The partition log is replicated to the followers.
> > > >    • The replicas and leader have local state built from metadata.
> > > >  • Original Diskless:
> > > >    • Metadata is in the batch coordinator, data is on remote storage.
> > > >    • The partition state is global in the batch coordinator.
> > > >  • New Diskless:
> > > >    • Metadata is in the partition log, data is on remote storage.
> > > >    • Partition log content: [Batch header (metadata)|Batch
> coordinates
> > on
> > > > remote storage].
> > > >    • The partition log is replicated to the followers.
> > > >    • The replicas and leader have local state built from metadata.
> > > >
> > > > Let’s consider the produce path. Here’s the reminder of the original
> > > > Diskless design:
> > > >
> > > >
> > > > The new approach could be depicted as the following:
> > > >
> > > >
> > > > As you can see, the main difference is that now instead of a single
> > commit
> > > > request to the batch coordinator, we send multiple parallel commit
> > requests
> > > > to all the leaders of each partition involved in the WAL file. Each
> of
> > them
> > > > will commit its batches independently, without coordinating with
> other
> > > > leaders and any other components. Batch data is addressed by the WAL
> > file
> > > > name, the byte offset and size, which allows partitions to know
> nothing
> > > > about other partitions to access their data in shared WAL files.
> > > >
> > > > The number of partitions involved in a single WAL file may be quite
> > large,
> > > > e.g. a hundred. A hundred network requests to commit one WAL file is
> > very
> > > > impractical. However, there are ways to reduce this number:
> > > >  1. Partition leaders are located on brokers. Requests to leaders on
> > one
> > > > broker could be grouped together into a single physical network
> request
> > > > (resembling the normal Produce request that may carry batches for
> many
> > > > partitions inside). This will cap the number of network requests to
> the
> > > > number of brokers in the cluster.
> > > >  2. If we craft the cluster metadata to make producers send their
> > requests
> > > > to the right brokers (with respect to AZs), we may achieve the higher
> > > > concentration of logical commit requests in physical network requests
> > > > reducing the number of the latter ones even further, ideally to one.
> > > >
> > > > Obviously, out of multiple commit requests some may fail or time out
> > for a
> > > > variety of reasons. This is fine. Some producers will receive totally
> > or
> > > > partially failed responses to their Produce requests, similar to what
> > they
> > > > would have received when appending to a classic topic fails or times
> > out.
> > > > If a partition experiences problems, other partitions will not be
> > affected
> > > > (again, like in classic topics). Of course, the uncommitted data will
> > be
> > > > garbage in WAL files. But WAL files are short-lived (batches are
> > constantly
> > > > assembled into segments and offloaded to tiered storage), so this
> > garbage
> > > > will be eventually deleted.
> > > >
> > > > For safely deleting WAL files we now need to centrally manage them,
> as
> > > > this is the only state and logic that spans multiple partitions. On
> the
> > > > diagram, you can see another commit request called “Commit file (best
> > > > effort)” going to the WAL File Manager. This manager will be
> > responsible
> > > > for the following:
> > > >  1. Collecting (by requests from brokers) and persisting information
> > about
> > > > committed WAL files.
> > > >  2. To handle potential failures in file information delivery, it
> will
> > be
> > > > doing prefix scan on the remote storage periodically to find and
> > register
> > > > unknown files. The period of this scan will be configurable and
> ideally
> > > > should be quite long.
> > > >  3. Checking with the relevant partition leaders (after a grace
> > period) if
> > > > they still have batches in a particular file.
> > > >  4. Physically deleting files when they aren’t anymore referred to by
> > any
> > > > partition.
> > > >
> > > > This new design offers the following advantages:
> > > >  1. It simplifies the implementation of many Kafka features such as
> > > > idempotence, transactions, queues, tiered storage, retention. Now we
> > don’t
> > > > need to abstract away and reuse the code from partition leaders in
> the
> > > > batch coordinator. Instead, we will literally use the same code paths
> > in
> > > > leaders, with little adaptation. Workflows from classic topics mostly
> > > > remain unchanged.
> > > > For example, it seems that
> > > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > > > KafkaApis.handleWriteTxnMarkersRequest used for transaction support
> on
> > the
> > > > partition leader side could be used for diskless topics with little
> > > > adaptation. ProducerStateManager, needed for both idempotent produce
> > and
> > > > transactions, would be reused.
> > > > Another example is share groups support, where the share partition
> > leader,
> > > > being co-located with the partition leader, would execute the same
> > logic
> > > > for both diskless and classic topics.
> > > >  2. It returns to the familiar partition-based scaling model, where
> > > > partitions are independent.
> > > >  3. It makes the operation and failure patterns closer to the
> familiar
> > > > ones from classic topics.
> > > >  4. It opens a straightforward path to seamless switching the topics
> > modes
> > > > between diskless and classic.
> > > >
> > > > The rest of the things remain unchanged compared to the previous
> > Diskless
> > > > design (after all previous discussions). Such things as local segment
> > > > materialization by replicas, the consume path, tiered storage
> > integration,
> > > > etc.
> > > >
> > > > If the community finds this design more suitable, we will update the
> > > > KIP(s) accordingly and continue working on it. Please let us know
> what
> > you
> > > > think.
> > > >
> > > > Best regards,
> > > > Ivan and Diskless team
> > > >
> > > > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > > Hi Justine,
> > > > >
> > > > > Yes, you're right. We need to track the aborted transactions for in
> > the
> > > > diskless coordinator for as long as the corresponding offsets are
> > there.
> > > > With the tiered storage unification Greg mentioned earlier, this will
> > be
> > > > finite time even for infinite data retention.
> > > > >
> > > > > Best,
> > > > > Ivan
> > > > >
> > > > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > > Hey Ivan,
> > > > > >
> > > > > > Thanks for the response. I think most of what you said made
> sense,
> > but
> > > > I
> > > > > > did have some questions about this part:
> > > > > >
> > > > > > > As we understand this, the partition leader in classic topics
> > forgets
> > > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > > transaction coordinator acts like the main guardian, allowing
> > partition
> > > > > > leaders to do this safely. Please correct me if this is wrong. We
> > think
> > > > > > about relying on this with the batch coordinator and delete the
> > > > information
> > > > > > about a transaction once it’s finished (as there’s no replication
> > and
> > > > HWM
> > > > > > advances immediately).
> > > > > >
> > > > > > I didn't quite understand this. In classic topics, we have maps
> for
> > > > ongoing
> > > > > > transactions which remove state when the transaction is completed
> > and
> > > > an
> > > > > > aborted transactions index which is retained for much longer.
> Once
> > the
> > > > > > transaction is completed, the coordinator is no longer involved
> in
> > > > > > maintaining this partition side state, and it is subject to
> > compaction
> > > > etc.
> > > > > > Looking back at the outline provided above, I didn't see much
> > about the
> > > > > > fetch path, so maybe that could be expanded a bit further. I saw
> > the
> > > > > > following in a response:
> > > > > > > When the broker constructs a fully valid local segment, all the
> > > > necessary
> > > > > > control batches will be inserted and indices, including the
> > transaction
> > > > > > index will be built to serve FetchRequests exactly as they are
> > today.
> > > > > >
> > > > > > Based on this, it seems like we need to retain the information
> > about
> > > > > > aborted txns for longer.
> > > > > >
> > > > > > Thanks,
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me>
> > wrote:
> > > > > >
> > > > > > > Hi Justine and all,
> > > > > > >
> > > > > > > Thank you for your questions!
> > > > > > >
> > > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > > producer ID
> > > > > > > > and epoch, the positive result of this check could be cached
> > > > locally
> > > > > > > > Are we saying that only new transaction version 2
> transactions
> > can
> > > > be
> > > > > > > used
> > > > > > > > here? If not, we can't uniquely identify transactions with
> > > > producer id +
> > > > > > > > epoch
> > > > > > >
> > > > > > > You’re right that we (probably unintentionally) focused only on
> > > > version 2.
> > > > > > > We can either limit the support to version 2 or consider using
> > some
> > > > > > > surrogates to support version 1.
> > > > > > >
> > > > > > > > JO 2. >The batch coordinator does the final transactional
> > checks
> > > > of the
> > > > > > > > batches. This procedure would output the same errors like the
> > > > partition
> > > > > > > > leader in classic topics would do.
> > > > > > > > Can you expand on what these checks are? Would you be
> checking
> > if
> > > > the
> > > > > > > > transaction was still ongoing for example?* *
> > > > > > >
> > > > > > > Yes, the producer epoch, that the transaction is ongoing, and
> of
> > > > course
> > > > > > > the normal idempotence checks. What the partition leader in the
> > > > classic
> > > > > > > topics does before appending a batch to the local log (e.g. in
> > > > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> > > > unfortunately
> > > > > > > cannot do these checks before appending the data to the WAL
> > segment
> > > > and
> > > > > > > uploading it, but we can “tombstone” these batches in the batch
> > > > coordinator
> > > > > > > during the final commit.
> > > > > > >
> > > > > > > > Is there state about ongoing
> > > > > > > > transactions in the batch coordinator? I see some other state
> > > > mentioned
> > > > > > > in
> > > > > > > > the End transaction section, but it's not super clear what
> > state is
> > > > > > > stored
> > > > > > > > and when it is stored.
> > > > > > >
> > > > > > > Right, this should have been more explicit. As the partition
> > leader
> > > > tracks
> > > > > > > ongoing transactions for classic topics, the batch coordinator
> > has
> > > > to as
> > > > > > > well. So when a transaction starts and ends, the transaction
> > > > coordinator
> > > > > > > must inform the batch coordinator about this.
> > > > > > >
> > > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> > that
> > > > would
> > > > > > > be
> > > > > > > > stored in the batch coordinator?
> > > > > > >
> > > > > > > Yes. This could be deduced from the committed batches and other
> > > > > > > information, but for the sake of performance we’d better store
> it
> > > > > > > explicitly.
> > > > > > >
> > > > > > > > JO 4. Are there any thoughts about how long transactional
> > state is
> > > > > > > > maintained in the batch coordinator and how it will be
> cleaned
> > up?
> > > > > > >
> > > > > > > As we understand this, the partition leader in classic topics
> > forgets
> > > > > > > about a transaction once it’s replicated (HWM overpasses it).
> The
> > > > > > > transaction coordinator acts like the main guardian, allowing
> > > > partition
> > > > > > > leaders to do this safely. Please correct me if this is wrong.
> We
> > > > think
> > > > > > > about relying on this with the batch coordinator and delete the
> > > > information
> > > > > > > about a transaction once it’s finished (as there’s no
> replication
> > > > and HWM
> > > > > > > advances immediately).
> > > > > > >
> > > > > > > Best,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > > > Hey folks,
> > > > > > > >
> > > > > > > > Excited to see some updates related to transactions!
> > > > > > > >
> > > > > > > > I had a few questions.
> > > > > > > >
> > > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > > producer ID
> > > > > > > > and epoch, the positive result of this check could be cached
> > > > locally
> > > > > > > > Are we saying that only new transaction version 2
> transactions
> > can
> > > > be
> > > > > > > used
> > > > > > > > here? If not, we can't uniquely identify transactions with
> > > > producer id +
> > > > > > > > epoch
> > > > > > > >
> > > > > > > > JO 2. >The batch coordinator does the final transactional
> > checks
> > > > of the
> > > > > > > > batches. This procedure would output the same errors like the
> > > > partition
> > > > > > > > leader in classic topics would do.
> > > > > > > > Can you expand on what these checks are? Would you be
> checking
> > if
> > > > the
> > > > > > > > transaction was still ongoing for example? Is there state
> about
> > > > ongoing
> > > > > > > > transactions in the batch coordinator? I see some other state
> > > > mentioned
> > > > > > > in
> > > > > > > > the End transaction section, but it's not super clear what
> > state is
> > > > > > > stored
> > > > > > > > and when it is stored.
> > > > > > > >
> > > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> > that
> > > > would
> > > > > > > be
> > > > > > > > stored in the batch coordinator?
> > > > > > > >
> > > > > > > > JO 4. Are there any thoughts about how long transactional
> > state is
> > > > > > > > maintained in the batch coordinator and how it will be
> cleaned
> > up?
> > > > > > > >
> > > > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> > <ju...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Greg and Ivan,
> > > > > > > > >
> > > > > > > > > Thanks for the update. A few comments.
> > > > > > > > >
> > > > > > > > > JR 10. "Consumer fetches are now served from local
> segments,
> > > > making
> > > > > > > use of
> > > > > > > > > the
> > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > functionality
> > > > > > > already
> > > > > > > > > built into classic topics."
> > > > > > > > > JR 10.1 Does the broker build the producer state for each
> > > > partition in
> > > > > > > > > diskless topics?
> > > > > > > > > JR 10.2 For transactional data, the consumer fetches need
> to
> > know
> > > > > > > aborted
> > > > > > > > > records. How is that achieved?
> > > > > > > > >
> > > > > > > > > JR 11. "The batch coordinator saves that the transaction is
> > > > finished
> > > > > > > and
> > > > > > > > > also inserts the control batches in the corresponding logs
> > of the
> > > > > > > involved
> > > > > > > > > Diskless topics. This happens only on the metadata level,
> no
> > > > actual
> > > > > > > control
> > > > > > > > > batches are written to any file. "
> > > > > > > > > A fetch response could include multiple transactional
> > batches.
> > > > How
> > > > > > > does the
> > > > > > > > > broker obtain the information about the ending control
> batch
> > for
> > > > each
> > > > > > > > > batch? Does that mean that a fetch response needs to be
> > built by
> > > > > > > > > stitching record batches and generated control batches
> > together?
> > > > > > > > >
> > > > > > > > > JR 12. Queues: Is there still a share partition leader that
> > all
> > > > > > > consumers
> > > > > > > > > are routed to?
> > > > > > > > >
> > > > > > > > > JR 13. "Should the KIPs be modified to include this or it's
> > too
> > > > > > > > > implementation-focused?" It would be useful to include
> enough
> > > > details
> > > > > > > to
> > > > > > > > > understand correctness and performance impact.
> > > > > > > > >
> > > > > > > > > HC5. Henry has a valid point. Requests from a given
> producer
> > > > contain a
> > > > > > > > > sequence number, which is ordered. If a producer sends
> every
> > > > Produce
> > > > > > > > > request to an arbitrary broker, those requests could reach
> > the
> > > > batch
> > > > > > > > > coordinator in different order and lead to rejection of the
> > > > produce
> > > > > > > > > requests.
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> > ivan@ivanyu.me>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > We have also thought in a bit more details about
> > transactions
> > > > and
> > > > > > > queues,
> > > > > > > > > > here's the plan.
> > > > > > > > > >
> > > > > > > > > > *Transactions*
> > > > > > > > > >
> > > > > > > > > > The support for transactions in *classic topics* is based
> > on
> > > > precise
> > > > > > > > > > interactions between three actors: clients (mostly
> > producers,
> > > > but
> > > > > > > also
> > > > > > > > > > consumers), brokers (ReplicaManager and other classes),
> and
> > > > > > > transaction
> > > > > > > > > > coordinators. Brokers also run partition leaders with
> their
> > > > local
> > > > > > > state
> > > > > > > > > > (ProducerStateManager and others).
> > > > > > > > > >
> > > > > > > > > > The high level (some details skipped) workflow is the
> > > > following.
> > > > > > > When a
> > > > > > > > > > transactional Produce request is received by the broker:
> > > > > > > > > > 1. For each partition, the partition leader checks if a
> > > > non-empty
> > > > > > > > > > transaction is running for this partition. This is done
> > using
> > > > its
> > > > > > > local
> > > > > > > > > > state derived from the log metadata
> (ProducerStateManager,
> > > > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > > > 2. The transaction coordinator is informed about all the
> > > > partitions
> > > > > > > that
> > > > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > > > 3. The partition leaders do additional transactional
> > checks.
> > > > > > > > > > 4. The partition leaders append the transactional data to
> > > > their logs
> > > > > > > and
> > > > > > > > > > update some of their state (for example, log the fact
> that
> > the
> > > > > > > > > transaction
> > > > > > > > > > is running for the partition and its first offset).
> > > > > > > > > >
> > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > 1. The producer contacts the transaction coordinator
> > directly
> > > > with
> > > > > > > > > > EndTxnRequest.
> > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > > PREPARE_ABORT to
> > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > 3. The transaction coordinator sends
> > WriteTxnMarkersRequest to
> > > > the
> > > > > > > > > leaders
> > > > > > > > > > of the involved partitions.
> > > > > > > > > > 4. The partition leaders write the transaction markers to
> > > > their logs
> > > > > > > and
> > > > > > > > > > respond to the coordinator.
> > > > > > > > > > 5. The coordinator writes the final transaction state
> > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > >
> > > > > > > > > > In classic topics, partitions have leaders and lots of
> > > > important
> > > > > > > state
> > > > > > > > > > necessary for supporting this workflow is local. The main
> > > > challenge
> > > > > > > in
> > > > > > > > > > mapping this to Diskless comes from the fact there are no
> > > > partition
> > > > > > > > > > leaders, so the corresponding pieces of state need to be
> > > > globalized
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > batch coordinator. We are already doing this to support
> > > > idempotent
> > > > > > > > > produce.
> > > > > > > > > >
> > > > > > > > > > The high level workflow for *diskless topics* would look
> > very
> > > > > > > similar:
> > > > > > > > > > 1. For each partition, the broker checks if a non-empty
> > > > transaction
> > > > > > > is
> > > > > > > > > > running for this partition. In contrast to classic
> topics,
> > > > this is
> > > > > > > > > checked
> > > > > > > > > > against the batch coordinator with a single RPC. Since a
> > > > transaction
> > > > > > > > > could
> > > > > > > > > > be uniquely identified with producer ID and epoch, the
> > positive
> > > > > > > result of
> > > > > > > > > > this check could be cached locally (for the double
> > configured
> > > > > > > duration
> > > > > > > > > of a
> > > > > > > > > > transaction, for example).
> > > > > > > > > > 2. The same: The transaction coordinator is informed
> about
> > all
> > > > the
> > > > > > > > > > partitions that aren’t part of the transaction to include
> > them.
> > > > > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > > > > 4. The broker appends the transactional data to the
> current
> > > > shared
> > > > > > > WAL
> > > > > > > > > > segment. It doesn’t update any transaction-related state
> > for
> > > > Diskless
> > > > > > > > > > topics, because it doesn’t have any.
> > > > > > > > > > 5. The WAL segment is committed to the batch coordinator
> > like
> > > > in the
> > > > > > > > > > normal produce flow.
> > > > > > > > > > 6. The batch coordinator does the final transactional
> > checks
> > > > of the
> > > > > > > > > > batches. This procedure would output the same errors like
> > the
> > > > > > > partition
> > > > > > > > > > leader in classic topics would do. I.e. some batches
> could
> > be
> > > > > > > rejected.
> > > > > > > > > > This means, there will potentially be garbage in the WAL
> > > > segment
> > > > > > > file in
> > > > > > > > > > case of transactional errors. This is preferable to doing
> > more
> > > > > > > network
> > > > > > > > > > round trips, especially considering the WAL segments will
> > be
> > > > > > > relatively
> > > > > > > > > > short-living (see the Greg's update above).
> > > > > > > > > >
> > > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > > 1. The producer contacts the transaction coordinator
> > directly
> > > > with
> > > > > > > > > > EndTxnRequest.
> > > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > > PREPARE_ABORT to
> > > > > > > > > > its log and responds to the producer.
> > > > > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> > > > coordinator
> > > > > > > that
> > > > > > > > > > the transaction is finished.
> > > > > > > > > > 4. *[NEW]* The batch coordinator saves that the
> > transaction is
> > > > > > > finished
> > > > > > > > > > and also inserts the control batches in the corresponding
> > logs
> > > > of the
> > > > > > > > > > involved Diskless topics. This happens only on the
> metadata
> > > > level, no
> > > > > > > > > > actual control batches are written to any file. They will
> > be
> > > > > > > dynamically
> > > > > > > > > > created on Fetch and other read operations. We could
> > > > technically
> > > > > > > write
> > > > > > > > > > these control batches for real, but this would mean extra
> > > > produce
> > > > > > > > > latency,
> > > > > > > > > > so it's better just to mark them in the batch coordinator
> > and
> > > > save
> > > > > > > these
> > > > > > > > > > milliseconds.
> > > > > > > > > > 5. The transaction coordinator sends
> > WriteTxnMarkersRequest to
> > > > the
> > > > > > > > > leaders
> > > > > > > > > > of the involved partitions. – Now only to classic topics
> > now.
> > > > > > > > > > 6. The partition leaders of classic topics write the
> > > > transaction
> > > > > > > markers
> > > > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > > > 7. The coordinator writes the final transaction state
> > > > > > > COMPLETE_COMMIT or
> > > > > > > > > > COMPLETE_ABORT.
> > > > > > > > > >
> > > > > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > > > > 1. An extra network round trip between brokers and the
> > batch
> > > > > > > coordinator
> > > > > > > > > > when a new partition appear in the transaction. To
> > mitigate the
> > > > > > > impact of
> > > > > > > > > > them:
> > > > > > > > > >   - The results will be cached.
> > > > > > > > > >   - The calls for multiple partitions in one Produce
> > request
> > > > will be
> > > > > > > > > > grouped.
> > > > > > > > > >   - The batch coordinator should be optimized for fast
> > > > response to
> > > > > > > these
> > > > > > > > > > RPCs.
> > > > > > > > > >   - The fact that a single producer normally will
> > communicate
> > > > with a
> > > > > > > > > > single broker for the duration of the transaction further
> > > > reduces the
> > > > > > > > > > expected number of round trips.
> > > > > > > > > > 2. An extra round trip between the transaction
> coordinator
> > and
> > > > batch
> > > > > > > > > > coordinator when a transaction is finished.
> > > > > > > > > >
> > > > > > > > > > With this proposal, transactions will also be able to
> span
> > both
> > > > > > > classic
> > > > > > > > > > and Diskless topics.
> > > > > > > > > >
> > > > > > > > > > *Queues*
> > > > > > > > > >
> > > > > > > > > > The share group coordination and management is a side job
> > that
> > > > > > > doesn't
> > > > > > > > > > interfere with the topic itself (leadership, replicas,
> > physical
> > > > > > > storage
> > > > > > > > > of
> > > > > > > > > > records, etc.) and non-queue producers and consumers
> > (Fetch and
> > > > > > > Produce
> > > > > > > > > > RPCs, consumer group-related RPCs are not affected.) We
> > don't
> > > > see any
> > > > > > > > > > reason why we can't make Diskless topics compatible with
> > share
> > > > > > > groups the
> > > > > > > > > > same way as classic topics are. Even on the code level,
> we
> > > > don't
> > > > > > > expect
> > > > > > > > > any
> > > > > > > > > > serious refactoring: the same reading routines are used
> > that
> > > > are
> > > > > > > used for
> > > > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > > > > implementation-focused?
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Ivan
> > > > > > > > > >
> > > > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Thank you all for your questions and design input on
> > > > KIP-1150.
> > > > > > > > > > >
> > > > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> > > > design. To
> > > > > > > > > > summarize
> > > > > > > > > > > the changes:
> > > > > > > > > > >
> > > > > > > > > > > 1. The design prioritizes integrating with the existing
> > > > KIP-405
> > > > > > > Tiered
> > > > > > > > > > > Storage interfaces, permitting data produced to a
> > Diskless
> > > > topic
> > > > > > > to be
> > > > > > > > > > > moved to tiered storage.
> > > > > > > > > > > This lowers the scalability requirements for the Batch
> > > > Coordinator
> > > > > > > > > > > component, and allows Diskless to compose with Tiered
> > Storage
> > > > > > > plugin
> > > > > > > > > > > features such as encryption and alternative data
> formats.
> > > > > > > > > > >
> > > > > > > > > > > 2. Consumer fetches are now served from local segments,
> > > > making use
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > > functionality
> > > > > > > > > > already
> > > > > > > > > > > built into classic topics.
> > > > > > > > > > > However, local segments are now considered cache
> > elements,
> > > > do not
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > be durably stored, and can be built without contacting
> > any
> > > > other
> > > > > > > > > > replicas.
> > > > > > > > > > >
> > > > > > > > > > > 3. The design has been simplified substantially, by
> > removing
> > > > the
> > > > > > > > > previous
> > > > > > > > > > > Diskless consume flow, distributed cache component, and
> > > > "object
> > > > > > > > > > > compaction/merging" step.
> > > > > > > > > > >
> > > > > > > > > > > The design maintains leaderless produces as enabled by
> > the
> > > > Batch
> > > > > > > > > > > Coordinator, and the same latency profiles as the
> earlier
> > > > design,
> > > > > > > while
> > > > > > > > > > > being simpler and integrating better into the existing
> > > > ecosystem.
> > > > > > > > > > >
> > > > > > > > > > > Thanks, and we are eager to hear your feedback on the
> new
> > > > design.
> > > > > > > > > > > Greg Harris
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > > <ju...@confluent.io.invalid>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Jan,
> > > > > > > > > > > >
> > > > > > > > > > > > For me, the main gap of KIP-1150 is the support of
> all
> > > > existing
> > > > > > > > > client
> > > > > > > > > > > > APIs. Currently, there is no design for supporting
> APIs
> > > > like
> > > > > > > > > > transactions
> > > > > > > > > > > > and queues.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Would it be a good time to ask for the current
> > status of
> > > > this
> > > > > > > KIP?
> > > > > > > > > I
> > > > > > > > > > > > > haven't seen much activity here for the past 2
> > months,
> > > > the
> > > > > > > vote got
> > > > > > > > > > > > vetoed
> > > > > > > > > > > > > but I think the pending questions have been
> answered
> > > > since
> > > > > > > then.
> > > > > > > > > > KIP-1183
> > > > > > > > > > > > > (AutoMQ's proposal) also didn't have any activity
> > since
> > > > May.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real
> choices
> > > > that can
> > > > > > > be
> > > > > > > > > > > > > made, with a coordinator-based approach being by
> far
> > the
> > > > > > > dominant
> > > > > > > > > one
> > > > > > > > > > > > when
> > > > > > > > > > > > > it comes to market adoption - but all these are
> > > > standalone
> > > > > > > > > products.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm a big fan of both approaches, but would hate to
> > see a
> > > > > > > stall. So
> > > > > > > > > > the
> > > > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Maybe it's time to start another vote? Colin
> McCabe -
> > > > have your
> > > > > > > > > > questions
> > > > > > > > > > > > > been answered? If not, is there anything I can do
> to
> > > > help? I'm
> > > > > > > > > deeply
> > > > > > > > > > > > > familiar with both architectures and have written
> > about
> > > > both?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > Jan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav
> Kozlovski
> > <
> > > > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > a) group all the KIP email threads in the main
> one
> > > > (just a
> > > > > > > bunch
> > > > > > > > > of
> > > > > > > > > > > > links
> > > > > > > > > > > > > > to everything)
> > > > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It's a bit hard to track it all - for example, I
> > was
> > > > > > > searching
> > > > > > > > > for
> > > > > > > > > > a
> > > > > > > > > > > > > > discuss thread for KIP-1165 for a while; As far
> as
> > I
> > > > can
> > > > > > > tell, it
> > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > exist yet.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Since the KIPs are published (by virtue of having
> > the
> > > > root
> > > > > > > KIP be
> > > > > > > > > > > > > > published, having a DISCUSS thread and links to
> > > > sub-KIPs
> > > > > > > where
> > > > > > > > > were
> > > > > > > > > > > > aimed
> > > > > > > > > > > > > > to move the discussion towards), I think it would
> > be
> > > > good to
> > > > > > > > > create
> > > > > > > > > > > > > DISCUSS
> > > > > > > > > > > > > > threads for them all.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Stan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We want to start a new KIP discussion about
> > > > introducing a
> > > > > > > new
> > > > > > > > > > type of
> > > > > > > > > > > > > > > topics that would make use of Object Storage as
> > the
> > > > primary
> > > > > > > > > > source of
> > > > > > > > > > > > > > > storage. However, as this KIP is big we decided
> > to
> > > > split it
> > > > > > > > > into
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > that aims to discuss if Apache Kafka should aim
> > to
> > > > have
> > > > > > > this
> > > > > > > > > > type of
> > > > > > > > > > > > > > > feature at all. This KIP doesn't go onto
> details
> > on
> > > > how to
> > > > > > > > > > implement
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > This follows the same approach used when we
> > discussed
> > > > > > > KRaft.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > But as we know that it is sometimes really hard
> > to
> > > > discuss
> > > > > > > on
> > > > > > > > > > that
> > > > > > > > > > > > meta
> > > > > > > > > > > > > > > level, we also created several sub-kips (linked
> > in
> > > > > > > KIP-1150)
> > > > > > > > > that
> > > > > > > > > > > > offer
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS
> > threads
> > > > for
> > > > > > > each
> > > > > > > > > > type of
> > > > > > > > > > > > > > > concern and keep this one to discuss whether
> > Apache
> > > > Kafka
> > > > > > > wants
> > > > > > > > > > to
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks in advance on behalf of all the authors
> of
> > > > this KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 |
> > aiven.io
> > > > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu
> > Valtonen,
> > > > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Ivan,

Thanks for the explanation.

"we write the reference to the WAL file with the batch data"

I understand the approach now, but I think it is a hacky one. There are
multiple short comings with this design. First, it degrades the durability.
For each partition, now there are two files being actively written at a
given point of time, one for the data and another for the metadata.
Flushing each file requires a separate IO. If the disk has 1K IOPS and we
have 5K partitions in a broker, currently we can afford to flush each
partition every 5 seconds, achieving an RPO of 5 seconds. If we double the
number of files per partition, we can only flush each partition every 10
seconds, which makes RPO twice as bad. Second, if we ever need this
metadata somewhere else, say in the WAL file manager, the consumer needs to
subscribe to every partition in the cluster, which is inefficient. The
actual benefit of this approach is also questionable. On the surface, it
might seem that we could reduce the number of lines that need to be changed
for this KIP. However, the changes are quite intrusive to the classic
partition's code path and will probably make the code base harder to
maintain in the long run. I like the original approach based on the batch
coordinator much better than this one. We could probably refactor the
producer state code so that it could be reused in the batch coordinator.

Thanks for addressing the concerns on the number of RPCs in the produce
path. I agree that with the metadata crafting mechanism, we could mitigate
the PRC problem. However, since we now require the metadata to be
collocated with the data on the same set of brokers, it's weird that they
are now managed by different mechanisms. The data assignment now uses the
metadata crafting mechanism, but the metadata is stored in the classic
partition using its own assignment strategy. It will be complicated to keep
them collocated.

I am also concerned about the removal of the object compaction/merging
step. My first concern is on the amount of metadata that need to be kept.
Without object compcation, the metadata generated in the produce path can
only be deleted after remote tiering kicks in. Let's say for every 250ms we
produce 100 byte of metadata per partition. Let's say remoting tiering
kicks in after 5 hours. In a cluster with 100K partitions, we need to keep
about 100 * (1 / 0.2)  * 5 * 3600 * 100K = 720 GB metadata, quite
signficant. A second concern is on performance. Every time we need to
rebuild the caching data, we need to read a bunch of small objects from S3,
slowing down the building process. If a consumer happens to need such data,
it could slow down the application.

I see a tendency toward primarily optimizing for the fewest code changes in
the KIP. Instead, our primary goal should be a clean design that can last
for the long term.

Thanks,

Jun

On Tue, Oct 14, 2025 at 11:02 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi Jun,
>
> Thank you for your message. I’m sorry that I failed to clearly explain the
> idea. Let me try to fix this.
>
> > Does each partition now have a metadata partition and a separate data
> > partition? If so, I am concerned that it essentially doubles the number
> of
> > partitions, which impacts the number of open file descriptors and the
> > required IOPS, and so on. It also seems wasteful to have a separate
> > partition just to store the metadata. It's as if we are creating an
> > internal topic with an unbounded number of partitions.
>
> No. There will be only one physical partition per diskless partition. Let
> me explain this with an example. Let’s say we have a diskless partition
> topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce some
> batches to this partition. The content of the segment file will be
> something like this (for each batch):
>
> BaseOffset: 00000000000000000000 (like in classic)
> Length: 123456 (like in classic)
> PartitionLeaderEpoch: like in classic
> Magic: like in classic
> CRC: like in classic
> Attributes: like in classic
> LastOffsetDelta: like in classic
> BaseTimestamp: like in classic
> MaxTimestamp: like in classic
> ProducerId: like in classic
> ProducerEpoch: like in classic
> BaseSequence: like in classic
> RecordsCount: like in classic
> Records:
>   path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset
> 123456
>
> It looks very much like classic log entries. The only difference is that
> instead of writing real Records, we write the reference to the WAL file
> with the batch data (I guess we need only the name and the byte offset,
> because the byte length is the standard field above). Otherwise, it’s a
> normal Kafka log with the leader and replicas.
>
> So we have as many partitions for diskless as for classic. As of open file
> descriptors, let’s proceed to the following:
>
> > Are the metadata and
> > the data for the same partition always collocated on the same broker? If
> > so, how do we enforce that when replicas are reassigned?
>
> The source of truth for the data is still in WAL files on object storage.
> The source of truth for the metadata is in segment files on the brokers in
> the replica set. Two new mechanisms are planned, both independent of this
> new proposal, but I want to present them to give the idea that only a
> limited amount of data files will be operated locally:
> - We want to assemble batches into segment files and offload them to
> tiered storage in order to prevent the unbounded growth of batch metadata.
> For this, we need to open only  a few file descriptors (for the segment
> file itself + the necessary indexes) before the segment is fully written
> and handed over to RemoteLogManager.
> - We want to assemble local segment files for caching purposes as well,
> i.e. to speed up fetching. This will not materialize the full content of
> the log, but only the hot set according to some policy (or configurable
> policies), i.e. the number of segments and file descriptors will also be
> limited.
>
> > The number of RPCs in the produce path is significantly higher. For
> > example, if a produce request has 100 partitions, in a cluster with 100
> > brokers, each produce request could generate 100 more RPC requests. This
> > will significantly increase the request rate.
>
> This is a valid concern that we considered, but this issue can be
> mitigated. I’ll try to explain the approach.
> The situation with a single broker is trivial: all the commit requests go
> from the broker to itself.
> Let’s scale this to a multi-broker cluster, but located in the single rack
> (AZ). Any broker can accept Produce requests for diskless partitions, but
> we can tell producers (through metadata) to always send Produce requests to
> leaders. For example, broker 0 hosts the leader replicas for diskless
> partitions t1-0, t2-1, t3-0. It will receive diskless Produce requests for
> these partitions in various combinations, but only for them.
>
>                     Broker 0
>               +-----------------+
>               |    t1-0         |
>               |    t2-1 <--------------------+
>               |    t3-0         |            |
>  produce      | +-------------+ |            |
>  requests     | |  diskless   | |            |
> --------------->|   produce   +--------------+
>  for these    | | WAL buffer  | |    commit requests
>  partitions   | +-------------+ |    for these partitions
>               |                 |
>               +-----------------+
>
> The same applies for other brokers in this cluster. Effectively, each
> broker will commit only to itself, which effectively means 1 commit request
> per WAL buffer (this may be 0 physical network calls, if we wish, just a
> local function call).
>
> Now let’s scale this to multiple racks (AZs). Obviously, we cannot always
> send Produce requests to the designated leaders of diskless partitions:
> this would mean inter-AZ network traffic, which we would like to avoid. To
> avoid it, we say that every broker has a “diskless produce representative”
> in every AZ. If we continue our example: when a Produce request for t1-0,
> t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0 (in the
> broker’s AZ the representative is the broker itself). However, if it comes
> from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2.
>
>   |produce requests         |produce requests        |produce requests
>   |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0
>   |from AZ 0                |from AZ 1               |from AZ 2
>   v                         v                        v
>  Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
> +---------------+      +---------------+      +---------------+
> |     t1-0      |      |               |      |               |
> |     t2-1      |      |               |      |               |
> |     t3-0      |      |               |      |               |
> +---------------+      +--------+------+      +--------+------+
>      ^     ^                    |                      |
>      |     +--------------------+                      |
>      |     commit requests for these partitions        |
>      |                                                 |
>      +-------------------------------------------------+
>            commit requests for these partitions
>
> All the partitions that broker 0 is the leader of will be “represented” by
> brokers 1 and 2 in their AZs.
>
> Of course, this relationship goes both ways between AZs (not necessarily
> between the same brokers). It means that provided the cluster is balanced
> by the number of brokers per AZ, each broker will represent (number_of_azs
> - 1) other brokers. This will result in the situation that for the majority
> of commits, each broker will do up to (number_of_azs - 1) network commit
> requests (plus one local). Cloud regions tend to have 3 AZs, very rarely
> more. That means, brokers will be doing up to 2 network commit requests per
> WAL file.
>
> There are the following exceptions:
> 1. Broker count imbalance between AZs. For example, when we have 2 AZs and
> one has three brokers and another AZ has one. This one broker will do
> between 1 and 3 commit requests per WAL file. This is not an extreme
> amplification. Such an imbalance is not healthy in most practical setups
> and should be avoided anyway.
> 2. Leadership changes and metadata propagation period. When the partition
> t3-0 is relocated from broker 0 to some broker 3, the producers will not
> know this immediately (unless we want to be strict and respond with
> NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come together in a
> WAL buffer on broker 2, it will have to send two commit requests: to broker
> 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This situation
> is not permanent and as producers update the cluster metadata, it will be
> resolved.
>
> This all could be built with the metadata crafting mechanism only (which
> is anyway needed for Diskless in one way or another to direct producers and
> consumers where we need to avoid inter-AZ traffic), just with the right
> policy for it (for example, some deterministic hash-based formula). I.e. no
> explicit support for “produce representative” or anything like this is
> needed on the cluster level, in KRaft, etc.
>
> > The same WAL file metadata is now duplicated into two places, partition
> > leader and WAL File Manager. Which one is the source of truth, and how do
> > we maintain consistency between the two places?
>
> We do only two operations on WAL files that span multiple diskless
> partitions: committing and deleting. Commits can be done independently as
> described above. But deletes are different, because when a file is deleted,
> this affects all the partitions that still have alive batches in this file
> (if any).
>
> The WAL file manager is a necessary point of coordination to delete WAL
> files safely. We can say it is the source of truth about files themselves,
> while the partition leaders and their logs hold the truth about whether a
> particular file contains live batches of this particular partition.
>
> The file manager will do this important task: be able to say for sure that
> a file does not contain any live batch of any existing partition. For this,
> it will have to periodically check against the partition leaders.
> Considering that batch deletion is irreversible, when we declare a file
> “empty”, this is guaranteed to be and stay so.
>
> The file manager has to know about files being committed to start track
> them and periodically check if they are empty. We can consider various ways
> to achieve this:
> 1. As was proposed in my previous message: best effort commit by brokers +
> periodic prefix scans of object storage to detect files that went below the
> radar due to network issue or the file manager temporary unavailability.
> We’re speaking about listing the file names only and opening only
> previously unknown files in order to find the partitions involved with them.
> 2. Only do scans without explicit commit, i.e. fill the list of files
> fully asynchronously and in the background. This may be not ideal due to
> costs and performance of scanning tons of files. However, the number of
> live WAL files should be limited due to tiered storage offloading + we can
> optimize this if we give files some global soft order in their names.
>
> > I am not sure how this design simplifies the implementation. The existing
> > producer/replication code can't be simply reused. Adjusting both the
> write
> > path in the leader and the replication path in the follower to understand
> > batch-header only data is quite intrusive to the existing logic.
>
> It is true that we’ll have to change LocalLog and UnifiedLog in order to
> support these changes. However, it seems that idempotence, transactions,
> queues, tiered storage will have to be changed less than with the original
> design. This is because the partition leader state would remain in the same
> place (on brokers) and existing workflows that involve it would have to be
> changed less compared to the situation where we globalize the partition
> leader state in the batch coordinator. I admit this is hard to make
> convincing without both real implementations to hand :)
>
> > I am also
> > not sure how this enables seamless switching the topic modes between
> > diskless and classic. Could you provide more details on those?
>
> Let’s consider the scenario of turning a classic topic into diskless. The
> user sets diskless.enabled=true, the leader receives this metadata update
> and does the following:
> 1. Stop accepting normal append writes.
> 2. Close the current active segment.
> 3. Start a new segment that will be written in the diskless format (i.e.
> without data).
> 4. Start accepting diskless commits.
>
> Since it’s the same log, the followers will know about that switch
> consistently. They will finish replicating the classic segments and start
> replicating the diskless ones. They will always know where each batch is
> located (either inside a classic segment or referenced by a diskless one).
> Switching back should be similar.
>
> Doing this with the coordinator is possible, but has some caveats. The
> leader must do the following:
> 1. Stop accepting normal append writes.
> 2. Close the current active segment.
> 3. Write a special control segment to persist and replicate the fact that
> from offset N the partition is now in the diskless mode.
> 4. Inform the coordinator about the first offset N of the “diskless era”.
> 5. Inform the controller quorum that the transition has finished and that
> brokers now can process diskless writes for this partition.
> This could fail at some points, so this will probably require some
> explicit state machine with replication either in the partition log or in
> KRaft.
>
> It seems that the coordinator-less approach makes this simpler because the
> “coordinator” for the partition and the partition leader are the same and
> they store the partition metadata in the same log, too. While in the
> coordinator approach we have to perform some kind of a distributed commit
> to handover metadata management from the classic partition leader to the
> batch coordinator.
>
> I hope these explanations help to clarify the idea. Please let me know if
> I should go deeper anywhere.
>
> Best,
> Ivan and the Diskless team
>
> On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> > Hi, Ivan,
> >
> > Thanks for the update.
> >
> > I am not sure that I fully understand the new design, but it seems less
> > clean than before.
> >
> > Does each partition now have a metadata partition and a separate data
> > partition? If so, I am concerned that it essentially doubles the number
> of
> > partitions, which impacts the number of open file descriptors and the
> > required IOPS, and so on. It also seems wasteful to have a separate
> > partition just to store the metadata. It's as if we are creating an
> > internal topic with an unbounded number of partitions. Are the metadata
> and
> > the data for the same partition always collocated on the same broker? If
> > so, how do we enforce that when replicas are reassigned?
> >
> > The number of RPCs in the produce path is significantly higher. For
> > example, if a produce request has 100 partitions, in a cluster with 100
> > brokers, each produce request could generate 100 more RPC requests. This
> > will significantly increase the request rate.
> >
> > The same WAL file metadata is now duplicated into two places, partition
> > leader and WAL File Manager. Which one is the source of truth, and how do
> > we maintain consistency between the two places?
> >
> > I am not sure how this design simplifies the implementation. The existing
> > producer/replication code can't be simply reused. Adjusting both the
> write
> > path in the leader and the replication path in the follower to understand
> > batch-header only data is quite intrusive to the existing logic. I am
> also
> > not sure how this enables seamless switching the topic modes between
> > diskless and classic. Could you provide more details on those?
> >
> > Jun
> >
> > On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi dear Kafka community,
> > >
> > > In the initial Diskless proposal, we proposed to have a separate
> > > component, batch/diskless coordinator, whose role would be to centrally
> > > manage the batch and WAL file metadata for diskless topics. This
> component
> > > drew many reasonable comments from the community about how it would
> support
> > > various Kafka features (transactions, queues) and its scalability.
> While we
> > > believe we have good answers to all the expressed concerns, we took a
> step
> > > back and looked at the problem from a different perspective.
> > >
> > > We would like to propose an alternative Diskless design *without a
> > > centralized coordinator*. We believe this approach has potential and
> > > propose to discuss it as it may be more appealing to the community.
> > >
> > > Let us explain the idea. Most of the complications with the original
> > > Diskless approach come from one necessary architecture change:
> globalizing
> > > the local state of partition leader in the batch coordinator. This
> causes
> > > deviations to the established workflows in various features like
> produce
> > > idempotence and transactions, queues, retention, etc. These deviations
> need
> > > to be carefully considered, designed, and later implemented and
> tested. In
> > > the new approach we want to avoid this by making partition leaders
> again
> > > responsible for managing their partitions, even in diskless topics.
> > >
> > > In classic Kafka topics, batch data and metadata are blended together
> in
> > > the one partition log. The crux of the Diskless idea is to decouple
> them
> > > and move data to the remote storage, while keeping metadata somewhere
> else.
> > > Using the central batch coordinator for managing batch metadata is one
> way,
> > > but not the only.
> > >
> > > Let’s now think about managing metadata for each user partition
> > > independently. Generally partitions are independent and don’t share
> > > anything apart from that their data are mixed in WAL files. If we
> figure
> > > out how to commit and later delete WAL files safely, we will achieve
> the
> > > necessary autonomy that allows us to get rid of the central batch
> > > coordinator. Instead, *each diskless user partition will be managed by
> its
> > > leader*, as in classic Kafka topics. Also like in classic topics, the
> > > leader uses the partition log as the way to persist batch metadata,
> i.e.
> > > the regular batch header + the information about how to find this
> batch on
> > > remote storage. In contrast to classic topics, batch data is in remote
> > > storage.
> > >
> > > For clarity, let’s compare the three designs:
> > >  • Classic topics:
> > >    • Data and metadata are co-located in the partition log.
> > >    • The partition log content: [Batch header (metadata)|Batch data].
> > >    • The partition log is replicated to the followers.
> > >    • The replicas and leader have local state built from metadata.
> > >  • Original Diskless:
> > >    • Metadata is in the batch coordinator, data is on remote storage.
> > >    • The partition state is global in the batch coordinator.
> > >  • New Diskless:
> > >    • Metadata is in the partition log, data is on remote storage.
> > >    • Partition log content: [Batch header (metadata)|Batch coordinates
> on
> > > remote storage].
> > >    • The partition log is replicated to the followers.
> > >    • The replicas and leader have local state built from metadata.
> > >
> > > Let’s consider the produce path. Here’s the reminder of the original
> > > Diskless design:
> > >
> > >
> > > The new approach could be depicted as the following:
> > >
> > >
> > > As you can see, the main difference is that now instead of a single
> commit
> > > request to the batch coordinator, we send multiple parallel commit
> requests
> > > to all the leaders of each partition involved in the WAL file. Each of
> them
> > > will commit its batches independently, without coordinating with other
> > > leaders and any other components. Batch data is addressed by the WAL
> file
> > > name, the byte offset and size, which allows partitions to know nothing
> > > about other partitions to access their data in shared WAL files.
> > >
> > > The number of partitions involved in a single WAL file may be quite
> large,
> > > e.g. a hundred. A hundred network requests to commit one WAL file is
> very
> > > impractical. However, there are ways to reduce this number:
> > >  1. Partition leaders are located on brokers. Requests to leaders on
> one
> > > broker could be grouped together into a single physical network request
> > > (resembling the normal Produce request that may carry batches for many
> > > partitions inside). This will cap the number of network requests to the
> > > number of brokers in the cluster.
> > >  2. If we craft the cluster metadata to make producers send their
> requests
> > > to the right brokers (with respect to AZs), we may achieve the higher
> > > concentration of logical commit requests in physical network requests
> > > reducing the number of the latter ones even further, ideally to one.
> > >
> > > Obviously, out of multiple commit requests some may fail or time out
> for a
> > > variety of reasons. This is fine. Some producers will receive totally
> or
> > > partially failed responses to their Produce requests, similar to what
> they
> > > would have received when appending to a classic topic fails or times
> out.
> > > If a partition experiences problems, other partitions will not be
> affected
> > > (again, like in classic topics). Of course, the uncommitted data will
> be
> > > garbage in WAL files. But WAL files are short-lived (batches are
> constantly
> > > assembled into segments and offloaded to tiered storage), so this
> garbage
> > > will be eventually deleted.
> > >
> > > For safely deleting WAL files we now need to centrally manage them, as
> > > this is the only state and logic that spans multiple partitions. On the
> > > diagram, you can see another commit request called “Commit file (best
> > > effort)” going to the WAL File Manager. This manager will be
> responsible
> > > for the following:
> > >  1. Collecting (by requests from brokers) and persisting information
> about
> > > committed WAL files.
> > >  2. To handle potential failures in file information delivery, it will
> be
> > > doing prefix scan on the remote storage periodically to find and
> register
> > > unknown files. The period of this scan will be configurable and ideally
> > > should be quite long.
> > >  3. Checking with the relevant partition leaders (after a grace
> period) if
> > > they still have batches in a particular file.
> > >  4. Physically deleting files when they aren’t anymore referred to by
> any
> > > partition.
> > >
> > > This new design offers the following advantages:
> > >  1. It simplifies the implementation of many Kafka features such as
> > > idempotence, transactions, queues, tiered storage, retention. Now we
> don’t
> > > need to abstract away and reuse the code from partition leaders in the
> > > batch coordinator. Instead, we will literally use the same code paths
> in
> > > leaders, with little adaptation. Workflows from classic topics mostly
> > > remain unchanged.
> > > For example, it seems that
> > > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > > KafkaApis.handleWriteTxnMarkersRequest used for transaction support on
> the
> > > partition leader side could be used for diskless topics with little
> > > adaptation. ProducerStateManager, needed for both idempotent produce
> and
> > > transactions, would be reused.
> > > Another example is share groups support, where the share partition
> leader,
> > > being co-located with the partition leader, would execute the same
> logic
> > > for both diskless and classic topics.
> > >  2. It returns to the familiar partition-based scaling model, where
> > > partitions are independent.
> > >  3. It makes the operation and failure patterns closer to the familiar
> > > ones from classic topics.
> > >  4. It opens a straightforward path to seamless switching the topics
> modes
> > > between diskless and classic.
> > >
> > > The rest of the things remain unchanged compared to the previous
> Diskless
> > > design (after all previous discussions). Such things as local segment
> > > materialization by replicas, the consume path, tiered storage
> integration,
> > > etc.
> > >
> > > If the community finds this design more suitable, we will update the
> > > KIP(s) accordingly and continue working on it. Please let us know what
> you
> > > think.
> > >
> > > Best regards,
> > > Ivan and Diskless team
> > >
> > > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > > Hi Justine,
> > > >
> > > > Yes, you're right. We need to track the aborted transactions for in
> the
> > > diskless coordinator for as long as the corresponding offsets are
> there.
> > > With the tiered storage unification Greg mentioned earlier, this will
> be
> > > finite time even for infinite data retention.
> > > >
> > > > Best,
> > > > Ivan
> > > >
> > > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > > Hey Ivan,
> > > > >
> > > > > Thanks for the response. I think most of what you said made sense,
> but
> > > I
> > > > > did have some questions about this part:
> > > > >
> > > > > > As we understand this, the partition leader in classic topics
> forgets
> > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > transaction coordinator acts like the main guardian, allowing
> partition
> > > > > leaders to do this safely. Please correct me if this is wrong. We
> think
> > > > > about relying on this with the batch coordinator and delete the
> > > information
> > > > > about a transaction once it’s finished (as there’s no replication
> and
> > > HWM
> > > > > advances immediately).
> > > > >
> > > > > I didn't quite understand this. In classic topics, we have maps for
> > > ongoing
> > > > > transactions which remove state when the transaction is completed
> and
> > > an
> > > > > aborted transactions index which is retained for much longer. Once
> the
> > > > > transaction is completed, the coordinator is no longer involved in
> > > > > maintaining this partition side state, and it is subject to
> compaction
> > > etc.
> > > > > Looking back at the outline provided above, I didn't see much
> about the
> > > > > fetch path, so maybe that could be expanded a bit further. I saw
> the
> > > > > following in a response:
> > > > > > When the broker constructs a fully valid local segment, all the
> > > necessary
> > > > > control batches will be inserted and indices, including the
> transaction
> > > > > index will be built to serve FetchRequests exactly as they are
> today.
> > > > >
> > > > > Based on this, it seems like we need to retain the information
> about
> > > > > aborted txns for longer.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> > > > >
> > > > > > Hi Justine and all,
> > > > > >
> > > > > > Thank you for your questions!
> > > > > >
> > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > producer ID
> > > > > > > and epoch, the positive result of this check could be cached
> > > locally
> > > > > > > Are we saying that only new transaction version 2 transactions
> can
> > > be
> > > > > > used
> > > > > > > here? If not, we can't uniquely identify transactions with
> > > producer id +
> > > > > > > epoch
> > > > > >
> > > > > > You’re right that we (probably unintentionally) focused only on
> > > version 2.
> > > > > > We can either limit the support to version 2 or consider using
> some
> > > > > > surrogates to support version 1.
> > > > > >
> > > > > > > JO 2. >The batch coordinator does the final transactional
> checks
> > > of the
> > > > > > > batches. This procedure would output the same errors like the
> > > partition
> > > > > > > leader in classic topics would do.
> > > > > > > Can you expand on what these checks are? Would you be checking
> if
> > > the
> > > > > > > transaction was still ongoing for example?* *
> > > > > >
> > > > > > Yes, the producer epoch, that the transaction is ongoing, and of
> > > course
> > > > > > the normal idempotence checks. What the partition leader in the
> > > classic
> > > > > > topics does before appending a batch to the local log (e.g. in
> > > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> > > unfortunately
> > > > > > cannot do these checks before appending the data to the WAL
> segment
> > > and
> > > > > > uploading it, but we can “tombstone” these batches in the batch
> > > coordinator
> > > > > > during the final commit.
> > > > > >
> > > > > > > Is there state about ongoing
> > > > > > > transactions in the batch coordinator? I see some other state
> > > mentioned
> > > > > > in
> > > > > > > the End transaction section, but it's not super clear what
> state is
> > > > > > stored
> > > > > > > and when it is stored.
> > > > > >
> > > > > > Right, this should have been more explicit. As the partition
> leader
> > > tracks
> > > > > > ongoing transactions for classic topics, the batch coordinator
> has
> > > to as
> > > > > > well. So when a transaction starts and ends, the transaction
> > > coordinator
> > > > > > must inform the batch coordinator about this.
> > > > > >
> > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> that
> > > would
> > > > > > be
> > > > > > > stored in the batch coordinator?
> > > > > >
> > > > > > Yes. This could be deduced from the committed batches and other
> > > > > > information, but for the sake of performance we’d better store it
> > > > > > explicitly.
> > > > > >
> > > > > > > JO 4. Are there any thoughts about how long transactional
> state is
> > > > > > > maintained in the batch coordinator and how it will be cleaned
> up?
> > > > > >
> > > > > > As we understand this, the partition leader in classic topics
> forgets
> > > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > > transaction coordinator acts like the main guardian, allowing
> > > partition
> > > > > > leaders to do this safely. Please correct me if this is wrong. We
> > > think
> > > > > > about relying on this with the batch coordinator and delete the
> > > information
> > > > > > about a transaction once it’s finished (as there’s no replication
> > > and HWM
> > > > > > advances immediately).
> > > > > >
> > > > > > Best,
> > > > > > Ivan
> > > > > >
> > > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > > Hey folks,
> > > > > > >
> > > > > > > Excited to see some updates related to transactions!
> > > > > > >
> > > > > > > I had a few questions.
> > > > > > >
> > > > > > > JO 1. >Since a transaction could be uniquely identified with
> > > producer ID
> > > > > > > and epoch, the positive result of this check could be cached
> > > locally
> > > > > > > Are we saying that only new transaction version 2 transactions
> can
> > > be
> > > > > > used
> > > > > > > here? If not, we can't uniquely identify transactions with
> > > producer id +
> > > > > > > epoch
> > > > > > >
> > > > > > > JO 2. >The batch coordinator does the final transactional
> checks
> > > of the
> > > > > > > batches. This procedure would output the same errors like the
> > > partition
> > > > > > > leader in classic topics would do.
> > > > > > > Can you expand on what these checks are? Would you be checking
> if
> > > the
> > > > > > > transaction was still ongoing for example? Is there state about
> > > ongoing
> > > > > > > transactions in the batch coordinator? I see some other state
> > > mentioned
> > > > > > in
> > > > > > > the End transaction section, but it's not super clear what
> state is
> > > > > > stored
> > > > > > > and when it is stored.
> > > > > > >
> > > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps
> that
> > > would
> > > > > > be
> > > > > > > stored in the batch coordinator?
> > > > > > >
> > > > > > > JO 4. Are there any thoughts about how long transactional
> state is
> > > > > > > maintained in the batch coordinator and how it will be cleaned
> up?
> > > > > > >
> > > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao
> <ju...@confluent.io.invalid>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Greg and Ivan,
> > > > > > > >
> > > > > > > > Thanks for the update. A few comments.
> > > > > > > >
> > > > > > > > JR 10. "Consumer fetches are now served from local segments,
> > > making
> > > > > > use of
> > > > > > > > the
> > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > functionality
> > > > > > already
> > > > > > > > built into classic topics."
> > > > > > > > JR 10.1 Does the broker build the producer state for each
> > > partition in
> > > > > > > > diskless topics?
> > > > > > > > JR 10.2 For transactional data, the consumer fetches need to
> know
> > > > > > aborted
> > > > > > > > records. How is that achieved?
> > > > > > > >
> > > > > > > > JR 11. "The batch coordinator saves that the transaction is
> > > finished
> > > > > > and
> > > > > > > > also inserts the control batches in the corresponding logs
> of the
> > > > > > involved
> > > > > > > > Diskless topics. This happens only on the metadata level, no
> > > actual
> > > > > > control
> > > > > > > > batches are written to any file. "
> > > > > > > > A fetch response could include multiple transactional
> batches.
> > > How
> > > > > > does the
> > > > > > > > broker obtain the information about the ending control batch
> for
> > > each
> > > > > > > > batch? Does that mean that a fetch response needs to be
> built by
> > > > > > > > stitching record batches and generated control batches
> together?
> > > > > > > >
> > > > > > > > JR 12. Queues: Is there still a share partition leader that
> all
> > > > > > consumers
> > > > > > > > are routed to?
> > > > > > > >
> > > > > > > > JR 13. "Should the KIPs be modified to include this or it's
> too
> > > > > > > > implementation-focused?" It would be useful to include enough
> > > details
> > > > > > to
> > > > > > > > understand correctness and performance impact.
> > > > > > > >
> > > > > > > > HC5. Henry has a valid point. Requests from a given producer
> > > contain a
> > > > > > > > sequence number, which is ordered. If a producer sends every
> > > Produce
> > > > > > > > request to an arbitrary broker, those requests could reach
> the
> > > batch
> > > > > > > > coordinator in different order and lead to rejection of the
> > > produce
> > > > > > > > requests.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <
> ivan@ivanyu.me>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > We have also thought in a bit more details about
> transactions
> > > and
> > > > > > queues,
> > > > > > > > > here's the plan.
> > > > > > > > >
> > > > > > > > > *Transactions*
> > > > > > > > >
> > > > > > > > > The support for transactions in *classic topics* is based
> on
> > > precise
> > > > > > > > > interactions between three actors: clients (mostly
> producers,
> > > but
> > > > > > also
> > > > > > > > > consumers), brokers (ReplicaManager and other classes), and
> > > > > > transaction
> > > > > > > > > coordinators. Brokers also run partition leaders with their
> > > local
> > > > > > state
> > > > > > > > > (ProducerStateManager and others).
> > > > > > > > >
> > > > > > > > > The high level (some details skipped) workflow is the
> > > following.
> > > > > > When a
> > > > > > > > > transactional Produce request is received by the broker:
> > > > > > > > > 1. For each partition, the partition leader checks if a
> > > non-empty
> > > > > > > > > transaction is running for this partition. This is done
> using
> > > its
> > > > > > local
> > > > > > > > > state derived from the log metadata (ProducerStateManager,
> > > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > > 2. The transaction coordinator is informed about all the
> > > partitions
> > > > > > that
> > > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > > 3. The partition leaders do additional transactional
> checks.
> > > > > > > > > 4. The partition leaders append the transactional data to
> > > their logs
> > > > > > and
> > > > > > > > > update some of their state (for example, log the fact that
> the
> > > > > > > > transaction
> > > > > > > > > is running for the partition and its first offset).
> > > > > > > > >
> > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > 1. The producer contacts the transaction coordinator
> directly
> > > with
> > > > > > > > > EndTxnRequest.
> > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > PREPARE_ABORT to
> > > > > > > > > its log and responds to the producer.
> > > > > > > > > 3. The transaction coordinator sends
> WriteTxnMarkersRequest to
> > > the
> > > > > > > > leaders
> > > > > > > > > of the involved partitions.
> > > > > > > > > 4. The partition leaders write the transaction markers to
> > > their logs
> > > > > > and
> > > > > > > > > respond to the coordinator.
> > > > > > > > > 5. The coordinator writes the final transaction state
> > > > > > COMPLETE_COMMIT or
> > > > > > > > > COMPLETE_ABORT.
> > > > > > > > >
> > > > > > > > > In classic topics, partitions have leaders and lots of
> > > important
> > > > > > state
> > > > > > > > > necessary for supporting this workflow is local. The main
> > > challenge
> > > > > > in
> > > > > > > > > mapping this to Diskless comes from the fact there are no
> > > partition
> > > > > > > > > leaders, so the corresponding pieces of state need to be
> > > globalized
> > > > > > in
> > > > > > > > the
> > > > > > > > > batch coordinator. We are already doing this to support
> > > idempotent
> > > > > > > > produce.
> > > > > > > > >
> > > > > > > > > The high level workflow for *diskless topics* would look
> very
> > > > > > similar:
> > > > > > > > > 1. For each partition, the broker checks if a non-empty
> > > transaction
> > > > > > is
> > > > > > > > > running for this partition. In contrast to classic topics,
> > > this is
> > > > > > > > checked
> > > > > > > > > against the batch coordinator with a single RPC. Since a
> > > transaction
> > > > > > > > could
> > > > > > > > > be uniquely identified with producer ID and epoch, the
> positive
> > > > > > result of
> > > > > > > > > this check could be cached locally (for the double
> configured
> > > > > > duration
> > > > > > > > of a
> > > > > > > > > transaction, for example).
> > > > > > > > > 2. The same: The transaction coordinator is informed about
> all
> > > the
> > > > > > > > > partitions that aren’t part of the transaction to include
> them.
> > > > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > > > 4. The broker appends the transactional data to the current
> > > shared
> > > > > > WAL
> > > > > > > > > segment. It doesn’t update any transaction-related state
> for
> > > Diskless
> > > > > > > > > topics, because it doesn’t have any.
> > > > > > > > > 5. The WAL segment is committed to the batch coordinator
> like
> > > in the
> > > > > > > > > normal produce flow.
> > > > > > > > > 6. The batch coordinator does the final transactional
> checks
> > > of the
> > > > > > > > > batches. This procedure would output the same errors like
> the
> > > > > > partition
> > > > > > > > > leader in classic topics would do. I.e. some batches could
> be
> > > > > > rejected.
> > > > > > > > > This means, there will potentially be garbage in the WAL
> > > segment
> > > > > > file in
> > > > > > > > > case of transactional errors. This is preferable to doing
> more
> > > > > > network
> > > > > > > > > round trips, especially considering the WAL segments will
> be
> > > > > > relatively
> > > > > > > > > short-living (see the Greg's update above).
> > > > > > > > >
> > > > > > > > > When the transaction is committed or aborted:
> > > > > > > > > 1. The producer contacts the transaction coordinator
> directly
> > > with
> > > > > > > > > EndTxnRequest.
> > > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > > PREPARE_ABORT to
> > > > > > > > > its log and responds to the producer.
> > > > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> > > coordinator
> > > > > > that
> > > > > > > > > the transaction is finished.
> > > > > > > > > 4. *[NEW]* The batch coordinator saves that the
> transaction is
> > > > > > finished
> > > > > > > > > and also inserts the control batches in the corresponding
> logs
> > > of the
> > > > > > > > > involved Diskless topics. This happens only on the metadata
> > > level, no
> > > > > > > > > actual control batches are written to any file. They will
> be
> > > > > > dynamically
> > > > > > > > > created on Fetch and other read operations. We could
> > > technically
> > > > > > write
> > > > > > > > > these control batches for real, but this would mean extra
> > > produce
> > > > > > > > latency,
> > > > > > > > > so it's better just to mark them in the batch coordinator
> and
> > > save
> > > > > > these
> > > > > > > > > milliseconds.
> > > > > > > > > 5. The transaction coordinator sends
> WriteTxnMarkersRequest to
> > > the
> > > > > > > > leaders
> > > > > > > > > of the involved partitions. – Now only to classic topics
> now.
> > > > > > > > > 6. The partition leaders of classic topics write the
> > > transaction
> > > > > > markers
> > > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > > 7. The coordinator writes the final transaction state
> > > > > > COMPLETE_COMMIT or
> > > > > > > > > COMPLETE_ABORT.
> > > > > > > > >
> > > > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > > > 1. An extra network round trip between brokers and the
> batch
> > > > > > coordinator
> > > > > > > > > when a new partition appear in the transaction. To
> mitigate the
> > > > > > impact of
> > > > > > > > > them:
> > > > > > > > >   - The results will be cached.
> > > > > > > > >   - The calls for multiple partitions in one Produce
> request
> > > will be
> > > > > > > > > grouped.
> > > > > > > > >   - The batch coordinator should be optimized for fast
> > > response to
> > > > > > these
> > > > > > > > > RPCs.
> > > > > > > > >   - The fact that a single producer normally will
> communicate
> > > with a
> > > > > > > > > single broker for the duration of the transaction further
> > > reduces the
> > > > > > > > > expected number of round trips.
> > > > > > > > > 2. An extra round trip between the transaction coordinator
> and
> > > batch
> > > > > > > > > coordinator when a transaction is finished.
> > > > > > > > >
> > > > > > > > > With this proposal, transactions will also be able to span
> both
> > > > > > classic
> > > > > > > > > and Diskless topics.
> > > > > > > > >
> > > > > > > > > *Queues*
> > > > > > > > >
> > > > > > > > > The share group coordination and management is a side job
> that
> > > > > > doesn't
> > > > > > > > > interfere with the topic itself (leadership, replicas,
> physical
> > > > > > storage
> > > > > > > > of
> > > > > > > > > records, etc.) and non-queue producers and consumers
> (Fetch and
> > > > > > Produce
> > > > > > > > > RPCs, consumer group-related RPCs are not affected.) We
> don't
> > > see any
> > > > > > > > > reason why we can't make Diskless topics compatible with
> share
> > > > > > groups the
> > > > > > > > > same way as classic topics are. Even on the code level, we
> > > don't
> > > > > > expect
> > > > > > > > any
> > > > > > > > > serious refactoring: the same reading routines are used
> that
> > > are
> > > > > > used for
> > > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > > > implementation-focused?
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Ivan
> > > > > > > > >
> > > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Thank you all for your questions and design input on
> > > KIP-1150.
> > > > > > > > > >
> > > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> > > design. To
> > > > > > > > > summarize
> > > > > > > > > > the changes:
> > > > > > > > > >
> > > > > > > > > > 1. The design prioritizes integrating with the existing
> > > KIP-405
> > > > > > Tiered
> > > > > > > > > > Storage interfaces, permitting data produced to a
> Diskless
> > > topic
> > > > > > to be
> > > > > > > > > > moved to tiered storage.
> > > > > > > > > > This lowers the scalability requirements for the Batch
> > > Coordinator
> > > > > > > > > > component, and allows Diskless to compose with Tiered
> Storage
> > > > > > plugin
> > > > > > > > > > features such as encryption and alternative data formats.
> > > > > > > > > >
> > > > > > > > > > 2. Consumer fetches are now served from local segments,
> > > making use
> > > > > > of
> > > > > > > > the
> > > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > > functionality
> > > > > > > > > already
> > > > > > > > > > built into classic topics.
> > > > > > > > > > However, local segments are now considered cache
> elements,
> > > do not
> > > > > > need
> > > > > > > > to
> > > > > > > > > > be durably stored, and can be built without contacting
> any
> > > other
> > > > > > > > > replicas.
> > > > > > > > > >
> > > > > > > > > > 3. The design has been simplified substantially, by
> removing
> > > the
> > > > > > > > previous
> > > > > > > > > > Diskless consume flow, distributed cache component, and
> > > "object
> > > > > > > > > > compaction/merging" step.
> > > > > > > > > >
> > > > > > > > > > The design maintains leaderless produces as enabled by
> the
> > > Batch
> > > > > > > > > > Coordinator, and the same latency profiles as the earlier
> > > design,
> > > > > > while
> > > > > > > > > > being simpler and integrating better into the existing
> > > ecosystem.
> > > > > > > > > >
> > > > > > > > > > Thanks, and we are eager to hear your feedback on the new
> > > design.
> > > > > > > > > > Greg Harris
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > > <ju...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Jan,
> > > > > > > > > > >
> > > > > > > > > > > For me, the main gap of KIP-1150 is the support of all
> > > existing
> > > > > > > > client
> > > > > > > > > > > APIs. Currently, there is no design for supporting APIs
> > > like
> > > > > > > > > transactions
> > > > > > > > > > > and queues.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Would it be a good time to ask for the current
> status of
> > > this
> > > > > > KIP?
> > > > > > > > I
> > > > > > > > > > > > haven't seen much activity here for the past 2
> months,
> > > the
> > > > > > vote got
> > > > > > > > > > > vetoed
> > > > > > > > > > > > but I think the pending questions have been answered
> > > since
> > > > > > then.
> > > > > > > > > KIP-1183
> > > > > > > > > > > > (AutoMQ's proposal) also didn't have any activity
> since
> > > May.
> > > > > > > > > > > >
> > > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices
> > > that can
> > > > > > be
> > > > > > > > > > > > made, with a coordinator-based approach being by far
> the
> > > > > > dominant
> > > > > > > > one
> > > > > > > > > > > when
> > > > > > > > > > > > it comes to market adoption - but all these are
> > > standalone
> > > > > > > > products.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm a big fan of both approaches, but would hate to
> see a
> > > > > > stall. So
> > > > > > > > > the
> > > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > > >
> > > > > > > > > > > > Maybe it's time to start another vote? Colin McCabe -
> > > have your
> > > > > > > > > questions
> > > > > > > > > > > > been answered? If not, is there anything I can do to
> > > help? I'm
> > > > > > > > deeply
> > > > > > > > > > > > familiar with both architectures and have written
> about
> > > both?
> > > > > > > > > > > >
> > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > Jan
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski
> <
> > > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > > >
> > > > > > > > > > > > > a) group all the KIP email threads in the main one
> > > (just a
> > > > > > bunch
> > > > > > > > of
> > > > > > > > > > > links
> > > > > > > > > > > > > to everything)
> > > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's a bit hard to track it all - for example, I
> was
> > > > > > searching
> > > > > > > > for
> > > > > > > > > a
> > > > > > > > > > > > > discuss thread for KIP-1165 for a while; As far as
> I
> > > can
> > > > > > tell, it
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > exist yet.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Since the KIPs are published (by virtue of having
> the
> > > root
> > > > > > KIP be
> > > > > > > > > > > > > published, having a DISCUSS thread and links to
> > > sub-KIPs
> > > > > > where
> > > > > > > > were
> > > > > > > > > > > aimed
> > > > > > > > > > > > > to move the discussion towards), I think it would
> be
> > > good to
> > > > > > > > create
> > > > > > > > > > > > DISCUSS
> > > > > > > > > > > > > threads for them all.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Stan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We want to start a new KIP discussion about
> > > introducing a
> > > > > > new
> > > > > > > > > type of
> > > > > > > > > > > > > > topics that would make use of Object Storage as
> the
> > > primary
> > > > > > > > > source of
> > > > > > > > > > > > > > storage. However, as this KIP is big we decided
> to
> > > split it
> > > > > > > > into
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > that aims to discuss if Apache Kafka should aim
> to
> > > have
> > > > > > this
> > > > > > > > > type of
> > > > > > > > > > > > > > feature at all. This KIP doesn't go onto details
> on
> > > how to
> > > > > > > > > implement
> > > > > > > > > > > > it.
> > > > > > > > > > > > > > This follows the same approach used when we
> discussed
> > > > > > KRaft.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > But as we know that it is sometimes really hard
> to
> > > discuss
> > > > > > on
> > > > > > > > > that
> > > > > > > > > > > meta
> > > > > > > > > > > > > > level, we also created several sub-kips (linked
> in
> > > > > > KIP-1150)
> > > > > > > > that
> > > > > > > > > > > offer
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS
> threads
> > > for
> > > > > > each
> > > > > > > > > type of
> > > > > > > > > > > > > > concern and keep this one to discuss whether
> Apache
> > > Kafka
> > > > > > wants
> > > > > > > > > to
> > > > > > > > > > > have
> > > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks in advance on behalf of all the authors of
> > > this KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 |
> aiven.io
> > > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu
> Valtonen,
> > > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Jun,

Thank you for your message. I’m sorry that I failed to clearly explain the idea. Let me try to fix this.

> Does each partition now have a metadata partition and a separate data
> partition? If so, I am concerned that it essentially doubles the number of
> partitions, which impacts the number of open file descriptors and the
> required IOPS, and so on. It also seems wasteful to have a separate
> partition just to store the metadata. It's as if we are creating an
> internal topic with an unbounded number of partitions.

No. There will be only one physical partition per diskless partition. Let me explain this with an example. Let’s say we have a diskless partition topic-0. It has three replicas 0, 1, 2; 0 is the leader. We produce some batches to this partition. The content of the segment file will be something like this (for each batch):

BaseOffset: 00000000000000000000 (like in classic)
Length: 123456 (like in classic)
PartitionLeaderEpoch: like in classic
Magic: like in classic
CRC: like in classic
Attributes: like in classic
LastOffsetDelta: like in classic
BaseTimestamp: like in classic
MaxTimestamp: like in classic
ProducerId: like in classic
ProducerEpoch: like in classic
BaseSequence: like in classic
RecordsCount: like in classic
Records:
  path/to/wal/files/5b55c4bb-f52a-4204-aea6-81226895158a; byte offset 123456

It looks very much like classic log entries. The only difference is that instead of writing real Records, we write the reference to the WAL file with the batch data (I guess we need only the name and the byte offset, because the byte length is the standard field above). Otherwise, it’s a normal Kafka log with the leader and replicas. 

So we have as many partitions for diskless as for classic. As of open file descriptors, let’s proceed to the following:

> Are the metadata and
> the data for the same partition always collocated on the same broker? If
> so, how do we enforce that when replicas are reassigned?

The source of truth for the data is still in WAL files on object storage. The source of truth for the metadata is in segment files on the brokers in the replica set. Two new mechanisms are planned, both independent of this new proposal, but I want to present them to give the idea that only a limited amount of data files will be operated locally:
- We want to assemble batches into segment files and offload them to tiered storage in order to prevent the unbounded growth of batch metadata. For this, we need to open only  a few file descriptors (for the segment file itself + the necessary indexes) before the segment is fully written and handed over to RemoteLogManager.
- We want to assemble local segment files for caching purposes as well, i.e. to speed up fetching. This will not materialize the full content of the log, but only the hot set according to some policy (or configurable policies), i.e. the number of segments and file descriptors will also be limited.

> The number of RPCs in the produce path is significantly higher. For
> example, if a produce request has 100 partitions, in a cluster with 100
> brokers, each produce request could generate 100 more RPC requests. This
> will significantly increase the request rate.

This is a valid concern that we considered, but this issue can be mitigated. I’ll try to explain the approach.
The situation with a single broker is trivial: all the commit requests go from the broker to itself. 
Let’s scale this to a multi-broker cluster, but located in the single rack (AZ). Any broker can accept Produce requests for diskless partitions, but we can tell producers (through metadata) to always send Produce requests to leaders. For example, broker 0 hosts the leader replicas for diskless partitions t1-0, t2-1, t3-0. It will receive diskless Produce requests for these partitions in various combinations, but only for them.

                    Broker 0
              +-----------------+
              |    t1-0         |
              |    t2-1 <--------------------+
              |    t3-0         |            |
 produce      | +-------------+ |            |
 requests     | |  diskless   | |            |
--------------->|   produce   +--------------+
 for these    | | WAL buffer  | |    commit requests
 partitions   | +-------------+ |    for these partitions
              |                 |
              +-----------------+

The same applies for other brokers in this cluster. Effectively, each broker will commit only to itself, which effectively means 1 commit request per WAL buffer (this may be 0 physical network calls, if we wish, just a local function call).

Now let’s scale this to multiple racks (AZs). Obviously, we cannot always send Produce requests to the designated leaders of diskless partitions: this would mean inter-AZ network traffic, which we would like to avoid. To avoid it, we say that every broker has a “diskless produce representative” in every AZ. If we continue our example: when a Produce request for t1-0, t2-1, or t3-0 comes from a producer in AZ 0, it lands on broker 0 (in the broker’s AZ the representative is the broker itself). However, if it comes from AZ 1, it lands on broker 1; in AZ 2, it’s broker 2. 

  |produce requests         |produce requests        |produce requests
  |for t1-0, t2-1, t3-0     |for t1-0, t2-1, t3-0    |for t1-0, t2-1, t3-0
  |from AZ 0                |from AZ 1               |from AZ 2
  v                         v                        v
 Broker 0 (AZ 0)        Broker 1 (AZ 1)        Broker 2 (AZ 2)
+---------------+      +---------------+      +---------------+
|     t1-0      |      |               |      |               |
|     t2-1      |      |               |      |               |
|     t3-0      |      |               |      |               |
+---------------+      +--------+------+      +--------+------+
     ^     ^                    |                      |
     |     +--------------------+                      |
     |     commit requests for these partitions        |
     |                                                 |
     +-------------------------------------------------+
           commit requests for these partitions

All the partitions that broker 0 is the leader of will be “represented” by brokers 1 and 2 in their AZs.

Of course, this relationship goes both ways between AZs (not necessarily between the same brokers). It means that provided the cluster is balanced by the number of brokers per AZ, each broker will represent (number_of_azs - 1) other brokers. This will result in the situation that for the majority of commits, each broker will do up to (number_of_azs - 1) network commit requests (plus one local). Cloud regions tend to have 3 AZs, very rarely more. That means, brokers will be doing up to 2 network commit requests per WAL file.

There are the following exceptions:
1. Broker count imbalance between AZs. For example, when we have 2 AZs and one has three brokers and another AZ has one. This one broker will do between 1 and 3 commit requests per WAL file. This is not an extreme amplification. Such an imbalance is not healthy in most practical setups and should be avoided anyway.
2. Leadership changes and metadata propagation period. When the partition t3-0 is relocated from broker 0 to some broker 3, the producers will not know this immediately (unless we want to be strict and respond with NOT_LEADER_OR_FOLLOWER). So if t1-0, t2-1, and t3-0 will come together in a WAL buffer on broker 2, it will have to send two commit requests: to broker 0 to commit t1-0 and t2-1, and to broker 3 to commit t3-0. This situation is not permanent and as producers update the cluster metadata, it will be resolved.

This all could be built with the metadata crafting mechanism only (which is anyway needed for Diskless in one way or another to direct producers and consumers where we need to avoid inter-AZ traffic), just with the right policy for it (for example, some deterministic hash-based formula). I.e. no explicit support for “produce representative” or anything like this is needed on the cluster level, in KRaft, etc.

> The same WAL file metadata is now duplicated into two places, partition
> leader and WAL File Manager. Which one is the source of truth, and how do
> we maintain consistency between the two places?

We do only two operations on WAL files that span multiple diskless partitions: committing and deleting. Commits can be done independently as described above. But deletes are different, because when a file is deleted, this affects all the partitions that still have alive batches in this file (if any).

The WAL file manager is a necessary point of coordination to delete WAL files safely. We can say it is the source of truth about files themselves, while the partition leaders and their logs hold the truth about whether a particular file contains live batches of this particular partition.

The file manager will do this important task: be able to say for sure that a file does not contain any live batch of any existing partition. For this, it will have to periodically check against the partition leaders. Considering that batch deletion is irreversible, when we declare a file “empty”, this is guaranteed to be and stay so.

The file manager has to know about files being committed to start track them and periodically check if they are empty. We can consider various ways to achieve this:
1. As was proposed in my previous message: best effort commit by brokers + periodic prefix scans of object storage to detect files that went below the radar due to network issue or the file manager temporary unavailability. We’re speaking about listing the file names only and opening only previously unknown files in order to find the partitions involved with them.
2. Only do scans without explicit commit, i.e. fill the list of files fully asynchronously and in the background. This may be not ideal due to costs and performance of scanning tons of files. However, the number of live WAL files should be limited due to tiered storage offloading + we can optimize this if we give files some global soft order in their names.

> I am not sure how this design simplifies the implementation. The existing
> producer/replication code can't be simply reused. Adjusting both the write
> path in the leader and the replication path in the follower to understand
> batch-header only data is quite intrusive to the existing logic.

It is true that we’ll have to change LocalLog and UnifiedLog in order to support these changes. However, it seems that idempotence, transactions, queues, tiered storage will have to be changed less than with the original design. This is because the partition leader state would remain in the same place (on brokers) and existing workflows that involve it would have to be changed less compared to the situation where we globalize the partition leader state in the batch coordinator. I admit this is hard to make convincing without both real implementations to hand :) 

> I am also
> not sure how this enables seamless switching the topic modes between
> diskless and classic. Could you provide more details on those?

Let’s consider the scenario of turning a classic topic into diskless. The user sets diskless.enabled=true, the leader receives this metadata update and does the following:
1. Stop accepting normal append writes.
2. Close the current active segment.
3. Start a new segment that will be written in the diskless format (i.e. without data).
4. Start accepting diskless commits.

Since it’s the same log, the followers will know about that switch consistently. They will finish replicating the classic segments and start replicating the diskless ones. They will always know where each batch is located (either inside a classic segment or referenced by a diskless one). Switching back should be similar.

Doing this with the coordinator is possible, but has some caveats. The leader must do the following:
1. Stop accepting normal append writes.
2. Close the current active segment.
3. Write a special control segment to persist and replicate the fact that from offset N the partition is now in the diskless mode.
4. Inform the coordinator about the first offset N of the “diskless era”.
5. Inform the controller quorum that the transition has finished and that brokers now can process diskless writes for this partition.
This could fail at some points, so this will probably require some explicit state machine with replication either in the partition log or in KRaft.

It seems that the coordinator-less approach makes this simpler because the “coordinator” for the partition and the partition leader are the same and they store the partition metadata in the same log, too. While in the coordinator approach we have to perform some kind of a distributed commit to handover metadata management from the classic partition leader to the batch coordinator.

I hope these explanations help to clarify the idea. Please let me know if I should go deeper anywhere.

Best,
Ivan and the Diskless team

On Tue, Oct 7, 2025, at 01:44, Jun Rao wrote:
> Hi, Ivan,
> 
> Thanks for the update.
> 
> I am not sure that I fully understand the new design, but it seems less
> clean than before.
> 
> Does each partition now have a metadata partition and a separate data
> partition? If so, I am concerned that it essentially doubles the number of
> partitions, which impacts the number of open file descriptors and the
> required IOPS, and so on. It also seems wasteful to have a separate
> partition just to store the metadata. It's as if we are creating an
> internal topic with an unbounded number of partitions. Are the metadata and
> the data for the same partition always collocated on the same broker? If
> so, how do we enforce that when replicas are reassigned?
> 
> The number of RPCs in the produce path is significantly higher. For
> example, if a produce request has 100 partitions, in a cluster with 100
> brokers, each produce request could generate 100 more RPC requests. This
> will significantly increase the request rate.
> 
> The same WAL file metadata is now duplicated into two places, partition
> leader and WAL File Manager. Which one is the source of truth, and how do
> we maintain consistency between the two places?
> 
> I am not sure how this design simplifies the implementation. The existing
> producer/replication code can't be simply reused. Adjusting both the write
> path in the leader and the replication path in the follower to understand
> batch-header only data is quite intrusive to the existing logic. I am also
> not sure how this enables seamless switching the topic modes between
> diskless and classic. Could you provide more details on those?
> 
> Jun
> 
> On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> 
> > Hi dear Kafka community,
> >
> > In the initial Diskless proposal, we proposed to have a separate
> > component, batch/diskless coordinator, whose role would be to centrally
> > manage the batch and WAL file metadata for diskless topics. This component
> > drew many reasonable comments from the community about how it would support
> > various Kafka features (transactions, queues) and its scalability. While we
> > believe we have good answers to all the expressed concerns, we took a step
> > back and looked at the problem from a different perspective.
> >
> > We would like to propose an alternative Diskless design *without a
> > centralized coordinator*. We believe this approach has potential and
> > propose to discuss it as it may be more appealing to the community.
> >
> > Let us explain the idea. Most of the complications with the original
> > Diskless approach come from one necessary architecture change: globalizing
> > the local state of partition leader in the batch coordinator. This causes
> > deviations to the established workflows in various features like produce
> > idempotence and transactions, queues, retention, etc. These deviations need
> > to be carefully considered, designed, and later implemented and tested. In
> > the new approach we want to avoid this by making partition leaders again
> > responsible for managing their partitions, even in diskless topics.
> >
> > In classic Kafka topics, batch data and metadata are blended together in
> > the one partition log. The crux of the Diskless idea is to decouple them
> > and move data to the remote storage, while keeping metadata somewhere else.
> > Using the central batch coordinator for managing batch metadata is one way,
> > but not the only.
> >
> > Let’s now think about managing metadata for each user partition
> > independently. Generally partitions are independent and don’t share
> > anything apart from that their data are mixed in WAL files. If we figure
> > out how to commit and later delete WAL files safely, we will achieve the
> > necessary autonomy that allows us to get rid of the central batch
> > coordinator. Instead, *each diskless user partition will be managed by its
> > leader*, as in classic Kafka topics. Also like in classic topics, the
> > leader uses the partition log as the way to persist batch metadata, i.e.
> > the regular batch header + the information about how to find this batch on
> > remote storage. In contrast to classic topics, batch data is in remote
> > storage.
> >
> > For clarity, let’s compare the three designs:
> >  • Classic topics:
> >    • Data and metadata are co-located in the partition log.
> >    • The partition log content: [Batch header (metadata)|Batch data].
> >    • The partition log is replicated to the followers.
> >    • The replicas and leader have local state built from metadata.
> >  • Original Diskless:
> >    • Metadata is in the batch coordinator, data is on remote storage.
> >    • The partition state is global in the batch coordinator.
> >  • New Diskless:
> >    • Metadata is in the partition log, data is on remote storage.
> >    • Partition log content: [Batch header (metadata)|Batch coordinates on
> > remote storage].
> >    • The partition log is replicated to the followers.
> >    • The replicas and leader have local state built from metadata.
> >
> > Let’s consider the produce path. Here’s the reminder of the original
> > Diskless design:
> >
> >
> > The new approach could be depicted as the following:
> >
> >
> > As you can see, the main difference is that now instead of a single commit
> > request to the batch coordinator, we send multiple parallel commit requests
> > to all the leaders of each partition involved in the WAL file. Each of them
> > will commit its batches independently, without coordinating with other
> > leaders and any other components. Batch data is addressed by the WAL file
> > name, the byte offset and size, which allows partitions to know nothing
> > about other partitions to access their data in shared WAL files.
> >
> > The number of partitions involved in a single WAL file may be quite large,
> > e.g. a hundred. A hundred network requests to commit one WAL file is very
> > impractical. However, there are ways to reduce this number:
> >  1. Partition leaders are located on brokers. Requests to leaders on one
> > broker could be grouped together into a single physical network request
> > (resembling the normal Produce request that may carry batches for many
> > partitions inside). This will cap the number of network requests to the
> > number of brokers in the cluster.
> >  2. If we craft the cluster metadata to make producers send their requests
> > to the right brokers (with respect to AZs), we may achieve the higher
> > concentration of logical commit requests in physical network requests
> > reducing the number of the latter ones even further, ideally to one.
> >
> > Obviously, out of multiple commit requests some may fail or time out for a
> > variety of reasons. This is fine. Some producers will receive totally or
> > partially failed responses to their Produce requests, similar to what they
> > would have received when appending to a classic topic fails or times out.
> > If a partition experiences problems, other partitions will not be affected
> > (again, like in classic topics). Of course, the uncommitted data will be
> > garbage in WAL files. But WAL files are short-lived (batches are constantly
> > assembled into segments and offloaded to tiered storage), so this garbage
> > will be eventually deleted.
> >
> > For safely deleting WAL files we now need to centrally manage them, as
> > this is the only state and logic that spans multiple partitions. On the
> > diagram, you can see another commit request called “Commit file (best
> > effort)” going to the WAL File Manager. This manager will be responsible
> > for the following:
> >  1. Collecting (by requests from brokers) and persisting information about
> > committed WAL files.
> >  2. To handle potential failures in file information delivery, it will be
> > doing prefix scan on the remote storage periodically to find and register
> > unknown files. The period of this scan will be configurable and ideally
> > should be quite long.
> >  3. Checking with the relevant partition leaders (after a grace period) if
> > they still have batches in a particular file.
> >  4. Physically deleting files when they aren’t anymore referred to by any
> > partition.
> >
> > This new design offers the following advantages:
> >  1. It simplifies the implementation of many Kafka features such as
> > idempotence, transactions, queues, tiered storage, retention. Now we don’t
> > need to abstract away and reuse the code from partition leaders in the
> > batch coordinator. Instead, we will literally use the same code paths in
> > leaders, with little adaptation. Workflows from classic topics mostly
> > remain unchanged.
> > For example, it seems that
> > ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> > KafkaApis.handleWriteTxnMarkersRequest used for transaction support on the
> > partition leader side could be used for diskless topics with little
> > adaptation. ProducerStateManager, needed for both idempotent produce and
> > transactions, would be reused.
> > Another example is share groups support, where the share partition leader,
> > being co-located with the partition leader, would execute the same logic
> > for both diskless and classic topics.
> >  2. It returns to the familiar partition-based scaling model, where
> > partitions are independent.
> >  3. It makes the operation and failure patterns closer to the familiar
> > ones from classic topics.
> >  4. It opens a straightforward path to seamless switching the topics modes
> > between diskless and classic.
> >
> > The rest of the things remain unchanged compared to the previous Diskless
> > design (after all previous discussions). Such things as local segment
> > materialization by replicas, the consume path, tiered storage integration,
> > etc.
> >
> > If the community finds this design more suitable, we will update the
> > KIP(s) accordingly and continue working on it. Please let us know what you
> > think.
> >
> > Best regards,
> > Ivan and Diskless team
> >
> > On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > > Hi Justine,
> > >
> > > Yes, you're right. We need to track the aborted transactions for in the
> > diskless coordinator for as long as the corresponding offsets are there.
> > With the tiered storage unification Greg mentioned earlier, this will be
> > finite time even for infinite data retention.
> > >
> > > Best,
> > > Ivan
> > >
> > > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > > Hey Ivan,
> > > >
> > > > Thanks for the response. I think most of what you said made sense, but
> > I
> > > > did have some questions about this part:
> > > >
> > > > > As we understand this, the partition leader in classic topics forgets
> > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > transaction coordinator acts like the main guardian, allowing partition
> > > > leaders to do this safely. Please correct me if this is wrong. We think
> > > > about relying on this with the batch coordinator and delete the
> > information
> > > > about a transaction once it’s finished (as there’s no replication and
> > HWM
> > > > advances immediately).
> > > >
> > > > I didn't quite understand this. In classic topics, we have maps for
> > ongoing
> > > > transactions which remove state when the transaction is completed and
> > an
> > > > aborted transactions index which is retained for much longer. Once the
> > > > transaction is completed, the coordinator is no longer involved in
> > > > maintaining this partition side state, and it is subject to compaction
> > etc.
> > > > Looking back at the outline provided above, I didn't see much about the
> > > > fetch path, so maybe that could be expanded a bit further. I saw the
> > > > following in a response:
> > > > > When the broker constructs a fully valid local segment, all the
> > necessary
> > > > control batches will be inserted and indices, including the transaction
> > > > index will be built to serve FetchRequests exactly as they are today.
> > > >
> > > > Based on this, it seems like we need to retain the information about
> > > > aborted txns for longer.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > >
> > > > > Hi Justine and all,
> > > > >
> > > > > Thank you for your questions!
> > > > >
> > > > > > JO 1. >Since a transaction could be uniquely identified with
> > producer ID
> > > > > > and epoch, the positive result of this check could be cached
> > locally
> > > > > > Are we saying that only new transaction version 2 transactions can
> > be
> > > > > used
> > > > > > here? If not, we can't uniquely identify transactions with
> > producer id +
> > > > > > epoch
> > > > >
> > > > > You’re right that we (probably unintentionally) focused only on
> > version 2.
> > > > > We can either limit the support to version 2 or consider using some
> > > > > surrogates to support version 1.
> > > > >
> > > > > > JO 2. >The batch coordinator does the final transactional checks
> > of the
> > > > > > batches. This procedure would output the same errors like the
> > partition
> > > > > > leader in classic topics would do.
> > > > > > Can you expand on what these checks are? Would you be checking if
> > the
> > > > > > transaction was still ongoing for example?* *
> > > > >
> > > > > Yes, the producer epoch, that the transaction is ongoing, and of
> > course
> > > > > the normal idempotence checks. What the partition leader in the
> > classic
> > > > > topics does before appending a batch to the local log (e.g. in
> > > > > UnifiedLog.maybeStartTransactionVerification and
> > > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> > unfortunately
> > > > > cannot do these checks before appending the data to the WAL segment
> > and
> > > > > uploading it, but we can “tombstone” these batches in the batch
> > coordinator
> > > > > during the final commit.
> > > > >
> > > > > > Is there state about ongoing
> > > > > > transactions in the batch coordinator? I see some other state
> > mentioned
> > > > > in
> > > > > > the End transaction section, but it's not super clear what state is
> > > > > stored
> > > > > > and when it is stored.
> > > > >
> > > > > Right, this should have been more explicit. As the partition leader
> > tracks
> > > > > ongoing transactions for classic topics, the batch coordinator has
> > to as
> > > > > well. So when a transaction starts and ends, the transaction
> > coordinator
> > > > > must inform the batch coordinator about this.
> > > > >
> > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> > would
> > > > > be
> > > > > > stored in the batch coordinator?
> > > > >
> > > > > Yes. This could be deduced from the committed batches and other
> > > > > information, but for the sake of performance we’d better store it
> > > > > explicitly.
> > > > >
> > > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > >
> > > > > As we understand this, the partition leader in classic topics forgets
> > > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > > transaction coordinator acts like the main guardian, allowing
> > partition
> > > > > leaders to do this safely. Please correct me if this is wrong. We
> > think
> > > > > about relying on this with the batch coordinator and delete the
> > information
> > > > > about a transaction once it’s finished (as there’s no replication
> > and HWM
> > > > > advances immediately).
> > > > >
> > > > > Best,
> > > > > Ivan
> > > > >
> > > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > > Hey folks,
> > > > > >
> > > > > > Excited to see some updates related to transactions!
> > > > > >
> > > > > > I had a few questions.
> > > > > >
> > > > > > JO 1. >Since a transaction could be uniquely identified with
> > producer ID
> > > > > > and epoch, the positive result of this check could be cached
> > locally
> > > > > > Are we saying that only new transaction version 2 transactions can
> > be
> > > > > used
> > > > > > here? If not, we can't uniquely identify transactions with
> > producer id +
> > > > > > epoch
> > > > > >
> > > > > > JO 2. >The batch coordinator does the final transactional checks
> > of the
> > > > > > batches. This procedure would output the same errors like the
> > partition
> > > > > > leader in classic topics would do.
> > > > > > Can you expand on what these checks are? Would you be checking if
> > the
> > > > > > transaction was still ongoing for example? Is there state about
> > ongoing
> > > > > > transactions in the batch coordinator? I see some other state
> > mentioned
> > > > > in
> > > > > > the End transaction section, but it's not super clear what state is
> > > > > stored
> > > > > > and when it is stored.
> > > > > >
> > > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> > would
> > > > > be
> > > > > > stored in the batch coordinator?
> > > > > >
> > > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > > >
> > > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Greg and Ivan,
> > > > > > >
> > > > > > > Thanks for the update. A few comments.
> > > > > > >
> > > > > > > JR 10. "Consumer fetches are now served from local segments,
> > making
> > > > > use of
> > > > > > > the
> > > > > > > indexes, page cache, request purgatory, and zero-copy
> > functionality
> > > > > already
> > > > > > > built into classic topics."
> > > > > > > JR 10.1 Does the broker build the producer state for each
> > partition in
> > > > > > > diskless topics?
> > > > > > > JR 10.2 For transactional data, the consumer fetches need to know
> > > > > aborted
> > > > > > > records. How is that achieved?
> > > > > > >
> > > > > > > JR 11. "The batch coordinator saves that the transaction is
> > finished
> > > > > and
> > > > > > > also inserts the control batches in the corresponding logs of the
> > > > > involved
> > > > > > > Diskless topics. This happens only on the metadata level, no
> > actual
> > > > > control
> > > > > > > batches are written to any file. "
> > > > > > > A fetch response could include multiple transactional batches.
> > How
> > > > > does the
> > > > > > > broker obtain the information about the ending control batch for
> > each
> > > > > > > batch? Does that mean that a fetch response needs to be built by
> > > > > > > stitching record batches and generated control batches together?
> > > > > > >
> > > > > > > JR 12. Queues: Is there still a share partition leader that all
> > > > > consumers
> > > > > > > are routed to?
> > > > > > >
> > > > > > > JR 13. "Should the KIPs be modified to include this or it's too
> > > > > > > implementation-focused?" It would be useful to include enough
> > details
> > > > > to
> > > > > > > understand correctness and performance impact.
> > > > > > >
> > > > > > > HC5. Henry has a valid point. Requests from a given producer
> > contain a
> > > > > > > sequence number, which is ordered. If a producer sends every
> > Produce
> > > > > > > request to an arbitrary broker, those requests could reach the
> > batch
> > > > > > > coordinator in different order and lead to rejection of the
> > produce
> > > > > > > requests.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me>
> > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > We have also thought in a bit more details about transactions
> > and
> > > > > queues,
> > > > > > > > here's the plan.
> > > > > > > >
> > > > > > > > *Transactions*
> > > > > > > >
> > > > > > > > The support for transactions in *classic topics* is based on
> > precise
> > > > > > > > interactions between three actors: clients (mostly producers,
> > but
> > > > > also
> > > > > > > > consumers), brokers (ReplicaManager and other classes), and
> > > > > transaction
> > > > > > > > coordinators. Brokers also run partition leaders with their
> > local
> > > > > state
> > > > > > > > (ProducerStateManager and others).
> > > > > > > >
> > > > > > > > The high level (some details skipped) workflow is the
> > following.
> > > > > When a
> > > > > > > > transactional Produce request is received by the broker:
> > > > > > > > 1. For each partition, the partition leader checks if a
> > non-empty
> > > > > > > > transaction is running for this partition. This is done using
> > its
> > > > > local
> > > > > > > > state derived from the log metadata (ProducerStateManager,
> > > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > > 2. The transaction coordinator is informed about all the
> > partitions
> > > > > that
> > > > > > > > aren’t part of the transaction to include them.
> > > > > > > > 3. The partition leaders do additional transactional checks.
> > > > > > > > 4. The partition leaders append the transactional data to
> > their logs
> > > > > and
> > > > > > > > update some of their state (for example, log the fact that the
> > > > > > > transaction
> > > > > > > > is running for the partition and its first offset).
> > > > > > > >
> > > > > > > > When the transaction is committed or aborted:
> > > > > > > > 1. The producer contacts the transaction coordinator directly
> > with
> > > > > > > > EndTxnRequest.
> > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > PREPARE_ABORT to
> > > > > > > > its log and responds to the producer.
> > > > > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to
> > the
> > > > > > > leaders
> > > > > > > > of the involved partitions.
> > > > > > > > 4. The partition leaders write the transaction markers to
> > their logs
> > > > > and
> > > > > > > > respond to the coordinator.
> > > > > > > > 5. The coordinator writes the final transaction state
> > > > > COMPLETE_COMMIT or
> > > > > > > > COMPLETE_ABORT.
> > > > > > > >
> > > > > > > > In classic topics, partitions have leaders and lots of
> > important
> > > > > state
> > > > > > > > necessary for supporting this workflow is local. The main
> > challenge
> > > > > in
> > > > > > > > mapping this to Diskless comes from the fact there are no
> > partition
> > > > > > > > leaders, so the corresponding pieces of state need to be
> > globalized
> > > > > in
> > > > > > > the
> > > > > > > > batch coordinator. We are already doing this to support
> > idempotent
> > > > > > > produce.
> > > > > > > >
> > > > > > > > The high level workflow for *diskless topics* would look very
> > > > > similar:
> > > > > > > > 1. For each partition, the broker checks if a non-empty
> > transaction
> > > > > is
> > > > > > > > running for this partition. In contrast to classic topics,
> > this is
> > > > > > > checked
> > > > > > > > against the batch coordinator with a single RPC. Since a
> > transaction
> > > > > > > could
> > > > > > > > be uniquely identified with producer ID and epoch, the positive
> > > > > result of
> > > > > > > > this check could be cached locally (for the double configured
> > > > > duration
> > > > > > > of a
> > > > > > > > transaction, for example).
> > > > > > > > 2. The same: The transaction coordinator is informed about all
> > the
> > > > > > > > partitions that aren’t part of the transaction to include them.
> > > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > > 4. The broker appends the transactional data to the current
> > shared
> > > > > WAL
> > > > > > > > segment. It doesn’t update any transaction-related state for
> > Diskless
> > > > > > > > topics, because it doesn’t have any.
> > > > > > > > 5. The WAL segment is committed to the batch coordinator like
> > in the
> > > > > > > > normal produce flow.
> > > > > > > > 6. The batch coordinator does the final transactional checks
> > of the
> > > > > > > > batches. This procedure would output the same errors like the
> > > > > partition
> > > > > > > > leader in classic topics would do. I.e. some batches could be
> > > > > rejected.
> > > > > > > > This means, there will potentially be garbage in the WAL
> > segment
> > > > > file in
> > > > > > > > case of transactional errors. This is preferable to doing more
> > > > > network
> > > > > > > > round trips, especially considering the WAL segments will be
> > > > > relatively
> > > > > > > > short-living (see the Greg's update above).
> > > > > > > >
> > > > > > > > When the transaction is committed or aborted:
> > > > > > > > 1. The producer contacts the transaction coordinator directly
> > with
> > > > > > > > EndTxnRequest.
> > > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > > PREPARE_ABORT to
> > > > > > > > its log and responds to the producer.
> > > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> > coordinator
> > > > > that
> > > > > > > > the transaction is finished.
> > > > > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> > > > > finished
> > > > > > > > and also inserts the control batches in the corresponding logs
> > of the
> > > > > > > > involved Diskless topics. This happens only on the metadata
> > level, no
> > > > > > > > actual control batches are written to any file. They will be
> > > > > dynamically
> > > > > > > > created on Fetch and other read operations. We could
> > technically
> > > > > write
> > > > > > > > these control batches for real, but this would mean extra
> > produce
> > > > > > > latency,
> > > > > > > > so it's better just to mark them in the batch coordinator and
> > save
> > > > > these
> > > > > > > > milliseconds.
> > > > > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to
> > the
> > > > > > > leaders
> > > > > > > > of the involved partitions. – Now only to classic topics now.
> > > > > > > > 6. The partition leaders of classic topics write the
> > transaction
> > > > > markers
> > > > > > > > to their logs and respond to the coordinator.
> > > > > > > > 7. The coordinator writes the final transaction state
> > > > > COMPLETE_COMMIT or
> > > > > > > > COMPLETE_ABORT.
> > > > > > > >
> > > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > > 1. An extra network round trip between brokers and the batch
> > > > > coordinator
> > > > > > > > when a new partition appear in the transaction. To mitigate the
> > > > > impact of
> > > > > > > > them:
> > > > > > > >   - The results will be cached.
> > > > > > > >   - The calls for multiple partitions in one Produce request
> > will be
> > > > > > > > grouped.
> > > > > > > >   - The batch coordinator should be optimized for fast
> > response to
> > > > > these
> > > > > > > > RPCs.
> > > > > > > >   - The fact that a single producer normally will communicate
> > with a
> > > > > > > > single broker for the duration of the transaction further
> > reduces the
> > > > > > > > expected number of round trips.
> > > > > > > > 2. An extra round trip between the transaction coordinator and
> > batch
> > > > > > > > coordinator when a transaction is finished.
> > > > > > > >
> > > > > > > > With this proposal, transactions will also be able to span both
> > > > > classic
> > > > > > > > and Diskless topics.
> > > > > > > >
> > > > > > > > *Queues*
> > > > > > > >
> > > > > > > > The share group coordination and management is a side job that
> > > > > doesn't
> > > > > > > > interfere with the topic itself (leadership, replicas, physical
> > > > > storage
> > > > > > > of
> > > > > > > > records, etc.) and non-queue producers and consumers (Fetch and
> > > > > Produce
> > > > > > > > RPCs, consumer group-related RPCs are not affected.) We don't
> > see any
> > > > > > > > reason why we can't make Diskless topics compatible with share
> > > > > groups the
> > > > > > > > same way as classic topics are. Even on the code level, we
> > don't
> > > > > expect
> > > > > > > any
> > > > > > > > serious refactoring: the same reading routines are used that
> > are
> > > > > used for
> > > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > > >
> > > > > > > >
> > > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > > implementation-focused?
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Ivan
> > > > > > > >
> > > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Thank you all for your questions and design input on
> > KIP-1150.
> > > > > > > > >
> > > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> > design. To
> > > > > > > > summarize
> > > > > > > > > the changes:
> > > > > > > > >
> > > > > > > > > 1. The design prioritizes integrating with the existing
> > KIP-405
> > > > > Tiered
> > > > > > > > > Storage interfaces, permitting data produced to a Diskless
> > topic
> > > > > to be
> > > > > > > > > moved to tiered storage.
> > > > > > > > > This lowers the scalability requirements for the Batch
> > Coordinator
> > > > > > > > > component, and allows Diskless to compose with Tiered Storage
> > > > > plugin
> > > > > > > > > features such as encryption and alternative data formats.
> > > > > > > > >
> > > > > > > > > 2. Consumer fetches are now served from local segments,
> > making use
> > > > > of
> > > > > > > the
> > > > > > > > > indexes, page cache, request purgatory, and zero-copy
> > functionality
> > > > > > > > already
> > > > > > > > > built into classic topics.
> > > > > > > > > However, local segments are now considered cache elements,
> > do not
> > > > > need
> > > > > > > to
> > > > > > > > > be durably stored, and can be built without contacting any
> > other
> > > > > > > > replicas.
> > > > > > > > >
> > > > > > > > > 3. The design has been simplified substantially, by removing
> > the
> > > > > > > previous
> > > > > > > > > Diskless consume flow, distributed cache component, and
> > "object
> > > > > > > > > compaction/merging" step.
> > > > > > > > >
> > > > > > > > > The design maintains leaderless produces as enabled by the
> > Batch
> > > > > > > > > Coordinator, and the same latency profiles as the earlier
> > design,
> > > > > while
> > > > > > > > > being simpler and integrating better into the existing
> > ecosystem.
> > > > > > > > >
> > > > > > > > > Thanks, and we are eager to hear your feedback on the new
> > design.
> > > > > > > > > Greg Harris
> > > > > > > > >
> > > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> > <ju...@confluent.io.invalid>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Jan,
> > > > > > > > > >
> > > > > > > > > > For me, the main gap of KIP-1150 is the support of all
> > existing
> > > > > > > client
> > > > > > > > > > APIs. Currently, there is no design for supporting APIs
> > like
> > > > > > > > transactions
> > > > > > > > > > and queues.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > > Would it be a good time to ask for the current status of
> > this
> > > > > KIP?
> > > > > > > I
> > > > > > > > > > > haven't seen much activity here for the past 2 months,
> > the
> > > > > vote got
> > > > > > > > > > vetoed
> > > > > > > > > > > but I think the pending questions have been answered
> > since
> > > > > then.
> > > > > > > > KIP-1183
> > > > > > > > > > > (AutoMQ's proposal) also didn't have any activity since
> > May.
> > > > > > > > > > >
> > > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices
> > that can
> > > > > be
> > > > > > > > > > > made, with a coordinator-based approach being by far the
> > > > > dominant
> > > > > > > one
> > > > > > > > > > when
> > > > > > > > > > > it comes to market adoption - but all these are
> > standalone
> > > > > > > products.
> > > > > > > > > > >
> > > > > > > > > > > I'm a big fan of both approaches, but would hate to see a
> > > > > stall. So
> > > > > > > > the
> > > > > > > > > > > question is: can we get an update?
> > > > > > > > > > >
> > > > > > > > > > > Maybe it's time to start another vote? Colin McCabe -
> > have your
> > > > > > > > questions
> > > > > > > > > > > been answered? If not, is there anything I can do to
> > help? I'm
> > > > > > > deeply
> > > > > > > > > > > familiar with both architectures and have written about
> > both?
> > > > > > > > > > >
> > > > > > > > > > > Kind regards,
> > > > > > > > > > > Jan
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > > >
> > > > > > > > > > > > a) group all the KIP email threads in the main one
> > (just a
> > > > > bunch
> > > > > > > of
> > > > > > > > > > links
> > > > > > > > > > > > to everything)
> > > > > > > > > > > > b) create the email threads
> > > > > > > > > > > >
> > > > > > > > > > > > It's a bit hard to track it all - for example, I was
> > > > > searching
> > > > > > > for
> > > > > > > > a
> > > > > > > > > > > > discuss thread for KIP-1165 for a while; As far as I
> > can
> > > > > tell, it
> > > > > > > > > > doesn't
> > > > > > > > > > > > exist yet.
> > > > > > > > > > > >
> > > > > > > > > > > > Since the KIPs are published (by virtue of having the
> > root
> > > > > KIP be
> > > > > > > > > > > > published, having a DISCUSS thread and links to
> > sub-KIPs
> > > > > where
> > > > > > > were
> > > > > > > > > > aimed
> > > > > > > > > > > > to move the discussion towards), I think it would be
> > good to
> > > > > > > create
> > > > > > > > > > > DISCUSS
> > > > > > > > > > > > threads for them all.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Stan
> > > > > > > > > > > >
> > > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > > >
> > > > > > > > > > > > > We want to start a new KIP discussion about
> > introducing a
> > > > > new
> > > > > > > > type of
> > > > > > > > > > > > > topics that would make use of Object Storage as the
> > primary
> > > > > > > > source of
> > > > > > > > > > > > > storage. However, as this KIP is big we decided to
> > split it
> > > > > > > into
> > > > > > > > > > > multiple
> > > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > > )
> > > > > > > > > > > > > that aims to discuss if Apache Kafka should aim to
> > have
> > > > > this
> > > > > > > > type of
> > > > > > > > > > > > > feature at all. This KIP doesn't go onto details on
> > how to
> > > > > > > > implement
> > > > > > > > > > > it.
> > > > > > > > > > > > > This follows the same approach used when we discussed
> > > > > KRaft.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But as we know that it is sometimes really hard to
> > discuss
> > > > > on
> > > > > > > > that
> > > > > > > > > > meta
> > > > > > > > > > > > > level, we also created several sub-kips (linked in
> > > > > KIP-1150)
> > > > > > > that
> > > > > > > > > > offer
> > > > > > > > > > > > an
> > > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We kindly ask you to use the proper DISCUSS threads
> > for
> > > > > each
> > > > > > > > type of
> > > > > > > > > > > > > concern and keep this one to discuss whether Apache
> > Kafka
> > > > > wants
> > > > > > > > to
> > > > > > > > > > have
> > > > > > > > > > > > > this feature or not.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance on behalf of all the authors of
> > this KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ------------------
> > > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Ivan,

Thanks for the update.

I am not sure that I fully understand the new design, but it seems less
clean than before.

Does each partition now have a metadata partition and a separate data
partition? If so, I am concerned that it essentially doubles the number of
partitions, which impacts the number of open file descriptors and the
required IOPS, and so on. It also seems wasteful to have a separate
partition just to store the metadata. It's as if we are creating an
internal topic with an unbounded number of partitions. Are the metadata and
the data for the same partition always collocated on the same broker? If
so, how do we enforce that when replicas are reassigned?

The number of RPCs in the produce path is significantly higher. For
example, if a produce request has 100 partitions, in a cluster with 100
brokers, each produce request could generate 100 more RPC requests. This
will significantly increase the request rate.

The same WAL file metadata is now duplicated into two places, partition
leader and WAL File Manager. Which one is the source of truth, and how do
we maintain consistency between the two places?

I am not sure how this design simplifies the implementation. The existing
producer/replication code can't be simply reused. Adjusting both the write
path in the leader and the replication path in the follower to understand
batch-header only data is quite intrusive to the existing logic. I am also
not sure how this enables seamless switching the topic modes between
diskless and classic. Could you provide more details on those?

Jun

On Thu, Oct 2, 2025 at 5:08 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi dear Kafka community,
>
> In the initial Diskless proposal, we proposed to have a separate
> component, batch/diskless coordinator, whose role would be to centrally
> manage the batch and WAL file metadata for diskless topics. This component
> drew many reasonable comments from the community about how it would support
> various Kafka features (transactions, queues) and its scalability. While we
> believe we have good answers to all the expressed concerns, we took a step
> back and looked at the problem from a different perspective.
>
> We would like to propose an alternative Diskless design *without a
> centralized coordinator*. We believe this approach has potential and
> propose to discuss it as it may be more appealing to the community.
>
> Let us explain the idea. Most of the complications with the original
> Diskless approach come from one necessary architecture change: globalizing
> the local state of partition leader in the batch coordinator. This causes
> deviations to the established workflows in various features like produce
> idempotence and transactions, queues, retention, etc. These deviations need
> to be carefully considered, designed, and later implemented and tested. In
> the new approach we want to avoid this by making partition leaders again
> responsible for managing their partitions, even in diskless topics.
>
> In classic Kafka topics, batch data and metadata are blended together in
> the one partition log. The crux of the Diskless idea is to decouple them
> and move data to the remote storage, while keeping metadata somewhere else.
> Using the central batch coordinator for managing batch metadata is one way,
> but not the only.
>
> Let’s now think about managing metadata for each user partition
> independently. Generally partitions are independent and don’t share
> anything apart from that their data are mixed in WAL files. If we figure
> out how to commit and later delete WAL files safely, we will achieve the
> necessary autonomy that allows us to get rid of the central batch
> coordinator. Instead, *each diskless user partition will be managed by its
> leader*, as in classic Kafka topics. Also like in classic topics, the
> leader uses the partition log as the way to persist batch metadata, i.e.
> the regular batch header + the information about how to find this batch on
> remote storage. In contrast to classic topics, batch data is in remote
> storage.
>
> For clarity, let’s compare the three designs:
>  • Classic topics:
>    • Data and metadata are co-located in the partition log.
>    • The partition log content: [Batch header (metadata)|Batch data].
>    • The partition log is replicated to the followers.
>    • The replicas and leader have local state built from metadata.
>  • Original Diskless:
>    • Metadata is in the batch coordinator, data is on remote storage.
>    • The partition state is global in the batch coordinator.
>  • New Diskless:
>    • Metadata is in the partition log, data is on remote storage.
>    • Partition log content: [Batch header (metadata)|Batch coordinates on
> remote storage].
>    • The partition log is replicated to the followers.
>    • The replicas and leader have local state built from metadata.
>
> Let’s consider the produce path. Here’s the reminder of the original
> Diskless design:
>
>
> The new approach could be depicted as the following:
>
>
> As you can see, the main difference is that now instead of a single commit
> request to the batch coordinator, we send multiple parallel commit requests
> to all the leaders of each partition involved in the WAL file. Each of them
> will commit its batches independently, without coordinating with other
> leaders and any other components. Batch data is addressed by the WAL file
> name, the byte offset and size, which allows partitions to know nothing
> about other partitions to access their data in shared WAL files.
>
> The number of partitions involved in a single WAL file may be quite large,
> e.g. a hundred. A hundred network requests to commit one WAL file is very
> impractical. However, there are ways to reduce this number:
>  1. Partition leaders are located on brokers. Requests to leaders on one
> broker could be grouped together into a single physical network request
> (resembling the normal Produce request that may carry batches for many
> partitions inside). This will cap the number of network requests to the
> number of brokers in the cluster.
>  2. If we craft the cluster metadata to make producers send their requests
> to the right brokers (with respect to AZs), we may achieve the higher
> concentration of logical commit requests in physical network requests
> reducing the number of the latter ones even further, ideally to one.
>
> Obviously, out of multiple commit requests some may fail or time out for a
> variety of reasons. This is fine. Some producers will receive totally or
> partially failed responses to their Produce requests, similar to what they
> would have received when appending to a classic topic fails or times out.
> If a partition experiences problems, other partitions will not be affected
> (again, like in classic topics). Of course, the uncommitted data will be
> garbage in WAL files. But WAL files are short-lived (batches are constantly
> assembled into segments and offloaded to tiered storage), so this garbage
> will be eventually deleted.
>
> For safely deleting WAL files we now need to centrally manage them, as
> this is the only state and logic that spans multiple partitions. On the
> diagram, you can see another commit request called “Commit file (best
> effort)” going to the WAL File Manager. This manager will be responsible
> for the following:
>  1. Collecting (by requests from brokers) and persisting information about
> committed WAL files.
>  2. To handle potential failures in file information delivery, it will be
> doing prefix scan on the remote storage periodically to find and register
> unknown files. The period of this scan will be configurable and ideally
> should be quite long.
>  3. Checking with the relevant partition leaders (after a grace period) if
> they still have batches in a particular file.
>  4. Physically deleting files when they aren’t anymore referred to by any
> partition.
>
> This new design offers the following advantages:
>  1. It simplifies the implementation of many Kafka features such as
> idempotence, transactions, queues, tiered storage, retention. Now we don’t
> need to abstract away and reuse the code from partition leaders in the
> batch coordinator. Instead, we will literally use the same code paths in
> leaders, with little adaptation. Workflows from classic topics mostly
> remain unchanged.
> For example, it seems that
> ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and
> KafkaApis.handleWriteTxnMarkersRequest used for transaction support on the
> partition leader side could be used for diskless topics with little
> adaptation. ProducerStateManager, needed for both idempotent produce and
> transactions, would be reused.
> Another example is share groups support, where the share partition leader,
> being co-located with the partition leader, would execute the same logic
> for both diskless and classic topics.
>  2. It returns to the familiar partition-based scaling model, where
> partitions are independent.
>  3. It makes the operation and failure patterns closer to the familiar
> ones from classic topics.
>  4. It opens a straightforward path to seamless switching the topics modes
> between diskless and classic.
>
> The rest of the things remain unchanged compared to the previous Diskless
> design (after all previous discussions). Such things as local segment
> materialization by replicas, the consume path, tiered storage integration,
> etc.
>
> If the community finds this design more suitable, we will update the
> KIP(s) accordingly and continue working on it. Please let us know what you
> think.
>
> Best regards,
> Ivan and Diskless team
>
> On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> > Hi Justine,
> >
> > Yes, you're right. We need to track the aborted transactions for in the
> diskless coordinator for as long as the corresponding offsets are there.
> With the tiered storage unification Greg mentioned earlier, this will be
> finite time even for infinite data retention.
> >
> > Best,
> > Ivan
> >
> > On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > > Hey Ivan,
> > >
> > > Thanks for the response. I think most of what you said made sense, but
> I
> > > did have some questions about this part:
> > >
> > > > As we understand this, the partition leader in classic topics forgets
> > > about a transaction once it’s replicated (HWM overpasses it). The
> > > transaction coordinator acts like the main guardian, allowing partition
> > > leaders to do this safely. Please correct me if this is wrong. We think
> > > about relying on this with the batch coordinator and delete the
> information
> > > about a transaction once it’s finished (as there’s no replication and
> HWM
> > > advances immediately).
> > >
> > > I didn't quite understand this. In classic topics, we have maps for
> ongoing
> > > transactions which remove state when the transaction is completed and
> an
> > > aborted transactions index which is retained for much longer. Once the
> > > transaction is completed, the coordinator is no longer involved in
> > > maintaining this partition side state, and it is subject to compaction
> etc.
> > > Looking back at the outline provided above, I didn't see much about the
> > > fetch path, so maybe that could be expanded a bit further. I saw the
> > > following in a response:
> > > > When the broker constructs a fully valid local segment, all the
> necessary
> > > control batches will be inserted and indices, including the transaction
> > > index will be built to serve FetchRequests exactly as they are today.
> > >
> > > Based on this, it seems like we need to retain the information about
> > > aborted txns for longer.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > >
> > > > Hi Justine and all,
> > > >
> > > > Thank you for your questions!
> > > >
> > > > > JO 1. >Since a transaction could be uniquely identified with
> producer ID
> > > > > and epoch, the positive result of this check could be cached
> locally
> > > > > Are we saying that only new transaction version 2 transactions can
> be
> > > > used
> > > > > here? If not, we can't uniquely identify transactions with
> producer id +
> > > > > epoch
> > > >
> > > > You’re right that we (probably unintentionally) focused only on
> version 2.
> > > > We can either limit the support to version 2 or consider using some
> > > > surrogates to support version 1.
> > > >
> > > > > JO 2. >The batch coordinator does the final transactional checks
> of the
> > > > > batches. This procedure would output the same errors like the
> partition
> > > > > leader in classic topics would do.
> > > > > Can you expand on what these checks are? Would you be checking if
> the
> > > > > transaction was still ongoing for example?* *
> > > >
> > > > Yes, the producer epoch, that the transaction is ongoing, and of
> course
> > > > the normal idempotence checks. What the partition leader in the
> classic
> > > > topics does before appending a batch to the local log (e.g. in
> > > > UnifiedLog.maybeStartTransactionVerification and
> > > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we
> unfortunately
> > > > cannot do these checks before appending the data to the WAL segment
> and
> > > > uploading it, but we can “tombstone” these batches in the batch
> coordinator
> > > > during the final commit.
> > > >
> > > > > Is there state about ongoing
> > > > > transactions in the batch coordinator? I see some other state
> mentioned
> > > > in
> > > > > the End transaction section, but it's not super clear what state is
> > > > stored
> > > > > and when it is stored.
> > > >
> > > > Right, this should have been more explicit. As the partition leader
> tracks
> > > > ongoing transactions for classic topics, the batch coordinator has
> to as
> > > > well. So when a transaction starts and ends, the transaction
> coordinator
> > > > must inform the batch coordinator about this.
> > > >
> > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> would
> > > > be
> > > > > stored in the batch coordinator?
> > > >
> > > > Yes. This could be deduced from the committed batches and other
> > > > information, but for the sake of performance we’d better store it
> > > > explicitly.
> > > >
> > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > >
> > > > As we understand this, the partition leader in classic topics forgets
> > > > about a transaction once it’s replicated (HWM overpasses it). The
> > > > transaction coordinator acts like the main guardian, allowing
> partition
> > > > leaders to do this safely. Please correct me if this is wrong. We
> think
> > > > about relying on this with the batch coordinator and delete the
> information
> > > > about a transaction once it’s finished (as there’s no replication
> and HWM
> > > > advances immediately).
> > > >
> > > > Best,
> > > > Ivan
> > > >
> > > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > > Hey folks,
> > > > >
> > > > > Excited to see some updates related to transactions!
> > > > >
> > > > > I had a few questions.
> > > > >
> > > > > JO 1. >Since a transaction could be uniquely identified with
> producer ID
> > > > > and epoch, the positive result of this check could be cached
> locally
> > > > > Are we saying that only new transaction version 2 transactions can
> be
> > > > used
> > > > > here? If not, we can't uniquely identify transactions with
> producer id +
> > > > > epoch
> > > > >
> > > > > JO 2. >The batch coordinator does the final transactional checks
> of the
> > > > > batches. This procedure would output the same errors like the
> partition
> > > > > leader in classic topics would do.
> > > > > Can you expand on what these checks are? Would you be checking if
> the
> > > > > transaction was still ongoing for example? Is there state about
> ongoing
> > > > > transactions in the batch coordinator? I see some other state
> mentioned
> > > > in
> > > > > the End transaction section, but it's not super clear what state is
> > > > stored
> > > > > and when it is stored.
> > > > >
> > > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that
> would
> > > > be
> > > > > stored in the batch coordinator?
> > > > >
> > > > > JO 4. Are there any thoughts about how long transactional state is
> > > > > maintained in the batch coordinator and how it will be cleaned up?
> > > > >
> > > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi, Greg and Ivan,
> > > > > >
> > > > > > Thanks for the update. A few comments.
> > > > > >
> > > > > > JR 10. "Consumer fetches are now served from local segments,
> making
> > > > use of
> > > > > > the
> > > > > > indexes, page cache, request purgatory, and zero-copy
> functionality
> > > > already
> > > > > > built into classic topics."
> > > > > > JR 10.1 Does the broker build the producer state for each
> partition in
> > > > > > diskless topics?
> > > > > > JR 10.2 For transactional data, the consumer fetches need to know
> > > > aborted
> > > > > > records. How is that achieved?
> > > > > >
> > > > > > JR 11. "The batch coordinator saves that the transaction is
> finished
> > > > and
> > > > > > also inserts the control batches in the corresponding logs of the
> > > > involved
> > > > > > Diskless topics. This happens only on the metadata level, no
> actual
> > > > control
> > > > > > batches are written to any file. "
> > > > > > A fetch response could include multiple transactional batches.
> How
> > > > does the
> > > > > > broker obtain the information about the ending control batch for
> each
> > > > > > batch? Does that mean that a fetch response needs to be built by
> > > > > > stitching record batches and generated control batches together?
> > > > > >
> > > > > > JR 12. Queues: Is there still a share partition leader that all
> > > > consumers
> > > > > > are routed to?
> > > > > >
> > > > > > JR 13. "Should the KIPs be modified to include this or it's too
> > > > > > implementation-focused?" It would be useful to include enough
> details
> > > > to
> > > > > > understand correctness and performance impact.
> > > > > >
> > > > > > HC5. Henry has a valid point. Requests from a given producer
> contain a
> > > > > > sequence number, which is ordered. If a producer sends every
> Produce
> > > > > > request to an arbitrary broker, those requests could reach the
> batch
> > > > > > coordinator in different order and lead to rejection of the
> produce
> > > > > > requests.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me>
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have also thought in a bit more details about transactions
> and
> > > > queues,
> > > > > > > here's the plan.
> > > > > > >
> > > > > > > *Transactions*
> > > > > > >
> > > > > > > The support for transactions in *classic topics* is based on
> precise
> > > > > > > interactions between three actors: clients (mostly producers,
> but
> > > > also
> > > > > > > consumers), brokers (ReplicaManager and other classes), and
> > > > transaction
> > > > > > > coordinators. Brokers also run partition leaders with their
> local
> > > > state
> > > > > > > (ProducerStateManager and others).
> > > > > > >
> > > > > > > The high level (some details skipped) workflow is the
> following.
> > > > When a
> > > > > > > transactional Produce request is received by the broker:
> > > > > > > 1. For each partition, the partition leader checks if a
> non-empty
> > > > > > > transaction is running for this partition. This is done using
> its
> > > > local
> > > > > > > state derived from the log metadata (ProducerStateManager,
> > > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > > 2. The transaction coordinator is informed about all the
> partitions
> > > > that
> > > > > > > aren’t part of the transaction to include them.
> > > > > > > 3. The partition leaders do additional transactional checks.
> > > > > > > 4. The partition leaders append the transactional data to
> their logs
> > > > and
> > > > > > > update some of their state (for example, log the fact that the
> > > > > > transaction
> > > > > > > is running for the partition and its first offset).
> > > > > > >
> > > > > > > When the transaction is committed or aborted:
> > > > > > > 1. The producer contacts the transaction coordinator directly
> with
> > > > > > > EndTxnRequest.
> > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > PREPARE_ABORT to
> > > > > > > its log and responds to the producer.
> > > > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to
> the
> > > > > > leaders
> > > > > > > of the involved partitions.
> > > > > > > 4. The partition leaders write the transaction markers to
> their logs
> > > > and
> > > > > > > respond to the coordinator.
> > > > > > > 5. The coordinator writes the final transaction state
> > > > COMPLETE_COMMIT or
> > > > > > > COMPLETE_ABORT.
> > > > > > >
> > > > > > > In classic topics, partitions have leaders and lots of
> important
> > > > state
> > > > > > > necessary for supporting this workflow is local. The main
> challenge
> > > > in
> > > > > > > mapping this to Diskless comes from the fact there are no
> partition
> > > > > > > leaders, so the corresponding pieces of state need to be
> globalized
> > > > in
> > > > > > the
> > > > > > > batch coordinator. We are already doing this to support
> idempotent
> > > > > > produce.
> > > > > > >
> > > > > > > The high level workflow for *diskless topics* would look very
> > > > similar:
> > > > > > > 1. For each partition, the broker checks if a non-empty
> transaction
> > > > is
> > > > > > > running for this partition. In contrast to classic topics,
> this is
> > > > > > checked
> > > > > > > against the batch coordinator with a single RPC. Since a
> transaction
> > > > > > could
> > > > > > > be uniquely identified with producer ID and epoch, the positive
> > > > result of
> > > > > > > this check could be cached locally (for the double configured
> > > > duration
> > > > > > of a
> > > > > > > transaction, for example).
> > > > > > > 2. The same: The transaction coordinator is informed about all
> the
> > > > > > > partitions that aren’t part of the transaction to include them.
> > > > > > > 3. No transactional checks are done on the broker side.
> > > > > > > 4. The broker appends the transactional data to the current
> shared
> > > > WAL
> > > > > > > segment. It doesn’t update any transaction-related state for
> Diskless
> > > > > > > topics, because it doesn’t have any.
> > > > > > > 5. The WAL segment is committed to the batch coordinator like
> in the
> > > > > > > normal produce flow.
> > > > > > > 6. The batch coordinator does the final transactional checks
> of the
> > > > > > > batches. This procedure would output the same errors like the
> > > > partition
> > > > > > > leader in classic topics would do. I.e. some batches could be
> > > > rejected.
> > > > > > > This means, there will potentially be garbage in the WAL
> segment
> > > > file in
> > > > > > > case of transactional errors. This is preferable to doing more
> > > > network
> > > > > > > round trips, especially considering the WAL segments will be
> > > > relatively
> > > > > > > short-living (see the Greg's update above).
> > > > > > >
> > > > > > > When the transaction is committed or aborted:
> > > > > > > 1. The producer contacts the transaction coordinator directly
> with
> > > > > > > EndTxnRequest.
> > > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > > PREPARE_ABORT to
> > > > > > > its log and responds to the producer.
> > > > > > > 3. *[NEW]* The transaction coordinator informs the batch
> coordinator
> > > > that
> > > > > > > the transaction is finished.
> > > > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> > > > finished
> > > > > > > and also inserts the control batches in the corresponding logs
> of the
> > > > > > > involved Diskless topics. This happens only on the metadata
> level, no
> > > > > > > actual control batches are written to any file. They will be
> > > > dynamically
> > > > > > > created on Fetch and other read operations. We could
> technically
> > > > write
> > > > > > > these control batches for real, but this would mean extra
> produce
> > > > > > latency,
> > > > > > > so it's better just to mark them in the batch coordinator and
> save
> > > > these
> > > > > > > milliseconds.
> > > > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to
> the
> > > > > > leaders
> > > > > > > of the involved partitions. – Now only to classic topics now.
> > > > > > > 6. The partition leaders of classic topics write the
> transaction
> > > > markers
> > > > > > > to their logs and respond to the coordinator.
> > > > > > > 7. The coordinator writes the final transaction state
> > > > COMPLETE_COMMIT or
> > > > > > > COMPLETE_ABORT.
> > > > > > >
> > > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > > 1. An extra network round trip between brokers and the batch
> > > > coordinator
> > > > > > > when a new partition appear in the transaction. To mitigate the
> > > > impact of
> > > > > > > them:
> > > > > > >   - The results will be cached.
> > > > > > >   - The calls for multiple partitions in one Produce request
> will be
> > > > > > > grouped.
> > > > > > >   - The batch coordinator should be optimized for fast
> response to
> > > > these
> > > > > > > RPCs.
> > > > > > >   - The fact that a single producer normally will communicate
> with a
> > > > > > > single broker for the duration of the transaction further
> reduces the
> > > > > > > expected number of round trips.
> > > > > > > 2. An extra round trip between the transaction coordinator and
> batch
> > > > > > > coordinator when a transaction is finished.
> > > > > > >
> > > > > > > With this proposal, transactions will also be able to span both
> > > > classic
> > > > > > > and Diskless topics.
> > > > > > >
> > > > > > > *Queues*
> > > > > > >
> > > > > > > The share group coordination and management is a side job that
> > > > doesn't
> > > > > > > interfere with the topic itself (leadership, replicas, physical
> > > > storage
> > > > > > of
> > > > > > > records, etc.) and non-queue producers and consumers (Fetch and
> > > > Produce
> > > > > > > RPCs, consumer group-related RPCs are not affected.) We don't
> see any
> > > > > > > reason why we can't make Diskless topics compatible with share
> > > > groups the
> > > > > > > same way as classic topics are. Even on the code level, we
> don't
> > > > expect
> > > > > > any
> > > > > > > serious refactoring: the same reading routines are used that
> are
> > > > used for
> > > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > > >
> > > > > > >
> > > > > > > Should the KIPs be modified to include this or it's too
> > > > > > > implementation-focused?
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Thank you all for your questions and design input on
> KIP-1150.
> > > > > > > >
> > > > > > > > We have just updated KIP-1150 and KIP-1163 with a new
> design. To
> > > > > > > summarize
> > > > > > > > the changes:
> > > > > > > >
> > > > > > > > 1. The design prioritizes integrating with the existing
> KIP-405
> > > > Tiered
> > > > > > > > Storage interfaces, permitting data produced to a Diskless
> topic
> > > > to be
> > > > > > > > moved to tiered storage.
> > > > > > > > This lowers the scalability requirements for the Batch
> Coordinator
> > > > > > > > component, and allows Diskless to compose with Tiered Storage
> > > > plugin
> > > > > > > > features such as encryption and alternative data formats.
> > > > > > > >
> > > > > > > > 2. Consumer fetches are now served from local segments,
> making use
> > > > of
> > > > > > the
> > > > > > > > indexes, page cache, request purgatory, and zero-copy
> functionality
> > > > > > > already
> > > > > > > > built into classic topics.
> > > > > > > > However, local segments are now considered cache elements,
> do not
> > > > need
> > > > > > to
> > > > > > > > be durably stored, and can be built without contacting any
> other
> > > > > > > replicas.
> > > > > > > >
> > > > > > > > 3. The design has been simplified substantially, by removing
> the
> > > > > > previous
> > > > > > > > Diskless consume flow, distributed cache component, and
> "object
> > > > > > > > compaction/merging" step.
> > > > > > > >
> > > > > > > > The design maintains leaderless produces as enabled by the
> Batch
> > > > > > > > Coordinator, and the same latency profiles as the earlier
> design,
> > > > while
> > > > > > > > being simpler and integrating better into the existing
> ecosystem.
> > > > > > > >
> > > > > > > > Thanks, and we are eager to hear your feedback on the new
> design.
> > > > > > > > Greg Harris
> > > > > > > >
> > > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao
> <ju...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Jan,
> > > > > > > > >
> > > > > > > > > For me, the main gap of KIP-1150 is the support of all
> existing
> > > > > > client
> > > > > > > > > APIs. Currently, there is no design for supporting APIs
> like
> > > > > > > transactions
> > > > > > > > > and queues.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Would it be a good time to ask for the current status of
> this
> > > > KIP?
> > > > > > I
> > > > > > > > > > haven't seen much activity here for the past 2 months,
> the
> > > > vote got
> > > > > > > > > vetoed
> > > > > > > > > > but I think the pending questions have been answered
> since
> > > > then.
> > > > > > > KIP-1183
> > > > > > > > > > (AutoMQ's proposal) also didn't have any activity since
> May.
> > > > > > > > > >
> > > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices
> that can
> > > > be
> > > > > > > > > > made, with a coordinator-based approach being by far the
> > > > dominant
> > > > > > one
> > > > > > > > > when
> > > > > > > > > > it comes to market adoption - but all these are
> standalone
> > > > > > products.
> > > > > > > > > >
> > > > > > > > > > I'm a big fan of both approaches, but would hate to see a
> > > > stall. So
> > > > > > > the
> > > > > > > > > > question is: can we get an update?
> > > > > > > > > >
> > > > > > > > > > Maybe it's time to start another vote? Colin McCabe -
> have your
> > > > > > > questions
> > > > > > > > > > been answered? If not, is there anything I can do to
> help? I'm
> > > > > > deeply
> > > > > > > > > > familiar with both architectures and have written about
> both?
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > > Jan
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > > >
> > > > > > > > > > > a) group all the KIP email threads in the main one
> (just a
> > > > bunch
> > > > > > of
> > > > > > > > > links
> > > > > > > > > > > to everything)
> > > > > > > > > > > b) create the email threads
> > > > > > > > > > >
> > > > > > > > > > > It's a bit hard to track it all - for example, I was
> > > > searching
> > > > > > for
> > > > > > > a
> > > > > > > > > > > discuss thread for KIP-1165 for a while; As far as I
> can
> > > > tell, it
> > > > > > > > > doesn't
> > > > > > > > > > > exist yet.
> > > > > > > > > > >
> > > > > > > > > > > Since the KIPs are published (by virtue of having the
> root
> > > > KIP be
> > > > > > > > > > > published, having a DISCUSS thread and links to
> sub-KIPs
> > > > where
> > > > > > were
> > > > > > > > > aimed
> > > > > > > > > > > to move the discussion towards), I think it would be
> good to
> > > > > > create
> > > > > > > > > > DISCUSS
> > > > > > > > > > > threads for them all.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Stan
> > > > > > > > > > >
> > > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > > >
> > > > > > > > > > > > We want to start a new KIP discussion about
> introducing a
> > > > new
> > > > > > > type of
> > > > > > > > > > > > topics that would make use of Object Storage as the
> primary
> > > > > > > source of
> > > > > > > > > > > > storage. However, as this KIP is big we decided to
> split it
> > > > > > into
> > > > > > > > > > multiple
> > > > > > > > > > > > related KIPs.
> > > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > > )
> > > > > > > > > > > > that aims to discuss if Apache Kafka should aim to
> have
> > > > this
> > > > > > > type of
> > > > > > > > > > > > feature at all. This KIP doesn't go onto details on
> how to
> > > > > > > implement
> > > > > > > > > > it.
> > > > > > > > > > > > This follows the same approach used when we discussed
> > > > KRaft.
> > > > > > > > > > > >
> > > > > > > > > > > > But as we know that it is sometimes really hard to
> discuss
> > > > on
> > > > > > > that
> > > > > > > > > meta
> > > > > > > > > > > > level, we also created several sub-kips (linked in
> > > > KIP-1150)
> > > > > > that
> > > > > > > > > offer
> > > > > > > > > > > an
> > > > > > > > > > > > implementation of this feature.
> > > > > > > > > > > >
> > > > > > > > > > > > We kindly ask you to use the proper DISCUSS threads
> for
> > > > each
> > > > > > > type of
> > > > > > > > > > > > concern and keep this one to discuss whether Apache
> Kafka
> > > > wants
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > > this feature or not.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance on behalf of all the authors of
> this KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > ------------------
> > > > > > > > > > > > Josep Prat
> > > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi dear Kafka community,

In the initial Diskless proposal, we proposed to have a separate component, batch/diskless coordinator, whose role would be to centrally manage the batch and WAL file metadata for diskless topics. This component drew many reasonable comments from the community about how it would support various Kafka features (transactions, queues) and its scalability. While we believe we have good answers to all the expressed concerns, we took a step back and looked at the problem from a different perspective.

We would like to propose an alternative Diskless design *without a centralized coordinator*. We believe this approach has potential and propose to discuss it as it may be more appealing to the community.

Let us explain the idea. Most of the complications with the original Diskless approach come from one necessary architecture change: globalizing the local state of partition leader in the batch coordinator. This causes deviations to the established workflows in various features like produce idempotence and transactions, queues, retention, etc. These deviations need to be carefully considered, designed, and later implemented and tested. In the new approach we want to avoid this by making partition leaders again responsible for managing their partitions, even in diskless topics.

In classic Kafka topics, batch data and metadata are blended together in the one partition log. The crux of the Diskless idea is to decouple them and move data to the remote storage, while keeping metadata somewhere else. Using the central batch coordinator for managing batch metadata is one way, but not the only.

Let’s now think about managing metadata for each user partition independently. Generally partitions are independent and don’t share anything apart from that their data are mixed in WAL files. If we figure out how to commit and later delete WAL files safely, we will achieve the necessary autonomy that allows us to get rid of the central batch coordinator. Instead, *each diskless user partition will be managed by its leader*, as in classic Kafka topics. Also like in classic topics, the leader uses the partition log as the way to persist batch metadata, i.e. the regular batch header + the information about how to find this batch on remote storage. In contrast to classic topics, batch data is in remote storage. 

For clarity, let’s compare the three designs:
 • Classic topics:
   • Data and metadata are co-located in the partition log.
   • The partition log content: [Batch header (metadata)|Batch data].
   • The partition log is replicated to the followers.
   • The replicas and leader have local state built from metadata.
 • Original Diskless:
   • Metadata is in the batch coordinator, data is on remote storage.
   • The partition state is global in the batch coordinator.
 • New Diskless:
   • Metadata is in the partition log, data is on remote storage.
   • Partition log content: [Batch header (metadata)|Batch coordinates on remote storage].
   • The partition log is replicated to the followers.
   • The replicas and leader have local state built from metadata.

Let’s consider the produce path. Here’s the reminder of the original Diskless design:


The new approach could be depicted as the following:


As you can see, the main difference is that now instead of a single commit request to the batch coordinator, we send multiple parallel commit requests to all the leaders of each partition involved in the WAL file. Each of them will commit its batches independently, without coordinating with other leaders and any other components. Batch data is addressed by the WAL file name, the byte offset and size, which allows partitions to know nothing about other partitions to access their data in shared WAL files.

The number of partitions involved in a single WAL file may be quite large, e.g. a hundred. A hundred network requests to commit one WAL file is very impractical. However, there are ways to reduce this number:
 1. Partition leaders are located on brokers. Requests to leaders on one broker could be grouped together into a single physical network request (resembling the normal Produce request that may carry batches for many partitions inside). This will cap the number of network requests to the number of brokers in the cluster.
 2. If we craft the cluster metadata to make producers send their requests to the right brokers (with respect to AZs), we may achieve the higher concentration of logical commit requests in physical network requests reducing the number of the latter ones even further, ideally to one.

Obviously, out of multiple commit requests some may fail or time out for a variety of reasons. This is fine. Some producers will receive totally or partially failed responses to their Produce requests, similar to what they would have received when appending to a classic topic fails or times out. If a partition experiences problems, other partitions will not be affected (again, like in classic topics). Of course, the uncommitted data will be garbage in WAL files. But WAL files are short-lived (batches are constantly assembled into segments and offloaded to tiered storage), so this garbage will be eventually deleted.

For safely deleting WAL files we now need to centrally manage them, as this is the only state and logic that spans multiple partitions. On the diagram, you can see another commit request called “Commit file (best effort)” going to the WAL File Manager. This manager will be responsible for the following:
 1. Collecting (by requests from brokers) and persisting information about committed WAL files.
 2. To handle potential failures in file information delivery, it will be doing prefix scan on the remote storage periodically to find and register unknown files. The period of this scan will be configurable and ideally should be quite long.
 3. Checking with the relevant partition leaders (after a grace period) if they still have batches in a particular file.
 4. Physically deleting files when they aren’t anymore referred to by any partition.

This new design offers the following advantages:
 1. It simplifies the implementation of many Kafka features such as idempotence, transactions, queues, tiered storage, retention. Now we don’t need to abstract away and reuse the code from partition leaders in the batch coordinator. Instead, we will literally use the same code paths in leaders, with little adaptation. Workflows from classic topics mostly remain unchanged.
For example, it seems that ReplicaManager.maybeSendPartitionsToTransactionCoordinator  and KafkaApis.handleWriteTxnMarkersRequest used for transaction support on the partition leader side could be used for diskless topics with little adaptation. ProducerStateManager, needed for both idempotent produce and transactions, would be reused.
Another example is share groups support, where the share partition leader, being co-located with the partition leader, would execute the same logic for both diskless and classic topics.
 2. It returns to the familiar partition-based scaling model, where partitions are independent.
 3. It makes the operation and failure patterns closer to the familiar ones from classic topics.
 4. It opens a straightforward path to seamless switching the topics modes between diskless and classic.

The rest of the things remain unchanged compared to the previous Diskless design (after all previous discussions). Such things as local segment materialization by replicas, the consume path, tiered storage integration, etc.

If the community finds this design more suitable, we will update the KIP(s) accordingly and continue working on it. Please let us know what you think.

Best regards,
Ivan and Diskless team

On Mon, Sep 29, 2025, at 15:06, Ivan Yurchenko wrote:
> Hi Justine,
> 
> Yes, you're right. We need to track the aborted transactions for in the diskless coordinator for as long as the corresponding offsets are there. With the tiered storage unification Greg mentioned earlier, this will be finite time even for infinite data retention.
> 
> Best,
> Ivan
> 
> On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> > Hey Ivan,
> > 
> > Thanks for the response. I think most of what you said made sense, but I
> > did have some questions about this part:
> > 
> > > As we understand this, the partition leader in classic topics forgets
> > about a transaction once it’s replicated (HWM overpasses it). The
> > transaction coordinator acts like the main guardian, allowing partition
> > leaders to do this safely. Please correct me if this is wrong. We think
> > about relying on this with the batch coordinator and delete the information
> > about a transaction once it’s finished (as there’s no replication and HWM
> > advances immediately).
> > 
> > I didn't quite understand this. In classic topics, we have maps for ongoing
> > transactions which remove state when the transaction is completed and an
> > aborted transactions index which is retained for much longer. Once the
> > transaction is completed, the coordinator is no longer involved in
> > maintaining this partition side state, and it is subject to compaction etc.
> > Looking back at the outline provided above, I didn't see much about the
> > fetch path, so maybe that could be expanded a bit further. I saw the
> > following in a response:
> > > When the broker constructs a fully valid local segment, all the necessary
> > control batches will be inserted and indices, including the transaction
> > index will be built to serve FetchRequests exactly as they are today.
> > 
> > Based on this, it seems like we need to retain the information about
> > aborted txns for longer.
> > 
> > Thanks,
> > Justine
> > 
> > On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > 
> > > Hi Justine and all,
> > >
> > > Thank you for your questions!
> > >
> > > > JO 1. >Since a transaction could be uniquely identified with producer ID
> > > > and epoch, the positive result of this check could be cached locally
> > > > Are we saying that only new transaction version 2 transactions can be
> > > used
> > > > here? If not, we can't uniquely identify transactions with producer id +
> > > > epoch
> > >
> > > You’re right that we (probably unintentionally) focused only on version 2.
> > > We can either limit the support to version 2 or consider using some
> > > surrogates to support version 1.
> > >
> > > > JO 2. >The batch coordinator does the final transactional checks of the
> > > > batches. This procedure would output the same errors like the partition
> > > > leader in classic topics would do.
> > > > Can you expand on what these checks are? Would you be checking if the
> > > > transaction was still ongoing for example?* *
> > >
> > > Yes, the producer epoch, that the transaction is ongoing, and of course
> > > the normal idempotence checks. What the partition leader in the classic
> > > topics does before appending a batch to the local log (e.g. in
> > > UnifiedLog.maybeStartTransactionVerification and
> > > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we unfortunately
> > > cannot do these checks before appending the data to the WAL segment and
> > > uploading it, but we can “tombstone” these batches in the batch coordinator
> > > during the final commit.
> > >
> > > > Is there state about ongoing
> > > > transactions in the batch coordinator? I see some other state mentioned
> > > in
> > > > the End transaction section, but it's not super clear what state is
> > > stored
> > > > and when it is stored.
> > >
> > > Right, this should have been more explicit. As the partition leader tracks
> > > ongoing transactions for classic topics, the batch coordinator has to as
> > > well. So when a transaction starts and ends, the transaction coordinator
> > > must inform the batch coordinator about this.
> > >
> > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> > > be
> > > > stored in the batch coordinator?
> > >
> > > Yes. This could be deduced from the committed batches and other
> > > information, but for the sake of performance we’d better store it
> > > explicitly.
> > >
> > > > JO 4. Are there any thoughts about how long transactional state is
> > > > maintained in the batch coordinator and how it will be cleaned up?
> > >
> > > As we understand this, the partition leader in classic topics forgets
> > > about a transaction once it’s replicated (HWM overpasses it). The
> > > transaction coordinator acts like the main guardian, allowing partition
> > > leaders to do this safely. Please correct me if this is wrong. We think
> > > about relying on this with the batch coordinator and delete the information
> > > about a transaction once it’s finished (as there’s no replication and HWM
> > > advances immediately).
> > >
> > > Best,
> > > Ivan
> > >
> > > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > > Hey folks,
> > > >
> > > > Excited to see some updates related to transactions!
> > > >
> > > > I had a few questions.
> > > >
> > > > JO 1. >Since a transaction could be uniquely identified with producer ID
> > > > and epoch, the positive result of this check could be cached locally
> > > > Are we saying that only new transaction version 2 transactions can be
> > > used
> > > > here? If not, we can't uniquely identify transactions with producer id +
> > > > epoch
> > > >
> > > > JO 2. >The batch coordinator does the final transactional checks of the
> > > > batches. This procedure would output the same errors like the partition
> > > > leader in classic topics would do.
> > > > Can you expand on what these checks are? Would you be checking if the
> > > > transaction was still ongoing for example? Is there state about ongoing
> > > > transactions in the batch coordinator? I see some other state mentioned
> > > in
> > > > the End transaction section, but it's not super clear what state is
> > > stored
> > > > and when it is stored.
> > > >
> > > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> > > be
> > > > stored in the batch coordinator?
> > > >
> > > > JO 4. Are there any thoughts about how long transactional state is
> > > > maintained in the batch coordinator and how it will be cleaned up?
> > > >
> > > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Hi, Greg and Ivan,
> > > > >
> > > > > Thanks for the update. A few comments.
> > > > >
> > > > > JR 10. "Consumer fetches are now served from local segments, making
> > > use of
> > > > > the
> > > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > already
> > > > > built into classic topics."
> > > > > JR 10.1 Does the broker build the producer state for each partition in
> > > > > diskless topics?
> > > > > JR 10.2 For transactional data, the consumer fetches need to know
> > > aborted
> > > > > records. How is that achieved?
> > > > >
> > > > > JR 11. "The batch coordinator saves that the transaction is finished
> > > and
> > > > > also inserts the control batches in the corresponding logs of the
> > > involved
> > > > > Diskless topics. This happens only on the metadata level, no actual
> > > control
> > > > > batches are written to any file. "
> > > > > A fetch response could include multiple transactional batches. How
> > > does the
> > > > > broker obtain the information about the ending control batch for each
> > > > > batch? Does that mean that a fetch response needs to be built by
> > > > > stitching record batches and generated control batches together?
> > > > >
> > > > > JR 12. Queues: Is there still a share partition leader that all
> > > consumers
> > > > > are routed to?
> > > > >
> > > > > JR 13. "Should the KIPs be modified to include this or it's too
> > > > > implementation-focused?" It would be useful to include enough details
> > > to
> > > > > understand correctness and performance impact.
> > > > >
> > > > > HC5. Henry has a valid point. Requests from a given producer contain a
> > > > > sequence number, which is ordered. If a producer sends every Produce
> > > > > request to an arbitrary broker, those requests could reach the batch
> > > > > coordinator in different order and lead to rejection of the produce
> > > > > requests.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We have also thought in a bit more details about transactions and
> > > queues,
> > > > > > here's the plan.
> > > > > >
> > > > > > *Transactions*
> > > > > >
> > > > > > The support for transactions in *classic topics* is based on precise
> > > > > > interactions between three actors: clients (mostly producers, but
> > > also
> > > > > > consumers), brokers (ReplicaManager and other classes), and
> > > transaction
> > > > > > coordinators. Brokers also run partition leaders with their local
> > > state
> > > > > > (ProducerStateManager and others).
> > > > > >
> > > > > > The high level (some details skipped) workflow is the following.
> > > When a
> > > > > > transactional Produce request is received by the broker:
> > > > > > 1. For each partition, the partition leader checks if a non-empty
> > > > > > transaction is running for this partition. This is done using its
> > > local
> > > > > > state derived from the log metadata (ProducerStateManager,
> > > > > > VerificationStateEntry, VerificationGuard).
> > > > > > 2. The transaction coordinator is informed about all the partitions
> > > that
> > > > > > aren’t part of the transaction to include them.
> > > > > > 3. The partition leaders do additional transactional checks.
> > > > > > 4. The partition leaders append the transactional data to their logs
> > > and
> > > > > > update some of their state (for example, log the fact that the
> > > > > transaction
> > > > > > is running for the partition and its first offset).
> > > > > >
> > > > > > When the transaction is committed or aborted:
> > > > > > 1. The producer contacts the transaction coordinator directly with
> > > > > > EndTxnRequest.
> > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > PREPARE_ABORT to
> > > > > > its log and responds to the producer.
> > > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > > > leaders
> > > > > > of the involved partitions.
> > > > > > 4. The partition leaders write the transaction markers to their logs
> > > and
> > > > > > respond to the coordinator.
> > > > > > 5. The coordinator writes the final transaction state
> > > COMPLETE_COMMIT or
> > > > > > COMPLETE_ABORT.
> > > > > >
> > > > > > In classic topics, partitions have leaders and lots of important
> > > state
> > > > > > necessary for supporting this workflow is local. The main challenge
> > > in
> > > > > > mapping this to Diskless comes from the fact there are no partition
> > > > > > leaders, so the corresponding pieces of state need to be globalized
> > > in
> > > > > the
> > > > > > batch coordinator. We are already doing this to support idempotent
> > > > > produce.
> > > > > >
> > > > > > The high level workflow for *diskless topics* would look very
> > > similar:
> > > > > > 1. For each partition, the broker checks if a non-empty transaction
> > > is
> > > > > > running for this partition. In contrast to classic topics, this is
> > > > > checked
> > > > > > against the batch coordinator with a single RPC. Since a transaction
> > > > > could
> > > > > > be uniquely identified with producer ID and epoch, the positive
> > > result of
> > > > > > this check could be cached locally (for the double configured
> > > duration
> > > > > of a
> > > > > > transaction, for example).
> > > > > > 2. The same: The transaction coordinator is informed about all the
> > > > > > partitions that aren’t part of the transaction to include them.
> > > > > > 3. No transactional checks are done on the broker side.
> > > > > > 4. The broker appends the transactional data to the current shared
> > > WAL
> > > > > > segment. It doesn’t update any transaction-related state for Diskless
> > > > > > topics, because it doesn’t have any.
> > > > > > 5. The WAL segment is committed to the batch coordinator like in the
> > > > > > normal produce flow.
> > > > > > 6. The batch coordinator does the final transactional checks of the
> > > > > > batches. This procedure would output the same errors like the
> > > partition
> > > > > > leader in classic topics would do. I.e. some batches could be
> > > rejected.
> > > > > > This means, there will potentially be garbage in the WAL segment
> > > file in
> > > > > > case of transactional errors. This is preferable to doing more
> > > network
> > > > > > round trips, especially considering the WAL segments will be
> > > relatively
> > > > > > short-living (see the Greg's update above).
> > > > > >
> > > > > > When the transaction is committed or aborted:
> > > > > > 1. The producer contacts the transaction coordinator directly with
> > > > > > EndTxnRequest.
> > > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > > PREPARE_ABORT to
> > > > > > its log and responds to the producer.
> > > > > > 3. *[NEW]* The transaction coordinator informs the batch coordinator
> > > that
> > > > > > the transaction is finished.
> > > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> > > finished
> > > > > > and also inserts the control batches in the corresponding logs of the
> > > > > > involved Diskless topics. This happens only on the metadata level, no
> > > > > > actual control batches are written to any file. They will be
> > > dynamically
> > > > > > created on Fetch and other read operations. We could technically
> > > write
> > > > > > these control batches for real, but this would mean extra produce
> > > > > latency,
> > > > > > so it's better just to mark them in the batch coordinator and save
> > > these
> > > > > > milliseconds.
> > > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > > > leaders
> > > > > > of the involved partitions. – Now only to classic topics now.
> > > > > > 6. The partition leaders of classic topics write the transaction
> > > markers
> > > > > > to their logs and respond to the coordinator.
> > > > > > 7. The coordinator writes the final transaction state
> > > COMPLETE_COMMIT or
> > > > > > COMPLETE_ABORT.
> > > > > >
> > > > > > Compared to the non-transactional produce flow, we get:
> > > > > > 1. An extra network round trip between brokers and the batch
> > > coordinator
> > > > > > when a new partition appear in the transaction. To mitigate the
> > > impact of
> > > > > > them:
> > > > > >   - The results will be cached.
> > > > > >   - The calls for multiple partitions in one Produce request will be
> > > > > > grouped.
> > > > > >   - The batch coordinator should be optimized for fast response to
> > > these
> > > > > > RPCs.
> > > > > >   - The fact that a single producer normally will communicate with a
> > > > > > single broker for the duration of the transaction further reduces the
> > > > > > expected number of round trips.
> > > > > > 2. An extra round trip between the transaction coordinator and batch
> > > > > > coordinator when a transaction is finished.
> > > > > >
> > > > > > With this proposal, transactions will also be able to span both
> > > classic
> > > > > > and Diskless topics.
> > > > > >
> > > > > > *Queues*
> > > > > >
> > > > > > The share group coordination and management is a side job that
> > > doesn't
> > > > > > interfere with the topic itself (leadership, replicas, physical
> > > storage
> > > > > of
> > > > > > records, etc.) and non-queue producers and consumers (Fetch and
> > > Produce
> > > > > > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > > > > > reason why we can't make Diskless topics compatible with share
> > > groups the
> > > > > > same way as classic topics are. Even on the code level, we don't
> > > expect
> > > > > any
> > > > > > serious refactoring: the same reading routines are used that are
> > > used for
> > > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > > >
> > > > > >
> > > > > > Should the KIPs be modified to include this or it's too
> > > > > > implementation-focused?
> > > > > >
> > > > > > Best regards,
> > > > > > Ivan
> > > > > >
> > > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Thank you all for your questions and design input on KIP-1150.
> > > > > > >
> > > > > > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > > > > > summarize
> > > > > > > the changes:
> > > > > > >
> > > > > > > 1. The design prioritizes integrating with the existing KIP-405
> > > Tiered
> > > > > > > Storage interfaces, permitting data produced to a Diskless topic
> > > to be
> > > > > > > moved to tiered storage.
> > > > > > > This lowers the scalability requirements for the Batch Coordinator
> > > > > > > component, and allows Diskless to compose with Tiered Storage
> > > plugin
> > > > > > > features such as encryption and alternative data formats.
> > > > > > >
> > > > > > > 2. Consumer fetches are now served from local segments, making use
> > > of
> > > > > the
> > > > > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > > > > already
> > > > > > > built into classic topics.
> > > > > > > However, local segments are now considered cache elements, do not
> > > need
> > > > > to
> > > > > > > be durably stored, and can be built without contacting any other
> > > > > > replicas.
> > > > > > >
> > > > > > > 3. The design has been simplified substantially, by removing the
> > > > > previous
> > > > > > > Diskless consume flow, distributed cache component, and "object
> > > > > > > compaction/merging" step.
> > > > > > >
> > > > > > > The design maintains leaderless produces as enabled by the Batch
> > > > > > > Coordinator, and the same latency profiles as the earlier design,
> > > while
> > > > > > > being simpler and integrating better into the existing ecosystem.
> > > > > > >
> > > > > > > Thanks, and we are eager to hear your feedback on the new design.
> > > > > > > Greg Harris
> > > > > > >
> > > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Jan,
> > > > > > > >
> > > > > > > > For me, the main gap of KIP-1150 is the support of all existing
> > > > > client
> > > > > > > > APIs. Currently, there is no design for supporting APIs like
> > > > > > transactions
> > > > > > > > and queues.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > > >
> > > > > > > > > Would it be a good time to ask for the current status of this
> > > KIP?
> > > > > I
> > > > > > > > > haven't seen much activity here for the past 2 months, the
> > > vote got
> > > > > > > > vetoed
> > > > > > > > > but I think the pending questions have been answered since
> > > then.
> > > > > > KIP-1183
> > > > > > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > > > > > >
> > > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can
> > > be
> > > > > > > > > made, with a coordinator-based approach being by far the
> > > dominant
> > > > > one
> > > > > > > > when
> > > > > > > > > it comes to market adoption - but all these are standalone
> > > > > products.
> > > > > > > > >
> > > > > > > > > I'm a big fan of both approaches, but would hate to see a
> > > stall. So
> > > > > > the
> > > > > > > > > question is: can we get an update?
> > > > > > > > >
> > > > > > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > > > > > questions
> > > > > > > > > been answered? If not, is there anything I can do to help? I'm
> > > > > deeply
> > > > > > > > > familiar with both architectures and have written about both?
> > > > > > > > >
> > > > > > > > > Kind regards,
> > > > > > > > > Jan
> > > > > > > > >
> > > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > I have some nits - it may be useful to
> > > > > > > > > >
> > > > > > > > > > a) group all the KIP email threads in the main one (just a
> > > bunch
> > > > > of
> > > > > > > > links
> > > > > > > > > > to everything)
> > > > > > > > > > b) create the email threads
> > > > > > > > > >
> > > > > > > > > > It's a bit hard to track it all - for example, I was
> > > searching
> > > > > for
> > > > > > a
> > > > > > > > > > discuss thread for KIP-1165 for a while; As far as I can
> > > tell, it
> > > > > > > > doesn't
> > > > > > > > > > exist yet.
> > > > > > > > > >
> > > > > > > > > > Since the KIPs are published (by virtue of having the root
> > > KIP be
> > > > > > > > > > published, having a DISCUSS thread and links to sub-KIPs
> > > where
> > > > > were
> > > > > > > > aimed
> > > > > > > > > > to move the discussion towards), I think it would be good to
> > > > > create
> > > > > > > > > DISCUSS
> > > > > > > > > > threads for them all.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stan
> > > > > > > > > >
> > > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > > >
> > > > > > > > > > > We want to start a new KIP discussion about introducing a
> > > new
> > > > > > type of
> > > > > > > > > > > topics that would make use of Object Storage as the primary
> > > > > > source of
> > > > > > > > > > > storage. However, as this KIP is big we decided to split it
> > > > > into
> > > > > > > > > multiple
> > > > > > > > > > > related KIPs.
> > > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > > )
> > > > > > > > > > > that aims to discuss if Apache Kafka should aim to have
> > > this
> > > > > > type of
> > > > > > > > > > > feature at all. This KIP doesn't go onto details on how to
> > > > > > implement
> > > > > > > > > it.
> > > > > > > > > > > This follows the same approach used when we discussed
> > > KRaft.
> > > > > > > > > > >
> > > > > > > > > > > But as we know that it is sometimes really hard to discuss
> > > on
> > > > > > that
> > > > > > > > meta
> > > > > > > > > > > level, we also created several sub-kips (linked in
> > > KIP-1150)
> > > > > that
> > > > > > > > offer
> > > > > > > > > > an
> > > > > > > > > > > implementation of this feature.
> > > > > > > > > > >
> > > > > > > > > > > We kindly ask you to use the proper DISCUSS threads for
> > > each
> > > > > > type of
> > > > > > > > > > > concern and keep this one to discuss whether Apache Kafka
> > > wants
> > > > > > to
> > > > > > > > have
> > > > > > > > > > > this feature or not.
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > > > > > >
> > > > > > > > > > > ------------------
> > > > > > > > > > > Josep Prat
> > > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > 
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Justine,

Yes, you're right. We need to track the aborted transactions for in the diskless coordinator for as long as the corresponding offsets are there. With the tiered storage unification Greg mentioned earlier, this will be finite time even for infinite data retention.

Best,
Ivan

On Wed, Sep 17, 2025, at 19:41, Justine Olshan wrote:
> Hey Ivan,
> 
> Thanks for the response. I think most of what you said made sense, but I
> did have some questions about this part:
> 
> > As we understand this, the partition leader in classic topics forgets
> about a transaction once it’s replicated (HWM overpasses it). The
> transaction coordinator acts like the main guardian, allowing partition
> leaders to do this safely. Please correct me if this is wrong. We think
> about relying on this with the batch coordinator and delete the information
> about a transaction once it’s finished (as there’s no replication and HWM
> advances immediately).
> 
> I didn't quite understand this. In classic topics, we have maps for ongoing
> transactions which remove state when the transaction is completed and an
> aborted transactions index which is retained for much longer. Once the
> transaction is completed, the coordinator is no longer involved in
> maintaining this partition side state, and it is subject to compaction etc.
> Looking back at the outline provided above, I didn't see much about the
> fetch path, so maybe that could be expanded a bit further. I saw the
> following in a response:
> > When the broker constructs a fully valid local segment, all the necessary
> control batches will be inserted and indices, including the transaction
> index will be built to serve FetchRequests exactly as they are today.
> 
> Based on this, it seems like we need to retain the information about
> aborted txns for longer.
> 
> Thanks,
> Justine
> 
> On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> 
> > Hi Justine and all,
> >
> > Thank you for your questions!
> >
> > > JO 1. >Since a transaction could be uniquely identified with producer ID
> > > and epoch, the positive result of this check could be cached locally
> > > Are we saying that only new transaction version 2 transactions can be
> > used
> > > here? If not, we can't uniquely identify transactions with producer id +
> > > epoch
> >
> > You’re right that we (probably unintentionally) focused only on version 2.
> > We can either limit the support to version 2 or consider using some
> > surrogates to support version 1.
> >
> > > JO 2. >The batch coordinator does the final transactional checks of the
> > > batches. This procedure would output the same errors like the partition
> > > leader in classic topics would do.
> > > Can you expand on what these checks are? Would you be checking if the
> > > transaction was still ongoing for example?* *
> >
> > Yes, the producer epoch, that the transaction is ongoing, and of course
> > the normal idempotence checks. What the partition leader in the classic
> > topics does before appending a batch to the local log (e.g. in
> > UnifiedLog.maybeStartTransactionVerification and
> > UnifiedLog.analyzeAndValidateProducerState). In Diskless, we unfortunately
> > cannot do these checks before appending the data to the WAL segment and
> > uploading it, but we can “tombstone” these batches in the batch coordinator
> > during the final commit.
> >
> > > Is there state about ongoing
> > > transactions in the batch coordinator? I see some other state mentioned
> > in
> > > the End transaction section, but it's not super clear what state is
> > stored
> > > and when it is stored.
> >
> > Right, this should have been more explicit. As the partition leader tracks
> > ongoing transactions for classic topics, the batch coordinator has to as
> > well. So when a transaction starts and ends, the transaction coordinator
> > must inform the batch coordinator about this.
> >
> > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> > be
> > > stored in the batch coordinator?
> >
> > Yes. This could be deduced from the committed batches and other
> > information, but for the sake of performance we’d better store it
> > explicitly.
> >
> > > JO 4. Are there any thoughts about how long transactional state is
> > > maintained in the batch coordinator and how it will be cleaned up?
> >
> > As we understand this, the partition leader in classic topics forgets
> > about a transaction once it’s replicated (HWM overpasses it). The
> > transaction coordinator acts like the main guardian, allowing partition
> > leaders to do this safely. Please correct me if this is wrong. We think
> > about relying on this with the batch coordinator and delete the information
> > about a transaction once it’s finished (as there’s no replication and HWM
> > advances immediately).
> >
> > Best,
> > Ivan
> >
> > On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > > Hey folks,
> > >
> > > Excited to see some updates related to transactions!
> > >
> > > I had a few questions.
> > >
> > > JO 1. >Since a transaction could be uniquely identified with producer ID
> > > and epoch, the positive result of this check could be cached locally
> > > Are we saying that only new transaction version 2 transactions can be
> > used
> > > here? If not, we can't uniquely identify transactions with producer id +
> > > epoch
> > >
> > > JO 2. >The batch coordinator does the final transactional checks of the
> > > batches. This procedure would output the same errors like the partition
> > > leader in classic topics would do.
> > > Can you expand on what these checks are? Would you be checking if the
> > > transaction was still ongoing for example? Is there state about ongoing
> > > transactions in the batch coordinator? I see some other state mentioned
> > in
> > > the End transaction section, but it's not super clear what state is
> > stored
> > > and when it is stored.
> > >
> > > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> > be
> > > stored in the batch coordinator?
> > >
> > > JO 4. Are there any thoughts about how long transactional state is
> > > maintained in the batch coordinator and how it will be cleaned up?
> > >
> > > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Hi, Greg and Ivan,
> > > >
> > > > Thanks for the update. A few comments.
> > > >
> > > > JR 10. "Consumer fetches are now served from local segments, making
> > use of
> > > > the
> > > > indexes, page cache, request purgatory, and zero-copy functionality
> > already
> > > > built into classic topics."
> > > > JR 10.1 Does the broker build the producer state for each partition in
> > > > diskless topics?
> > > > JR 10.2 For transactional data, the consumer fetches need to know
> > aborted
> > > > records. How is that achieved?
> > > >
> > > > JR 11. "The batch coordinator saves that the transaction is finished
> > and
> > > > also inserts the control batches in the corresponding logs of the
> > involved
> > > > Diskless topics. This happens only on the metadata level, no actual
> > control
> > > > batches are written to any file. "
> > > > A fetch response could include multiple transactional batches. How
> > does the
> > > > broker obtain the information about the ending control batch for each
> > > > batch? Does that mean that a fetch response needs to be built by
> > > > stitching record batches and generated control batches together?
> > > >
> > > > JR 12. Queues: Is there still a share partition leader that all
> > consumers
> > > > are routed to?
> > > >
> > > > JR 13. "Should the KIPs be modified to include this or it's too
> > > > implementation-focused?" It would be useful to include enough details
> > to
> > > > understand correctness and performance impact.
> > > >
> > > > HC5. Henry has a valid point. Requests from a given producer contain a
> > > > sequence number, which is ordered. If a producer sends every Produce
> > > > request to an arbitrary broker, those requests could reach the batch
> > > > coordinator in different order and lead to rejection of the produce
> > > > requests.
> > > >
> > > > Jun
> > > >
> > > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We have also thought in a bit more details about transactions and
> > queues,
> > > > > here's the plan.
> > > > >
> > > > > *Transactions*
> > > > >
> > > > > The support for transactions in *classic topics* is based on precise
> > > > > interactions between three actors: clients (mostly producers, but
> > also
> > > > > consumers), brokers (ReplicaManager and other classes), and
> > transaction
> > > > > coordinators. Brokers also run partition leaders with their local
> > state
> > > > > (ProducerStateManager and others).
> > > > >
> > > > > The high level (some details skipped) workflow is the following.
> > When a
> > > > > transactional Produce request is received by the broker:
> > > > > 1. For each partition, the partition leader checks if a non-empty
> > > > > transaction is running for this partition. This is done using its
> > local
> > > > > state derived from the log metadata (ProducerStateManager,
> > > > > VerificationStateEntry, VerificationGuard).
> > > > > 2. The transaction coordinator is informed about all the partitions
> > that
> > > > > aren’t part of the transaction to include them.
> > > > > 3. The partition leaders do additional transactional checks.
> > > > > 4. The partition leaders append the transactional data to their logs
> > and
> > > > > update some of their state (for example, log the fact that the
> > > > transaction
> > > > > is running for the partition and its first offset).
> > > > >
> > > > > When the transaction is committed or aborted:
> > > > > 1. The producer contacts the transaction coordinator directly with
> > > > > EndTxnRequest.
> > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > PREPARE_ABORT to
> > > > > its log and responds to the producer.
> > > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > > leaders
> > > > > of the involved partitions.
> > > > > 4. The partition leaders write the transaction markers to their logs
> > and
> > > > > respond to the coordinator.
> > > > > 5. The coordinator writes the final transaction state
> > COMPLETE_COMMIT or
> > > > > COMPLETE_ABORT.
> > > > >
> > > > > In classic topics, partitions have leaders and lots of important
> > state
> > > > > necessary for supporting this workflow is local. The main challenge
> > in
> > > > > mapping this to Diskless comes from the fact there are no partition
> > > > > leaders, so the corresponding pieces of state need to be globalized
> > in
> > > > the
> > > > > batch coordinator. We are already doing this to support idempotent
> > > > produce.
> > > > >
> > > > > The high level workflow for *diskless topics* would look very
> > similar:
> > > > > 1. For each partition, the broker checks if a non-empty transaction
> > is
> > > > > running for this partition. In contrast to classic topics, this is
> > > > checked
> > > > > against the batch coordinator with a single RPC. Since a transaction
> > > > could
> > > > > be uniquely identified with producer ID and epoch, the positive
> > result of
> > > > > this check could be cached locally (for the double configured
> > duration
> > > > of a
> > > > > transaction, for example).
> > > > > 2. The same: The transaction coordinator is informed about all the
> > > > > partitions that aren’t part of the transaction to include them.
> > > > > 3. No transactional checks are done on the broker side.
> > > > > 4. The broker appends the transactional data to the current shared
> > WAL
> > > > > segment. It doesn’t update any transaction-related state for Diskless
> > > > > topics, because it doesn’t have any.
> > > > > 5. The WAL segment is committed to the batch coordinator like in the
> > > > > normal produce flow.
> > > > > 6. The batch coordinator does the final transactional checks of the
> > > > > batches. This procedure would output the same errors like the
> > partition
> > > > > leader in classic topics would do. I.e. some batches could be
> > rejected.
> > > > > This means, there will potentially be garbage in the WAL segment
> > file in
> > > > > case of transactional errors. This is preferable to doing more
> > network
> > > > > round trips, especially considering the WAL segments will be
> > relatively
> > > > > short-living (see the Greg's update above).
> > > > >
> > > > > When the transaction is committed or aborted:
> > > > > 1. The producer contacts the transaction coordinator directly with
> > > > > EndTxnRequest.
> > > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> > PREPARE_ABORT to
> > > > > its log and responds to the producer.
> > > > > 3. *[NEW]* The transaction coordinator informs the batch coordinator
> > that
> > > > > the transaction is finished.
> > > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> > finished
> > > > > and also inserts the control batches in the corresponding logs of the
> > > > > involved Diskless topics. This happens only on the metadata level, no
> > > > > actual control batches are written to any file. They will be
> > dynamically
> > > > > created on Fetch and other read operations. We could technically
> > write
> > > > > these control batches for real, but this would mean extra produce
> > > > latency,
> > > > > so it's better just to mark them in the batch coordinator and save
> > these
> > > > > milliseconds.
> > > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > > leaders
> > > > > of the involved partitions. – Now only to classic topics now.
> > > > > 6. The partition leaders of classic topics write the transaction
> > markers
> > > > > to their logs and respond to the coordinator.
> > > > > 7. The coordinator writes the final transaction state
> > COMPLETE_COMMIT or
> > > > > COMPLETE_ABORT.
> > > > >
> > > > > Compared to the non-transactional produce flow, we get:
> > > > > 1. An extra network round trip between brokers and the batch
> > coordinator
> > > > > when a new partition appear in the transaction. To mitigate the
> > impact of
> > > > > them:
> > > > >   - The results will be cached.
> > > > >   - The calls for multiple partitions in one Produce request will be
> > > > > grouped.
> > > > >   - The batch coordinator should be optimized for fast response to
> > these
> > > > > RPCs.
> > > > >   - The fact that a single producer normally will communicate with a
> > > > > single broker for the duration of the transaction further reduces the
> > > > > expected number of round trips.
> > > > > 2. An extra round trip between the transaction coordinator and batch
> > > > > coordinator when a transaction is finished.
> > > > >
> > > > > With this proposal, transactions will also be able to span both
> > classic
> > > > > and Diskless topics.
> > > > >
> > > > > *Queues*
> > > > >
> > > > > The share group coordination and management is a side job that
> > doesn't
> > > > > interfere with the topic itself (leadership, replicas, physical
> > storage
> > > > of
> > > > > records, etc.) and non-queue producers and consumers (Fetch and
> > Produce
> > > > > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > > > > reason why we can't make Diskless topics compatible with share
> > groups the
> > > > > same way as classic topics are. Even on the code level, we don't
> > expect
> > > > any
> > > > > serious refactoring: the same reading routines are used that are
> > used for
> > > > > fetching (e.g. ReplicaManager.readFromLog).
> > > > >
> > > > >
> > > > > Should the KIPs be modified to include this or it's too
> > > > > implementation-focused?
> > > > >
> > > > > Best regards,
> > > > > Ivan
> > > > >
> > > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > Thank you all for your questions and design input on KIP-1150.
> > > > > >
> > > > > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > > > > summarize
> > > > > > the changes:
> > > > > >
> > > > > > 1. The design prioritizes integrating with the existing KIP-405
> > Tiered
> > > > > > Storage interfaces, permitting data produced to a Diskless topic
> > to be
> > > > > > moved to tiered storage.
> > > > > > This lowers the scalability requirements for the Batch Coordinator
> > > > > > component, and allows Diskless to compose with Tiered Storage
> > plugin
> > > > > > features such as encryption and alternative data formats.
> > > > > >
> > > > > > 2. Consumer fetches are now served from local segments, making use
> > of
> > > > the
> > > > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > > > already
> > > > > > built into classic topics.
> > > > > > However, local segments are now considered cache elements, do not
> > need
> > > > to
> > > > > > be durably stored, and can be built without contacting any other
> > > > > replicas.
> > > > > >
> > > > > > 3. The design has been simplified substantially, by removing the
> > > > previous
> > > > > > Diskless consume flow, distributed cache component, and "object
> > > > > > compaction/merging" step.
> > > > > >
> > > > > > The design maintains leaderless produces as enabled by the Batch
> > > > > > Coordinator, and the same latency profiles as the earlier design,
> > while
> > > > > > being simpler and integrating better into the existing ecosystem.
> > > > > >
> > > > > > Thanks, and we are eager to hear your feedback on the new design.
> > > > > > Greg Harris
> > > > > >
> > > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Jan,
> > > > > > >
> > > > > > > For me, the main gap of KIP-1150 is the support of all existing
> > > > client
> > > > > > > APIs. Currently, there is no design for supporting APIs like
> > > > > transactions
> > > > > > > and queues.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > > >
> > > > > > > > Would it be a good time to ask for the current status of this
> > KIP?
> > > > I
> > > > > > > > haven't seen much activity here for the past 2 months, the
> > vote got
> > > > > > > vetoed
> > > > > > > > but I think the pending questions have been answered since
> > then.
> > > > > KIP-1183
> > > > > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > > > > >
> > > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can
> > be
> > > > > > > > made, with a coordinator-based approach being by far the
> > dominant
> > > > one
> > > > > > > when
> > > > > > > > it comes to market adoption - but all these are standalone
> > > > products.
> > > > > > > >
> > > > > > > > I'm a big fan of both approaches, but would hate to see a
> > stall. So
> > > > > the
> > > > > > > > question is: can we get an update?
> > > > > > > >
> > > > > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > > > > questions
> > > > > > > > been answered? If not, is there anything I can do to help? I'm
> > > > deeply
> > > > > > > > familiar with both architectures and have written about both?
> > > > > > > >
> > > > > > > > Kind regards,
> > > > > > > > Jan
> > > > > > > >
> > > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > > >
> > > > > > > > > I have some nits - it may be useful to
> > > > > > > > >
> > > > > > > > > a) group all the KIP email threads in the main one (just a
> > bunch
> > > > of
> > > > > > > links
> > > > > > > > > to everything)
> > > > > > > > > b) create the email threads
> > > > > > > > >
> > > > > > > > > It's a bit hard to track it all - for example, I was
> > searching
> > > > for
> > > > > a
> > > > > > > > > discuss thread for KIP-1165 for a while; As far as I can
> > tell, it
> > > > > > > doesn't
> > > > > > > > > exist yet.
> > > > > > > > >
> > > > > > > > > Since the KIPs are published (by virtue of having the root
> > KIP be
> > > > > > > > > published, having a DISCUSS thread and links to sub-KIPs
> > where
> > > > were
> > > > > > > aimed
> > > > > > > > > to move the discussion towards), I think it would be good to
> > > > create
> > > > > > > > DISCUSS
> > > > > > > > > threads for them all.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stan
> > > > > > > > >
> > > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > > Hi Kafka Devs!
> > > > > > > > > >
> > > > > > > > > > We want to start a new KIP discussion about introducing a
> > new
> > > > > type of
> > > > > > > > > > topics that would make use of Object Storage as the primary
> > > > > source of
> > > > > > > > > > storage. However, as this KIP is big we decided to split it
> > > > into
> > > > > > > > multiple
> > > > > > > > > > related KIPs.
> > > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > > )
> > > > > > > > > > that aims to discuss if Apache Kafka should aim to have
> > this
> > > > > type of
> > > > > > > > > > feature at all. This KIP doesn't go onto details on how to
> > > > > implement
> > > > > > > > it.
> > > > > > > > > > This follows the same approach used when we discussed
> > KRaft.
> > > > > > > > > >
> > > > > > > > > > But as we know that it is sometimes really hard to discuss
> > on
> > > > > that
> > > > > > > meta
> > > > > > > > > > level, we also created several sub-kips (linked in
> > KIP-1150)
> > > > that
> > > > > > > offer
> > > > > > > > > an
> > > > > > > > > > implementation of this feature.
> > > > > > > > > >
> > > > > > > > > > We kindly ask you to use the proper DISCUSS threads for
> > each
> > > > > type of
> > > > > > > > > > concern and keep this one to discuss whether Apache Kafka
> > wants
> > > > > to
> > > > > > > have
> > > > > > > > > > this feature or not.
> > > > > > > > > >
> > > > > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > > > > >
> > > > > > > > > > ------------------
> > > > > > > > > > Josep Prat
> > > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Justine Olshan <jo...@confluent.io.INVALID>]

Hey Ivan,

Thanks for the response. I think most of what you said made sense, but I
did have some questions about this part:

> As we understand this, the partition leader in classic topics forgets
about a transaction once it’s replicated (HWM overpasses it). The
transaction coordinator acts like the main guardian, allowing partition
leaders to do this safely. Please correct me if this is wrong. We think
about relying on this with the batch coordinator and delete the information
about a transaction once it’s finished (as there’s no replication and HWM
advances immediately).

I didn't quite understand this. In classic topics, we have maps for ongoing
transactions which remove state when the transaction is completed and an
aborted transactions index which is retained for much longer. Once the
transaction is completed, the coordinator is no longer involved in
maintaining this partition side state, and it is subject to compaction etc.
Looking back at the outline provided above, I didn't see much about the
fetch path, so maybe that could be expanded a bit further. I saw the
following in a response:
> When the broker constructs a fully valid local segment, all the necessary
control batches will be inserted and indices, including the transaction
index will be built to serve FetchRequests exactly as they are today.

Based on this, it seems like we need to retain the information about
aborted txns for longer.

Thanks,
Justine

On Mon, Sep 15, 2025 at 9:43 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi Justine and all,
>
> Thank you for your questions!
>
> > JO 1. >Since a transaction could be uniquely identified with producer ID
> > and epoch, the positive result of this check could be cached locally
> > Are we saying that only new transaction version 2 transactions can be
> used
> > here? If not, we can't uniquely identify transactions with producer id +
> > epoch
>
> You’re right that we (probably unintentionally) focused only on version 2.
> We can either limit the support to version 2 or consider using some
> surrogates to support version 1.
>
> > JO 2. >The batch coordinator does the final transactional checks of the
> > batches. This procedure would output the same errors like the partition
> > leader in classic topics would do.
> > Can you expand on what these checks are? Would you be checking if the
> > transaction was still ongoing for example?* *
>
> Yes, the producer epoch, that the transaction is ongoing, and of course
> the normal idempotence checks. What the partition leader in the classic
> topics does before appending a batch to the local log (e.g. in
> UnifiedLog.maybeStartTransactionVerification and
> UnifiedLog.analyzeAndValidateProducerState). In Diskless, we unfortunately
> cannot do these checks before appending the data to the WAL segment and
> uploading it, but we can “tombstone” these batches in the batch coordinator
> during the final commit.
>
> > Is there state about ongoing
> > transactions in the batch coordinator? I see some other state mentioned
> in
> > the End transaction section, but it's not super clear what state is
> stored
> > and when it is stored.
>
> Right, this should have been more explicit. As the partition leader tracks
> ongoing transactions for classic topics, the batch coordinator has to as
> well. So when a transaction starts and ends, the transaction coordinator
> must inform the batch coordinator about this.
>
> > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> be
> > stored in the batch coordinator?
>
> Yes. This could be deduced from the committed batches and other
> information, but for the sake of performance we’d better store it
> explicitly.
>
> > JO 4. Are there any thoughts about how long transactional state is
> > maintained in the batch coordinator and how it will be cleaned up?
>
> As we understand this, the partition leader in classic topics forgets
> about a transaction once it’s replicated (HWM overpasses it). The
> transaction coordinator acts like the main guardian, allowing partition
> leaders to do this safely. Please correct me if this is wrong. We think
> about relying on this with the batch coordinator and delete the information
> about a transaction once it’s finished (as there’s no replication and HWM
> advances immediately).
>
> Best,
> Ivan
>
> On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> > Hey folks,
> >
> > Excited to see some updates related to transactions!
> >
> > I had a few questions.
> >
> > JO 1. >Since a transaction could be uniquely identified with producer ID
> > and epoch, the positive result of this check could be cached locally
> > Are we saying that only new transaction version 2 transactions can be
> used
> > here? If not, we can't uniquely identify transactions with producer id +
> > epoch
> >
> > JO 2. >The batch coordinator does the final transactional checks of the
> > batches. This procedure would output the same errors like the partition
> > leader in classic topics would do.
> > Can you expand on what these checks are? Would you be checking if the
> > transaction was still ongoing for example? Is there state about ongoing
> > transactions in the batch coordinator? I see some other state mentioned
> in
> > the End transaction section, but it's not super clear what state is
> stored
> > and when it is stored.
> >
> > JO 3. I didn't see anything about maintaining LSO -- perhaps that would
> be
> > stored in the batch coordinator?
> >
> > JO 4. Are there any thoughts about how long transactional state is
> > maintained in the batch coordinator and how it will be cleaned up?
> >
> > On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Greg and Ivan,
> > >
> > > Thanks for the update. A few comments.
> > >
> > > JR 10. "Consumer fetches are now served from local segments, making
> use of
> > > the
> > > indexes, page cache, request purgatory, and zero-copy functionality
> already
> > > built into classic topics."
> > > JR 10.1 Does the broker build the producer state for each partition in
> > > diskless topics?
> > > JR 10.2 For transactional data, the consumer fetches need to know
> aborted
> > > records. How is that achieved?
> > >
> > > JR 11. "The batch coordinator saves that the transaction is finished
> and
> > > also inserts the control batches in the corresponding logs of the
> involved
> > > Diskless topics. This happens only on the metadata level, no actual
> control
> > > batches are written to any file. "
> > > A fetch response could include multiple transactional batches. How
> does the
> > > broker obtain the information about the ending control batch for each
> > > batch? Does that mean that a fetch response needs to be built by
> > > stitching record batches and generated control batches together?
> > >
> > > JR 12. Queues: Is there still a share partition leader that all
> consumers
> > > are routed to?
> > >
> > > JR 13. "Should the KIPs be modified to include this or it's too
> > > implementation-focused?" It would be useful to include enough details
> to
> > > understand correctness and performance impact.
> > >
> > > HC5. Henry has a valid point. Requests from a given producer contain a
> > > sequence number, which is ordered. If a producer sends every Produce
> > > request to an arbitrary broker, those requests could reach the batch
> > > coordinator in different order and lead to rejection of the produce
> > > requests.
> > >
> > > Jun
> > >
> > > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We have also thought in a bit more details about transactions and
> queues,
> > > > here's the plan.
> > > >
> > > > *Transactions*
> > > >
> > > > The support for transactions in *classic topics* is based on precise
> > > > interactions between three actors: clients (mostly producers, but
> also
> > > > consumers), brokers (ReplicaManager and other classes), and
> transaction
> > > > coordinators. Brokers also run partition leaders with their local
> state
> > > > (ProducerStateManager and others).
> > > >
> > > > The high level (some details skipped) workflow is the following.
> When a
> > > > transactional Produce request is received by the broker:
> > > > 1. For each partition, the partition leader checks if a non-empty
> > > > transaction is running for this partition. This is done using its
> local
> > > > state derived from the log metadata (ProducerStateManager,
> > > > VerificationStateEntry, VerificationGuard).
> > > > 2. The transaction coordinator is informed about all the partitions
> that
> > > > aren’t part of the transaction to include them.
> > > > 3. The partition leaders do additional transactional checks.
> > > > 4. The partition leaders append the transactional data to their logs
> and
> > > > update some of their state (for example, log the fact that the
> > > transaction
> > > > is running for the partition and its first offset).
> > > >
> > > > When the transaction is committed or aborted:
> > > > 1. The producer contacts the transaction coordinator directly with
> > > > EndTxnRequest.
> > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> PREPARE_ABORT to
> > > > its log and responds to the producer.
> > > > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > leaders
> > > > of the involved partitions.
> > > > 4. The partition leaders write the transaction markers to their logs
> and
> > > > respond to the coordinator.
> > > > 5. The coordinator writes the final transaction state
> COMPLETE_COMMIT or
> > > > COMPLETE_ABORT.
> > > >
> > > > In classic topics, partitions have leaders and lots of important
> state
> > > > necessary for supporting this workflow is local. The main challenge
> in
> > > > mapping this to Diskless comes from the fact there are no partition
> > > > leaders, so the corresponding pieces of state need to be globalized
> in
> > > the
> > > > batch coordinator. We are already doing this to support idempotent
> > > produce.
> > > >
> > > > The high level workflow for *diskless topics* would look very
> similar:
> > > > 1. For each partition, the broker checks if a non-empty transaction
> is
> > > > running for this partition. In contrast to classic topics, this is
> > > checked
> > > > against the batch coordinator with a single RPC. Since a transaction
> > > could
> > > > be uniquely identified with producer ID and epoch, the positive
> result of
> > > > this check could be cached locally (for the double configured
> duration
> > > of a
> > > > transaction, for example).
> > > > 2. The same: The transaction coordinator is informed about all the
> > > > partitions that aren’t part of the transaction to include them.
> > > > 3. No transactional checks are done on the broker side.
> > > > 4. The broker appends the transactional data to the current shared
> WAL
> > > > segment. It doesn’t update any transaction-related state for Diskless
> > > > topics, because it doesn’t have any.
> > > > 5. The WAL segment is committed to the batch coordinator like in the
> > > > normal produce flow.
> > > > 6. The batch coordinator does the final transactional checks of the
> > > > batches. This procedure would output the same errors like the
> partition
> > > > leader in classic topics would do. I.e. some batches could be
> rejected.
> > > > This means, there will potentially be garbage in the WAL segment
> file in
> > > > case of transactional errors. This is preferable to doing more
> network
> > > > round trips, especially considering the WAL segments will be
> relatively
> > > > short-living (see the Greg's update above).
> > > >
> > > > When the transaction is committed or aborted:
> > > > 1. The producer contacts the transaction coordinator directly with
> > > > EndTxnRequest.
> > > > 2. The transaction coordinator writes PREPARE_COMMIT or
> PREPARE_ABORT to
> > > > its log and responds to the producer.
> > > > 3. *[NEW]* The transaction coordinator informs the batch coordinator
> that
> > > > the transaction is finished.
> > > > 4. *[NEW]* The batch coordinator saves that the transaction is
> finished
> > > > and also inserts the control batches in the corresponding logs of the
> > > > involved Diskless topics. This happens only on the metadata level, no
> > > > actual control batches are written to any file. They will be
> dynamically
> > > > created on Fetch and other read operations. We could technically
> write
> > > > these control batches for real, but this would mean extra produce
> > > latency,
> > > > so it's better just to mark them in the batch coordinator and save
> these
> > > > milliseconds.
> > > > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
> > > leaders
> > > > of the involved partitions. – Now only to classic topics now.
> > > > 6. The partition leaders of classic topics write the transaction
> markers
> > > > to their logs and respond to the coordinator.
> > > > 7. The coordinator writes the final transaction state
> COMPLETE_COMMIT or
> > > > COMPLETE_ABORT.
> > > >
> > > > Compared to the non-transactional produce flow, we get:
> > > > 1. An extra network round trip between brokers and the batch
> coordinator
> > > > when a new partition appear in the transaction. To mitigate the
> impact of
> > > > them:
> > > >   - The results will be cached.
> > > >   - The calls for multiple partitions in one Produce request will be
> > > > grouped.
> > > >   - The batch coordinator should be optimized for fast response to
> these
> > > > RPCs.
> > > >   - The fact that a single producer normally will communicate with a
> > > > single broker for the duration of the transaction further reduces the
> > > > expected number of round trips.
> > > > 2. An extra round trip between the transaction coordinator and batch
> > > > coordinator when a transaction is finished.
> > > >
> > > > With this proposal, transactions will also be able to span both
> classic
> > > > and Diskless topics.
> > > >
> > > > *Queues*
> > > >
> > > > The share group coordination and management is a side job that
> doesn't
> > > > interfere with the topic itself (leadership, replicas, physical
> storage
> > > of
> > > > records, etc.) and non-queue producers and consumers (Fetch and
> Produce
> > > > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > > > reason why we can't make Diskless topics compatible with share
> groups the
> > > > same way as classic topics are. Even on the code level, we don't
> expect
> > > any
> > > > serious refactoring: the same reading routines are used that are
> used for
> > > > fetching (e.g. ReplicaManager.readFromLog).
> > > >
> > > >
> > > > Should the KIPs be modified to include this or it's too
> > > > implementation-focused?
> > > >
> > > > Best regards,
> > > > Ivan
> > > >
> > > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > > Hi all,
> > > > >
> > > > > Thank you all for your questions and design input on KIP-1150.
> > > > >
> > > > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > > > summarize
> > > > > the changes:
> > > > >
> > > > > 1. The design prioritizes integrating with the existing KIP-405
> Tiered
> > > > > Storage interfaces, permitting data produced to a Diskless topic
> to be
> > > > > moved to tiered storage.
> > > > > This lowers the scalability requirements for the Batch Coordinator
> > > > > component, and allows Diskless to compose with Tiered Storage
> plugin
> > > > > features such as encryption and alternative data formats.
> > > > >
> > > > > 2. Consumer fetches are now served from local segments, making use
> of
> > > the
> > > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > > already
> > > > > built into classic topics.
> > > > > However, local segments are now considered cache elements, do not
> need
> > > to
> > > > > be durably stored, and can be built without contacting any other
> > > > replicas.
> > > > >
> > > > > 3. The design has been simplified substantially, by removing the
> > > previous
> > > > > Diskless consume flow, distributed cache component, and "object
> > > > > compaction/merging" step.
> > > > >
> > > > > The design maintains leaderless produces as enabled by the Batch
> > > > > Coordinator, and the same latency profiles as the earlier design,
> while
> > > > > being simpler and integrating better into the existing ecosystem.
> > > > >
> > > > > Thanks, and we are eager to hear your feedback on the new design.
> > > > > Greg Harris
> > > > >
> > > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi, Jan,
> > > > > >
> > > > > > For me, the main gap of KIP-1150 is the support of all existing
> > > client
> > > > > > APIs. Currently, there is no design for supporting APIs like
> > > > transactions
> > > > > > and queues.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > > <ja...@kentra.io.invalid> wrote:
> > > > > >
> > > > > > > Would it be a good time to ask for the current status of this
> KIP?
> > > I
> > > > > > > haven't seen much activity here for the past 2 months, the
> vote got
> > > > > > vetoed
> > > > > > > but I think the pending questions have been answered since
> then.
> > > > KIP-1183
> > > > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > > > >
> > > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can
> be
> > > > > > > made, with a coordinator-based approach being by far the
> dominant
> > > one
> > > > > > when
> > > > > > > it comes to market adoption - but all these are standalone
> > > products.
> > > > > > >
> > > > > > > I'm a big fan of both approaches, but would hate to see a
> stall. So
> > > > the
> > > > > > > question is: can we get an update?
> > > > > > >
> > > > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > > > questions
> > > > > > > been answered? If not, is there anything I can do to help? I'm
> > > deeply
> > > > > > > familiar with both architectures and have written about both?
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > Jan
> > > > > > >
> > > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > > >
> > > > > > > > I have some nits - it may be useful to
> > > > > > > >
> > > > > > > > a) group all the KIP email threads in the main one (just a
> bunch
> > > of
> > > > > > links
> > > > > > > > to everything)
> > > > > > > > b) create the email threads
> > > > > > > >
> > > > > > > > It's a bit hard to track it all - for example, I was
> searching
> > > for
> > > > a
> > > > > > > > discuss thread for KIP-1165 for a while; As far as I can
> tell, it
> > > > > > doesn't
> > > > > > > > exist yet.
> > > > > > > >
> > > > > > > > Since the KIPs are published (by virtue of having the root
> KIP be
> > > > > > > > published, having a DISCUSS thread and links to sub-KIPs
> where
> > > were
> > > > > > aimed
> > > > > > > > to move the discussion towards), I think it would be good to
> > > create
> > > > > > > DISCUSS
> > > > > > > > threads for them all.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stan
> > > > > > > >
> > > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > > Hi Kafka Devs!
> > > > > > > > >
> > > > > > > > > We want to start a new KIP discussion about introducing a
> new
> > > > type of
> > > > > > > > > topics that would make use of Object Storage as the primary
> > > > source of
> > > > > > > > > storage. However, as this KIP is big we decided to split it
> > > into
> > > > > > > multiple
> > > > > > > > > related KIPs.
> > > > > > > > > We have the motivational KIP-1150 (
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > > )
> > > > > > > > > that aims to discuss if Apache Kafka should aim to have
> this
> > > > type of
> > > > > > > > > feature at all. This KIP doesn't go onto details on how to
> > > > implement
> > > > > > > it.
> > > > > > > > > This follows the same approach used when we discussed
> KRaft.
> > > > > > > > >
> > > > > > > > > But as we know that it is sometimes really hard to discuss
> on
> > > > that
> > > > > > meta
> > > > > > > > > level, we also created several sub-kips (linked in
> KIP-1150)
> > > that
> > > > > > offer
> > > > > > > > an
> > > > > > > > > implementation of this feature.
> > > > > > > > >
> > > > > > > > > We kindly ask you to use the proper DISCUSS threads for
> each
> > > > type of
> > > > > > > > > concern and keep this one to discuss whether Apache Kafka
> wants
> > > > to
> > > > > > have
> > > > > > > > > this feature or not.
> > > > > > > > >
> > > > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > > > >
> > > > > > > > > ------------------
> > > > > > > > > Josep Prat
> > > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi Justine and all,

Thank you for your questions!

> JO 1. >Since a transaction could be uniquely identified with producer ID
> and epoch, the positive result of this check could be cached locally
> Are we saying that only new transaction version 2 transactions can be used
> here? If not, we can't uniquely identify transactions with producer id +
> epoch

You’re right that we (probably unintentionally) focused only on version 2. We can either limit the support to version 2 or consider using some surrogates to support version 1.

> JO 2. >The batch coordinator does the final transactional checks of the
> batches. This procedure would output the same errors like the partition
> leader in classic topics would do.
> Can you expand on what these checks are? Would you be checking if the
> transaction was still ongoing for example?* *

Yes, the producer epoch, that the transaction is ongoing, and of course the normal idempotence checks. What the partition leader in the classic topics does before appending a batch to the local log (e.g. in UnifiedLog.maybeStartTransactionVerification and  UnifiedLog.analyzeAndValidateProducerState). In Diskless, we unfortunately cannot do these checks before appending the data to the WAL segment and uploading it, but we can “tombstone” these batches in the batch coordinator during the final commit.
 
> Is there state about ongoing
> transactions in the batch coordinator? I see some other state mentioned in
> the End transaction section, but it's not super clear what state is stored
> and when it is stored.

Right, this should have been more explicit. As the partition leader tracks ongoing transactions for classic topics, the batch coordinator has to as well. So when a transaction starts and ends, the transaction coordinator must inform the batch coordinator about this.

> JO 3. I didn't see anything about maintaining LSO -- perhaps that would be
> stored in the batch coordinator?

Yes. This could be deduced from the committed batches and other information, but for the sake of performance we’d better store it explicitly.

> JO 4. Are there any thoughts about how long transactional state is
> maintained in the batch coordinator and how it will be cleaned up?

As we understand this, the partition leader in classic topics forgets about a transaction once it’s replicated (HWM overpasses it). The transaction coordinator acts like the main guardian, allowing partition leaders to do this safely. Please correct me if this is wrong. We think about relying on this with the batch coordinator and delete the information about a transaction once it’s finished (as there’s no replication and HWM advances immediately).

Best,
Ivan

On Tue, Sep 9, 2025, at 00:38, Justine Olshan wrote:
> Hey folks,
> 
> Excited to see some updates related to transactions!
> 
> I had a few questions.
> 
> JO 1. >Since a transaction could be uniquely identified with producer ID
> and epoch, the positive result of this check could be cached locally
> Are we saying that only new transaction version 2 transactions can be used
> here? If not, we can't uniquely identify transactions with producer id +
> epoch
> 
> JO 2. >The batch coordinator does the final transactional checks of the
> batches. This procedure would output the same errors like the partition
> leader in classic topics would do.
> Can you expand on what these checks are? Would you be checking if the
> transaction was still ongoing for example? Is there state about ongoing
> transactions in the batch coordinator? I see some other state mentioned in
> the End transaction section, but it's not super clear what state is stored
> and when it is stored.
> 
> JO 3. I didn't see anything about maintaining LSO -- perhaps that would be
> stored in the batch coordinator?
> 
> JO 4. Are there any thoughts about how long transactional state is
> maintained in the batch coordinator and how it will be cleaned up?
> 
> On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid> wrote:
> 
> > Hi, Greg and Ivan,
> >
> > Thanks for the update. A few comments.
> >
> > JR 10. "Consumer fetches are now served from local segments, making use of
> > the
> > indexes, page cache, request purgatory, and zero-copy functionality already
> > built into classic topics."
> > JR 10.1 Does the broker build the producer state for each partition in
> > diskless topics?
> > JR 10.2 For transactional data, the consumer fetches need to know aborted
> > records. How is that achieved?
> >
> > JR 11. "The batch coordinator saves that the transaction is finished and
> > also inserts the control batches in the corresponding logs of the involved
> > Diskless topics. This happens only on the metadata level, no actual control
> > batches are written to any file. "
> > A fetch response could include multiple transactional batches. How does the
> > broker obtain the information about the ending control batch for each
> > batch? Does that mean that a fetch response needs to be built by
> > stitching record batches and generated control batches together?
> >
> > JR 12. Queues: Is there still a share partition leader that all consumers
> > are routed to?
> >
> > JR 13. "Should the KIPs be modified to include this or it's too
> > implementation-focused?" It would be useful to include enough details to
> > understand correctness and performance impact.
> >
> > HC5. Henry has a valid point. Requests from a given producer contain a
> > sequence number, which is ordered. If a producer sends every Produce
> > request to an arbitrary broker, those requests could reach the batch
> > coordinator in different order and lead to rejection of the produce
> > requests.
> >
> > Jun
> >
> > On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
> >
> > > Hi all,
> > >
> > > We have also thought in a bit more details about transactions and queues,
> > > here's the plan.
> > >
> > > *Transactions*
> > >
> > > The support for transactions in *classic topics* is based on precise
> > > interactions between three actors: clients (mostly producers, but also
> > > consumers), brokers (ReplicaManager and other classes), and transaction
> > > coordinators. Brokers also run partition leaders with their local state
> > > (ProducerStateManager and others).
> > >
> > > The high level (some details skipped) workflow is the following. When a
> > > transactional Produce request is received by the broker:
> > > 1. For each partition, the partition leader checks if a non-empty
> > > transaction is running for this partition. This is done using its local
> > > state derived from the log metadata (ProducerStateManager,
> > > VerificationStateEntry, VerificationGuard).
> > > 2. The transaction coordinator is informed about all the partitions that
> > > aren’t part of the transaction to include them.
> > > 3. The partition leaders do additional transactional checks.
> > > 4. The partition leaders append the transactional data to their logs and
> > > update some of their state (for example, log the fact that the
> > transaction
> > > is running for the partition and its first offset).
> > >
> > > When the transaction is committed or aborted:
> > > 1. The producer contacts the transaction coordinator directly with
> > > EndTxnRequest.
> > > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> > > its log and responds to the producer.
> > > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
> > leaders
> > > of the involved partitions.
> > > 4. The partition leaders write the transaction markers to their logs and
> > > respond to the coordinator.
> > > 5. The coordinator writes the final transaction state COMPLETE_COMMIT or
> > > COMPLETE_ABORT.
> > >
> > > In classic topics, partitions have leaders and lots of important state
> > > necessary for supporting this workflow is local. The main challenge in
> > > mapping this to Diskless comes from the fact there are no partition
> > > leaders, so the corresponding pieces of state need to be globalized in
> > the
> > > batch coordinator. We are already doing this to support idempotent
> > produce.
> > >
> > > The high level workflow for *diskless topics* would look very similar:
> > > 1. For each partition, the broker checks if a non-empty transaction is
> > > running for this partition. In contrast to classic topics, this is
> > checked
> > > against the batch coordinator with a single RPC. Since a transaction
> > could
> > > be uniquely identified with producer ID and epoch, the positive result of
> > > this check could be cached locally (for the double configured duration
> > of a
> > > transaction, for example).
> > > 2. The same: The transaction coordinator is informed about all the
> > > partitions that aren’t part of the transaction to include them.
> > > 3. No transactional checks are done on the broker side.
> > > 4. The broker appends the transactional data to the current shared WAL
> > > segment. It doesn’t update any transaction-related state for Diskless
> > > topics, because it doesn’t have any.
> > > 5. The WAL segment is committed to the batch coordinator like in the
> > > normal produce flow.
> > > 6. The batch coordinator does the final transactional checks of the
> > > batches. This procedure would output the same errors like the partition
> > > leader in classic topics would do. I.e. some batches could be rejected.
> > > This means, there will potentially be garbage in the WAL segment file in
> > > case of transactional errors. This is preferable to doing more network
> > > round trips, especially considering the WAL segments will be relatively
> > > short-living (see the Greg's update above).
> > >
> > > When the transaction is committed or aborted:
> > > 1. The producer contacts the transaction coordinator directly with
> > > EndTxnRequest.
> > > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> > > its log and responds to the producer.
> > > 3. *[NEW]* The transaction coordinator informs the batch coordinator that
> > > the transaction is finished.
> > > 4. *[NEW]* The batch coordinator saves that the transaction is finished
> > > and also inserts the control batches in the corresponding logs of the
> > > involved Diskless topics. This happens only on the metadata level, no
> > > actual control batches are written to any file. They will be dynamically
> > > created on Fetch and other read operations. We could technically write
> > > these control batches for real, but this would mean extra produce
> > latency,
> > > so it's better just to mark them in the batch coordinator and save these
> > > milliseconds.
> > > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
> > leaders
> > > of the involved partitions. – Now only to classic topics now.
> > > 6. The partition leaders of classic topics write the transaction markers
> > > to their logs and respond to the coordinator.
> > > 7. The coordinator writes the final transaction state COMPLETE_COMMIT or
> > > COMPLETE_ABORT.
> > >
> > > Compared to the non-transactional produce flow, we get:
> > > 1. An extra network round trip between brokers and the batch coordinator
> > > when a new partition appear in the transaction. To mitigate the impact of
> > > them:
> > >   - The results will be cached.
> > >   - The calls for multiple partitions in one Produce request will be
> > > grouped.
> > >   - The batch coordinator should be optimized for fast response to these
> > > RPCs.
> > >   - The fact that a single producer normally will communicate with a
> > > single broker for the duration of the transaction further reduces the
> > > expected number of round trips.
> > > 2. An extra round trip between the transaction coordinator and batch
> > > coordinator when a transaction is finished.
> > >
> > > With this proposal, transactions will also be able to span both classic
> > > and Diskless topics.
> > >
> > > *Queues*
> > >
> > > The share group coordination and management is a side job that doesn't
> > > interfere with the topic itself (leadership, replicas, physical storage
> > of
> > > records, etc.) and non-queue producers and consumers (Fetch and Produce
> > > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > > reason why we can't make Diskless topics compatible with share groups the
> > > same way as classic topics are. Even on the code level, we don't expect
> > any
> > > serious refactoring: the same reading routines are used that are used for
> > > fetching (e.g. ReplicaManager.readFromLog).
> > >
> > >
> > > Should the KIPs be modified to include this or it's too
> > > implementation-focused?
> > >
> > > Best regards,
> > > Ivan
> > >
> > > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > > Hi all,
> > > >
> > > > Thank you all for your questions and design input on KIP-1150.
> > > >
> > > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > > summarize
> > > > the changes:
> > > >
> > > > 1. The design prioritizes integrating with the existing KIP-405 Tiered
> > > > Storage interfaces, permitting data produced to a Diskless topic to be
> > > > moved to tiered storage.
> > > > This lowers the scalability requirements for the Batch Coordinator
> > > > component, and allows Diskless to compose with Tiered Storage plugin
> > > > features such as encryption and alternative data formats.
> > > >
> > > > 2. Consumer fetches are now served from local segments, making use of
> > the
> > > > indexes, page cache, request purgatory, and zero-copy functionality
> > > already
> > > > built into classic topics.
> > > > However, local segments are now considered cache elements, do not need
> > to
> > > > be durably stored, and can be built without contacting any other
> > > replicas.
> > > >
> > > > 3. The design has been simplified substantially, by removing the
> > previous
> > > > Diskless consume flow, distributed cache component, and "object
> > > > compaction/merging" step.
> > > >
> > > > The design maintains leaderless produces as enabled by the Batch
> > > > Coordinator, and the same latency profiles as the earlier design, while
> > > > being simpler and integrating better into the existing ecosystem.
> > > >
> > > > Thanks, and we are eager to hear your feedback on the new design.
> > > > Greg Harris
> > > >
> > > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Hi, Jan,
> > > > >
> > > > > For me, the main gap of KIP-1150 is the support of all existing
> > client
> > > > > APIs. Currently, there is no design for supporting APIs like
> > > transactions
> > > > > and queues.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > > <ja...@kentra.io.invalid> wrote:
> > > > >
> > > > > > Would it be a good time to ask for the current status of this KIP?
> > I
> > > > > > haven't seen much activity here for the past 2 months, the vote got
> > > > > vetoed
> > > > > > but I think the pending questions have been answered since then.
> > > KIP-1183
> > > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > > >
> > > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> > > > > > made, with a coordinator-based approach being by far the dominant
> > one
> > > > > when
> > > > > > it comes to market adoption - but all these are standalone
> > products.
> > > > > >
> > > > > > I'm a big fan of both approaches, but would hate to see a stall. So
> > > the
> > > > > > question is: can we get an update?
> > > > > >
> > > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > > questions
> > > > > > been answered? If not, is there anything I can do to help? I'm
> > deeply
> > > > > > familiar with both architectures and have written about both?
> > > > > >
> > > > > > Kind regards,
> > > > > > Jan
> > > > > >
> > > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > > stanislavkozlovski@apache.org> wrote:
> > > > > >
> > > > > > > I have some nits - it may be useful to
> > > > > > >
> > > > > > > a) group all the KIP email threads in the main one (just a bunch
> > of
> > > > > links
> > > > > > > to everything)
> > > > > > > b) create the email threads
> > > > > > >
> > > > > > > It's a bit hard to track it all - for example, I was searching
> > for
> > > a
> > > > > > > discuss thread for KIP-1165 for a while; As far as I can tell, it
> > > > > doesn't
> > > > > > > exist yet.
> > > > > > >
> > > > > > > Since the KIPs are published (by virtue of having the root KIP be
> > > > > > > published, having a DISCUSS thread and links to sub-KIPs where
> > were
> > > > > aimed
> > > > > > > to move the discussion towards), I think it would be good to
> > create
> > > > > > DISCUSS
> > > > > > > threads for them all.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stan
> > > > > > >
> > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > > Hi Kafka Devs!
> > > > > > > >
> > > > > > > > We want to start a new KIP discussion about introducing a new
> > > type of
> > > > > > > > topics that would make use of Object Storage as the primary
> > > source of
> > > > > > > > storage. However, as this KIP is big we decided to split it
> > into
> > > > > > multiple
> > > > > > > > related KIPs.
> > > > > > > > We have the motivational KIP-1150 (
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > > )
> > > > > > > > that aims to discuss if Apache Kafka should aim to have this
> > > type of
> > > > > > > > feature at all. This KIP doesn't go onto details on how to
> > > implement
> > > > > > it.
> > > > > > > > This follows the same approach used when we discussed KRaft.
> > > > > > > >
> > > > > > > > But as we know that it is sometimes really hard to discuss on
> > > that
> > > > > meta
> > > > > > > > level, we also created several sub-kips (linked in KIP-1150)
> > that
> > > > > offer
> > > > > > > an
> > > > > > > > implementation of this feature.
> > > > > > > >
> > > > > > > > We kindly ask you to use the proper DISCUSS threads for each
> > > type of
> > > > > > > > concern and keep this one to discuss whether Apache Kafka wants
> > > to
> > > > > have
> > > > > > > > this feature or not.
> > > > > > > >
> > > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > > >
> > > > > > > > ------------------
> > > > > > > > Josep Prat
> > > > > > > > Open Source Engineering Director, Aiven
> > > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > > Aiven Deutschland GmbH
> > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Justine Olshan <jo...@confluent.io.INVALID>]

Hey folks,

Excited to see some updates related to transactions!

I had a few questions.

JO 1. >Since a transaction could be uniquely identified with producer ID
and epoch, the positive result of this check could be cached locally
Are we saying that only new transaction version 2 transactions can be used
here? If not, we can't uniquely identify transactions with producer id +
epoch

JO 2. >The batch coordinator does the final transactional checks of the
batches. This procedure would output the same errors like the partition
leader in classic topics would do.
Can you expand on what these checks are? Would you be checking if the
transaction was still ongoing for example? Is there state about ongoing
transactions in the batch coordinator? I see some other state mentioned in
the End transaction section, but it's not super clear what state is stored
and when it is stored.

JO 3. I didn't see anything about maintaining LSO -- perhaps that would be
stored in the batch coordinator?

JO 4. Are there any thoughts about how long transactional state is
maintained in the batch coordinator and how it will be cleaned up?

On Mon, Sep 8, 2025 at 10:38 AM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Greg and Ivan,
>
> Thanks for the update. A few comments.
>
> JR 10. "Consumer fetches are now served from local segments, making use of
> the
> indexes, page cache, request purgatory, and zero-copy functionality already
> built into classic topics."
> JR 10.1 Does the broker build the producer state for each partition in
> diskless topics?
> JR 10.2 For transactional data, the consumer fetches need to know aborted
> records. How is that achieved?
>
> JR 11. "The batch coordinator saves that the transaction is finished and
> also inserts the control batches in the corresponding logs of the involved
> Diskless topics. This happens only on the metadata level, no actual control
> batches are written to any file. "
> A fetch response could include multiple transactional batches. How does the
> broker obtain the information about the ending control batch for each
> batch? Does that mean that a fetch response needs to be built by
> stitching record batches and generated control batches together?
>
> JR 12. Queues: Is there still a share partition leader that all consumers
> are routed to?
>
> JR 13. "Should the KIPs be modified to include this or it's too
> implementation-focused?" It would be useful to include enough details to
> understand correctness and performance impact.
>
> HC5. Henry has a valid point. Requests from a given producer contain a
> sequence number, which is ordered. If a producer sends every Produce
> request to an arbitrary broker, those requests could reach the batch
> coordinator in different order and lead to rejection of the produce
> requests.
>
> Jun
>
> On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:
>
> > Hi all,
> >
> > We have also thought in a bit more details about transactions and queues,
> > here's the plan.
> >
> > *Transactions*
> >
> > The support for transactions in *classic topics* is based on precise
> > interactions between three actors: clients (mostly producers, but also
> > consumers), brokers (ReplicaManager and other classes), and transaction
> > coordinators. Brokers also run partition leaders with their local state
> > (ProducerStateManager and others).
> >
> > The high level (some details skipped) workflow is the following. When a
> > transactional Produce request is received by the broker:
> > 1. For each partition, the partition leader checks if a non-empty
> > transaction is running for this partition. This is done using its local
> > state derived from the log metadata (ProducerStateManager,
> > VerificationStateEntry, VerificationGuard).
> > 2. The transaction coordinator is informed about all the partitions that
> > aren’t part of the transaction to include them.
> > 3. The partition leaders do additional transactional checks.
> > 4. The partition leaders append the transactional data to their logs and
> > update some of their state (for example, log the fact that the
> transaction
> > is running for the partition and its first offset).
> >
> > When the transaction is committed or aborted:
> > 1. The producer contacts the transaction coordinator directly with
> > EndTxnRequest.
> > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> > its log and responds to the producer.
> > 3. The transaction coordinator sends WriteTxnMarkersRequest to the
> leaders
> > of the involved partitions.
> > 4. The partition leaders write the transaction markers to their logs and
> > respond to the coordinator.
> > 5. The coordinator writes the final transaction state COMPLETE_COMMIT or
> > COMPLETE_ABORT.
> >
> > In classic topics, partitions have leaders and lots of important state
> > necessary for supporting this workflow is local. The main challenge in
> > mapping this to Diskless comes from the fact there are no partition
> > leaders, so the corresponding pieces of state need to be globalized in
> the
> > batch coordinator. We are already doing this to support idempotent
> produce.
> >
> > The high level workflow for *diskless topics* would look very similar:
> > 1. For each partition, the broker checks if a non-empty transaction is
> > running for this partition. In contrast to classic topics, this is
> checked
> > against the batch coordinator with a single RPC. Since a transaction
> could
> > be uniquely identified with producer ID and epoch, the positive result of
> > this check could be cached locally (for the double configured duration
> of a
> > transaction, for example).
> > 2. The same: The transaction coordinator is informed about all the
> > partitions that aren’t part of the transaction to include them.
> > 3. No transactional checks are done on the broker side.
> > 4. The broker appends the transactional data to the current shared WAL
> > segment. It doesn’t update any transaction-related state for Diskless
> > topics, because it doesn’t have any.
> > 5. The WAL segment is committed to the batch coordinator like in the
> > normal produce flow.
> > 6. The batch coordinator does the final transactional checks of the
> > batches. This procedure would output the same errors like the partition
> > leader in classic topics would do. I.e. some batches could be rejected.
> > This means, there will potentially be garbage in the WAL segment file in
> > case of transactional errors. This is preferable to doing more network
> > round trips, especially considering the WAL segments will be relatively
> > short-living (see the Greg's update above).
> >
> > When the transaction is committed or aborted:
> > 1. The producer contacts the transaction coordinator directly with
> > EndTxnRequest.
> > 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> > its log and responds to the producer.
> > 3. *[NEW]* The transaction coordinator informs the batch coordinator that
> > the transaction is finished.
> > 4. *[NEW]* The batch coordinator saves that the transaction is finished
> > and also inserts the control batches in the corresponding logs of the
> > involved Diskless topics. This happens only on the metadata level, no
> > actual control batches are written to any file. They will be dynamically
> > created on Fetch and other read operations. We could technically write
> > these control batches for real, but this would mean extra produce
> latency,
> > so it's better just to mark them in the batch coordinator and save these
> > milliseconds.
> > 5. The transaction coordinator sends WriteTxnMarkersRequest to the
> leaders
> > of the involved partitions. – Now only to classic topics now.
> > 6. The partition leaders of classic topics write the transaction markers
> > to their logs and respond to the coordinator.
> > 7. The coordinator writes the final transaction state COMPLETE_COMMIT or
> > COMPLETE_ABORT.
> >
> > Compared to the non-transactional produce flow, we get:
> > 1. An extra network round trip between brokers and the batch coordinator
> > when a new partition appear in the transaction. To mitigate the impact of
> > them:
> >   - The results will be cached.
> >   - The calls for multiple partitions in one Produce request will be
> > grouped.
> >   - The batch coordinator should be optimized for fast response to these
> > RPCs.
> >   - The fact that a single producer normally will communicate with a
> > single broker for the duration of the transaction further reduces the
> > expected number of round trips.
> > 2. An extra round trip between the transaction coordinator and batch
> > coordinator when a transaction is finished.
> >
> > With this proposal, transactions will also be able to span both classic
> > and Diskless topics.
> >
> > *Queues*
> >
> > The share group coordination and management is a side job that doesn't
> > interfere with the topic itself (leadership, replicas, physical storage
> of
> > records, etc.) and non-queue producers and consumers (Fetch and Produce
> > RPCs, consumer group-related RPCs are not affected.) We don't see any
> > reason why we can't make Diskless topics compatible with share groups the
> > same way as classic topics are. Even on the code level, we don't expect
> any
> > serious refactoring: the same reading routines are used that are used for
> > fetching (e.g. ReplicaManager.readFromLog).
> >
> >
> > Should the KIPs be modified to include this or it's too
> > implementation-focused?
> >
> > Best regards,
> > Ivan
> >
> > On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > > Hi all,
> > >
> > > Thank you all for your questions and design input on KIP-1150.
> > >
> > > We have just updated KIP-1150 and KIP-1163 with a new design. To
> > summarize
> > > the changes:
> > >
> > > 1. The design prioritizes integrating with the existing KIP-405 Tiered
> > > Storage interfaces, permitting data produced to a Diskless topic to be
> > > moved to tiered storage.
> > > This lowers the scalability requirements for the Batch Coordinator
> > > component, and allows Diskless to compose with Tiered Storage plugin
> > > features such as encryption and alternative data formats.
> > >
> > > 2. Consumer fetches are now served from local segments, making use of
> the
> > > indexes, page cache, request purgatory, and zero-copy functionality
> > already
> > > built into classic topics.
> > > However, local segments are now considered cache elements, do not need
> to
> > > be durably stored, and can be built without contacting any other
> > replicas.
> > >
> > > 3. The design has been simplified substantially, by removing the
> previous
> > > Diskless consume flow, distributed cache component, and "object
> > > compaction/merging" step.
> > >
> > > The design maintains leaderless produces as enabled by the Batch
> > > Coordinator, and the same latency profiles as the earlier design, while
> > > being simpler and integrating better into the existing ecosystem.
> > >
> > > Thanks, and we are eager to hear your feedback on the new design.
> > > Greg Harris
> > >
> > > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Hi, Jan,
> > > >
> > > > For me, the main gap of KIP-1150 is the support of all existing
> client
> > > > APIs. Currently, there is no design for supporting APIs like
> > transactions
> > > > and queues.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > > <ja...@kentra.io.invalid> wrote:
> > > >
> > > > > Would it be a good time to ask for the current status of this KIP?
> I
> > > > > haven't seen much activity here for the past 2 months, the vote got
> > > > vetoed
> > > > > but I think the pending questions have been answered since then.
> > KIP-1183
> > > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > > >
> > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> > > > > made, with a coordinator-based approach being by far the dominant
> one
> > > > when
> > > > > it comes to market adoption - but all these are standalone
> products.
> > > > >
> > > > > I'm a big fan of both approaches, but would hate to see a stall. So
> > the
> > > > > question is: can we get an update?
> > > > >
> > > > > Maybe it's time to start another vote? Colin McCabe - have your
> > questions
> > > > > been answered? If not, is there anything I can do to help? I'm
> deeply
> > > > > familiar with both architectures and have written about both?
> > > > >
> > > > > Kind regards,
> > > > > Jan
> > > > >
> > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > > stanislavkozlovski@apache.org> wrote:
> > > > >
> > > > > > I have some nits - it may be useful to
> > > > > >
> > > > > > a) group all the KIP email threads in the main one (just a bunch
> of
> > > > links
> > > > > > to everything)
> > > > > > b) create the email threads
> > > > > >
> > > > > > It's a bit hard to track it all - for example, I was searching
> for
> > a
> > > > > > discuss thread for KIP-1165 for a while; As far as I can tell, it
> > > > doesn't
> > > > > > exist yet.
> > > > > >
> > > > > > Since the KIPs are published (by virtue of having the root KIP be
> > > > > > published, having a DISCUSS thread and links to sub-KIPs where
> were
> > > > aimed
> > > > > > to move the discussion towards), I think it would be good to
> create
> > > > > DISCUSS
> > > > > > threads for them all.
> > > > > >
> > > > > > Best,
> > > > > > Stan
> > > > > >
> > > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > > Hi Kafka Devs!
> > > > > > >
> > > > > > > We want to start a new KIP discussion about introducing a new
> > type of
> > > > > > > topics that would make use of Object Storage as the primary
> > source of
> > > > > > > storage. However, as this KIP is big we decided to split it
> into
> > > > > multiple
> > > > > > > related KIPs.
> > > > > > > We have the motivational KIP-1150 (
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > > )
> > > > > > > that aims to discuss if Apache Kafka should aim to have this
> > type of
> > > > > > > feature at all. This KIP doesn't go onto details on how to
> > implement
> > > > > it.
> > > > > > > This follows the same approach used when we discussed KRaft.
> > > > > > >
> > > > > > > But as we know that it is sometimes really hard to discuss on
> > that
> > > > meta
> > > > > > > level, we also created several sub-kips (linked in KIP-1150)
> that
> > > > offer
> > > > > > an
> > > > > > > implementation of this feature.
> > > > > > >
> > > > > > > We kindly ask you to use the proper DISCUSS threads for each
> > type of
> > > > > > > concern and keep this one to discuss whether Apache Kafka wants
> > to
> > > > have
> > > > > > > this feature or not.
> > > > > > >
> > > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > > >
> > > > > > > ------------------
> > > > > > > Josep Prat
> > > > > > > Open Source Engineering Director, Aiven
> > > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > > Aiven Deutschland GmbH
> > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > Anna Richardson, Kenneth Chen
> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Greg and Ivan,

Thanks for the update. A few comments.

JR 10. "Consumer fetches are now served from local segments, making use of
the
indexes, page cache, request purgatory, and zero-copy functionality already
built into classic topics."
JR 10.1 Does the broker build the producer state for each partition in
diskless topics?
JR 10.2 For transactional data, the consumer fetches need to know aborted
records. How is that achieved?

JR 11. "The batch coordinator saves that the transaction is finished and
also inserts the control batches in the corresponding logs of the involved
Diskless topics. This happens only on the metadata level, no actual control
batches are written to any file. "
A fetch response could include multiple transactional batches. How does the
broker obtain the information about the ending control batch for each
batch? Does that mean that a fetch response needs to be built by
stitching record batches and generated control batches together?

JR 12. Queues: Is there still a share partition leader that all consumers
are routed to?

JR 13. "Should the KIPs be modified to include this or it's too
implementation-focused?" It would be useful to include enough details to
understand correctness and performance impact.

HC5. Henry has a valid point. Requests from a given producer contain a
sequence number, which is ordered. If a producer sends every Produce
request to an arbitrary broker, those requests could reach the batch
coordinator in different order and lead to rejection of the produce
requests.

Jun

On Thu, Sep 4, 2025 at 12:00 AM Ivan Yurchenko <iv...@ivanyu.me> wrote:

> Hi all,
>
> We have also thought in a bit more details about transactions and queues,
> here's the plan.
>
> *Transactions*
>
> The support for transactions in *classic topics* is based on precise
> interactions between three actors: clients (mostly producers, but also
> consumers), brokers (ReplicaManager and other classes), and transaction
> coordinators. Brokers also run partition leaders with their local state
> (ProducerStateManager and others).
>
> The high level (some details skipped) workflow is the following. When a
> transactional Produce request is received by the broker:
> 1. For each partition, the partition leader checks if a non-empty
> transaction is running for this partition. This is done using its local
> state derived from the log metadata (ProducerStateManager,
> VerificationStateEntry, VerificationGuard).
> 2. The transaction coordinator is informed about all the partitions that
> aren’t part of the transaction to include them.
> 3. The partition leaders do additional transactional checks.
> 4. The partition leaders append the transactional data to their logs and
> update some of their state (for example, log the fact that the transaction
> is running for the partition and its first offset).
>
> When the transaction is committed or aborted:
> 1. The producer contacts the transaction coordinator directly with
> EndTxnRequest.
> 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> its log and responds to the producer.
> 3. The transaction coordinator sends WriteTxnMarkersRequest to the leaders
> of the involved partitions.
> 4. The partition leaders write the transaction markers to their logs and
> respond to the coordinator.
> 5. The coordinator writes the final transaction state COMPLETE_COMMIT or
> COMPLETE_ABORT.
>
> In classic topics, partitions have leaders and lots of important state
> necessary for supporting this workflow is local. The main challenge in
> mapping this to Diskless comes from the fact there are no partition
> leaders, so the corresponding pieces of state need to be globalized in the
> batch coordinator. We are already doing this to support idempotent produce.
>
> The high level workflow for *diskless topics* would look very similar:
> 1. For each partition, the broker checks if a non-empty transaction is
> running for this partition. In contrast to classic topics, this is checked
> against the batch coordinator with a single RPC. Since a transaction could
> be uniquely identified with producer ID and epoch, the positive result of
> this check could be cached locally (for the double configured duration of a
> transaction, for example).
> 2. The same: The transaction coordinator is informed about all the
> partitions that aren’t part of the transaction to include them.
> 3. No transactional checks are done on the broker side.
> 4. The broker appends the transactional data to the current shared WAL
> segment. It doesn’t update any transaction-related state for Diskless
> topics, because it doesn’t have any.
> 5. The WAL segment is committed to the batch coordinator like in the
> normal produce flow.
> 6. The batch coordinator does the final transactional checks of the
> batches. This procedure would output the same errors like the partition
> leader in classic topics would do. I.e. some batches could be rejected.
> This means, there will potentially be garbage in the WAL segment file in
> case of transactional errors. This is preferable to doing more network
> round trips, especially considering the WAL segments will be relatively
> short-living (see the Greg's update above).
>
> When the transaction is committed or aborted:
> 1. The producer contacts the transaction coordinator directly with
> EndTxnRequest.
> 2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to
> its log and responds to the producer.
> 3. *[NEW]* The transaction coordinator informs the batch coordinator that
> the transaction is finished.
> 4. *[NEW]* The batch coordinator saves that the transaction is finished
> and also inserts the control batches in the corresponding logs of the
> involved Diskless topics. This happens only on the metadata level, no
> actual control batches are written to any file. They will be dynamically
> created on Fetch and other read operations. We could technically write
> these control batches for real, but this would mean extra produce latency,
> so it's better just to mark them in the batch coordinator and save these
> milliseconds.
> 5. The transaction coordinator sends WriteTxnMarkersRequest to the leaders
> of the involved partitions. – Now only to classic topics now.
> 6. The partition leaders of classic topics write the transaction markers
> to their logs and respond to the coordinator.
> 7. The coordinator writes the final transaction state COMPLETE_COMMIT or
> COMPLETE_ABORT.
>
> Compared to the non-transactional produce flow, we get:
> 1. An extra network round trip between brokers and the batch coordinator
> when a new partition appear in the transaction. To mitigate the impact of
> them:
>   - The results will be cached.
>   - The calls for multiple partitions in one Produce request will be
> grouped.
>   - The batch coordinator should be optimized for fast response to these
> RPCs.
>   - The fact that a single producer normally will communicate with a
> single broker for the duration of the transaction further reduces the
> expected number of round trips.
> 2. An extra round trip between the transaction coordinator and batch
> coordinator when a transaction is finished.
>
> With this proposal, transactions will also be able to span both classic
> and Diskless topics.
>
> *Queues*
>
> The share group coordination and management is a side job that doesn't
> interfere with the topic itself (leadership, replicas, physical storage of
> records, etc.) and non-queue producers and consumers (Fetch and Produce
> RPCs, consumer group-related RPCs are not affected.) We don't see any
> reason why we can't make Diskless topics compatible with share groups the
> same way as classic topics are. Even on the code level, we don't expect any
> serious refactoring: the same reading routines are used that are used for
> fetching (e.g. ReplicaManager.readFromLog).
>
>
> Should the KIPs be modified to include this or it's too
> implementation-focused?
>
> Best regards,
> Ivan
>
> On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> > Hi all,
> >
> > Thank you all for your questions and design input on KIP-1150.
> >
> > We have just updated KIP-1150 and KIP-1163 with a new design. To
> summarize
> > the changes:
> >
> > 1. The design prioritizes integrating with the existing KIP-405 Tiered
> > Storage interfaces, permitting data produced to a Diskless topic to be
> > moved to tiered storage.
> > This lowers the scalability requirements for the Batch Coordinator
> > component, and allows Diskless to compose with Tiered Storage plugin
> > features such as encryption and alternative data formats.
> >
> > 2. Consumer fetches are now served from local segments, making use of the
> > indexes, page cache, request purgatory, and zero-copy functionality
> already
> > built into classic topics.
> > However, local segments are now considered cache elements, do not need to
> > be durably stored, and can be built without contacting any other
> replicas.
> >
> > 3. The design has been simplified substantially, by removing the previous
> > Diskless consume flow, distributed cache component, and "object
> > compaction/merging" step.
> >
> > The design maintains leaderless produces as enabled by the Batch
> > Coordinator, and the same latency profiles as the earlier design, while
> > being simpler and integrating better into the existing ecosystem.
> >
> > Thanks, and we are eager to hear your feedback on the new design.
> > Greg Harris
> >
> > On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Jan,
> > >
> > > For me, the main gap of KIP-1150 is the support of all existing client
> > > APIs. Currently, there is no design for supporting APIs like
> transactions
> > > and queues.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > > <ja...@kentra.io.invalid> wrote:
> > >
> > > > Would it be a good time to ask for the current status of this KIP? I
> > > > haven't seen much activity here for the past 2 months, the vote got
> > > vetoed
> > > > but I think the pending questions have been answered since then.
> KIP-1183
> > > > (AutoMQ's proposal) also didn't have any activity since May.
> > > >
> > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> > > > made, with a coordinator-based approach being by far the dominant one
> > > when
> > > > it comes to market adoption - but all these are standalone products.
> > > >
> > > > I'm a big fan of both approaches, but would hate to see a stall. So
> the
> > > > question is: can we get an update?
> > > >
> > > > Maybe it's time to start another vote? Colin McCabe - have your
> questions
> > > > been answered? If not, is there anything I can do to help? I'm deeply
> > > > familiar with both architectures and have written about both?
> > > >
> > > > Kind regards,
> > > > Jan
> > > >
> > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > > stanislavkozlovski@apache.org> wrote:
> > > >
> > > > > I have some nits - it may be useful to
> > > > >
> > > > > a) group all the KIP email threads in the main one (just a bunch of
> > > links
> > > > > to everything)
> > > > > b) create the email threads
> > > > >
> > > > > It's a bit hard to track it all - for example, I was searching for
> a
> > > > > discuss thread for KIP-1165 for a while; As far as I can tell, it
> > > doesn't
> > > > > exist yet.
> > > > >
> > > > > Since the KIPs are published (by virtue of having the root KIP be
> > > > > published, having a DISCUSS thread and links to sub-KIPs where were
> > > aimed
> > > > > to move the discussion towards), I think it would be good to create
> > > > DISCUSS
> > > > > threads for them all.
> > > > >
> > > > > Best,
> > > > > Stan
> > > > >
> > > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > > Hi Kafka Devs!
> > > > > >
> > > > > > We want to start a new KIP discussion about introducing a new
> type of
> > > > > > topics that would make use of Object Storage as the primary
> source of
> > > > > > storage. However, as this KIP is big we decided to split it into
> > > > multiple
> > > > > > related KIPs.
> > > > > > We have the motivational KIP-1150 (
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > > )
> > > > > > that aims to discuss if Apache Kafka should aim to have this
> type of
> > > > > > feature at all. This KIP doesn't go onto details on how to
> implement
> > > > it.
> > > > > > This follows the same approach used when we discussed KRaft.
> > > > > >
> > > > > > But as we know that it is sometimes really hard to discuss on
> that
> > > meta
> > > > > > level, we also created several sub-kips (linked in KIP-1150) that
> > > offer
> > > > > an
> > > > > > implementation of this feature.
> > > > > >
> > > > > > We kindly ask you to use the proper DISCUSS threads for each
> type of
> > > > > > concern and keep this one to discuss whether Apache Kafka wants
> to
> > > have
> > > > > > this feature or not.
> > > > > >
> > > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > > >
> > > > > > ------------------
> > > > > > Josep Prat
> > > > > > Open Source Engineering Director, Aiven
> > > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > > Aiven Deutschland GmbH
> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > Anna Richardson, Kenneth Chen
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Ivan Yurchenko <iv...@ivanyu.me>]

Hi all,

We have also thought in a bit more details about transactions and queues, here's the plan.

*Transactions*

The support for transactions in *classic topics* is based on precise interactions between three actors: clients (mostly producers, but also consumers), brokers (ReplicaManager and other classes), and transaction coordinators. Brokers also run partition leaders with their local state (ProducerStateManager and others). 

The high level (some details skipped) workflow is the following. When a transactional Produce request is received by the broker:
1. For each partition, the partition leader checks if a non-empty transaction is running for this partition. This is done using its local state derived from the log metadata (ProducerStateManager, VerificationStateEntry, VerificationGuard).
2. The transaction coordinator is informed about all the partitions that aren’t part of the transaction to include them.
3. The partition leaders do additional transactional checks.
4. The partition leaders append the transactional data to their logs and update some of their state (for example, log the fact that the transaction is running for the partition and its first offset).

When the transaction is committed or aborted:
1. The producer contacts the transaction coordinator directly with EndTxnRequest.
2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to its log and responds to the producer.
3. The transaction coordinator sends WriteTxnMarkersRequest to the leaders of the involved partitions.
4. The partition leaders write the transaction markers to their logs and respond to the coordinator.
5. The coordinator writes the final transaction state COMPLETE_COMMIT or COMPLETE_ABORT.

In classic topics, partitions have leaders and lots of important state necessary for supporting this workflow is local. The main challenge in mapping this to Diskless comes from the fact there are no partition leaders, so the corresponding pieces of state need to be globalized in the batch coordinator. We are already doing this to support idempotent produce.

The high level workflow for *diskless topics* would look very similar:
1. For each partition, the broker checks if a non-empty transaction is running for this partition. In contrast to classic topics, this is checked against the batch coordinator with a single RPC. Since a transaction could be uniquely identified with producer ID and epoch, the positive result of this check could be cached locally (for the double configured duration of a transaction, for example).
2. The same: The transaction coordinator is informed about all the partitions that aren’t part of the transaction to include them.
3. No transactional checks are done on the broker side.
4. The broker appends the transactional data to the current shared WAL segment. It doesn’t update any transaction-related state for Diskless topics, because it doesn’t have any.
5. The WAL segment is committed to the batch coordinator like in the normal produce flow.
6. The batch coordinator does the final transactional checks of the batches. This procedure would output the same errors like the partition leader in classic topics would do. I.e. some batches could be rejected. This means, there will potentially be garbage in the WAL segment file in case of transactional errors. This is preferable to doing more network round trips, especially considering the WAL segments will be relatively short-living (see the Greg's update above).

When the transaction is committed or aborted:
1. The producer contacts the transaction coordinator directly with EndTxnRequest.
2. The transaction coordinator writes PREPARE_COMMIT or PREPARE_ABORT to its log and responds to the producer.
3. *[NEW]* The transaction coordinator informs the batch coordinator that the transaction is finished.
4. *[NEW]* The batch coordinator saves that the transaction is finished and also inserts the control batches in the corresponding logs of the involved Diskless topics. This happens only on the metadata level, no actual control batches are written to any file. They will be dynamically created on Fetch and other read operations. We could technically write these control batches for real, but this would mean extra produce latency, so it's better just to mark them in the batch coordinator and save these milliseconds.
5. The transaction coordinator sends WriteTxnMarkersRequest to the leaders of the involved partitions. – Now only to classic topics now.
6. The partition leaders of classic topics write the transaction markers to their logs and respond to the coordinator.
7. The coordinator writes the final transaction state COMPLETE_COMMIT or COMPLETE_ABORT.

Compared to the non-transactional produce flow, we get:
1. An extra network round trip between brokers and the batch coordinator when a new partition appear in the transaction. To mitigate the impact of them:
  - The results will be cached.
  - The calls for multiple partitions in one Produce request will be grouped.
  - The batch coordinator should be optimized for fast response to these RPCs.
  - The fact that a single producer normally will communicate with a single broker for the duration of the transaction further reduces the expected number of round trips.
2. An extra round trip between the transaction coordinator and batch coordinator when a transaction is finished.

With this proposal, transactions will also be able to span both classic and Diskless topics.

*Queues*

The share group coordination and management is a side job that doesn't interfere with the topic itself (leadership, replicas, physical storage of records, etc.) and non-queue producers and consumers (Fetch and Produce RPCs, consumer group-related RPCs are not affected.) We don't see any reason why we can't make Diskless topics compatible with share groups the same way as classic topics are. Even on the code level, we don't expect any serious refactoring: the same reading routines are used that are used for fetching (e.g. ReplicaManager.readFromLog).


Should the KIPs be modified to include this or it's too implementation-focused?

Best regards,
Ivan

On Wed, Sep 3, 2025, at 21:59, Greg Harris wrote:
> Hi all,
> 
> Thank you all for your questions and design input on KIP-1150.
> 
> We have just updated KIP-1150 and KIP-1163 with a new design. To summarize
> the changes:
> 
> 1. The design prioritizes integrating with the existing KIP-405 Tiered
> Storage interfaces, permitting data produced to a Diskless topic to be
> moved to tiered storage.
> This lowers the scalability requirements for the Batch Coordinator
> component, and allows Diskless to compose with Tiered Storage plugin
> features such as encryption and alternative data formats.
> 
> 2. Consumer fetches are now served from local segments, making use of the
> indexes, page cache, request purgatory, and zero-copy functionality already
> built into classic topics.
> However, local segments are now considered cache elements, do not need to
> be durably stored, and can be built without contacting any other replicas.
> 
> 3. The design has been simplified substantially, by removing the previous
> Diskless consume flow, distributed cache component, and "object
> compaction/merging" step.
> 
> The design maintains leaderless produces as enabled by the Batch
> Coordinator, and the same latency profiles as the earlier design, while
> being simpler and integrating better into the existing ecosystem.
> 
> Thanks, and we are eager to hear your feedback on the new design.
> Greg Harris
> 
> On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid> wrote:
> 
> > Hi, Jan,
> >
> > For me, the main gap of KIP-1150 is the support of all existing client
> > APIs. Currently, there is no design for supporting APIs like transactions
> > and queues.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> > <ja...@kentra.io.invalid> wrote:
> >
> > > Would it be a good time to ask for the current status of this KIP? I
> > > haven't seen much activity here for the past 2 months, the vote got
> > vetoed
> > > but I think the pending questions have been answered since then. KIP-1183
> > > (AutoMQ's proposal) also didn't have any activity since May.
> > >
> > > In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> > > made, with a coordinator-based approach being by far the dominant one
> > when
> > > it comes to market adoption - but all these are standalone products.
> > >
> > > I'm a big fan of both approaches, but would hate to see a stall. So the
> > > question is: can we get an update?
> > >
> > > Maybe it's time to start another vote? Colin McCabe - have your questions
> > > been answered? If not, is there anything I can do to help? I'm deeply
> > > familiar with both architectures and have written about both?
> > >
> > > Kind regards,
> > > Jan
> > >
> > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > > stanislavkozlovski@apache.org> wrote:
> > >
> > > > I have some nits - it may be useful to
> > > >
> > > > a) group all the KIP email threads in the main one (just a bunch of
> > links
> > > > to everything)
> > > > b) create the email threads
> > > >
> > > > It's a bit hard to track it all - for example, I was searching for a
> > > > discuss thread for KIP-1165 for a while; As far as I can tell, it
> > doesn't
> > > > exist yet.
> > > >
> > > > Since the KIPs are published (by virtue of having the root KIP be
> > > > published, having a DISCUSS thread and links to sub-KIPs where were
> > aimed
> > > > to move the discussion towards), I think it would be good to create
> > > DISCUSS
> > > > threads for them all.
> > > >
> > > > Best,
> > > > Stan
> > > >
> > > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > > Hi Kafka Devs!
> > > > >
> > > > > We want to start a new KIP discussion about introducing a new type of
> > > > > topics that would make use of Object Storage as the primary source of
> > > > > storage. However, as this KIP is big we decided to split it into
> > > multiple
> > > > > related KIPs.
> > > > > We have the motivational KIP-1150 (
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > > )
> > > > > that aims to discuss if Apache Kafka should aim to have this type of
> > > > > feature at all. This KIP doesn't go onto details on how to implement
> > > it.
> > > > > This follows the same approach used when we discussed KRaft.
> > > > >
> > > > > But as we know that it is sometimes really hard to discuss on that
> > meta
> > > > > level, we also created several sub-kips (linked in KIP-1150) that
> > offer
> > > > an
> > > > > implementation of this feature.
> > > > >
> > > > > We kindly ask you to use the proper DISCUSS threads for each type of
> > > > > concern and keep this one to discuss whether Apache Kafka wants to
> > have
> > > > > this feature or not.
> > > > >
> > > > > Thanks in advance on behalf of all the authors of this KIP.
> > > > >
> > > > > ------------------
> > > > > Josep Prat
> > > > > Open Source Engineering Director, Aiven
> > > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > > Aiven Deutschland GmbH
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > Anna Richardson, Kenneth Chen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1150 Diskless Topics

[Greg Harris <gr...@aiven.io.INVALID>]

Hi all,

Thank you all for your questions and design input on KIP-1150.

We have just updated KIP-1150 and KIP-1163 with a new design. To summarize
the changes:

1. The design prioritizes integrating with the existing KIP-405 Tiered
Storage interfaces, permitting data produced to a Diskless topic to be
moved to tiered storage.
This lowers the scalability requirements for the Batch Coordinator
component, and allows Diskless to compose with Tiered Storage plugin
features such as encryption and alternative data formats.

2. Consumer fetches are now served from local segments, making use of the
indexes, page cache, request purgatory, and zero-copy functionality already
built into classic topics.
However, local segments are now considered cache elements, do not need to
be durably stored, and can be built without contacting any other replicas.

3. The design has been simplified substantially, by removing the previous
Diskless consume flow, distributed cache component, and "object
compaction/merging" step.

The design maintains leaderless produces as enabled by the Batch
Coordinator, and the same latency profiles as the earlier design, while
being simpler and integrating better into the existing ecosystem.

Thanks, and we are eager to hear your feedback on the new design.
Greg Harris

On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Jan,
>
> For me, the main gap of KIP-1150 is the support of all existing client
> APIs. Currently, there is no design for supporting APIs like transactions
> and queues.
>
> Thanks,
>
> Jun
>
> On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
> <ja...@kentra.io.invalid> wrote:
>
> > Would it be a good time to ask for the current status of this KIP? I
> > haven't seen much activity here for the past 2 months, the vote got
> vetoed
> > but I think the pending questions have been answered since then. KIP-1183
> > (AutoMQ's proposal) also didn't have any activity since May.
> >
> > In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> > made, with a coordinator-based approach being by far the dominant one
> when
> > it comes to market adoption - but all these are standalone products.
> >
> > I'm a big fan of both approaches, but would hate to see a stall. So the
> > question is: can we get an update?
> >
> > Maybe it's time to start another vote? Colin McCabe - have your questions
> > been answered? If not, is there anything I can do to help? I'm deeply
> > familiar with both architectures and have written about both?
> >
> > Kind regards,
> > Jan
> >
> > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> > stanislavkozlovski@apache.org> wrote:
> >
> > > I have some nits - it may be useful to
> > >
> > > a) group all the KIP email threads in the main one (just a bunch of
> links
> > > to everything)
> > > b) create the email threads
> > >
> > > It's a bit hard to track it all - for example, I was searching for a
> > > discuss thread for KIP-1165 for a while; As far as I can tell, it
> doesn't
> > > exist yet.
> > >
> > > Since the KIPs are published (by virtue of having the root KIP be
> > > published, having a DISCUSS thread and links to sub-KIPs where were
> aimed
> > > to move the discussion towards), I think it would be good to create
> > DISCUSS
> > > threads for them all.
> > >
> > > Best,
> > > Stan
> > >
> > > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > > Hi Kafka Devs!
> > > >
> > > > We want to start a new KIP discussion about introducing a new type of
> > > > topics that would make use of Object Storage as the primary source of
> > > > storage. However, as this KIP is big we decided to split it into
> > multiple
> > > > related KIPs.
> > > > We have the motivational KIP-1150 (
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > > )
> > > > that aims to discuss if Apache Kafka should aim to have this type of
> > > > feature at all. This KIP doesn't go onto details on how to implement
> > it.
> > > > This follows the same approach used when we discussed KRaft.
> > > >
> > > > But as we know that it is sometimes really hard to discuss on that
> meta
> > > > level, we also created several sub-kips (linked in KIP-1150) that
> offer
> > > an
> > > > implementation of this feature.
> > > >
> > > > We kindly ask you to use the proper DISCUSS threads for each type of
> > > > concern and keep this one to discuss whether Apache Kafka wants to
> have
> > > > this feature or not.
> > > >
> > > > Thanks in advance on behalf of all the authors of this KIP.
> > > >
> > > > ------------------
> > > > Josep Prat
> > > > Open Source Engineering Director, Aiven
> > > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > > Aiven Deutschland GmbH
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > Anna Richardson, Kenneth Chen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jan,

For me, the main gap of KIP-1150 is the support of all existing client
APIs. Currently, there is no design for supporting APIs like transactions
and queues.

Thanks,

Jun

On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
<ja...@kentra.io.invalid> wrote:

> Would it be a good time to ask for the current status of this KIP? I
> haven't seen much activity here for the past 2 months, the vote got vetoed
> but I think the pending questions have been answered since then. KIP-1183
> (AutoMQ's proposal) also didn't have any activity since May.
>
> In my eyes KIP-1150 and KIP-1183 are two real choices that can be
> made, with a coordinator-based approach being by far the dominant one when
> it comes to market adoption - but all these are standalone products.
>
> I'm a big fan of both approaches, but would hate to see a stall. So the
> question is: can we get an update?
>
> Maybe it's time to start another vote? Colin McCabe - have your questions
> been answered? If not, is there anything I can do to help? I'm deeply
> familiar with both architectures and have written about both?
>
> Kind regards,
> Jan
>
> On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
> stanislavkozlovski@apache.org> wrote:
>
> > I have some nits - it may be useful to
> >
> > a) group all the KIP email threads in the main one (just a bunch of links
> > to everything)
> > b) create the email threads
> >
> > It's a bit hard to track it all - for example, I was searching for a
> > discuss thread for KIP-1165 for a while; As far as I can tell, it doesn't
> > exist yet.
> >
> > Since the KIPs are published (by virtue of having the root KIP be
> > published, having a DISCUSS thread and links to sub-KIPs where were aimed
> > to move the discussion towards), I think it would be good to create
> DISCUSS
> > threads for them all.
> >
> > Best,
> > Stan
> >
> > On 2025/04/16 11:58:22 Josep Prat wrote:
> > > Hi Kafka Devs!
> > >
> > > We want to start a new KIP discussion about introducing a new type of
> > > topics that would make use of Object Storage as the primary source of
> > > storage. However, as this KIP is big we decided to split it into
> multiple
> > > related KIPs.
> > > We have the motivational KIP-1150 (
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > )
> > > that aims to discuss if Apache Kafka should aim to have this type of
> > > feature at all. This KIP doesn't go onto details on how to implement
> it.
> > > This follows the same approach used when we discussed KRaft.
> > >
> > > But as we know that it is sometimes really hard to discuss on that meta
> > > level, we also created several sub-kips (linked in KIP-1150) that offer
> > an
> > > implementation of this feature.
> > >
> > > We kindly ask you to use the proper DISCUSS threads for each type of
> > > concern and keep this one to discuss whether Apache Kafka wants to have
> > > this feature or not.
> > >
> > > Thanks in advance on behalf of all the authors of this KIP.
> > >
> > > ------------------
> > > Josep Prat
> > > Open Source Engineering Director, Aiven
> > > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > > Aiven Deutschland GmbH
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > Anna Richardson, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: [DISCUSS] KIP-1150 Diskless Topics

[Jan Siekierski <ja...@kentra.io.INVALID>]

Would it be a good time to ask for the current status of this KIP? I
haven't seen much activity here for the past 2 months, the vote got vetoed
but I think the pending questions have been answered since then. KIP-1183
(AutoMQ's proposal) also didn't have any activity since May.

In my eyes KIP-1150 and KIP-1183 are two real choices that can be
made, with a coordinator-based approach being by far the dominant one when
it comes to market adoption - but all these are standalone products.

I'm a big fan of both approaches, but would hate to see a stall. So the
question is: can we get an update?

Maybe it's time to start another vote? Colin McCabe - have your questions
been answered? If not, is there anything I can do to help? I'm deeply
familiar with both architectures and have written about both?

Kind regards,
Jan

On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski <
stanislavkozlovski@apache.org> wrote:

> I have some nits - it may be useful to
>
> a) group all the KIP email threads in the main one (just a bunch of links
> to everything)
> b) create the email threads
>
> It's a bit hard to track it all - for example, I was searching for a
> discuss thread for KIP-1165 for a while; As far as I can tell, it doesn't
> exist yet.
>
> Since the KIPs are published (by virtue of having the root KIP be
> published, having a DISCUSS thread and links to sub-KIPs where were aimed
> to move the discussion towards), I think it would be good to create DISCUSS
> threads for them all.
>
> Best,
> Stan
>
> On 2025/04/16 11:58:22 Josep Prat wrote:
> > Hi Kafka Devs!
> >
> > We want to start a new KIP discussion about introducing a new type of
> > topics that would make use of Object Storage as the primary source of
> > storage. However, as this KIP is big we decided to split it into multiple
> > related KIPs.
> > We have the motivational KIP-1150 (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> )
> > that aims to discuss if Apache Kafka should aim to have this type of
> > feature at all. This KIP doesn't go onto details on how to implement it.
> > This follows the same approach used when we discussed KRaft.
> >
> > But as we know that it is sometimes really hard to discuss on that meta
> > level, we also created several sub-kips (linked in KIP-1150) that offer
> an
> > implementation of this feature.
> >
> > We kindly ask you to use the proper DISCUSS threads for each type of
> > concern and keep this one to discuss whether Apache Kafka wants to have
> > this feature or not.
> >
> > Thanks in advance on behalf of all the authors of this KIP.
> >
> > ------------------
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.prat@aiven.io   |   +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>