You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Chris Egerton <fe...@gmail.com> on 2026/02/25 03:09:36 UTC
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Hi all,
Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as the
relevant discussion threads. I may have granular comments about 1163 and
1164 but the overall approach suggested in 1150 looks good to me. I
especially like that the approach covers two main pain points of operating
and paying for Kafka today: it allows cross-AZ traffic to be reduced (even
eliminated in some cases), and it also allows local disk usage by brokers
to be reduced (if operators opt for a small local cache on follower brokers
for non-tiered segments).
+1 (binding)
Cheers,
Chris
On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <va...@gmail.com> wrote:
> Hi Josep,
>
> Thank you for the detailed response. I appreciate the clarification
> regarding the distinction between the Inkless POC and the KIP design.
>
> However, my objection is not based on temporary bugs in the fork, but *on
> architectural gaps in the KIPs themselves* that these implementation issues
> highlighted. If we are voting to approve the design, the design documents
> must be structurally complete regarding data safety.
>
> *1. Regarding Storage Leaks (The Missing Design)* You mentioned that
> cleanup logic "can be defined later." However, KIP-1163 explicitly
> delegates this responsibility to a separate process, and KIP-1165 (Object
> Compaction/GC) is currently marked as "Discarded" in the wiki.
>
> We cannot vote to approve a storage engine that has no specified mechanism
> for garbage collection. The "Upload-then-Commit" pattern described in
> KIP-1163 structurally creates orphaned segments during broker failures.
> Without an active KIP defining the reconciliation protocol (since KIP-1165
> was withdrawn), the proposal effectively describes a system with unbounded
> storage growth during failure modes. This is a blocking design gap, not an
> implementation detail.
>
> *2. Regarding EOS (The Coordinator Synchronization Gap)* This is not a
> misunderstanding of standard Kafka transactions; it is a critique of how
> KIP-1150 changes them. Standard EOS relies on the Partition Leader to
> sequence markers and calculate the LSO (Last Stable Offset) in memory.
> KIP-1150 removes the Leader.
>
> KIP-1164 (Batch Coordinator) must explicitly define the RPC flow between
> the Transaction Coordinator and the Batch Coordinator to replace the
> leader's role. Currently, the KIP does not specify how the system prevents
> a "Split Brain" scenario where a consumer reads ahead of a transaction
> marker that hasn't yet been sequenced by the Batch Coordinator. This is a
> protocol-level correctness issue that must be resolved in the text before
> adoption.
>
> Please note - I am maintaining my objection based on missing
> specifications, not code bugs.
>
> I respectfully request that we pause the vote until:
>
> A valid design for Garbage Collection (replacing the discarded
> KIP-1165) is added to the proposal.
>
> The Transaction/LSO synchronization protocol is explicitly documented
> in KIP-1164.
>
> Regards,
>
> Vaquar Khan
> Sr Data Architect
> https://www.linkedin.com/in/vaquar-khan-b695577/
>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Andrew Schofield <sc...@apache.org>.
I'm convinced by the principle of reducing inter-zone cloud costs for Apache Kafka. There are still a lot of missing details in 1163 and 1164, but this foundational KIP looks good to me.
+1 (binding)
Thanks,
Andrew
On 2026/02/25 16:32:27 Greg Harris via dev wrote:
> Hey all,
>
> I'm excited to discuss more details in 1163 and 1164 with everyone.
>
> +1 (binding)
>
> Thanks!
> Greg
>
> On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi all,
> >
> > Given the importance of this KIP, we want to keep the vote open for a few
> > more days to give time to people who had comments in the DISCUSS thread to
> > cast their vote if they want.
> >
> > On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi all,
> > > As a co-author of the KIP, I want to explicitly cast my vote for this
> > KIP.
> > >
> > > +1 (binding)
> > >
> > >
> > > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > I've re-read KIP-1150, and still agree this is what we need for Apache
> > > > Kafka.
> > > >
> > > > +1 (binding) from me.
> > > >
> > > > Thank you,
> > > > Luke
> > > >
> > > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> > fearthecellos@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as the
> > > >> relevant discussion threads. I may have granular comments about 1163
> > and
> > > >> 1164 but the overall approach suggested in 1150 looks good to me. I
> > > >> especially like that the approach covers two main pain points of
> > > operating
> > > >> and paying for Kafka today: it allows cross-AZ traffic to be reduced
> > > (even
> > > >> eliminated in some cases), and it also allows local disk usage by
> > > brokers
> > > >> to be reduced (if operators opt for a small local cache on follower
> > > >> brokers
> > > >> for non-tiered segments).
> > > >>
> > > >> +1 (binding)
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Chris
> > > >>
> > > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <va...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Josep,
> > > >> >
> > > >> > Thank you for the detailed response. I appreciate the clarification
> > > >> > regarding the distinction between the Inkless POC and the KIP
> > design.
> > > >> >
> > > >> > However, my objection is not based on temporary bugs in the fork,
> > but
> > > >> *on
> > > >> > architectural gaps in the KIPs themselves* that these implementation
> > > >> issues
> > > >> > highlighted. If we are voting to approve the design, the design
> > > >> documents
> > > >> > must be structurally complete regarding data safety.
> > > >> >
> > > >> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned that
> > > >> > cleanup logic "can be defined later." However, KIP-1163 explicitly
> > > >> > delegates this responsibility to a separate process, and KIP-1165
> > > >> (Object
> > > >> > Compaction/GC) is currently marked as "Discarded" in the wiki.
> > > >> >
> > > >> > We cannot vote to approve a storage engine that has no specified
> > > >> mechanism
> > > >> > for garbage collection. The "Upload-then-Commit" pattern described
> > in
> > > >> > KIP-1163 structurally creates orphaned segments during broker
> > > failures.
> > > >> > Without an active KIP defining the reconciliation protocol (since
> > > >> KIP-1165
> > > >> > was withdrawn), the proposal effectively describes a system with
> > > >> unbounded
> > > >> > storage growth during failure modes. This is a blocking design gap,
> > > not
> > > >> an
> > > >> > implementation detail.
> > > >> >
> > > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is
> > not a
> > > >> > misunderstanding of standard Kafka transactions; it is a critique of
> > > how
> > > >> > KIP-1150 changes them. Standard EOS relies on the Partition Leader
> > to
> > > >> > sequence markers and calculate the LSO (Last Stable Offset) in
> > memory.
> > > >> > KIP-1150 removes the Leader.
> > > >> >
> > > >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow
> > > between
> > > >> > the Transaction Coordinator and the Batch Coordinator to replace the
> > > >> > leader's role. Currently, the KIP does not specify how the system
> > > >> prevents
> > > >> > a "Split Brain" scenario where a consumer reads ahead of a
> > transaction
> > > >> > marker that hasn't yet been sequenced by the Batch Coordinator. This
> > > is
> > > >> a
> > > >> > protocol-level correctness issue that must be resolved in the text
> > > >> before
> > > >> > adoption.
> > > >> >
> > > >> > Please note - I am maintaining my objection based on missing
> > > >> > specifications, not code bugs.
> > > >> >
> > > >> > I respectfully request that we pause the vote until:
> > > >> >
> > > >> > A valid design for Garbage Collection (replacing the discarded
> > > >> > KIP-1165) is added to the proposal.
> > > >> >
> > > >> > The Transaction/LSO synchronization protocol is explicitly
> > > >> documented
> > > >> > in KIP-1164.
> > > >> >
> > > >> > Regards,
> > > >> >
> > > >> > Vaquar Khan
> > > >> > Sr Data Architect
> > > >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> > > >> >
> > > >>
> > > >
> > >
> > > --
> > > [image: Aiven] <https://www.aiven.io>
> > >
> > > *Josep Prat*
> > > Sr. Engineering Director, Streaming Services, *Aiven*
> > > josep.prat@aiven.io | +491715557497
> > > aiven.io <https://www.aiven.io> | <
> > https://www.facebook.com/aivencloud
> > > >
> > > <https://www.linkedin.com/company/aiven/> <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > >
> > > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
> >
> > --
> > Anatolii Popov
> > Senior Software Developer, *Aiven OY*
> > m: +358505126242
> > w: aiven.io e: anatolii.popov@aiven.io
> > <https://www.facebook.com/aivencloud>
> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
> >
>
Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Anatolii Popov via dev <de...@kafka.apache.org>.
Hi all,
Thanks everyone for the votes and discussion.
I'm happy to announce that KIP-1150 is now accepted with
+9 binding (Stanislav Kozlovski, Chris Egerton, Luke Chen, Josep Prat, Greg
Harris, Andrew Schofield, Jun Rao, Satish Duggana, Chia-Ping Tsai) votes
and
+5 non-binding(Henry Cai, Jian Fu, Andrew Mills, Varun Ghai, Vaquar Khan)
votes.
As a reminder, KIP-1150 is a motivational KIP — it establishes community
consensus on whether Apache Kafka should pursue object storage as the
primary storage backend for a new topic type, without prescribing a
specific implementation. All implementation details will be defined and
discussed in the follow-up KIPs:
- KIP-1163: Diskless Core — DISCUSS thread : https://lists.
apache.org/thread/3dj67w04r7pcmlytl912gv69j22o3g4j
- KIP-1164: Diskless Coordinator — DISCUSS thrread:
https://lists.apache.org/thread/m9l6lbqv2cffxtz5frypylmqjd7bsqoz
We encourage everyone to continue the conversation in those threads. Each of
those KIPs will have its own discussion and voting process.
Thanks again to all who participated!
~Anatolii.
On Mon, Mar 2, 2026 at 7:31 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
> +1 (binding)
>
> > Satish Duggana <sa...@gmail.com> 於 2026年2月28日 上午11:00 寫道：
> >
> > Thanks for the KIP.
> > I've reviewed the updated KIP and agree with the motivation behind
> > KIP-1150, overall LGTM.
> > It seems KIP-1163 and KIP-1164 require more details, which we can discuss
> > in those respective threads.
> >
> > +1(binding) for KIP-1150.
> >
> > ~Satish.
> >
> >> On Fri, 27 Feb 2026 at 23:28, Jun Rao via dev <de...@kafka.apache.org>
> wrote:
> >>
> >> Hi, Anatolii,
> >>
> >> Thanks for the KIP. The link you posted for KIP-1150 seems incorrect
> and it
> >> points to KIP-1163. Otherwise, +1.
> >>
> >> Jun
> >>
> >>> On Wed, Feb 25, 2026 at 2:59 PM vaquar khan <va...@gmail.com>
> wrote:
> >>>
> >>> Fair point, Chris. I agree with that architectural boundary. KIP-1150
> >>> successfully sets the high-level mandate , and we can rigorously tackle
> >> the
> >>> exact EOS and RPC mechanics over in the KIP-1164 thread .
> >>>
> >>> Andrew, I am fully aligned with you on the massive operational value of
> >>> eliminating those cross-AZ replication costs. It is absolutely the
> right
> >>> strategic direction for Kafka.
> >>>
> >>> Since my initial concerns on the storage side are resolved, and we are
> >>> aligned on where the transactional interfaces will be finalized, I am
> >>> officially withdrawing my objection.
> >>> +1 (non-binding) for KIP-1150.
> >>>
> >>> I will migrate my open questions over to the KIP-1164 discussion thread
> >> so
> >>> we can lock down the data safety details there.
> >>>
> >>> Regards,
> >>> Vaquar Khan
> >>>
> >>> On Wed, 25 Feb 2026 at 15:24, Chris Egerton <fe...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Vaquar,
> >>>>
> >>>>> Let me know what you guys think about locking down the text for these
> >>>> interfaces.
> >>>>
> >>>> I think this KIP has the appropriate level of detail and any concerns
> >>> about
> >>>> EOS can be addressed in the relevant sub-KIP.
> >>>>
> >>>> Chris
> >>>>
> >>>> On Wed, Feb 25, 2026 at 4:20 PM vaquar khan <va...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Hi everyone,
> >>>>>
> >>>>> First off, thanks to the authors for the Feb 12th updates to
> >> KIP-1163 .
> >>>>> Adding the periodic reconciliation loop clears up my concerns about
> >> the
> >>>>> orphaned "Upload-then-Commit" segments, so I'm officially withdrawing
> >>> my
> >>>>> objection on the storage leak issue .
> >>>>>
> >>>>> Chris and Greg- since you both mentioned digging into the 1164
> >>> details, I
> >>>>> wanted to pick your brains on how Exactly-Once Semantics (EOS) is
> >> going
> >>>> to
> >>>>> safely operate here. In standard Kafka, the Partition Leader is our
> >>>> single
> >>>>> serialization point. It receives the data, tracks ongoing
> >> transactions
> >>>> via
> >>>>> the ProducerStateManager, and calculates the Last Stable Offset (LSO)
> >>>>> locally . Since KIP-1150 removes the leader, the Batch Coordinator
> >>> takes
> >>>>> over. But as I read through the current text, a few critical
> >>>>> synchronization barriers seem to be missing to me:
> >>>>>
> >>>>> 1. LSO Calculation: How exactly will the Batch Coordinator maintain
> >> and
> >>>>> calculate the LSO? Justine Olshan brought this up earlier too . Will
> >>> the
> >>>>> coordinator run its own ProducerStateManager to track ongoing
> >>>> transactions,
> >>>>> or is there a totally different state machine planned?
> >>>>>
> >>>>> 2. RPC Protocol: What's the exact synchronization protocol between
> >> the
> >>>>> legacy Transaction Coordinator and the new Batch Coordinator? When
> >> the
> >>>> Txn
> >>>>> Coordinator sends a commit marker, how does the Batch Coordinator
> >>>> actually
> >>>>> verify it has received all the prerequisite data batches for that
> >>>> specific
> >>>>> transaction epoch?
> >>>>>
> >>>>> 3. Delayed Data Race Condition: Let's say a broker hits a GC pause
> >>> right
> >>>>> *after
> >>>>> *uploading a batch to object storage, but *before* committing the
> >>>>> coordinates . If the transaction commit marker arrives at the
> >>> Coordinator
> >>>>> first, what happens? Does the Coordinator wait? If not, couldn't the
> >>>>> transaction commit with missing data, completely violating
> >>> read_committed
> >>>>> isolation?
> >>>>>
> >>>>> The KIP vaguely mentions *transactional checks* but leaves the actual
> >>>>> commit protocol and public interfaces undefined right now . I'm not
> >>>> saying
> >>>>> the design itself is broken, but I really think myself and others
> >> need
> >>> to
> >>>>> see these RPC flows explicitly documented before we implement and
> >>> adopt
> >>>>> this. Otherwise, we risk baking in some severe data isolation
> >> headaches
> >>>>> down the line.
> >>>>>
> >>>>> Let me know what you guys think about locking down the text for these
> >>>>> interfaces.
> >>>>>
> >>>>> Regards,
> >>>>> Vaquar Khan
> >>>>>
> >>>>> On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <
> >>> dev@kafka.apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> Hey all,
> >>>>>>
> >>>>>> I'm excited to discuss more details in 1163 and 1164 with everyone.
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>> Thanks!
> >>>>>> Greg
> >>>>>>
> >>>>>> On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <
> >>>>>> dev@kafka.apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> Given the importance of this KIP, we want to keep the vote open
> >>> for a
> >>>>> few
> >>>>>>> more days to give time to people who had comments in the DISCUSS
> >>>> thread
> >>>>>> to
> >>>>>>> cast their vote if they want.
> >>>>>>>
> >>>>>>> On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <
> >>>>>> dev@kafka.apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>> As a co-author of the KIP, I want to explicitly cast my vote
> >> for
> >>>> this
> >>>>>>> KIP.
> >>>>>>>>
> >>>>>>>> +1 (binding)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> I've re-read KIP-1150, and still agree this is what we need
> >> for
> >>>>>> Apache
> >>>>>>>>> Kafka.
> >>>>>>>>>
> >>>>>>>>> +1 (binding) from me.
> >>>>>>>>>
> >>>>>>>>> Thank you,
> >>>>>>>>> Luke
> >>>>>>>>>
> >>>>>>>>> On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> >>>>>>> fearthecellos@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as
> >>> well
> >>>> as
> >>>>>> the
> >>>>>>>>>> relevant discussion threads. I may have granular comments
> >>> about
> >>>>> 1163
> >>>>>>> and
> >>>>>>>>>> 1164 but the overall approach suggested in 1150 looks good
> >> to
> >>>> me.
> >>>>> I
> >>>>>>>>>> especially like that the approach covers two main pain
> >> points
> >>> of
> >>>>>>>> operating
> >>>>>>>>>> and paying for Kafka today: it allows cross-AZ traffic to be
> >>>>> reduced
> >>>>>>>> (even
> >>>>>>>>>> eliminated in some cases), and it also allows local disk
> >> usage
> >>>> by
> >>>>>>>> brokers
> >>>>>>>>>> to be reduced (if operators opt for a small local cache on
> >>>>> follower
> >>>>>>>>>> brokers
> >>>>>>>>>> for non-tiered segments).
> >>>>>>>>>>
> >>>>>>>>>> +1 (binding)
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>>
> >>>>>>>>>> Chris
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <
> >>>>> vaquar.khan@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Josep,
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you for the detailed response. I appreciate the
> >>>>>> clarification
> >>>>>>>>>>> regarding the distinction between the Inkless POC and the
> >>> KIP
> >>>>>>> design.
> >>>>>>>>>>>
> >>>>>>>>>>> However, my objection is not based on temporary bugs in
> >> the
> >>>>> fork,
> >>>>>>> but
> >>>>>>>>>> *on
> >>>>>>>>>>> architectural gaps in the KIPs themselves* that these
> >>>>>> implementation
> >>>>>>>>>> issues
> >>>>>>>>>>> highlighted. If we are voting to approve the design, the
> >>>> design
> >>>>>>>>>> documents
> >>>>>>>>>>> must be structurally complete regarding data safety.
> >>>>>>>>>>>
> >>>>>>>>>>> *1. Regarding Storage Leaks (The Missing Design)* You
> >>>> mentioned
> >>>>>> that
> >>>>>>>>>>> cleanup logic "can be defined later." However, KIP-1163
> >>>>> explicitly
> >>>>>>>>>>> delegates this responsibility to a separate process, and
> >>>>> KIP-1165
> >>>>>>>>>> (Object
> >>>>>>>>>>> Compaction/GC) is currently marked as "Discarded" in the
> >>> wiki.
> >>>>>>>>>>>
> >>>>>>>>>>> We cannot vote to approve a storage engine that has no
> >>>> specified
> >>>>>>>>>> mechanism
> >>>>>>>>>>> for garbage collection. The "Upload-then-Commit" pattern
> >>>>> described
> >>>>>>> in
> >>>>>>>>>>> KIP-1163 structurally creates orphaned segments during
> >>> broker
> >>>>>>>> failures.
> >>>>>>>>>>> Without an active KIP defining the reconciliation protocol
> >>>>> (since
> >>>>>>>>>> KIP-1165
> >>>>>>>>>>> was withdrawn), the proposal effectively describes a
> >> system
> >>>> with
> >>>>>>>>>> unbounded
> >>>>>>>>>>> storage growth during failure modes. This is a blocking
> >>> design
> >>>>>> gap,
> >>>>>>>> not
> >>>>>>>>>> an
> >>>>>>>>>>> implementation detail.
> >>>>>>>>>>>
> >>>>>>>>>>> *2. Regarding EOS (The Coordinator Synchronization Gap)*
> >>> This
> >>>> is
> >>>>>>> not a
> >>>>>>>>>>> misunderstanding of standard Kafka transactions; it is a
> >>>>> critique
> >>>>>> of
> >>>>>>>> how
> >>>>>>>>>>> KIP-1150 changes them. Standard EOS relies on the
> >> Partition
> >>>>> Leader
> >>>>>>> to
> >>>>>>>>>>> sequence markers and calculate the LSO (Last Stable
> >> Offset)
> >>> in
> >>>>>>> memory.
> >>>>>>>>>>> KIP-1150 removes the Leader.
> >>>>>>>>>>>
> >>>>>>>>>>> KIP-1164 (Batch Coordinator) must explicitly define the
> >> RPC
> >>>> flow
> >>>>>>>> between
> >>>>>>>>>>> the Transaction Coordinator and the Batch Coordinator to
> >>>> replace
> >>>>>> the
> >>>>>>>>>>> leader's role. Currently, the KIP does not specify how the
> >>>>> system
> >>>>>>>>>> prevents
> >>>>>>>>>>> a "Split Brain" scenario where a consumer reads ahead of a
> >>>>>>> transaction
> >>>>>>>>>>> marker that hasn't yet been sequenced by the Batch
> >>>> Coordinator.
> >>>>>> This
> >>>>>>>> is
> >>>>>>>>>> a
> >>>>>>>>>>> protocol-level correctness issue that must be resolved in
> >>> the
> >>>>> text
> >>>>>>>>>> before
> >>>>>>>>>>> adoption.
> >>>>>>>>>>>
> >>>>>>>>>>> Please note - I am maintaining my objection based on
> >> missing
> >>>>>>>>>>> specifications, not code bugs.
> >>>>>>>>>>>
> >>>>>>>>>>> I respectfully request that we pause the vote until:
> >>>>>>>>>>>
> >>>>>>>>>>> A valid design for Garbage Collection (replacing the
> >>>>> discarded
> >>>>>>>>>>> KIP-1165) is added to the proposal.
> >>>>>>>>>>>
> >>>>>>>>>>> The Transaction/LSO synchronization protocol is
> >>> explicitly
> >>>>>>>>>> documented
> >>>>>>>>>>> in KIP-1164.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>>
> >>>>>>>>>>> Vaquar Khan
> >>>>>>>>>>> Sr Data Architect
> >>>>>>>>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> [image: Aiven] <https://www.aiven.io>
> >>>>>>>>
> >>>>>>>> *Josep Prat*
> >>>>>>>> Sr. Engineering Director, Streaming Services, *Aiven*
> >>>>>>>> josep.prat@aiven.io | +491715557497
> >>>>>>>> aiven.io <https://www.aiven.io> | <
> >>>>>>> https://www.facebook.com/aivencloud
> >>>>>>>>>
> >>>>>>>> <https://www.linkedin.com/company/aiven/> <
> >>>>>>>> https://twitter.com/aiven_io>
> >>>>>>>> *Aiven Deutschland GmbH*
> >>>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>>>
> >>>>>>>> Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> >>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Anatolii Popov
> >>>>>>> Senior Software Developer, *Aiven OY*
> >>>>>>> m: +358505126242
> >>>>>>> w: aiven.io e: anatolii.popov@aiven.io
> >>>>>>> <https://www.facebook.com/aivencloud>
> >>>>>>> <https://www.linkedin.com/company/aiven/> <
> >>>>>> https://twitter.com/aiven_io>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
--
Anatolii Popov
Senior Software Developer, *Aiven OY*
m: +358505126242
w: aiven.io e: anatolii.popov@aiven.io
<https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Chia-Ping Tsai <ch...@gmail.com>.
+1 (binding)
> Satish Duggana <sa...@gmail.com> 於 2026年2月28日 上午11:00 寫道：
>
> Thanks for the KIP.
> I've reviewed the updated KIP and agree with the motivation behind
> KIP-1150, overall LGTM.
> It seems KIP-1163 and KIP-1164 require more details, which we can discuss
> in those respective threads.
>
> +1(binding) for KIP-1150.
>
> ~Satish.
>
>> On Fri, 27 Feb 2026 at 23:28, Jun Rao via dev <de...@kafka.apache.org> wrote:
>>
>> Hi, Anatolii,
>>
>> Thanks for the KIP. The link you posted for KIP-1150 seems incorrect and it
>> points to KIP-1163. Otherwise, +1.
>>
>> Jun
>>
>>> On Wed, Feb 25, 2026 at 2:59 PM vaquar khan <va...@gmail.com> wrote:
>>>
>>> Fair point, Chris. I agree with that architectural boundary. KIP-1150
>>> successfully sets the high-level mandate , and we can rigorously tackle
>> the
>>> exact EOS and RPC mechanics over in the KIP-1164 thread .
>>>
>>> Andrew, I am fully aligned with you on the massive operational value of
>>> eliminating those cross-AZ replication costs. It is absolutely the right
>>> strategic direction for Kafka.
>>>
>>> Since my initial concerns on the storage side are resolved, and we are
>>> aligned on where the transactional interfaces will be finalized, I am
>>> officially withdrawing my objection.
>>> +1 (non-binding) for KIP-1150.
>>>
>>> I will migrate my open questions over to the KIP-1164 discussion thread
>> so
>>> we can lock down the data safety details there.
>>>
>>> Regards,
>>> Vaquar Khan
>>>
>>> On Wed, 25 Feb 2026 at 15:24, Chris Egerton <fe...@gmail.com>
>>> wrote:
>>>
>>>> Hi Vaquar,
>>>>
>>>>> Let me know what you guys think about locking down the text for these
>>>> interfaces.
>>>>
>>>> I think this KIP has the appropriate level of detail and any concerns
>>> about
>>>> EOS can be addressed in the relevant sub-KIP.
>>>>
>>>> Chris
>>>>
>>>> On Wed, Feb 25, 2026 at 4:20 PM vaquar khan <va...@gmail.com>
>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> First off, thanks to the authors for the Feb 12th updates to
>> KIP-1163 .
>>>>> Adding the periodic reconciliation loop clears up my concerns about
>> the
>>>>> orphaned "Upload-then-Commit" segments, so I'm officially withdrawing
>>> my
>>>>> objection on the storage leak issue .
>>>>>
>>>>> Chris and Greg- since you both mentioned digging into the 1164
>>> details, I
>>>>> wanted to pick your brains on how Exactly-Once Semantics (EOS) is
>> going
>>>> to
>>>>> safely operate here. In standard Kafka, the Partition Leader is our
>>>> single
>>>>> serialization point. It receives the data, tracks ongoing
>> transactions
>>>> via
>>>>> the ProducerStateManager, and calculates the Last Stable Offset (LSO)
>>>>> locally . Since KIP-1150 removes the leader, the Batch Coordinator
>>> takes
>>>>> over. But as I read through the current text, a few critical
>>>>> synchronization barriers seem to be missing to me:
>>>>>
>>>>> 1. LSO Calculation: How exactly will the Batch Coordinator maintain
>> and
>>>>> calculate the LSO? Justine Olshan brought this up earlier too . Will
>>> the
>>>>> coordinator run its own ProducerStateManager to track ongoing
>>>> transactions,
>>>>> or is there a totally different state machine planned?
>>>>>
>>>>> 2. RPC Protocol: What's the exact synchronization protocol between
>> the
>>>>> legacy Transaction Coordinator and the new Batch Coordinator? When
>> the
>>>> Txn
>>>>> Coordinator sends a commit marker, how does the Batch Coordinator
>>>> actually
>>>>> verify it has received all the prerequisite data batches for that
>>>> specific
>>>>> transaction epoch?
>>>>>
>>>>> 3. Delayed Data Race Condition: Let's say a broker hits a GC pause
>>> right
>>>>> *after
>>>>> *uploading a batch to object storage, but *before* committing the
>>>>> coordinates . If the transaction commit marker arrives at the
>>> Coordinator
>>>>> first, what happens? Does the Coordinator wait? If not, couldn't the
>>>>> transaction commit with missing data, completely violating
>>> read_committed
>>>>> isolation?
>>>>>
>>>>> The KIP vaguely mentions *transactional checks* but leaves the actual
>>>>> commit protocol and public interfaces undefined right now . I'm not
>>>> saying
>>>>> the design itself is broken, but I really think myself and others
>> need
>>> to
>>>>> see these RPC flows explicitly documented before we implement and
>>> adopt
>>>>> this. Otherwise, we risk baking in some severe data isolation
>> headaches
>>>>> down the line.
>>>>>
>>>>> Let me know what you guys think about locking down the text for these
>>>>> interfaces.
>>>>>
>>>>> Regards,
>>>>> Vaquar Khan
>>>>>
>>>>> On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <
>>> dev@kafka.apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> I'm excited to discuss more details in 1163 and 1164 with everyone.
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> Thanks!
>>>>>> Greg
>>>>>>
>>>>>> On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <
>>>>>> dev@kafka.apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Given the importance of this KIP, we want to keep the vote open
>>> for a
>>>>> few
>>>>>>> more days to give time to people who had comments in the DISCUSS
>>>> thread
>>>>>> to
>>>>>>> cast their vote if they want.
>>>>>>>
>>>>>>> On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <
>>>>>> dev@kafka.apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>> As a co-author of the KIP, I want to explicitly cast my vote
>> for
>>>> this
>>>>>>> KIP.
>>>>>>>>
>>>>>>>> +1 (binding)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com>
>>>> wrote:
>>>>>>>>
>>>>>>>>> I've re-read KIP-1150, and still agree this is what we need
>> for
>>>>>> Apache
>>>>>>>>> Kafka.
>>>>>>>>>
>>>>>>>>> +1 (binding) from me.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>> Luke
>>>>>>>>>
>>>>>>>>> On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
>>>>>>> fearthecellos@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as
>>> well
>>>> as
>>>>>> the
>>>>>>>>>> relevant discussion threads. I may have granular comments
>>> about
>>>>> 1163
>>>>>>> and
>>>>>>>>>> 1164 but the overall approach suggested in 1150 looks good
>> to
>>>> me.
>>>>> I
>>>>>>>>>> especially like that the approach covers two main pain
>> points
>>> of
>>>>>>>> operating
>>>>>>>>>> and paying for Kafka today: it allows cross-AZ traffic to be
>>>>> reduced
>>>>>>>> (even
>>>>>>>>>> eliminated in some cases), and it also allows local disk
>> usage
>>>> by
>>>>>>>> brokers
>>>>>>>>>> to be reduced (if operators opt for a small local cache on
>>>>> follower
>>>>>>>>>> brokers
>>>>>>>>>> for non-tiered segments).
>>>>>>>>>>
>>>>>>>>>> +1 (binding)
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Chris
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <
>>>>> vaquar.khan@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Josep,
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the detailed response. I appreciate the
>>>>>> clarification
>>>>>>>>>>> regarding the distinction between the Inkless POC and the
>>> KIP
>>>>>>> design.
>>>>>>>>>>>
>>>>>>>>>>> However, my objection is not based on temporary bugs in
>> the
>>>>> fork,
>>>>>>> but
>>>>>>>>>> *on
>>>>>>>>>>> architectural gaps in the KIPs themselves* that these
>>>>>> implementation
>>>>>>>>>> issues
>>>>>>>>>>> highlighted. If we are voting to approve the design, the
>>>> design
>>>>>>>>>> documents
>>>>>>>>>>> must be structurally complete regarding data safety.
>>>>>>>>>>>
>>>>>>>>>>> *1. Regarding Storage Leaks (The Missing Design)* You
>>>> mentioned
>>>>>> that
>>>>>>>>>>> cleanup logic "can be defined later." However, KIP-1163
>>>>> explicitly
>>>>>>>>>>> delegates this responsibility to a separate process, and
>>>>> KIP-1165
>>>>>>>>>> (Object
>>>>>>>>>>> Compaction/GC) is currently marked as "Discarded" in the
>>> wiki.
>>>>>>>>>>>
>>>>>>>>>>> We cannot vote to approve a storage engine that has no
>>>> specified
>>>>>>>>>> mechanism
>>>>>>>>>>> for garbage collection. The "Upload-then-Commit" pattern
>>>>> described
>>>>>>> in
>>>>>>>>>>> KIP-1163 structurally creates orphaned segments during
>>> broker
>>>>>>>> failures.
>>>>>>>>>>> Without an active KIP defining the reconciliation protocol
>>>>> (since
>>>>>>>>>> KIP-1165
>>>>>>>>>>> was withdrawn), the proposal effectively describes a
>> system
>>>> with
>>>>>>>>>> unbounded
>>>>>>>>>>> storage growth during failure modes. This is a blocking
>>> design
>>>>>> gap,
>>>>>>>> not
>>>>>>>>>> an
>>>>>>>>>>> implementation detail.
>>>>>>>>>>>
>>>>>>>>>>> *2. Regarding EOS (The Coordinator Synchronization Gap)*
>>> This
>>>> is
>>>>>>> not a
>>>>>>>>>>> misunderstanding of standard Kafka transactions; it is a
>>>>> critique
>>>>>> of
>>>>>>>> how
>>>>>>>>>>> KIP-1150 changes them. Standard EOS relies on the
>> Partition
>>>>> Leader
>>>>>>> to
>>>>>>>>>>> sequence markers and calculate the LSO (Last Stable
>> Offset)
>>> in
>>>>>>> memory.
>>>>>>>>>>> KIP-1150 removes the Leader.
>>>>>>>>>>>
>>>>>>>>>>> KIP-1164 (Batch Coordinator) must explicitly define the
>> RPC
>>>> flow
>>>>>>>> between
>>>>>>>>>>> the Transaction Coordinator and the Batch Coordinator to
>>>> replace
>>>>>> the
>>>>>>>>>>> leader's role. Currently, the KIP does not specify how the
>>>>> system
>>>>>>>>>> prevents
>>>>>>>>>>> a "Split Brain" scenario where a consumer reads ahead of a
>>>>>>> transaction
>>>>>>>>>>> marker that hasn't yet been sequenced by the Batch
>>>> Coordinator.
>>>>>> This
>>>>>>>> is
>>>>>>>>>> a
>>>>>>>>>>> protocol-level correctness issue that must be resolved in
>>> the
>>>>> text
>>>>>>>>>> before
>>>>>>>>>>> adoption.
>>>>>>>>>>>
>>>>>>>>>>> Please note - I am maintaining my objection based on
>> missing
>>>>>>>>>>> specifications, not code bugs.
>>>>>>>>>>>
>>>>>>>>>>> I respectfully request that we pause the vote until:
>>>>>>>>>>>
>>>>>>>>>>> A valid design for Garbage Collection (replacing the
>>>>> discarded
>>>>>>>>>>> KIP-1165) is added to the proposal.
>>>>>>>>>>>
>>>>>>>>>>> The Transaction/LSO synchronization protocol is
>>> explicitly
>>>>>>>>>> documented
>>>>>>>>>>> in KIP-1164.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Vaquar Khan
>>>>>>>>>>> Sr Data Architect
>>>>>>>>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> [image: Aiven] <https://www.aiven.io>
>>>>>>>>
>>>>>>>> *Josep Prat*
>>>>>>>> Sr. Engineering Director, Streaming Services, *Aiven*
>>>>>>>> josep.prat@aiven.io | +491715557497
>>>>>>>> aiven.io <https://www.aiven.io> | <
>>>>>>> https://www.facebook.com/aivencloud
>>>>>>>>>
>>>>>>>> <https://www.linkedin.com/company/aiven/> <
>>>>>>>> https://twitter.com/aiven_io>
>>>>>>>> *Aiven Deutschland GmbH*
>>>>>>>> Alexanderufer 3-7, 10117 Berlin
>>>>>>>>
>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Anatolii Popov
>>>>>>> Senior Software Developer, *Aiven OY*
>>>>>>> m: +358505126242
>>>>>>> w: aiven.io e: anatolii.popov@aiven.io
>>>>>>> <https://www.facebook.com/aivencloud>
>>>>>>> <https://www.linkedin.com/company/aiven/> <
>>>>>> https://twitter.com/aiven_io>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Satish Duggana <sa...@gmail.com>.
Thanks for the KIP.
I've reviewed the updated KIP and agree with the motivation behind
KIP-1150, overall LGTM.
It seems KIP-1163 and KIP-1164 require more details, which we can discuss
in those respective threads.
+1(binding) for KIP-1150.
~Satish.
On Fri, 27 Feb 2026 at 23:28, Jun Rao via dev <de...@kafka.apache.org> wrote:
> Hi, Anatolii,
>
> Thanks for the KIP. The link you posted for KIP-1150 seems incorrect and it
> points to KIP-1163. Otherwise, +1.
>
> Jun
>
> On Wed, Feb 25, 2026 at 2:59 PM vaquar khan <va...@gmail.com> wrote:
>
> > Fair point, Chris. I agree with that architectural boundary. KIP-1150
> > successfully sets the high-level mandate , and we can rigorously tackle
> the
> > exact EOS and RPC mechanics over in the KIP-1164 thread .
> >
> > Andrew, I am fully aligned with you on the massive operational value of
> > eliminating those cross-AZ replication costs. It is absolutely the right
> > strategic direction for Kafka.
> >
> > Since my initial concerns on the storage side are resolved, and we are
> > aligned on where the transactional interfaces will be finalized, I am
> > officially withdrawing my objection.
> > +1 (non-binding) for KIP-1150.
> >
> > I will migrate my open questions over to the KIP-1164 discussion thread
> so
> > we can lock down the data safety details there.
> >
> > Regards,
> > Vaquar Khan
> >
> > On Wed, 25 Feb 2026 at 15:24, Chris Egerton <fe...@gmail.com>
> > wrote:
> >
> > > Hi Vaquar,
> > >
> > > > Let me know what you guys think about locking down the text for these
> > > interfaces.
> > >
> > > I think this KIP has the appropriate level of detail and any concerns
> > about
> > > EOS can be addressed in the relevant sub-KIP.
> > >
> > > Chris
> > >
> > > On Wed, Feb 25, 2026 at 4:20 PM vaquar khan <va...@gmail.com>
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > First off, thanks to the authors for the Feb 12th updates to
> KIP-1163 .
> > > > Adding the periodic reconciliation loop clears up my concerns about
> the
> > > > orphaned "Upload-then-Commit" segments, so I'm officially withdrawing
> > my
> > > > objection on the storage leak issue .
> > > >
> > > > Chris and Greg- since you both mentioned digging into the 1164
> > details, I
> > > > wanted to pick your brains on how Exactly-Once Semantics (EOS) is
> going
> > > to
> > > > safely operate here. In standard Kafka, the Partition Leader is our
> > > single
> > > > serialization point. It receives the data, tracks ongoing
> transactions
> > > via
> > > > the ProducerStateManager, and calculates the Last Stable Offset (LSO)
> > > > locally . Since KIP-1150 removes the leader, the Batch Coordinator
> > takes
> > > > over. But as I read through the current text, a few critical
> > > > synchronization barriers seem to be missing to me:
> > > >
> > > > 1. LSO Calculation: How exactly will the Batch Coordinator maintain
> and
> > > > calculate the LSO? Justine Olshan brought this up earlier too . Will
> > the
> > > > coordinator run its own ProducerStateManager to track ongoing
> > > transactions,
> > > > or is there a totally different state machine planned?
> > > >
> > > > 2. RPC Protocol: What's the exact synchronization protocol between
> the
> > > > legacy Transaction Coordinator and the new Batch Coordinator? When
> the
> > > Txn
> > > > Coordinator sends a commit marker, how does the Batch Coordinator
> > > actually
> > > > verify it has received all the prerequisite data batches for that
> > > specific
> > > > transaction epoch?
> > > >
> > > > 3. Delayed Data Race Condition: Let's say a broker hits a GC pause
> > right
> > > > *after
> > > > *uploading a batch to object storage, but *before* committing the
> > > > coordinates . If the transaction commit marker arrives at the
> > Coordinator
> > > > first, what happens? Does the Coordinator wait? If not, couldn't the
> > > > transaction commit with missing data, completely violating
> > read_committed
> > > > isolation?
> > > >
> > > > The KIP vaguely mentions *transactional checks* but leaves the actual
> > > > commit protocol and public interfaces undefined right now . I'm not
> > > saying
> > > > the design itself is broken, but I really think myself and others
> need
> > to
> > > > see these RPC flows explicitly documented before we implement and
> > adopt
> > > > this. Otherwise, we risk baking in some severe data isolation
> headaches
> > > > down the line.
> > > >
> > > > Let me know what you guys think about locking down the text for these
> > > > interfaces.
> > > >
> > > > Regards,
> > > > Vaquar Khan
> > > >
> > > > On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <
> > dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > I'm excited to discuss more details in 1163 and 1164 with everyone.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks!
> > > > > Greg
> > > > >
> > > > > On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <
> > > > > dev@kafka.apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Given the importance of this KIP, we want to keep the vote open
> > for a
> > > > few
> > > > > > more days to give time to people who had comments in the DISCUSS
> > > thread
> > > > > to
> > > > > > cast their vote if they want.
> > > > > >
> > > > > > On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <
> > > > > dev@kafka.apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > > As a co-author of the KIP, I want to explicitly cast my vote
> for
> > > this
> > > > > > KIP.
> > > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > I've re-read KIP-1150, and still agree this is what we need
> for
> > > > > Apache
> > > > > > > > Kafka.
> > > > > > > >
> > > > > > > > +1 (binding) from me.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > > Luke
> > > > > > > >
> > > > > > > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> > > > > > fearthecellos@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi all,
> > > > > > > >>
> > > > > > > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as
> > well
> > > as
> > > > > the
> > > > > > > >> relevant discussion threads. I may have granular comments
> > about
> > > > 1163
> > > > > > and
> > > > > > > >> 1164 but the overall approach suggested in 1150 looks good
> to
> > > me.
> > > > I
> > > > > > > >> especially like that the approach covers two main pain
> points
> > of
> > > > > > > operating
> > > > > > > >> and paying for Kafka today: it allows cross-AZ traffic to be
> > > > reduced
> > > > > > > (even
> > > > > > > >> eliminated in some cases), and it also allows local disk
> usage
> > > by
> > > > > > > brokers
> > > > > > > >> to be reduced (if operators opt for a small local cache on
> > > > follower
> > > > > > > >> brokers
> > > > > > > >> for non-tiered segments).
> > > > > > > >>
> > > > > > > >> +1 (binding)
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >>
> > > > > > > >> Chris
> > > > > > > >>
> > > > > > > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <
> > > > vaquar.khan@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Josep,
> > > > > > > >> >
> > > > > > > >> > Thank you for the detailed response. I appreciate the
> > > > > clarification
> > > > > > > >> > regarding the distinction between the Inkless POC and the
> > KIP
> > > > > > design.
> > > > > > > >> >
> > > > > > > >> > However, my objection is not based on temporary bugs in
> the
> > > > fork,
> > > > > > but
> > > > > > > >> *on
> > > > > > > >> > architectural gaps in the KIPs themselves* that these
> > > > > implementation
> > > > > > > >> issues
> > > > > > > >> > highlighted. If we are voting to approve the design, the
> > > design
> > > > > > > >> documents
> > > > > > > >> > must be structurally complete regarding data safety.
> > > > > > > >> >
> > > > > > > >> > *1. Regarding Storage Leaks (The Missing Design)* You
> > > mentioned
> > > > > that
> > > > > > > >> > cleanup logic "can be defined later." However, KIP-1163
> > > > explicitly
> > > > > > > >> > delegates this responsibility to a separate process, and
> > > > KIP-1165
> > > > > > > >> (Object
> > > > > > > >> > Compaction/GC) is currently marked as "Discarded" in the
> > wiki.
> > > > > > > >> >
> > > > > > > >> > We cannot vote to approve a storage engine that has no
> > > specified
> > > > > > > >> mechanism
> > > > > > > >> > for garbage collection. The "Upload-then-Commit" pattern
> > > > described
> > > > > > in
> > > > > > > >> > KIP-1163 structurally creates orphaned segments during
> > broker
> > > > > > > failures.
> > > > > > > >> > Without an active KIP defining the reconciliation protocol
> > > > (since
> > > > > > > >> KIP-1165
> > > > > > > >> > was withdrawn), the proposal effectively describes a
> system
> > > with
> > > > > > > >> unbounded
> > > > > > > >> > storage growth during failure modes. This is a blocking
> > design
> > > > > gap,
> > > > > > > not
> > > > > > > >> an
> > > > > > > >> > implementation detail.
> > > > > > > >> >
> > > > > > > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)*
> > This
> > > is
> > > > > > not a
> > > > > > > >> > misunderstanding of standard Kafka transactions; it is a
> > > > critique
> > > > > of
> > > > > > > how
> > > > > > > >> > KIP-1150 changes them. Standard EOS relies on the
> Partition
> > > > Leader
> > > > > > to
> > > > > > > >> > sequence markers and calculate the LSO (Last Stable
> Offset)
> > in
> > > > > > memory.
> > > > > > > >> > KIP-1150 removes the Leader.
> > > > > > > >> >
> > > > > > > >> > KIP-1164 (Batch Coordinator) must explicitly define the
> RPC
> > > flow
> > > > > > > between
> > > > > > > >> > the Transaction Coordinator and the Batch Coordinator to
> > > replace
> > > > > the
> > > > > > > >> > leader's role. Currently, the KIP does not specify how the
> > > > system
> > > > > > > >> prevents
> > > > > > > >> > a "Split Brain" scenario where a consumer reads ahead of a
> > > > > > transaction
> > > > > > > >> > marker that hasn't yet been sequenced by the Batch
> > > Coordinator.
> > > > > This
> > > > > > > is
> > > > > > > >> a
> > > > > > > >> > protocol-level correctness issue that must be resolved in
> > the
> > > > text
> > > > > > > >> before
> > > > > > > >> > adoption.
> > > > > > > >> >
> > > > > > > >> > Please note - I am maintaining my objection based on
> missing
> > > > > > > >> > specifications, not code bugs.
> > > > > > > >> >
> > > > > > > >> > I respectfully request that we pause the vote until:
> > > > > > > >> >
> > > > > > > >> > A valid design for Garbage Collection (replacing the
> > > > discarded
> > > > > > > >> > KIP-1165) is added to the proposal.
> > > > > > > >> >
> > > > > > > >> > The Transaction/LSO synchronization protocol is
> > explicitly
> > > > > > > >> documented
> > > > > > > >> > in KIP-1164.
> > > > > > > >> >
> > > > > > > >> > Regards,
> > > > > > > >> >
> > > > > > > >> > Vaquar Khan
> > > > > > > >> > Sr Data Architect
> > > > > > > >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > >
> > > > > > > *Josep Prat*
> > > > > > > Sr. Engineering Director, Streaming Services, *Aiven*
> > > > > > > josep.prat@aiven.io | +491715557497
> > > > > > > aiven.io <https://www.aiven.io> | <
> > > > > > https://www.facebook.com/aivencloud
> > > > > > > >
> > > > > > > <https://www.linkedin.com/company/aiven/> <
> > > > > > > https://twitter.com/aiven_io>
> > > > > > > *Aiven Deutschland GmbH*
> > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > >
> > > > > > > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Anatolii Popov
> > > > > > Senior Software Developer, *Aiven OY*
> > > > > > m: +358505126242
> > > > > > w: aiven.io e: anatolii.popov@aiven.io
> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > <https://www.linkedin.com/company/aiven/> <
> > > > > https://twitter.com/aiven_io>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Anatolii,
Thanks for the KIP. The link you posted for KIP-1150 seems incorrect and it
points to KIP-1163. Otherwise, +1.
Jun
On Wed, Feb 25, 2026 at 2:59 PM vaquar khan <va...@gmail.com> wrote:
> Fair point, Chris. I agree with that architectural boundary. KIP-1150
> successfully sets the high-level mandate , and we can rigorously tackle the
> exact EOS and RPC mechanics over in the KIP-1164 thread .
>
> Andrew, I am fully aligned with you on the massive operational value of
> eliminating those cross-AZ replication costs. It is absolutely the right
> strategic direction for Kafka.
>
> Since my initial concerns on the storage side are resolved, and we are
> aligned on where the transactional interfaces will be finalized, I am
> officially withdrawing my objection.
> +1 (non-binding) for KIP-1150.
>
> I will migrate my open questions over to the KIP-1164 discussion thread so
> we can lock down the data safety details there.
>
> Regards,
> Vaquar Khan
>
> On Wed, 25 Feb 2026 at 15:24, Chris Egerton <fe...@gmail.com>
> wrote:
>
> > Hi Vaquar,
> >
> > > Let me know what you guys think about locking down the text for these
> > interfaces.
> >
> > I think this KIP has the appropriate level of detail and any concerns
> about
> > EOS can be addressed in the relevant sub-KIP.
> >
> > Chris
> >
> > On Wed, Feb 25, 2026 at 4:20 PM vaquar khan <va...@gmail.com>
> wrote:
> >
> > > Hi everyone,
> > >
> > > First off, thanks to the authors for the Feb 12th updates to KIP-1163 .
> > > Adding the periodic reconciliation loop clears up my concerns about the
> > > orphaned "Upload-then-Commit" segments, so I'm officially withdrawing
> my
> > > objection on the storage leak issue .
> > >
> > > Chris and Greg- since you both mentioned digging into the 1164
> details, I
> > > wanted to pick your brains on how Exactly-Once Semantics (EOS) is going
> > to
> > > safely operate here. In standard Kafka, the Partition Leader is our
> > single
> > > serialization point. It receives the data, tracks ongoing transactions
> > via
> > > the ProducerStateManager, and calculates the Last Stable Offset (LSO)
> > > locally . Since KIP-1150 removes the leader, the Batch Coordinator
> takes
> > > over. But as I read through the current text, a few critical
> > > synchronization barriers seem to be missing to me:
> > >
> > > 1. LSO Calculation: How exactly will the Batch Coordinator maintain and
> > > calculate the LSO? Justine Olshan brought this up earlier too . Will
> the
> > > coordinator run its own ProducerStateManager to track ongoing
> > transactions,
> > > or is there a totally different state machine planned?
> > >
> > > 2. RPC Protocol: What's the exact synchronization protocol between the
> > > legacy Transaction Coordinator and the new Batch Coordinator? When the
> > Txn
> > > Coordinator sends a commit marker, how does the Batch Coordinator
> > actually
> > > verify it has received all the prerequisite data batches for that
> > specific
> > > transaction epoch?
> > >
> > > 3. Delayed Data Race Condition: Let's say a broker hits a GC pause
> right
> > > *after
> > > *uploading a batch to object storage, but *before* committing the
> > > coordinates . If the transaction commit marker arrives at the
> Coordinator
> > > first, what happens? Does the Coordinator wait? If not, couldn't the
> > > transaction commit with missing data, completely violating
> read_committed
> > > isolation?
> > >
> > > The KIP vaguely mentions *transactional checks* but leaves the actual
> > > commit protocol and public interfaces undefined right now . I'm not
> > saying
> > > the design itself is broken, but I really think myself and others need
> to
> > > see these RPC flows explicitly documented before we implement and
> adopt
> > > this. Otherwise, we risk baking in some severe data isolation headaches
> > > down the line.
> > >
> > > Let me know what you guys think about locking down the text for these
> > > interfaces.
> > >
> > > Regards,
> > > Vaquar Khan
> > >
> > > On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <
> dev@kafka.apache.org>
> > > wrote:
> > >
> > > > Hey all,
> > > >
> > > > I'm excited to discuss more details in 1163 and 1164 with everyone.
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks!
> > > > Greg
> > > >
> > > > On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <
> > > > dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Given the importance of this KIP, we want to keep the vote open
> for a
> > > few
> > > > > more days to give time to people who had comments in the DISCUSS
> > thread
> > > > to
> > > > > cast their vote if they want.
> > > > >
> > > > > On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <
> > > > dev@kafka.apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > > As a co-author of the KIP, I want to explicitly cast my vote for
> > this
> > > > > KIP.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > I've re-read KIP-1150, and still agree this is what we need for
> > > > Apache
> > > > > > > Kafka.
> > > > > > >
> > > > > > > +1 (binding) from me.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> > > > > fearthecellos@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as
> well
> > as
> > > > the
> > > > > > >> relevant discussion threads. I may have granular comments
> about
> > > 1163
> > > > > and
> > > > > > >> 1164 but the overall approach suggested in 1150 looks good to
> > me.
> > > I
> > > > > > >> especially like that the approach covers two main pain points
> of
> > > > > > operating
> > > > > > >> and paying for Kafka today: it allows cross-AZ traffic to be
> > > reduced
> > > > > > (even
> > > > > > >> eliminated in some cases), and it also allows local disk usage
> > by
> > > > > > brokers
> > > > > > >> to be reduced (if operators opt for a small local cache on
> > > follower
> > > > > > >> brokers
> > > > > > >> for non-tiered segments).
> > > > > > >>
> > > > > > >> +1 (binding)
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >>
> > > > > > >> Chris
> > > > > > >>
> > > > > > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <
> > > vaquar.khan@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Josep,
> > > > > > >> >
> > > > > > >> > Thank you for the detailed response. I appreciate the
> > > > clarification
> > > > > > >> > regarding the distinction between the Inkless POC and the
> KIP
> > > > > design.
> > > > > > >> >
> > > > > > >> > However, my objection is not based on temporary bugs in the
> > > fork,
> > > > > but
> > > > > > >> *on
> > > > > > >> > architectural gaps in the KIPs themselves* that these
> > > > implementation
> > > > > > >> issues
> > > > > > >> > highlighted. If we are voting to approve the design, the
> > design
> > > > > > >> documents
> > > > > > >> > must be structurally complete regarding data safety.
> > > > > > >> >
> > > > > > >> > *1. Regarding Storage Leaks (The Missing Design)* You
> > mentioned
> > > > that
> > > > > > >> > cleanup logic "can be defined later." However, KIP-1163
> > > explicitly
> > > > > > >> > delegates this responsibility to a separate process, and
> > > KIP-1165
> > > > > > >> (Object
> > > > > > >> > Compaction/GC) is currently marked as "Discarded" in the
> wiki.
> > > > > > >> >
> > > > > > >> > We cannot vote to approve a storage engine that has no
> > specified
> > > > > > >> mechanism
> > > > > > >> > for garbage collection. The "Upload-then-Commit" pattern
> > > described
> > > > > in
> > > > > > >> > KIP-1163 structurally creates orphaned segments during
> broker
> > > > > > failures.
> > > > > > >> > Without an active KIP defining the reconciliation protocol
> > > (since
> > > > > > >> KIP-1165
> > > > > > >> > was withdrawn), the proposal effectively describes a system
> > with
> > > > > > >> unbounded
> > > > > > >> > storage growth during failure modes. This is a blocking
> design
> > > > gap,
> > > > > > not
> > > > > > >> an
> > > > > > >> > implementation detail.
> > > > > > >> >
> > > > > > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)*
> This
> > is
> > > > > not a
> > > > > > >> > misunderstanding of standard Kafka transactions; it is a
> > > critique
> > > > of
> > > > > > how
> > > > > > >> > KIP-1150 changes them. Standard EOS relies on the Partition
> > > Leader
> > > > > to
> > > > > > >> > sequence markers and calculate the LSO (Last Stable Offset)
> in
> > > > > memory.
> > > > > > >> > KIP-1150 removes the Leader.
> > > > > > >> >
> > > > > > >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC
> > flow
> > > > > > between
> > > > > > >> > the Transaction Coordinator and the Batch Coordinator to
> > replace
> > > > the
> > > > > > >> > leader's role. Currently, the KIP does not specify how the
> > > system
> > > > > > >> prevents
> > > > > > >> > a "Split Brain" scenario where a consumer reads ahead of a
> > > > > transaction
> > > > > > >> > marker that hasn't yet been sequenced by the Batch
> > Coordinator.
> > > > This
> > > > > > is
> > > > > > >> a
> > > > > > >> > protocol-level correctness issue that must be resolved in
> the
> > > text
> > > > > > >> before
> > > > > > >> > adoption.
> > > > > > >> >
> > > > > > >> > Please note - I am maintaining my objection based on missing
> > > > > > >> > specifications, not code bugs.
> > > > > > >> >
> > > > > > >> > I respectfully request that we pause the vote until:
> > > > > > >> >
> > > > > > >> > A valid design for Garbage Collection (replacing the
> > > discarded
> > > > > > >> > KIP-1165) is added to the proposal.
> > > > > > >> >
> > > > > > >> > The Transaction/LSO synchronization protocol is
> explicitly
> > > > > > >> documented
> > > > > > >> > in KIP-1164.
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> >
> > > > > > >> > Vaquar Khan
> > > > > > >> > Sr Data Architect
> > > > > > >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > >
> > > > > > *Josep Prat*
> > > > > > Sr. Engineering Director, Streaming Services, *Aiven*
> > > > > > josep.prat@aiven.io | +491715557497
> > > > > > aiven.io <https://www.aiven.io> | <
> > > > > https://www.facebook.com/aivencloud
> > > > > > >
> > > > > > <https://www.linkedin.com/company/aiven/> <
> > > > > > https://twitter.com/aiven_io>
> > > > > > *Aiven Deutschland GmbH*
> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > >
> > > > > > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Anatolii Popov
> > > > > Senior Software Developer, *Aiven OY*
> > > > > m: +358505126242
> > > > > w: aiven.io e: anatolii.popov@aiven.io
> > > > > <https://www.facebook.com/aivencloud>
> > > > > <https://www.linkedin.com/company/aiven/> <
> > > > https://twitter.com/aiven_io>
> > > > >
> > > >
> > >
> >
>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by vaquar khan <va...@gmail.com>.
Fair point, Chris. I agree with that architectural boundary. KIP-1150
successfully sets the high-level mandate , and we can rigorously tackle the
exact EOS and RPC mechanics over in the KIP-1164 thread .
Andrew, I am fully aligned with you on the massive operational value of
eliminating those cross-AZ replication costs. It is absolutely the right
strategic direction for Kafka.
Since my initial concerns on the storage side are resolved, and we are
aligned on where the transactional interfaces will be finalized, I am
officially withdrawing my objection.
+1 (non-binding) for KIP-1150.
I will migrate my open questions over to the KIP-1164 discussion thread so
we can lock down the data safety details there.
Regards,
Vaquar Khan
On Wed, 25 Feb 2026 at 15:24, Chris Egerton <fe...@gmail.com> wrote:
> Hi Vaquar,
>
> > Let me know what you guys think about locking down the text for these
> interfaces.
>
> I think this KIP has the appropriate level of detail and any concerns about
> EOS can be addressed in the relevant sub-KIP.
>
> Chris
>
> On Wed, Feb 25, 2026 at 4:20 PM vaquar khan <va...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > First off, thanks to the authors for the Feb 12th updates to KIP-1163 .
> > Adding the periodic reconciliation loop clears up my concerns about the
> > orphaned "Upload-then-Commit" segments, so I'm officially withdrawing my
> > objection on the storage leak issue .
> >
> > Chris and Greg- since you both mentioned digging into the 1164 details, I
> > wanted to pick your brains on how Exactly-Once Semantics (EOS) is going
> to
> > safely operate here. In standard Kafka, the Partition Leader is our
> single
> > serialization point. It receives the data, tracks ongoing transactions
> via
> > the ProducerStateManager, and calculates the Last Stable Offset (LSO)
> > locally . Since KIP-1150 removes the leader, the Batch Coordinator takes
> > over. But as I read through the current text, a few critical
> > synchronization barriers seem to be missing to me:
> >
> > 1. LSO Calculation: How exactly will the Batch Coordinator maintain and
> > calculate the LSO? Justine Olshan brought this up earlier too . Will the
> > coordinator run its own ProducerStateManager to track ongoing
> transactions,
> > or is there a totally different state machine planned?
> >
> > 2. RPC Protocol: What's the exact synchronization protocol between the
> > legacy Transaction Coordinator and the new Batch Coordinator? When the
> Txn
> > Coordinator sends a commit marker, how does the Batch Coordinator
> actually
> > verify it has received all the prerequisite data batches for that
> specific
> > transaction epoch?
> >
> > 3. Delayed Data Race Condition: Let's say a broker hits a GC pause right
> > *after
> > *uploading a batch to object storage, but *before* committing the
> > coordinates . If the transaction commit marker arrives at the Coordinator
> > first, what happens? Does the Coordinator wait? If not, couldn't the
> > transaction commit with missing data, completely violating read_committed
> > isolation?
> >
> > The KIP vaguely mentions *transactional checks* but leaves the actual
> > commit protocol and public interfaces undefined right now . I'm not
> saying
> > the design itself is broken, but I really think myself and others need to
> > see these RPC flows explicitly documented before we implement and adopt
> > this. Otherwise, we risk baking in some severe data isolation headaches
> > down the line.
> >
> > Let me know what you guys think about locking down the text for these
> > interfaces.
> >
> > Regards,
> > Vaquar Khan
> >
> > On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hey all,
> > >
> > > I'm excited to discuss more details in 1163 and 1164 with everyone.
> > >
> > > +1 (binding)
> > >
> > > Thanks!
> > > Greg
> > >
> > > On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <
> > > dev@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Given the importance of this KIP, we want to keep the vote open for a
> > few
> > > > more days to give time to people who had comments in the DISCUSS
> thread
> > > to
> > > > cast their vote if they want.
> > > >
> > > > On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <
> > > dev@kafka.apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > > As a co-author of the KIP, I want to explicitly cast my vote for
> this
> > > > KIP.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > >
> > > > > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > I've re-read KIP-1150, and still agree this is what we need for
> > > Apache
> > > > > > Kafka.
> > > > > >
> > > > > > +1 (binding) from me.
> > > > > >
> > > > > > Thank you,
> > > > > > Luke
> > > > > >
> > > > > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> > > > fearthecellos@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well
> as
> > > the
> > > > > >> relevant discussion threads. I may have granular comments about
> > 1163
> > > > and
> > > > > >> 1164 but the overall approach suggested in 1150 looks good to
> me.
> > I
> > > > > >> especially like that the approach covers two main pain points of
> > > > > operating
> > > > > >> and paying for Kafka today: it allows cross-AZ traffic to be
> > reduced
> > > > > (even
> > > > > >> eliminated in some cases), and it also allows local disk usage
> by
> > > > > brokers
> > > > > >> to be reduced (if operators opt for a small local cache on
> > follower
> > > > > >> brokers
> > > > > >> for non-tiered segments).
> > > > > >>
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> Cheers,
> > > > > >>
> > > > > >> Chris
> > > > > >>
> > > > > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <
> > vaquar.khan@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Josep,
> > > > > >> >
> > > > > >> > Thank you for the detailed response. I appreciate the
> > > clarification
> > > > > >> > regarding the distinction between the Inkless POC and the KIP
> > > > design.
> > > > > >> >
> > > > > >> > However, my objection is not based on temporary bugs in the
> > fork,
> > > > but
> > > > > >> *on
> > > > > >> > architectural gaps in the KIPs themselves* that these
> > > implementation
> > > > > >> issues
> > > > > >> > highlighted. If we are voting to approve the design, the
> design
> > > > > >> documents
> > > > > >> > must be structurally complete regarding data safety.
> > > > > >> >
> > > > > >> > *1. Regarding Storage Leaks (The Missing Design)* You
> mentioned
> > > that
> > > > > >> > cleanup logic "can be defined later." However, KIP-1163
> > explicitly
> > > > > >> > delegates this responsibility to a separate process, and
> > KIP-1165
> > > > > >> (Object
> > > > > >> > Compaction/GC) is currently marked as "Discarded" in the wiki.
> > > > > >> >
> > > > > >> > We cannot vote to approve a storage engine that has no
> specified
> > > > > >> mechanism
> > > > > >> > for garbage collection. The "Upload-then-Commit" pattern
> > described
> > > > in
> > > > > >> > KIP-1163 structurally creates orphaned segments during broker
> > > > > failures.
> > > > > >> > Without an active KIP defining the reconciliation protocol
> > (since
> > > > > >> KIP-1165
> > > > > >> > was withdrawn), the proposal effectively describes a system
> with
> > > > > >> unbounded
> > > > > >> > storage growth during failure modes. This is a blocking design
> > > gap,
> > > > > not
> > > > > >> an
> > > > > >> > implementation detail.
> > > > > >> >
> > > > > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This
> is
> > > > not a
> > > > > >> > misunderstanding of standard Kafka transactions; it is a
> > critique
> > > of
> > > > > how
> > > > > >> > KIP-1150 changes them. Standard EOS relies on the Partition
> > Leader
> > > > to
> > > > > >> > sequence markers and calculate the LSO (Last Stable Offset) in
> > > > memory.
> > > > > >> > KIP-1150 removes the Leader.
> > > > > >> >
> > > > > >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC
> flow
> > > > > between
> > > > > >> > the Transaction Coordinator and the Batch Coordinator to
> replace
> > > the
> > > > > >> > leader's role. Currently, the KIP does not specify how the
> > system
> > > > > >> prevents
> > > > > >> > a "Split Brain" scenario where a consumer reads ahead of a
> > > > transaction
> > > > > >> > marker that hasn't yet been sequenced by the Batch
> Coordinator.
> > > This
> > > > > is
> > > > > >> a
> > > > > >> > protocol-level correctness issue that must be resolved in the
> > text
> > > > > >> before
> > > > > >> > adoption.
> > > > > >> >
> > > > > >> > Please note - I am maintaining my objection based on missing
> > > > > >> > specifications, not code bugs.
> > > > > >> >
> > > > > >> > I respectfully request that we pause the vote until:
> > > > > >> >
> > > > > >> > A valid design for Garbage Collection (replacing the
> > discarded
> > > > > >> > KIP-1165) is added to the proposal.
> > > > > >> >
> > > > > >> > The Transaction/LSO synchronization protocol is explicitly
> > > > > >> documented
> > > > > >> > in KIP-1164.
> > > > > >> >
> > > > > >> > Regards,
> > > > > >> >
> > > > > >> > Vaquar Khan
> > > > > >> > Sr Data Architect
> > > > > >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > > > --
> > > > > [image: Aiven] <https://www.aiven.io>
> > > > >
> > > > > *Josep Prat*
> > > > > Sr. Engineering Director, Streaming Services, *Aiven*
> > > > > josep.prat@aiven.io | +491715557497
> > > > > aiven.io <https://www.aiven.io> | <
> > > > https://www.facebook.com/aivencloud
> > > > > >
> > > > > <https://www.linkedin.com/company/aiven/> <
> > > > > https://twitter.com/aiven_io>
> > > > > *Aiven Deutschland GmbH*
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > >
> > > >
> > > > --
> > > > Anatolii Popov
> > > > Senior Software Developer, *Aiven OY*
> > > > m: +358505126242
> > > > w: aiven.io e: anatolii.popov@aiven.io
> > > > <https://www.facebook.com/aivencloud>
> > > > <https://www.linkedin.com/company/aiven/> <
> > > https://twitter.com/aiven_io>
> > > >
> > >
> >
>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Chris Egerton <fe...@gmail.com>.
Hi Vaquar,
> Let me know what you guys think about locking down the text for these
interfaces.
I think this KIP has the appropriate level of detail and any concerns about
EOS can be addressed in the relevant sub-KIP.
Chris
On Wed, Feb 25, 2026 at 4:20 PM vaquar khan <va...@gmail.com> wrote:
> Hi everyone,
>
> First off, thanks to the authors for the Feb 12th updates to KIP-1163 .
> Adding the periodic reconciliation loop clears up my concerns about the
> orphaned "Upload-then-Commit" segments, so I'm officially withdrawing my
> objection on the storage leak issue .
>
> Chris and Greg- since you both mentioned digging into the 1164 details, I
> wanted to pick your brains on how Exactly-Once Semantics (EOS) is going to
> safely operate here. In standard Kafka, the Partition Leader is our single
> serialization point. It receives the data, tracks ongoing transactions via
> the ProducerStateManager, and calculates the Last Stable Offset (LSO)
> locally . Since KIP-1150 removes the leader, the Batch Coordinator takes
> over. But as I read through the current text, a few critical
> synchronization barriers seem to be missing to me:
>
> 1. LSO Calculation: How exactly will the Batch Coordinator maintain and
> calculate the LSO? Justine Olshan brought this up earlier too . Will the
> coordinator run its own ProducerStateManager to track ongoing transactions,
> or is there a totally different state machine planned?
>
> 2. RPC Protocol: What's the exact synchronization protocol between the
> legacy Transaction Coordinator and the new Batch Coordinator? When the Txn
> Coordinator sends a commit marker, how does the Batch Coordinator actually
> verify it has received all the prerequisite data batches for that specific
> transaction epoch?
>
> 3. Delayed Data Race Condition: Let's say a broker hits a GC pause right
> *after
> *uploading a batch to object storage, but *before* committing the
> coordinates . If the transaction commit marker arrives at the Coordinator
> first, what happens? Does the Coordinator wait? If not, couldn't the
> transaction commit with missing data, completely violating read_committed
> isolation?
>
> The KIP vaguely mentions *transactional checks* but leaves the actual
> commit protocol and public interfaces undefined right now . I'm not saying
> the design itself is broken, but I really think myself and others need to
> see these RPC flows explicitly documented before we implement and adopt
> this. Otherwise, we risk baking in some severe data isolation headaches
> down the line.
>
> Let me know what you guys think about locking down the text for these
> interfaces.
>
> Regards,
> Vaquar Khan
>
> On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <de...@kafka.apache.org>
> wrote:
>
> > Hey all,
> >
> > I'm excited to discuss more details in 1163 and 1164 with everyone.
> >
> > +1 (binding)
> >
> > Thanks!
> > Greg
> >
> > On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <
> > dev@kafka.apache.org>
> > wrote:
> >
> > > Hi all,
> > >
> > > Given the importance of this KIP, we want to keep the vote open for a
> few
> > > more days to give time to people who had comments in the DISCUSS thread
> > to
> > > cast their vote if they want.
> > >
> > > On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <
> > dev@kafka.apache.org>
> > > wrote:
> > >
> > > > Hi all,
> > > > As a co-author of the KIP, I want to explicitly cast my vote for this
> > > KIP.
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com> wrote:
> > > >
> > > > > I've re-read KIP-1150, and still agree this is what we need for
> > Apache
> > > > > Kafka.
> > > > >
> > > > > +1 (binding) from me.
> > > > >
> > > > > Thank you,
> > > > > Luke
> > > > >
> > > > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> > > fearthecellos@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as
> > the
> > > > >> relevant discussion threads. I may have granular comments about
> 1163
> > > and
> > > > >> 1164 but the overall approach suggested in 1150 looks good to me.
> I
> > > > >> especially like that the approach covers two main pain points of
> > > > operating
> > > > >> and paying for Kafka today: it allows cross-AZ traffic to be
> reduced
> > > > (even
> > > > >> eliminated in some cases), and it also allows local disk usage by
> > > > brokers
> > > > >> to be reduced (if operators opt for a small local cache on
> follower
> > > > >> brokers
> > > > >> for non-tiered segments).
> > > > >>
> > > > >> +1 (binding)
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Chris
> > > > >>
> > > > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <
> vaquar.khan@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Josep,
> > > > >> >
> > > > >> > Thank you for the detailed response. I appreciate the
> > clarification
> > > > >> > regarding the distinction between the Inkless POC and the KIP
> > > design.
> > > > >> >
> > > > >> > However, my objection is not based on temporary bugs in the
> fork,
> > > but
> > > > >> *on
> > > > >> > architectural gaps in the KIPs themselves* that these
> > implementation
> > > > >> issues
> > > > >> > highlighted. If we are voting to approve the design, the design
> > > > >> documents
> > > > >> > must be structurally complete regarding data safety.
> > > > >> >
> > > > >> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned
> > that
> > > > >> > cleanup logic "can be defined later." However, KIP-1163
> explicitly
> > > > >> > delegates this responsibility to a separate process, and
> KIP-1165
> > > > >> (Object
> > > > >> > Compaction/GC) is currently marked as "Discarded" in the wiki.
> > > > >> >
> > > > >> > We cannot vote to approve a storage engine that has no specified
> > > > >> mechanism
> > > > >> > for garbage collection. The "Upload-then-Commit" pattern
> described
> > > in
> > > > >> > KIP-1163 structurally creates orphaned segments during broker
> > > > failures.
> > > > >> > Without an active KIP defining the reconciliation protocol
> (since
> > > > >> KIP-1165
> > > > >> > was withdrawn), the proposal effectively describes a system with
> > > > >> unbounded
> > > > >> > storage growth during failure modes. This is a blocking design
> > gap,
> > > > not
> > > > >> an
> > > > >> > implementation detail.
> > > > >> >
> > > > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is
> > > not a
> > > > >> > misunderstanding of standard Kafka transactions; it is a
> critique
> > of
> > > > how
> > > > >> > KIP-1150 changes them. Standard EOS relies on the Partition
> Leader
> > > to
> > > > >> > sequence markers and calculate the LSO (Last Stable Offset) in
> > > memory.
> > > > >> > KIP-1150 removes the Leader.
> > > > >> >
> > > > >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow
> > > > between
> > > > >> > the Transaction Coordinator and the Batch Coordinator to replace
> > the
> > > > >> > leader's role. Currently, the KIP does not specify how the
> system
> > > > >> prevents
> > > > >> > a "Split Brain" scenario where a consumer reads ahead of a
> > > transaction
> > > > >> > marker that hasn't yet been sequenced by the Batch Coordinator.
> > This
> > > > is
> > > > >> a
> > > > >> > protocol-level correctness issue that must be resolved in the
> text
> > > > >> before
> > > > >> > adoption.
> > > > >> >
> > > > >> > Please note - I am maintaining my objection based on missing
> > > > >> > specifications, not code bugs.
> > > > >> >
> > > > >> > I respectfully request that we pause the vote until:
> > > > >> >
> > > > >> > A valid design for Garbage Collection (replacing the
> discarded
> > > > >> > KIP-1165) is added to the proposal.
> > > > >> >
> > > > >> > The Transaction/LSO synchronization protocol is explicitly
> > > > >> documented
> > > > >> > in KIP-1164.
> > > > >> >
> > > > >> > Regards,
> > > > >> >
> > > > >> > Vaquar Khan
> > > > >> > Sr Data Architect
> > > > >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> > > > >> >
> > > > >>
> > > > >
> > > >
> > > > --
> > > > [image: Aiven] <https://www.aiven.io>
> > > >
> > > > *Josep Prat*
> > > > Sr. Engineering Director, Streaming Services, *Aiven*
> > > > josep.prat@aiven.io | +491715557497
> > > > aiven.io <https://www.aiven.io> | <
> > > https://www.facebook.com/aivencloud
> > > > >
> > > > <https://www.linkedin.com/company/aiven/> <
> > > > https://twitter.com/aiven_io>
> > > > *Aiven Deutschland GmbH*
> > > > Alexanderufer 3-7, 10117 Berlin
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
> > >
> > > --
> > > Anatolii Popov
> > > Senior Software Developer, *Aiven OY*
> > > m: +358505126242
> > > w: aiven.io e: anatolii.popov@aiven.io
> > > <https://www.facebook.com/aivencloud>
> > > <https://www.linkedin.com/company/aiven/> <
> > https://twitter.com/aiven_io>
> > >
> >
>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by vaquar khan <va...@gmail.com>.
Hi everyone,
First off, thanks to the authors for the Feb 12th updates to KIP-1163 .
Adding the periodic reconciliation loop clears up my concerns about the
orphaned "Upload-then-Commit" segments, so I'm officially withdrawing my
objection on the storage leak issue .
Chris and Greg- since you both mentioned digging into the 1164 details, I
wanted to pick your brains on how Exactly-Once Semantics (EOS) is going to
safely operate here. In standard Kafka, the Partition Leader is our single
serialization point. It receives the data, tracks ongoing transactions via
the ProducerStateManager, and calculates the Last Stable Offset (LSO)
locally . Since KIP-1150 removes the leader, the Batch Coordinator takes
over. But as I read through the current text, a few critical
synchronization barriers seem to be missing to me:
1. LSO Calculation: How exactly will the Batch Coordinator maintain and
calculate the LSO? Justine Olshan brought this up earlier too . Will the
coordinator run its own ProducerStateManager to track ongoing transactions,
or is there a totally different state machine planned?
2. RPC Protocol: What's the exact synchronization protocol between the
legacy Transaction Coordinator and the new Batch Coordinator? When the Txn
Coordinator sends a commit marker, how does the Batch Coordinator actually
verify it has received all the prerequisite data batches for that specific
transaction epoch?
3. Delayed Data Race Condition: Let's say a broker hits a GC pause right *after
*uploading a batch to object storage, but *before* committing the
coordinates . If the transaction commit marker arrives at the Coordinator
first, what happens? Does the Coordinator wait? If not, couldn't the
transaction commit with missing data, completely violating read_committed
isolation?
The KIP vaguely mentions *transactional checks* but leaves the actual
commit protocol and public interfaces undefined right now . I'm not saying
the design itself is broken, but I really think myself and others need to
see these RPC flows explicitly documented before we implement and adopt
this. Otherwise, we risk baking in some severe data isolation headaches
down the line.
Let me know what you guys think about locking down the text for these
interfaces.
Regards,
Vaquar Khan
On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <de...@kafka.apache.org>
wrote:
> Hey all,
>
> I'm excited to discuss more details in 1163 and 1164 with everyone.
>
> +1 (binding)
>
> Thanks!
> Greg
>
> On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <
> dev@kafka.apache.org>
> wrote:
>
> > Hi all,
> >
> > Given the importance of this KIP, we want to keep the vote open for a few
> > more days to give time to people who had comments in the DISCUSS thread
> to
> > cast their vote if they want.
> >
> > On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <
> dev@kafka.apache.org>
> > wrote:
> >
> > > Hi all,
> > > As a co-author of the KIP, I want to explicitly cast my vote for this
> > KIP.
> > >
> > > +1 (binding)
> > >
> > >
> > > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > I've re-read KIP-1150, and still agree this is what we need for
> Apache
> > > > Kafka.
> > > >
> > > > +1 (binding) from me.
> > > >
> > > > Thank you,
> > > > Luke
> > > >
> > > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> > fearthecellos@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as
> the
> > > >> relevant discussion threads. I may have granular comments about 1163
> > and
> > > >> 1164 but the overall approach suggested in 1150 looks good to me. I
> > > >> especially like that the approach covers two main pain points of
> > > operating
> > > >> and paying for Kafka today: it allows cross-AZ traffic to be reduced
> > > (even
> > > >> eliminated in some cases), and it also allows local disk usage by
> > > brokers
> > > >> to be reduced (if operators opt for a small local cache on follower
> > > >> brokers
> > > >> for non-tiered segments).
> > > >>
> > > >> +1 (binding)
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Chris
> > > >>
> > > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <va...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Josep,
> > > >> >
> > > >> > Thank you for the detailed response. I appreciate the
> clarification
> > > >> > regarding the distinction between the Inkless POC and the KIP
> > design.
> > > >> >
> > > >> > However, my objection is not based on temporary bugs in the fork,
> > but
> > > >> *on
> > > >> > architectural gaps in the KIPs themselves* that these
> implementation
> > > >> issues
> > > >> > highlighted. If we are voting to approve the design, the design
> > > >> documents
> > > >> > must be structurally complete regarding data safety.
> > > >> >
> > > >> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned
> that
> > > >> > cleanup logic "can be defined later." However, KIP-1163 explicitly
> > > >> > delegates this responsibility to a separate process, and KIP-1165
> > > >> (Object
> > > >> > Compaction/GC) is currently marked as "Discarded" in the wiki.
> > > >> >
> > > >> > We cannot vote to approve a storage engine that has no specified
> > > >> mechanism
> > > >> > for garbage collection. The "Upload-then-Commit" pattern described
> > in
> > > >> > KIP-1163 structurally creates orphaned segments during broker
> > > failures.
> > > >> > Without an active KIP defining the reconciliation protocol (since
> > > >> KIP-1165
> > > >> > was withdrawn), the proposal effectively describes a system with
> > > >> unbounded
> > > >> > storage growth during failure modes. This is a blocking design
> gap,
> > > not
> > > >> an
> > > >> > implementation detail.
> > > >> >
> > > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is
> > not a
> > > >> > misunderstanding of standard Kafka transactions; it is a critique
> of
> > > how
> > > >> > KIP-1150 changes them. Standard EOS relies on the Partition Leader
> > to
> > > >> > sequence markers and calculate the LSO (Last Stable Offset) in
> > memory.
> > > >> > KIP-1150 removes the Leader.
> > > >> >
> > > >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow
> > > between
> > > >> > the Transaction Coordinator and the Batch Coordinator to replace
> the
> > > >> > leader's role. Currently, the KIP does not specify how the system
> > > >> prevents
> > > >> > a "Split Brain" scenario where a consumer reads ahead of a
> > transaction
> > > >> > marker that hasn't yet been sequenced by the Batch Coordinator.
> This
> > > is
> > > >> a
> > > >> > protocol-level correctness issue that must be resolved in the text
> > > >> before
> > > >> > adoption.
> > > >> >
> > > >> > Please note - I am maintaining my objection based on missing
> > > >> > specifications, not code bugs.
> > > >> >
> > > >> > I respectfully request that we pause the vote until:
> > > >> >
> > > >> > A valid design for Garbage Collection (replacing the discarded
> > > >> > KIP-1165) is added to the proposal.
> > > >> >
> > > >> > The Transaction/LSO synchronization protocol is explicitly
> > > >> documented
> > > >> > in KIP-1164.
> > > >> >
> > > >> > Regards,
> > > >> >
> > > >> > Vaquar Khan
> > > >> > Sr Data Architect
> > > >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> > > >> >
> > > >>
> > > >
> > >
> > > --
> > > [image: Aiven] <https://www.aiven.io>
> > >
> > > *Josep Prat*
> > > Sr. Engineering Director, Streaming Services, *Aiven*
> > > josep.prat@aiven.io | +491715557497
> > > aiven.io <https://www.aiven.io> | <
> > https://www.facebook.com/aivencloud
> > > >
> > > <https://www.linkedin.com/company/aiven/> <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > >
> > > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
> >
> > --
> > Anatolii Popov
> > Senior Software Developer, *Aiven OY*
> > m: +358505126242
> > w: aiven.io e: anatolii.popov@aiven.io
> > <https://www.facebook.com/aivencloud>
> > <https://www.linkedin.com/company/aiven/> <
> https://twitter.com/aiven_io>
> >
>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Greg Harris via dev <de...@kafka.apache.org>.
Hey all,
I'm excited to discuss more details in 1163 and 1164 with everyone.
+1 (binding)
Thanks!
Greg
On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev <de...@kafka.apache.org>
wrote:
> Hi all,
>
> Given the importance of this KIP, we want to keep the vote open for a few
> more days to give time to people who had comments in the DISCUSS thread to
> cast their vote if they want.
>
> On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi all,
> > As a co-author of the KIP, I want to explicitly cast my vote for this
> KIP.
> >
> > +1 (binding)
> >
> >
> > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com> wrote:
> >
> > > I've re-read KIP-1150, and still agree this is what we need for Apache
> > > Kafka.
> > >
> > > +1 (binding) from me.
> > >
> > > Thank you,
> > > Luke
> > >
> > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <
> fearthecellos@gmail.com>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as the
> > >> relevant discussion threads. I may have granular comments about 1163
> and
> > >> 1164 but the overall approach suggested in 1150 looks good to me. I
> > >> especially like that the approach covers two main pain points of
> > operating
> > >> and paying for Kafka today: it allows cross-AZ traffic to be reduced
> > (even
> > >> eliminated in some cases), and it also allows local disk usage by
> > brokers
> > >> to be reduced (if operators opt for a small local cache on follower
> > >> brokers
> > >> for non-tiered segments).
> > >>
> > >> +1 (binding)
> > >>
> > >> Cheers,
> > >>
> > >> Chris
> > >>
> > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <va...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Josep,
> > >> >
> > >> > Thank you for the detailed response. I appreciate the clarification
> > >> > regarding the distinction between the Inkless POC and the KIP
> design.
> > >> >
> > >> > However, my objection is not based on temporary bugs in the fork,
> but
> > >> *on
> > >> > architectural gaps in the KIPs themselves* that these implementation
> > >> issues
> > >> > highlighted. If we are voting to approve the design, the design
> > >> documents
> > >> > must be structurally complete regarding data safety.
> > >> >
> > >> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned that
> > >> > cleanup logic "can be defined later." However, KIP-1163 explicitly
> > >> > delegates this responsibility to a separate process, and KIP-1165
> > >> (Object
> > >> > Compaction/GC) is currently marked as "Discarded" in the wiki.
> > >> >
> > >> > We cannot vote to approve a storage engine that has no specified
> > >> mechanism
> > >> > for garbage collection. The "Upload-then-Commit" pattern described
> in
> > >> > KIP-1163 structurally creates orphaned segments during broker
> > failures.
> > >> > Without an active KIP defining the reconciliation protocol (since
> > >> KIP-1165
> > >> > was withdrawn), the proposal effectively describes a system with
> > >> unbounded
> > >> > storage growth during failure modes. This is a blocking design gap,
> > not
> > >> an
> > >> > implementation detail.
> > >> >
> > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is
> not a
> > >> > misunderstanding of standard Kafka transactions; it is a critique of
> > how
> > >> > KIP-1150 changes them. Standard EOS relies on the Partition Leader
> to
> > >> > sequence markers and calculate the LSO (Last Stable Offset) in
> memory.
> > >> > KIP-1150 removes the Leader.
> > >> >
> > >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow
> > between
> > >> > the Transaction Coordinator and the Batch Coordinator to replace the
> > >> > leader's role. Currently, the KIP does not specify how the system
> > >> prevents
> > >> > a "Split Brain" scenario where a consumer reads ahead of a
> transaction
> > >> > marker that hasn't yet been sequenced by the Batch Coordinator. This
> > is
> > >> a
> > >> > protocol-level correctness issue that must be resolved in the text
> > >> before
> > >> > adoption.
> > >> >
> > >> > Please note - I am maintaining my objection based on missing
> > >> > specifications, not code bugs.
> > >> >
> > >> > I respectfully request that we pause the vote until:
> > >> >
> > >> > A valid design for Garbage Collection (replacing the discarded
> > >> > KIP-1165) is added to the proposal.
> > >> >
> > >> > The Transaction/LSO synchronization protocol is explicitly
> > >> documented
> > >> > in KIP-1164.
> > >> >
> > >> > Regards,
> > >> >
> > >> > Vaquar Khan
> > >> > Sr Data Architect
> > >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> > >> >
> > >>
> > >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Sr. Engineering Director, Streaming Services, *Aiven*
> > josep.prat@aiven.io | +491715557497
> > aiven.io <https://www.aiven.io> | <
> https://www.facebook.com/aivencloud
> > >
> > <https://www.linkedin.com/company/aiven/> <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> Anatolii Popov
> Senior Software Developer, *Aiven OY*
> m: +358505126242
> w: aiven.io e: anatolii.popov@aiven.io
> <https://www.facebook.com/aivencloud>
> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Anatolii Popov via dev <de...@kafka.apache.org>.
Hi all,
Given the importance of this KIP, we want to keep the vote open for a few
more days to give time to people who had comments in the DISCUSS thread to
cast their vote if they want.
On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev <de...@kafka.apache.org>
wrote:
> Hi all,
> As a co-author of the KIP, I want to explicitly cast my vote for this KIP.
>
> +1 (binding)
>
>
> On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com> wrote:
>
> > I've re-read KIP-1150, and still agree this is what we need for Apache
> > Kafka.
> >
> > +1 (binding) from me.
> >
> > Thank you,
> > Luke
> >
> > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <fe...@gmail.com>
> > wrote:
> >
> >> Hi all,
> >>
> >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as the
> >> relevant discussion threads. I may have granular comments about 1163 and
> >> 1164 but the overall approach suggested in 1150 looks good to me. I
> >> especially like that the approach covers two main pain points of
> operating
> >> and paying for Kafka today: it allows cross-AZ traffic to be reduced
> (even
> >> eliminated in some cases), and it also allows local disk usage by
> brokers
> >> to be reduced (if operators opt for a small local cache on follower
> >> brokers
> >> for non-tiered segments).
> >>
> >> +1 (binding)
> >>
> >> Cheers,
> >>
> >> Chris
> >>
> >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <va...@gmail.com>
> >> wrote:
> >>
> >> > Hi Josep,
> >> >
> >> > Thank you for the detailed response. I appreciate the clarification
> >> > regarding the distinction between the Inkless POC and the KIP design.
> >> >
> >> > However, my objection is not based on temporary bugs in the fork, but
> >> *on
> >> > architectural gaps in the KIPs themselves* that these implementation
> >> issues
> >> > highlighted. If we are voting to approve the design, the design
> >> documents
> >> > must be structurally complete regarding data safety.
> >> >
> >> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned that
> >> > cleanup logic "can be defined later." However, KIP-1163 explicitly
> >> > delegates this responsibility to a separate process, and KIP-1165
> >> (Object
> >> > Compaction/GC) is currently marked as "Discarded" in the wiki.
> >> >
> >> > We cannot vote to approve a storage engine that has no specified
> >> mechanism
> >> > for garbage collection. The "Upload-then-Commit" pattern described in
> >> > KIP-1163 structurally creates orphaned segments during broker
> failures.
> >> > Without an active KIP defining the reconciliation protocol (since
> >> KIP-1165
> >> > was withdrawn), the proposal effectively describes a system with
> >> unbounded
> >> > storage growth during failure modes. This is a blocking design gap,
> not
> >> an
> >> > implementation detail.
> >> >
> >> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is not a
> >> > misunderstanding of standard Kafka transactions; it is a critique of
> how
> >> > KIP-1150 changes them. Standard EOS relies on the Partition Leader to
> >> > sequence markers and calculate the LSO (Last Stable Offset) in memory.
> >> > KIP-1150 removes the Leader.
> >> >
> >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow
> between
> >> > the Transaction Coordinator and the Batch Coordinator to replace the
> >> > leader's role. Currently, the KIP does not specify how the system
> >> prevents
> >> > a "Split Brain" scenario where a consumer reads ahead of a transaction
> >> > marker that hasn't yet been sequenced by the Batch Coordinator. This
> is
> >> a
> >> > protocol-level correctness issue that must be resolved in the text
> >> before
> >> > adoption.
> >> >
> >> > Please note - I am maintaining my objection based on missing
> >> > specifications, not code bugs.
> >> >
> >> > I respectfully request that we pause the vote until:
> >> >
> >> > A valid design for Garbage Collection (replacing the discarded
> >> > KIP-1165) is added to the proposal.
> >> >
> >> > The Transaction/LSO synchronization protocol is explicitly
> >> documented
> >> > in KIP-1164.
> >> >
> >> > Regards,
> >> >
> >> > Vaquar Khan
> >> > Sr Data Architect
> >> > https://www.linkedin.com/in/vaquar-khan-b695577/
> >> >
> >>
> >
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Sr. Engineering Director, Streaming Services, *Aiven*
> josep.prat@aiven.io | +491715557497
> aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud
> >
> <https://www.linkedin.com/company/aiven/> <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
>
> Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>
--
Anatolii Popov
Senior Software Developer, *Aiven OY*
m: +358505126242
w: aiven.io e: anatolii.popov@aiven.io
<https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Josep Prat via dev <de...@kafka.apache.org>.
Hi all,
As a co-author of the KIP, I want to explicitly cast my vote for this KIP.
+1 (binding)
On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <sh...@gmail.com> wrote:
> I've re-read KIP-1150, and still agree this is what we need for Apache
> Kafka.
>
> +1 (binding) from me.
>
> Thank you,
> Luke
>
> On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <fe...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as the
>> relevant discussion threads. I may have granular comments about 1163 and
>> 1164 but the overall approach suggested in 1150 looks good to me. I
>> especially like that the approach covers two main pain points of operating
>> and paying for Kafka today: it allows cross-AZ traffic to be reduced (even
>> eliminated in some cases), and it also allows local disk usage by brokers
>> to be reduced (if operators opt for a small local cache on follower
>> brokers
>> for non-tiered segments).
>>
>> +1 (binding)
>>
>> Cheers,
>>
>> Chris
>>
>> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <va...@gmail.com>
>> wrote:
>>
>> > Hi Josep,
>> >
>> > Thank you for the detailed response. I appreciate the clarification
>> > regarding the distinction between the Inkless POC and the KIP design.
>> >
>> > However, my objection is not based on temporary bugs in the fork, but
>> *on
>> > architectural gaps in the KIPs themselves* that these implementation
>> issues
>> > highlighted. If we are voting to approve the design, the design
>> documents
>> > must be structurally complete regarding data safety.
>> >
>> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned that
>> > cleanup logic "can be defined later." However, KIP-1163 explicitly
>> > delegates this responsibility to a separate process, and KIP-1165
>> (Object
>> > Compaction/GC) is currently marked as "Discarded" in the wiki.
>> >
>> > We cannot vote to approve a storage engine that has no specified
>> mechanism
>> > for garbage collection. The "Upload-then-Commit" pattern described in
>> > KIP-1163 structurally creates orphaned segments during broker failures.
>> > Without an active KIP defining the reconciliation protocol (since
>> KIP-1165
>> > was withdrawn), the proposal effectively describes a system with
>> unbounded
>> > storage growth during failure modes. This is a blocking design gap, not
>> an
>> > implementation detail.
>> >
>> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is not a
>> > misunderstanding of standard Kafka transactions; it is a critique of how
>> > KIP-1150 changes them. Standard EOS relies on the Partition Leader to
>> > sequence markers and calculate the LSO (Last Stable Offset) in memory.
>> > KIP-1150 removes the Leader.
>> >
>> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow between
>> > the Transaction Coordinator and the Batch Coordinator to replace the
>> > leader's role. Currently, the KIP does not specify how the system
>> prevents
>> > a "Split Brain" scenario where a consumer reads ahead of a transaction
>> > marker that hasn't yet been sequenced by the Batch Coordinator. This is
>> a
>> > protocol-level correctness issue that must be resolved in the text
>> before
>> > adoption.
>> >
>> > Please note - I am maintaining my objection based on missing
>> > specifications, not code bugs.
>> >
>> > I respectfully request that we pause the vote until:
>> >
>> > A valid design for Garbage Collection (replacing the discarded
>> > KIP-1165) is added to the proposal.
>> >
>> > The Transaction/LSO synchronization protocol is explicitly
>> documented
>> > in KIP-1164.
>> >
>> > Regards,
>> >
>> > Vaquar Khan
>> > Sr Data Architect
>> > https://www.linkedin.com/in/vaquar-khan-b695577/
>> >
>>
>
--
[image: Aiven] <https://www.aiven.io>
*Josep Prat*
Sr. Engineering Director, Streaming Services, *Aiven*
josep.prat@aiven.io | +491715557497
aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B
Re: Re: [VOTE] KIP-1150: Diskless Topics (Restarted)
Posted by Luke Chen <sh...@gmail.com>.
I've re-read KIP-1150, and still agree this is what we need for Apache
Kafka.
+1 (binding) from me.
Thank you,
Luke
On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton <fe...@gmail.com>
wrote:
> Hi all,
>
> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as the
> relevant discussion threads. I may have granular comments about 1163 and
> 1164 but the overall approach suggested in 1150 looks good to me. I
> especially like that the approach covers two main pain points of operating
> and paying for Kafka today: it allows cross-AZ traffic to be reduced (even
> eliminated in some cases), and it also allows local disk usage by brokers
> to be reduced (if operators opt for a small local cache on follower brokers
> for non-tiered segments).
>
> +1 (binding)
>
> Cheers,
>
> Chris
>
> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <va...@gmail.com> wrote:
>
> > Hi Josep,
> >
> > Thank you for the detailed response. I appreciate the clarification
> > regarding the distinction between the Inkless POC and the KIP design.
> >
> > However, my objection is not based on temporary bugs in the fork, but *on
> > architectural gaps in the KIPs themselves* that these implementation
> issues
> > highlighted. If we are voting to approve the design, the design documents
> > must be structurally complete regarding data safety.
> >
> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned that
> > cleanup logic "can be defined later." However, KIP-1163 explicitly
> > delegates this responsibility to a separate process, and KIP-1165 (Object
> > Compaction/GC) is currently marked as "Discarded" in the wiki.
> >
> > We cannot vote to approve a storage engine that has no specified
> mechanism
> > for garbage collection. The "Upload-then-Commit" pattern described in
> > KIP-1163 structurally creates orphaned segments during broker failures.
> > Without an active KIP defining the reconciliation protocol (since
> KIP-1165
> > was withdrawn), the proposal effectively describes a system with
> unbounded
> > storage growth during failure modes. This is a blocking design gap, not
> an
> > implementation detail.
> >
> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is not a
> > misunderstanding of standard Kafka transactions; it is a critique of how
> > KIP-1150 changes them. Standard EOS relies on the Partition Leader to
> > sequence markers and calculate the LSO (Last Stable Offset) in memory.
> > KIP-1150 removes the Leader.
> >
> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow between
> > the Transaction Coordinator and the Batch Coordinator to replace the
> > leader's role. Currently, the KIP does not specify how the system
> prevents
> > a "Split Brain" scenario where a consumer reads ahead of a transaction
> > marker that hasn't yet been sequenced by the Batch Coordinator. This is a
> > protocol-level correctness issue that must be resolved in the text before
> > adoption.
> >
> > Please note - I am maintaining my objection based on missing
> > specifications, not code bugs.
> >
> > I respectfully request that we pause the vote until:
> >
> > A valid design for Garbage Collection (replacing the discarded
> > KIP-1165) is added to the proposal.
> >
> > The Transaction/LSO synchronization protocol is explicitly documented
> > in KIP-1164.
> >
> > Regards,
> >
> > Vaquar Khan
> > Sr Data Architect
> > https://www.linkedin.com/in/vaquar-khan-b695577/
> >
>