[DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Josep Prat <jo...@aiven.io.INVALID>]

Hi all!
As agreed on the "Road to Kafka 4.0" email thread, I created KIP-1012 to
discuss and I'd like to open it up for discussion:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release

Let's use this KIP to:
a) Leave a papertrail agreement for the need of a 3.8 version
b) Define which KIPs are the must-haves in regards to KRaft that should be
included there.

Please let me know your feedback and suggestions.

Best,

-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe <cm...@apache.org>]

On Fri, Jan 5, 2024, at 14:50, Greg Harris wrote:
> Hi Colin,
>
> Thanks for the quick reply!
>
>> If we cannot get KIP-853 done in time for 3.8, then we'd move to have another 3.x release.
>
> This is the crux of my concern, and this is satisfactory. This means
> that the "must-haves" are advisory only, and don't constitute a
> binding feature list or a feature-based release. Thank you!
>

Hi Greg,

I agree that you could say that the release is still time-based rather than feature-based since we will release on time even if we haven't finished everything. (And I think this is a good thing)

However, I wouldn't phrase the KIP as "advisory only" ... I would say it just spells out the conditions for the last 3.x release. Once those conditions are met, we will move on to 4.0.

>>The intention behind saying the timeline was "rough" was to make the obvious comment that if we shipped 3.7 in February rather than in January, it would still be in the spirit of following the KIP.
>> I kind of regret putting that comment in there now, since apparently there was a lot of misinterpretation! It didn't mean that the KIP itself was just a suggestion and not binding, or that we hadn't come to a consensus about 3.7 being the last release.
>
> I certainly did not get that sense from reading KIP-833 or the
> discussion thread after-the-fact. I was reading it using the
> contemporary interpretation you posted here: [1]. If I were voting on
> that KIP, I would not think that I was voting for 3.7 to be the last
> 3.x release.
>
>> To make this totally clear:
>> NO vote for this KIP ===> 3.7 is the last 3.x release, as per KIP-833.
>> YES vote for this KIP ===> 3.8 is the last 3.x release.
>
> This was not the precedent set by previous major version bump
> discussions. We need a mutual consensus to advance the major version,
> and lack of consensus means by default the next release will be a
> minor release.
> If KIP-833 actually changed this precedent, then I'll vote for this
> KIP just to restore the norm, as long as it is clear to everyone that
> after 3.8 the version will bump to 3.9 (and etc) unless another
> discussion reaches consensus about the project being ready for 4.0.
>

Just to be totally clear. KIP-1012 states that once we have a 3.x release with KIP-853 and a mechanism for configuring perpetual unclean leader election, we can move on to 4.0. There will not need to be additional discussions or KIPs to do that. If there is more stuff you feel is needed in 3.x to make migration possible, now is the time to speak up.

best,
Colin

>
> [1] https://lists.apache.org/thread/k3bbz7k0hsb4y0b2xn1lh7bjrmjowmcq
>
> Thanks!
> Greg
>
> On Fri, Jan 5, 2024 at 2:20 PM Colin McCabe <cm...@apache.org> wrote:
>>
>> Hi Justine,
>>
>> If there are more things you think are needed in 3.8 for migration, let's discuss in the VOTE thread.
>>
>> best,
>> Colin
>>
>>
>> On Fri, Jan 5, 2024, at 09:23, Justine Olshan wrote:
>> > While I agree we should have this release and should vote on it soon, is it
>> > worth determining the exact items we need before we vote? Just so we are
>> > all in agreement?
>> > There is still some discussion on the road to 4.0 thread that may be worth
>> > having here.
>> >
>> > On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <jo...@aiven.io.invalid>
>> > wrote:
>> >
>> >> Hi Colin,
>> >> Sorry for being quiet these last days (PTO).
>> >>
>> >> I will start the vote thread right away.
>> >>
>> >> Best,
>> >>
>> >>
>> >> ---
>> >> Josep Prat
>> >> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
>> >> +491715557497 | aiven.io
>> >> Aiven Deutschland GmbH
>> >> Alexanderufer 3-7, 10117 Berlin
>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> >>
>> >> On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:
>> >>
>> >> > Hi all,
>> >> >
>> >> > Since this has been open for a few weeks, are there any objections to
>> >> > starting the vote? What do you think, Josep?
>> >> >
>> >> > Since 3.8 is going to be the next release (according to the KIP) we
>> >> should
>> >> > really vote this in as soon as possible.
>> >> >
>> >> > Also, I created a wiki page about the 3.8 release with a tentative
>> >> > schedule.
>> >> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
>> >> >
>> >> > Please let me know if these dates make sense -- they're just proposals
>> >> > right now.
>> >> >
>> >> > best,
>> >> > Colin
>> >> >
>> >> >
>> >> > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
>> >> > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
>> >> > >> Hey Colin,
>> >> > >>
>> >> > >> Some folks were concerned about the lack of automatic unclean leader
>> >> > >> election. I mentioned that KIP-966 would actually be better with its
>> >> > >> aggressive recovery option.
>> >> > >> I think folks were hoping for some availability over durability
>> >> solution
>> >> > >> for KRaft, so if we don't do KIP-966 we should provide an alternative
>> >> > or be
>> >> > >> able to convince ourselves it is not needed.
>> >> > >
>> >> > > Hi Justine,
>> >> > >
>> >> > > That's a fair point. We should specify in KIP-1012 that we need to have
>> >> > > some way to configure the system to automatically do unclean leader
>> >> > > election. If we run out of time implementing KIP-966, this could be
>> >> > > something quite simple, like honoring the static
>> >> > > unclean.leader.election = true configuration.
>> >> > >
>> >> > >>
>> >> > >> I think while many folks decided KIP-853 was a blocker, there were a
>> >> > lot of
>> >> > >> other features that many folks were expecting so I don't think we can
>> >> > say
>> >> > >> definitively the only must-have is KIP-853 (and hence the discussion
>> >> > thread
>> >> > >> here :) )
>> >> > >>
>> >> > >> Also as an aside, I filed a ticket to remove ZK from the top of the
>> >> > >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
>> >> > >>
>> >> > >
>> >> > > Yeah. There is a bunch of docs and quickstart cleanup that we should
>> >> > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new
>> >> > > year is a good time to clean things up.
>> >> > >
>> >> > > best,
>> >> > > Colin
>> >> > >
>> >> > >
>> >> > >> Justine
>> >> > >>
>> >> > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org>
>> >> > wrote:
>> >> > >>
>> >> > >>> Hi Josep,
>> >> > >>>
>> >> > >>> Thanks for the KIP. Based on the discussions we had previously, I
>> >> agree
>> >> > >>> that we need a 3.8.
>> >> > >>>
>> >> > >>> It would be good to link to KIP-833 in the motivation section, since
>> >> > this
>> >> > >>> KIP builds on that one.
>> >> > >>>
>> >> > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
>> >> > >>> general-purpose release that may add some new features. This was
>> >> > something
>> >> > >>> that we were on the fence about previously, so it would be good to
>> >> > clarify
>> >> > >>> it here.
>> >> > >>>
>> >> > >>> On another note. I don't think KIP-966 is a "must-have" for Kafka
>> >> 3.8,
>> >> > as
>> >> > >>> the KIP currently states. I certainly hope that it makes it for 3.8,
>> >> > but if
>> >> > >>> it doesn't, it can go into 4.0. It's not needed for migration, so it
>> >> > could
>> >> > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really
>> >> > needs
>> >> > >>> is "KIP-853: KRaft Controller Membership Changes."
>> >> > >>>
>> >> > >>> Along these lines, I think we should drop the language about
>> >> "strategic
>> >> > >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper,
>> >> > and
>> >> > >>> doesn't need feature parity with it. For example, ZK implemented
>> >> > >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably
>> >> > won't
>> >> > >>> in 3.8). We probably won't add this -- or if we do, it won't be so
>> >> > that we
>> >> > >>> can have "parity with ZK." Really the only must-have in 3.8 is
>> >> > KIP-853, and
>> >> > >>> we should be clear about that.
>> >> > >>>
>> >> > >>> I think we should start issuing a deprecation log message at ERROR
>> >> > level
>> >> > >>> when brokers start up in ZK mode. This message could point out that
>> >> > some
>> >> > >>> safety mechanisms and new features will not be available in ZK mode,
>> >> > and
>> >> > >>> give a link to our documentation about migration.
>> >> > >>>
>> >> > >>> We should probably also move the example configurations for kraft
>> >> from
>> >> > >>> config/kraft to config. And move the zk ones into config/zk. Or maybe
>> >> > even
>> >> > >>> drop the ZK ones altogether, since they're not needed for migration
>> >> or
>> >> > >>> upgrade.
>> >> > >>>
>> >> > >>> best,
>> >> > >>> Colin
>> >> > >>>
>> >> > >>>
>> >> > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
>> >> > >>> > On this note, I'd like to add that I would volunteer to be the
>> >> > release
>> >> > >>> > manager of such release 3.8.0.
>> >> > >>> >
>> >> > >>> > Best,
>> >> > >>> >
>> >> > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io>
>> >> > wrote:
>> >> > >>> >
>> >> > >>> >> Hi all!
>> >> > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
>> >> > KIP-1012 to
>> >> > >>> >> discuss and I'd like to open it up for discussion:
>> >> > >>> >>
>> >> > >>>
>> >> >
>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>> >> > >>> >>
>> >> > >>> >> Let's use this KIP to:
>> >> > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
>> >> > >>> >> b) Define which KIPs are the must-haves in regards to KRaft that
>> >> > should
>> >> > >>> be
>> >> > >>> >> included there.
>> >> > >>> >>
>> >> > >>> >> Please let me know your feedback and suggestions.
>> >> > >>> >>
>> >> > >>> >> Best,
>> >> > >>> >>
>> >> > >>> >> --
>> >> > >>> >> [image: Aiven] <https://www.aiven.io>
>> >> > >>> >>
>> >> > >>> >> *Josep Prat*
>> >> > >>> >> Open Source Engineering Director, *Aiven*
>> >> > >>> >> josep.prat@aiven.io   |   +491715557497
>> >> > >>> >> aiven.io <https://www.aiven.io>   |
>> >> > >>> >> <https://www.facebook.com/aivencloud>
>> >> > >>> >> <https://www.linkedin.com/company/aiven/>   <
>> >> > >>> https://twitter.com/aiven_io>
>> >> > >>> >> *Aiven Deutschland GmbH*
>> >> > >>> >> Alexanderufer 3-7, 10117 Berlin
>> >> > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> > >>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> >> > >>> >>
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > --
>> >> > >>> > [image: Aiven] <https://www.aiven.io>
>> >> > >>> >
>> >> > >>> > *Josep Prat*
>> >> > >>> > Open Source Engineering Director, *Aiven*
>> >> > >>> > josep.prat@aiven.io   |   +491715557497
>> >> > >>> > aiven.io <https://www.aiven.io>   |   <
>> >> > >>> https://www.facebook.com/aivencloud>
>> >> > >>> >   <https://www.linkedin.com/company/aiven/>   <
>> >> > >>> https://twitter.com/aiven_io>
>> >> > >>> > *Aiven Deutschland GmbH*
>> >> > >>> > Alexanderufer 3-7, 10117 Berlin
>> >> > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> > >>> > Amtsgericht Charlottenburg, HRB 209739 B
>> >> > >>>
>> >> >
>> >>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Josep Prat <jo...@aiven.io.INVALID>]

Hi all,

Copying here the same message as in the VOTE thread just in case someone is
not following it (yet).
"*Thanks for your comments,*
*I reworded some parts of the KIP to express that:*
*- The KIP is to agree that we need at least one more minor version in the
3.x series*

*- Explicitly saying that the list of KIPs is not exhaustive and that if
some are not done, we would need another minor version*
*- Which are the KIPs/Features the community identified that should be
present in a 3.x version so they can safely migrate to a potential 4.0
version*
*- The timeline of 4.0 (starting after branching, not after release)*
*- Wording is now focusing more on the need to have at least another minor
release in 3.x to enable and ease migration to a potential 4.0 version*

*I always mention potential in terms of 4.0 as we don't know what will be
in there yet, and this KIP's scope is not meant to define this.*"

Best,

On Fri, Jan 5, 2024 at 11:50 PM Greg Harris <gr...@aiven.io.invalid>
wrote:

> Hi Colin,
>
> Thanks for the quick reply!
>
> > If we cannot get KIP-853 done in time for 3.8, then we'd move to have
> another 3.x release.
>
> This is the crux of my concern, and this is satisfactory. This means
> that the "must-haves" are advisory only, and don't constitute a
> binding feature list or a feature-based release. Thank you!
>
> >The intention behind saying the timeline was "rough" was to make the
> obvious comment that if we shipped 3.7 in February rather than in January,
> it would still be in the spirit of following the KIP.
> > I kind of regret putting that comment in there now, since apparently
> there was a lot of misinterpretation! It didn't mean that the KIP itself
> was just a suggestion and not binding, or that we hadn't come to a
> consensus about 3.7 being the last release.
>
> I certainly did not get that sense from reading KIP-833 or the
> discussion thread after-the-fact. I was reading it using the
> contemporary interpretation you posted here: [1]. If I were voting on
> that KIP, I would not think that I was voting for 3.7 to be the last
> 3.x release.
>
> > To make this totally clear:
> > NO vote for this KIP ===> 3.7 is the last 3.x release, as per KIP-833.
> > YES vote for this KIP ===> 3.8 is the last 3.x release.
>
> This was not the precedent set by previous major version bump
> discussions. We need a mutual consensus to advance the major version,
> and lack of consensus means by default the next release will be a
> minor release.
> If KIP-833 actually changed this precedent, then I'll vote for this
> KIP just to restore the norm, as long as it is clear to everyone that
> after 3.8 the version will bump to 3.9 (and etc) unless another
> discussion reaches consensus about the project being ready for 4.0.
>
> [1] https://lists.apache.org/thread/k3bbz7k0hsb4y0b2xn1lh7bjrmjowmcq
>
> Thanks!
> Greg
>
> On Fri, Jan 5, 2024 at 2:20 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > Hi Justine,
> >
> > If there are more things you think are needed in 3.8 for migration,
> let's discuss in the VOTE thread.
> >
> > best,
> > Colin
> >
> >
> > On Fri, Jan 5, 2024, at 09:23, Justine Olshan wrote:
> > > While I agree we should have this release and should vote on it soon,
> is it
> > > worth determining the exact items we need before we vote? Just so we
> are
> > > all in agreement?
> > > There is still some discussion on the road to 4.0 thread that may be
> worth
> > > having here.
> > >
> > > On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <josep.prat@aiven.io.invalid
> >
> > > wrote:
> > >
> > >> Hi Colin,
> > >> Sorry for being quiet these last days (PTO).
> > >>
> > >> I will start the vote thread right away.
> > >>
> > >> Best,
> > >>
> > >>
> > >> ---
> > >> Josep Prat
> > >> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > >> +491715557497 | aiven.io
> > >> Aiven Deutschland GmbH
> > >> Alexanderufer 3-7, 10117 Berlin
> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>
> > >> On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > Since this has been open for a few weeks, are there any objections
> to
> > >> > starting the vote? What do you think, Josep?
> > >> >
> > >> > Since 3.8 is going to be the next release (according to the KIP) we
> > >> should
> > >> > really vote this in as soon as possible.
> > >> >
> > >> > Also, I created a wiki page about the 3.8 release with a tentative
> > >> > schedule.
> > >> >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
> > >> >
> > >> > Please let me know if these dates make sense -- they're just
> proposals
> > >> > right now.
> > >> >
> > >> > best,
> > >> > Colin
> > >> >
> > >> >
> > >> > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
> > >> > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
> > >> > >> Hey Colin,
> > >> > >>
> > >> > >> Some folks were concerned about the lack of automatic unclean
> leader
> > >> > >> election. I mentioned that KIP-966 would actually be better with
> its
> > >> > >> aggressive recovery option.
> > >> > >> I think folks were hoping for some availability over durability
> > >> solution
> > >> > >> for KRaft, so if we don't do KIP-966 we should provide an
> alternative
> > >> > or be
> > >> > >> able to convince ourselves it is not needed.
> > >> > >
> > >> > > Hi Justine,
> > >> > >
> > >> > > That's a fair point. We should specify in KIP-1012 that we need
> to have
> > >> > > some way to configure the system to automatically do unclean
> leader
> > >> > > election. If we run out of time implementing KIP-966, this could
> be
> > >> > > something quite simple, like honoring the static
> > >> > > unclean.leader.election = true configuration.
> > >> > >
> > >> > >>
> > >> > >> I think while many folks decided KIP-853 was a blocker, there
> were a
> > >> > lot of
> > >> > >> other features that many folks were expecting so I don't think
> we can
> > >> > say
> > >> > >> definitively the only must-have is KIP-853 (and hence the
> discussion
> > >> > thread
> > >> > >> here :) )
> > >> > >>
> > >> > >> Also as an aside, I filed a ticket to remove ZK from the top of
> the
> > >> > >> quickstart guide.
> https://issues.apache.org/jira/browse/KAFKA-15975
> > >> > >>
> > >> > >
> > >> > > Yeah. There is a bunch of docs and quickstart cleanup that we
> should
> > >> > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the
> new
> > >> > > year is a good time to clean things up.
> > >> > >
> > >> > > best,
> > >> > > Colin
> > >> > >
> > >> > >
> > >> > >> Justine
> > >> > >>
> > >> > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cmccabe@apache.org
> >
> > >> > wrote:
> > >> > >>
> > >> > >>> Hi Josep,
> > >> > >>>
> > >> > >>> Thanks for the KIP. Based on the discussions we had previously,
> I
> > >> agree
> > >> > >>> that we need a 3.8.
> > >> > >>>
> > >> > >>> It would be good to link to KIP-833 in the motivation section,
> since
> > >> > this
> > >> > >>> KIP builds on that one.
> > >> > >>>
> > >> > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
> > >> > >>> general-purpose release that may add some new features. This was
> > >> > something
> > >> > >>> that we were on the fence about previously, so it would be good
> to
> > >> > clarify
> > >> > >>> it here.
> > >> > >>>
> > >> > >>> On another note. I don't think KIP-966 is a "must-have" for
> Kafka
> > >> 3.8,
> > >> > as
> > >> > >>> the KIP currently states. I certainly hope that it makes it for
> 3.8,
> > >> > but if
> > >> > >>> it doesn't, it can go into 4.0. It's not needed for migration,
> so it
> > >> > could
> > >> > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966
> really
> > >> > needs
> > >> > >>> is "KIP-853: KRaft Controller Membership Changes."
> > >> > >>>
> > >> > >>> Along these lines, I think we should drop the language about
> > >> "strategic
> > >> > >>> feature parity with Zookeeper." Kafka isn't competing with
> ZooKeeper,
> > >> > and
> > >> > >>> doesn't need feature parity with it. For example, ZK implemented
> > >> > >>> Netty-TcNative OpenSSL Support, but we don't have that (and
> probably
> > >> > won't
> > >> > >>> in 3.8). We probably won't add this -- or if we do, it won't be
> so
> > >> > that we
> > >> > >>> can have "parity with ZK." Really the only must-have in 3.8 is
> > >> > KIP-853, and
> > >> > >>> we should be clear about that.
> > >> > >>>
> > >> > >>> I think we should start issuing a deprecation log message at
> ERROR
> > >> > level
> > >> > >>> when brokers start up in ZK mode. This message could point out
> that
> > >> > some
> > >> > >>> safety mechanisms and new features will not be available in ZK
> mode,
> > >> > and
> > >> > >>> give a link to our documentation about migration.
> > >> > >>>
> > >> > >>> We should probably also move the example configurations for
> kraft
> > >> from
> > >> > >>> config/kraft to config. And move the zk ones into config/zk. Or
> maybe
> > >> > even
> > >> > >>> drop the ZK ones altogether, since they're not needed for
> migration
> > >> or
> > >> > >>> upgrade.
> > >> > >>>
> > >> > >>> best,
> > >> > >>> Colin
> > >> > >>>
> > >> > >>>
> > >> > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
> > >> > >>> > On this note, I'd like to add that I would volunteer to be the
> > >> > release
> > >> > >>> > manager of such release 3.8.0.
> > >> > >>> >
> > >> > >>> > Best,
> > >> > >>> >
> > >> > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <
> josep.prat@aiven.io>
> > >> > wrote:
> > >> > >>> >
> > >> > >>> >> Hi all!
> > >> > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
> > >> > KIP-1012 to
> > >> > >>> >> discuss and I'd like to open it up for discussion:
> > >> > >>> >>
> > >> > >>>
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > >> > >>> >>
> > >> > >>> >> Let's use this KIP to:
> > >> > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
> > >> > >>> >> b) Define which KIPs are the must-haves in regards to KRaft
> that
> > >> > should
> > >> > >>> be
> > >> > >>> >> included there.
> > >> > >>> >>
> > >> > >>> >> Please let me know your feedback and suggestions.
> > >> > >>> >>
> > >> > >>> >> Best,
> > >> > >>> >>
> > >> > >>> >> --
> > >> > >>> >> [image: Aiven] <https://www.aiven.io>
> > >> > >>> >>
> > >> > >>> >> *Josep Prat*
> > >> > >>> >> Open Source Engineering Director, *Aiven*
> > >> > >>> >> josep.prat@aiven.io   |   +491715557497
> > >> > >>> >> aiven.io <https://www.aiven.io>   |
> > >> > >>> >> <https://www.facebook.com/aivencloud>
> > >> > >>> >> <https://www.linkedin.com/company/aiven/>   <
> > >> > >>> https://twitter.com/aiven_io>
> > >> > >>> >> *Aiven Deutschland GmbH*
> > >> > >>> >> Alexanderufer 3-7, 10117 Berlin
> > >> > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> > >>> >> Amtsgericht Charlottenburg, HRB 209739 B
> > >> > >>> >>
> > >> > >>> >
> > >> > >>> >
> > >> > >>> > --
> > >> > >>> > [image: Aiven] <https://www.aiven.io>
> > >> > >>> >
> > >> > >>> > *Josep Prat*
> > >> > >>> > Open Source Engineering Director, *Aiven*
> > >> > >>> > josep.prat@aiven.io   |   +491715557497
> > >> > >>> > aiven.io <https://www.aiven.io>   |   <
> > >> > >>> https://www.facebook.com/aivencloud>
> > >> > >>> >   <https://www.linkedin.com/company/aiven/>   <
> > >> > >>> https://twitter.com/aiven_io>
> > >> > >>> > *Aiven Deutschland GmbH*
> > >> > >>> > Alexanderufer 3-7, 10117 Berlin
> > >> > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> > >>> > Amtsgericht Charlottenburg, HRB 209739 B
> > >> > >>>
> > >> >
> > >>
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Greg Harris <gr...@aiven.io.INVALID>]

Hi Colin,

Thanks for the quick reply!

> If we cannot get KIP-853 done in time for 3.8, then we'd move to have another 3.x release.

This is the crux of my concern, and this is satisfactory. This means
that the "must-haves" are advisory only, and don't constitute a
binding feature list or a feature-based release. Thank you!

>The intention behind saying the timeline was "rough" was to make the obvious comment that if we shipped 3.7 in February rather than in January, it would still be in the spirit of following the KIP.
> I kind of regret putting that comment in there now, since apparently there was a lot of misinterpretation! It didn't mean that the KIP itself was just a suggestion and not binding, or that we hadn't come to a consensus about 3.7 being the last release.

I certainly did not get that sense from reading KIP-833 or the
discussion thread after-the-fact. I was reading it using the
contemporary interpretation you posted here: [1]. If I were voting on
that KIP, I would not think that I was voting for 3.7 to be the last
3.x release.

> To make this totally clear:
> NO vote for this KIP ===> 3.7 is the last 3.x release, as per KIP-833.
> YES vote for this KIP ===> 3.8 is the last 3.x release.

This was not the precedent set by previous major version bump
discussions. We need a mutual consensus to advance the major version,
and lack of consensus means by default the next release will be a
minor release.
If KIP-833 actually changed this precedent, then I'll vote for this
KIP just to restore the norm, as long as it is clear to everyone that
after 3.8 the version will bump to 3.9 (and etc) unless another
discussion reaches consensus about the project being ready for 4.0.

[1] https://lists.apache.org/thread/k3bbz7k0hsb4y0b2xn1lh7bjrmjowmcq

Thanks!
Greg

On Fri, Jan 5, 2024 at 2:20 PM Colin McCabe <cm...@apache.org> wrote:
>
> Hi Justine,
>
> If there are more things you think are needed in 3.8 for migration, let's discuss in the VOTE thread.
>
> best,
> Colin
>
>
> On Fri, Jan 5, 2024, at 09:23, Justine Olshan wrote:
> > While I agree we should have this release and should vote on it soon, is it
> > worth determining the exact items we need before we vote? Just so we are
> > all in agreement?
> > There is still some discussion on the road to 4.0 thread that may be worth
> > having here.
> >
> > On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <jo...@aiven.io.invalid>
> > wrote:
> >
> >> Hi Colin,
> >> Sorry for being quiet these last days (PTO).
> >>
> >> I will start the vote thread right away.
> >>
> >> Best,
> >>
> >>
> >> ---
> >> Josep Prat
> >> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> >> +491715557497 | aiven.io
> >> Aiven Deutschland GmbH
> >> Alexanderufer 3-7, 10117 Berlin
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:
> >>
> >> > Hi all,
> >> >
> >> > Since this has been open for a few weeks, are there any objections to
> >> > starting the vote? What do you think, Josep?
> >> >
> >> > Since 3.8 is going to be the next release (according to the KIP) we
> >> should
> >> > really vote this in as soon as possible.
> >> >
> >> > Also, I created a wiki page about the 3.8 release with a tentative
> >> > schedule.
> >> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
> >> >
> >> > Please let me know if these dates make sense -- they're just proposals
> >> > right now.
> >> >
> >> > best,
> >> > Colin
> >> >
> >> >
> >> > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
> >> > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
> >> > >> Hey Colin,
> >> > >>
> >> > >> Some folks were concerned about the lack of automatic unclean leader
> >> > >> election. I mentioned that KIP-966 would actually be better with its
> >> > >> aggressive recovery option.
> >> > >> I think folks were hoping for some availability over durability
> >> solution
> >> > >> for KRaft, so if we don't do KIP-966 we should provide an alternative
> >> > or be
> >> > >> able to convince ourselves it is not needed.
> >> > >
> >> > > Hi Justine,
> >> > >
> >> > > That's a fair point. We should specify in KIP-1012 that we need to have
> >> > > some way to configure the system to automatically do unclean leader
> >> > > election. If we run out of time implementing KIP-966, this could be
> >> > > something quite simple, like honoring the static
> >> > > unclean.leader.election = true configuration.
> >> > >
> >> > >>
> >> > >> I think while many folks decided KIP-853 was a blocker, there were a
> >> > lot of
> >> > >> other features that many folks were expecting so I don't think we can
> >> > say
> >> > >> definitively the only must-have is KIP-853 (and hence the discussion
> >> > thread
> >> > >> here :) )
> >> > >>
> >> > >> Also as an aside, I filed a ticket to remove ZK from the top of the
> >> > >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
> >> > >>
> >> > >
> >> > > Yeah. There is a bunch of docs and quickstart cleanup that we should
> >> > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new
> >> > > year is a good time to clean things up.
> >> > >
> >> > > best,
> >> > > Colin
> >> > >
> >> > >
> >> > >> Justine
> >> > >>
> >> > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org>
> >> > wrote:
> >> > >>
> >> > >>> Hi Josep,
> >> > >>>
> >> > >>> Thanks for the KIP. Based on the discussions we had previously, I
> >> agree
> >> > >>> that we need a 3.8.
> >> > >>>
> >> > >>> It would be good to link to KIP-833 in the motivation section, since
> >> > this
> >> > >>> KIP builds on that one.
> >> > >>>
> >> > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
> >> > >>> general-purpose release that may add some new features. This was
> >> > something
> >> > >>> that we were on the fence about previously, so it would be good to
> >> > clarify
> >> > >>> it here.
> >> > >>>
> >> > >>> On another note. I don't think KIP-966 is a "must-have" for Kafka
> >> 3.8,
> >> > as
> >> > >>> the KIP currently states. I certainly hope that it makes it for 3.8,
> >> > but if
> >> > >>> it doesn't, it can go into 4.0. It's not needed for migration, so it
> >> > could
> >> > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really
> >> > needs
> >> > >>> is "KIP-853: KRaft Controller Membership Changes."
> >> > >>>
> >> > >>> Along these lines, I think we should drop the language about
> >> "strategic
> >> > >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper,
> >> > and
> >> > >>> doesn't need feature parity with it. For example, ZK implemented
> >> > >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably
> >> > won't
> >> > >>> in 3.8). We probably won't add this -- or if we do, it won't be so
> >> > that we
> >> > >>> can have "parity with ZK." Really the only must-have in 3.8 is
> >> > KIP-853, and
> >> > >>> we should be clear about that.
> >> > >>>
> >> > >>> I think we should start issuing a deprecation log message at ERROR
> >> > level
> >> > >>> when brokers start up in ZK mode. This message could point out that
> >> > some
> >> > >>> safety mechanisms and new features will not be available in ZK mode,
> >> > and
> >> > >>> give a link to our documentation about migration.
> >> > >>>
> >> > >>> We should probably also move the example configurations for kraft
> >> from
> >> > >>> config/kraft to config. And move the zk ones into config/zk. Or maybe
> >> > even
> >> > >>> drop the ZK ones altogether, since they're not needed for migration
> >> or
> >> > >>> upgrade.
> >> > >>>
> >> > >>> best,
> >> > >>> Colin
> >> > >>>
> >> > >>>
> >> > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
> >> > >>> > On this note, I'd like to add that I would volunteer to be the
> >> > release
> >> > >>> > manager of such release 3.8.0.
> >> > >>> >
> >> > >>> > Best,
> >> > >>> >
> >> > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io>
> >> > wrote:
> >> > >>> >
> >> > >>> >> Hi all!
> >> > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
> >> > KIP-1012 to
> >> > >>> >> discuss and I'd like to open it up for discussion:
> >> > >>> >>
> >> > >>>
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> >> > >>> >>
> >> > >>> >> Let's use this KIP to:
> >> > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
> >> > >>> >> b) Define which KIPs are the must-haves in regards to KRaft that
> >> > should
> >> > >>> be
> >> > >>> >> included there.
> >> > >>> >>
> >> > >>> >> Please let me know your feedback and suggestions.
> >> > >>> >>
> >> > >>> >> Best,
> >> > >>> >>
> >> > >>> >> --
> >> > >>> >> [image: Aiven] <https://www.aiven.io>
> >> > >>> >>
> >> > >>> >> *Josep Prat*
> >> > >>> >> Open Source Engineering Director, *Aiven*
> >> > >>> >> josep.prat@aiven.io   |   +491715557497
> >> > >>> >> aiven.io <https://www.aiven.io>   |
> >> > >>> >> <https://www.facebook.com/aivencloud>
> >> > >>> >> <https://www.linkedin.com/company/aiven/>   <
> >> > >>> https://twitter.com/aiven_io>
> >> > >>> >> *Aiven Deutschland GmbH*
> >> > >>> >> Alexanderufer 3-7, 10117 Berlin
> >> > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > >>> >> Amtsgericht Charlottenburg, HRB 209739 B
> >> > >>> >>
> >> > >>> >
> >> > >>> >
> >> > >>> > --
> >> > >>> > [image: Aiven] <https://www.aiven.io>
> >> > >>> >
> >> > >>> > *Josep Prat*
> >> > >>> > Open Source Engineering Director, *Aiven*
> >> > >>> > josep.prat@aiven.io   |   +491715557497
> >> > >>> > aiven.io <https://www.aiven.io>   |   <
> >> > >>> https://www.facebook.com/aivencloud>
> >> > >>> >   <https://www.linkedin.com/company/aiven/>   <
> >> > >>> https://twitter.com/aiven_io>
> >> > >>> > *Aiven Deutschland GmbH*
> >> > >>> > Alexanderufer 3-7, 10117 Berlin
> >> > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > >>> > Amtsgericht Charlottenburg, HRB 209739 B
> >> > >>>
> >> >
> >>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe <cm...@apache.org>]

Hi Justine,

If there are more things you think are needed in 3.8 for migration, let's discuss in the VOTE thread.

best,
Colin


On Fri, Jan 5, 2024, at 09:23, Justine Olshan wrote:
> While I agree we should have this release and should vote on it soon, is it
> worth determining the exact items we need before we vote? Just so we are
> all in agreement?
> There is still some discussion on the road to 4.0 thread that may be worth
> having here.
>
> On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <jo...@aiven.io.invalid>
> wrote:
>
>> Hi Colin,
>> Sorry for being quiet these last days (PTO).
>>
>> I will start the vote thread right away.
>>
>> Best,
>>
>>
>> ---
>> Josep Prat
>> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
>> +491715557497 | aiven.io
>> Aiven Deutschland GmbH
>> Alexanderufer 3-7, 10117 Berlin
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>> On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:
>>
>> > Hi all,
>> >
>> > Since this has been open for a few weeks, are there any objections to
>> > starting the vote? What do you think, Josep?
>> >
>> > Since 3.8 is going to be the next release (according to the KIP) we
>> should
>> > really vote this in as soon as possible.
>> >
>> > Also, I created a wiki page about the 3.8 release with a tentative
>> > schedule.
>> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
>> >
>> > Please let me know if these dates make sense -- they're just proposals
>> > right now.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
>> > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
>> > >> Hey Colin,
>> > >>
>> > >> Some folks were concerned about the lack of automatic unclean leader
>> > >> election. I mentioned that KIP-966 would actually be better with its
>> > >> aggressive recovery option.
>> > >> I think folks were hoping for some availability over durability
>> solution
>> > >> for KRaft, so if we don't do KIP-966 we should provide an alternative
>> > or be
>> > >> able to convince ourselves it is not needed.
>> > >
>> > > Hi Justine,
>> > >
>> > > That's a fair point. We should specify in KIP-1012 that we need to have
>> > > some way to configure the system to automatically do unclean leader
>> > > election. If we run out of time implementing KIP-966, this could be
>> > > something quite simple, like honoring the static
>> > > unclean.leader.election = true configuration.
>> > >
>> > >>
>> > >> I think while many folks decided KIP-853 was a blocker, there were a
>> > lot of
>> > >> other features that many folks were expecting so I don't think we can
>> > say
>> > >> definitively the only must-have is KIP-853 (and hence the discussion
>> > thread
>> > >> here :) )
>> > >>
>> > >> Also as an aside, I filed a ticket to remove ZK from the top of the
>> > >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
>> > >>
>> > >
>> > > Yeah. There is a bunch of docs and quickstart cleanup that we should
>> > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new
>> > > year is a good time to clean things up.
>> > >
>> > > best,
>> > > Colin
>> > >
>> > >
>> > >> Justine
>> > >>
>> > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org>
>> > wrote:
>> > >>
>> > >>> Hi Josep,
>> > >>>
>> > >>> Thanks for the KIP. Based on the discussions we had previously, I
>> agree
>> > >>> that we need a 3.8.
>> > >>>
>> > >>> It would be good to link to KIP-833 in the motivation section, since
>> > this
>> > >>> KIP builds on that one.
>> > >>>
>> > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
>> > >>> general-purpose release that may add some new features. This was
>> > something
>> > >>> that we were on the fence about previously, so it would be good to
>> > clarify
>> > >>> it here.
>> > >>>
>> > >>> On another note. I don't think KIP-966 is a "must-have" for Kafka
>> 3.8,
>> > as
>> > >>> the KIP currently states. I certainly hope that it makes it for 3.8,
>> > but if
>> > >>> it doesn't, it can go into 4.0. It's not needed for migration, so it
>> > could
>> > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really
>> > needs
>> > >>> is "KIP-853: KRaft Controller Membership Changes."
>> > >>>
>> > >>> Along these lines, I think we should drop the language about
>> "strategic
>> > >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper,
>> > and
>> > >>> doesn't need feature parity with it. For example, ZK implemented
>> > >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably
>> > won't
>> > >>> in 3.8). We probably won't add this -- or if we do, it won't be so
>> > that we
>> > >>> can have "parity with ZK." Really the only must-have in 3.8 is
>> > KIP-853, and
>> > >>> we should be clear about that.
>> > >>>
>> > >>> I think we should start issuing a deprecation log message at ERROR
>> > level
>> > >>> when brokers start up in ZK mode. This message could point out that
>> > some
>> > >>> safety mechanisms and new features will not be available in ZK mode,
>> > and
>> > >>> give a link to our documentation about migration.
>> > >>>
>> > >>> We should probably also move the example configurations for kraft
>> from
>> > >>> config/kraft to config. And move the zk ones into config/zk. Or maybe
>> > even
>> > >>> drop the ZK ones altogether, since they're not needed for migration
>> or
>> > >>> upgrade.
>> > >>>
>> > >>> best,
>> > >>> Colin
>> > >>>
>> > >>>
>> > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
>> > >>> > On this note, I'd like to add that I would volunteer to be the
>> > release
>> > >>> > manager of such release 3.8.0.
>> > >>> >
>> > >>> > Best,
>> > >>> >
>> > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io>
>> > wrote:
>> > >>> >
>> > >>> >> Hi all!
>> > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
>> > KIP-1012 to
>> > >>> >> discuss and I'd like to open it up for discussion:
>> > >>> >>
>> > >>>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>> > >>> >>
>> > >>> >> Let's use this KIP to:
>> > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
>> > >>> >> b) Define which KIPs are the must-haves in regards to KRaft that
>> > should
>> > >>> be
>> > >>> >> included there.
>> > >>> >>
>> > >>> >> Please let me know your feedback and suggestions.
>> > >>> >>
>> > >>> >> Best,
>> > >>> >>
>> > >>> >> --
>> > >>> >> [image: Aiven] <https://www.aiven.io>
>> > >>> >>
>> > >>> >> *Josep Prat*
>> > >>> >> Open Source Engineering Director, *Aiven*
>> > >>> >> josep.prat@aiven.io   |   +491715557497
>> > >>> >> aiven.io <https://www.aiven.io>   |
>> > >>> >> <https://www.facebook.com/aivencloud>
>> > >>> >> <https://www.linkedin.com/company/aiven/>   <
>> > >>> https://twitter.com/aiven_io>
>> > >>> >> *Aiven Deutschland GmbH*
>> > >>> >> Alexanderufer 3-7, 10117 Berlin
>> > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > >>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> > >>> >>
>> > >>> >
>> > >>> >
>> > >>> > --
>> > >>> > [image: Aiven] <https://www.aiven.io>
>> > >>> >
>> > >>> > *Josep Prat*
>> > >>> > Open Source Engineering Director, *Aiven*
>> > >>> > josep.prat@aiven.io   |   +491715557497
>> > >>> > aiven.io <https://www.aiven.io>   |   <
>> > >>> https://www.facebook.com/aivencloud>
>> > >>> >   <https://www.linkedin.com/company/aiven/>   <
>> > >>> https://twitter.com/aiven_io>
>> > >>> > *Aiven Deutschland GmbH*
>> > >>> > Alexanderufer 3-7, 10117 Berlin
>> > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > >>> > Amtsgericht Charlottenburg, HRB 209739 B
>> > >>>
>> >
>>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe <cm...@apache.org>]

On Fri, Jan 5, 2024, at 11:28, Greg Harris wrote:
> Hi Josep,
>
> Thanks for the KIP! I think that the motivations for having another
> feature release before 4.0 are valid, as it seems that 3.7 is not
> suitable as the last 3.x release.
> I had a concern about the "Scope" section, in particular the language
> about "must haves", and the prevention of further 3.x releases.
>
> As I understand it, the Kafka project uses Time based releases [1], in
> which the release cadence has a fixed time between releases, and the
> set of features in a release is flexible. This is in contrast to
> feature-based releases where the release cadence is flexible, but the
> set of features in a release is fixed.
> If we are describing particular KIPs as "must-haves" for 3.8, does
> that mean that 3.8 is not a time-based release, but a feature-based
> release? For example, if KIP-853 was not ready by a deadline in the
> release plan, what would happen to the 3.8 release? Would it be
> delayed until KIP-853 was ready, or would KIP-853 be postponed until a
> 3.9 release?
> Is KIP-853 sufficiently low-risk that slipping from 3.8 is impossible?
> I am not familiar enough with that KIP to answer that question.
>

Hi Greg,

If we cannot get KIP-853 done in time for 3.8, then we'd move to have another 3.x release.

If you like, we can write that into the KIP. (I don't see why not).

>
> Looking back at the discussion on KIP-833 [2], it looks like the "last
> 3.x version" was interpreted to be an estimation, and the KIP itself
> includes "Note: this timeline is very rough and subject to change."
> I'll take that to mean that accepting KIP-833 did not count as
> explicitly voting that 3.7 would be the last release.
>

The intention behind saying the timeline was "rough" was to make the obvious comment that if we shipped 3.7 in February rather than in January, it would still be in the spirit of following the KIP.

I kind of regret putting that comment in there now, since apparently there was a lot of misinterpretation! It didn't mean that the KIP itself was just a suggestion and not binding, or that we hadn't come to a consensus about 3.7 being the last release.

>
> It looks like in the past, we have had loose discussion threads about
> bumping to the next major version (ex 1.0 [3] 2.0 [4] and 3.0 [5]) but
> I don't see anything labeled VOTE with formal consensus. It looks like
> each of the discussions reached informal consensus after the final
> minor release reached feature freeze.
>
>
> If we were to approve this KIP, we would be formally voting that 4.0
> is the next release after 3.8. If we were to follow a similar process
> today, we would (1) vote down a bump to 4.0 after 3.7, (2) proceed
> with a 3.8 release, and (3) after 3.8 feature freeze vote whether to
> have 4.0.
>

To make this totally clear:

NO vote for this KIP ===> 3.7 is the last 3.x release, as per KIP-833.
YES vote for this KIP ===> 3.8 is the last 3.x release.

>
> There are two separate things to reach consensus about here: Whether
> the 3.8 release will exist, and when the 4.0 release will appear. We
> can vote for a 3.8 release with or without also agreeing on the timing
> for 4.0, and I think that committing to 4.0 now is in violation of our
> time-based release policy.
> I am whole-heartedly +1 for a 3.8, but I'm hesitant about voting on
> 4.0 before the feature set of 3.8 is finalized. I think there is a
> risk that features that are on-time and eligible for a 3.8 release
> could be delayed by some KIPs which are given special treatment.

Nobody is proposing changing away from time-based releases. The vote is about what we need to have in the last 3.x release. The dates are there to give people a sense of when that release will be.

Also, if you have other stuff you want to get into 3.8, please submit PRs / KIPs and we'll follow the normal process as usual. Actually, this is something we should state explicitly in the KIP : there may be other features besides the "required" ones. I did ask Josep to make that change to the KIP. I will mention it again in the VOTE thread.

best,
Colin

>
> [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> [2] https://lists.apache.org/thread/90zkqvmmw3y8j6tkgbg3md78m7hs4yn6
> [3] https://lists.apache.org/thread/vr1pwbz3q0dpkphb4m4b1snxc4cy8otm
> [4] https://lists.apache.org/thread/9vc5q3m6y68czt929tgnk0frcf12oypt
> [5] https://lists.apache.org/thread/76dt0s95s73j7rwtccwjf0nh6zkn2sft
>
> Thanks all!
> Greg
>
>
> On Fri, Jan 5, 2024 at 9:25 AM Justine Olshan
> <jo...@confluent.io.invalid> wrote:
>>
>> While I agree we should have this release and should vote on it soon, is it
>> worth determining the exact items we need before we vote? Just so we are
>> all in agreement?
>> There is still some discussion on the road to 4.0 thread that may be worth
>> having here.
>>
>> On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <jo...@aiven.io.invalid>
>> wrote:
>>
>> > Hi Colin,
>> > Sorry for being quiet these last days (PTO).
>> >
>> > I will start the vote thread right away.
>> >
>> > Best,
>> >
>> >
>> > ---
>> > Josep Prat
>> > Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
>> > +491715557497 | aiven.io
>> > Aiven Deutschland GmbH
>> > Alexanderufer 3-7, 10117 Berlin
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > Amtsgericht Charlottenburg, HRB 209739 B
>> >
>> > On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:
>> >
>> > > Hi all,
>> > >
>> > > Since this has been open for a few weeks, are there any objections to
>> > > starting the vote? What do you think, Josep?
>> > >
>> > > Since 3.8 is going to be the next release (according to the KIP) we
>> > should
>> > > really vote this in as soon as possible.
>> > >
>> > > Also, I created a wiki page about the 3.8 release with a tentative
>> > > schedule.
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
>> > >
>> > > Please let me know if these dates make sense -- they're just proposals
>> > > right now.
>> > >
>> > > best,
>> > > Colin
>> > >
>> > >
>> > > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
>> > > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
>> > > >> Hey Colin,
>> > > >>
>> > > >> Some folks were concerned about the lack of automatic unclean leader
>> > > >> election. I mentioned that KIP-966 would actually be better with its
>> > > >> aggressive recovery option.
>> > > >> I think folks were hoping for some availability over durability
>> > solution
>> > > >> for KRaft, so if we don't do KIP-966 we should provide an alternative
>> > > or be
>> > > >> able to convince ourselves it is not needed.
>> > > >
>> > > > Hi Justine,
>> > > >
>> > > > That's a fair point. We should specify in KIP-1012 that we need to have
>> > > > some way to configure the system to automatically do unclean leader
>> > > > election. If we run out of time implementing KIP-966, this could be
>> > > > something quite simple, like honoring the static
>> > > > unclean.leader.election = true configuration.
>> > > >
>> > > >>
>> > > >> I think while many folks decided KIP-853 was a blocker, there were a
>> > > lot of
>> > > >> other features that many folks were expecting so I don't think we can
>> > > say
>> > > >> definitively the only must-have is KIP-853 (and hence the discussion
>> > > thread
>> > > >> here :) )
>> > > >>
>> > > >> Also as an aside, I filed a ticket to remove ZK from the top of the
>> > > >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
>> > > >>
>> > > >
>> > > > Yeah. There is a bunch of docs and quickstart cleanup that we should
>> > > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new
>> > > > year is a good time to clean things up.
>> > > >
>> > > > best,
>> > > > Colin
>> > > >
>> > > >
>> > > >> Justine
>> > > >>
>> > > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> Hi Josep,
>> > > >>>
>> > > >>> Thanks for the KIP. Based on the discussions we had previously, I
>> > agree
>> > > >>> that we need a 3.8.
>> > > >>>
>> > > >>> It would be good to link to KIP-833 in the motivation section, since
>> > > this
>> > > >>> KIP builds on that one.
>> > > >>>
>> > > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
>> > > >>> general-purpose release that may add some new features. This was
>> > > something
>> > > >>> that we were on the fence about previously, so it would be good to
>> > > clarify
>> > > >>> it here.
>> > > >>>
>> > > >>> On another note. I don't think KIP-966 is a "must-have" for Kafka
>> > 3.8,
>> > > as
>> > > >>> the KIP currently states. I certainly hope that it makes it for 3.8,
>> > > but if
>> > > >>> it doesn't, it can go into 4.0. It's not needed for migration, so it
>> > > could
>> > > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really
>> > > needs
>> > > >>> is "KIP-853: KRaft Controller Membership Changes."
>> > > >>>
>> > > >>> Along these lines, I think we should drop the language about
>> > "strategic
>> > > >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper,
>> > > and
>> > > >>> doesn't need feature parity with it. For example, ZK implemented
>> > > >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably
>> > > won't
>> > > >>> in 3.8). We probably won't add this -- or if we do, it won't be so
>> > > that we
>> > > >>> can have "parity with ZK." Really the only must-have in 3.8 is
>> > > KIP-853, and
>> > > >>> we should be clear about that.
>> > > >>>
>> > > >>> I think we should start issuing a deprecation log message at ERROR
>> > > level
>> > > >>> when brokers start up in ZK mode. This message could point out that
>> > > some
>> > > >>> safety mechanisms and new features will not be available in ZK mode,
>> > > and
>> > > >>> give a link to our documentation about migration.
>> > > >>>
>> > > >>> We should probably also move the example configurations for kraft
>> > from
>> > > >>> config/kraft to config. And move the zk ones into config/zk. Or maybe
>> > > even
>> > > >>> drop the ZK ones altogether, since they're not needed for migration
>> > or
>> > > >>> upgrade.
>> > > >>>
>> > > >>> best,
>> > > >>> Colin
>> > > >>>
>> > > >>>
>> > > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
>> > > >>> > On this note, I'd like to add that I would volunteer to be the
>> > > release
>> > > >>> > manager of such release 3.8.0.
>> > > >>> >
>> > > >>> > Best,
>> > > >>> >
>> > > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io>
>> > > wrote:
>> > > >>> >
>> > > >>> >> Hi all!
>> > > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
>> > > KIP-1012 to
>> > > >>> >> discuss and I'd like to open it up for discussion:
>> > > >>> >>
>> > > >>>
>> > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>> > > >>> >>
>> > > >>> >> Let's use this KIP to:
>> > > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
>> > > >>> >> b) Define which KIPs are the must-haves in regards to KRaft that
>> > > should
>> > > >>> be
>> > > >>> >> included there.
>> > > >>> >>
>> > > >>> >> Please let me know your feedback and suggestions.
>> > > >>> >>
>> > > >>> >> Best,
>> > > >>> >>
>> > > >>> >> --
>> > > >>> >> [image: Aiven] <https://www.aiven.io>
>> > > >>> >>
>> > > >>> >> *Josep Prat*
>> > > >>> >> Open Source Engineering Director, *Aiven*
>> > > >>> >> josep.prat@aiven.io   |   +491715557497
>> > > >>> >> aiven.io <https://www.aiven.io>   |
>> > > >>> >> <https://www.facebook.com/aivencloud>
>> > > >>> >> <https://www.linkedin.com/company/aiven/>   <
>> > > >>> https://twitter.com/aiven_io>
>> > > >>> >> *Aiven Deutschland GmbH*
>> > > >>> >> Alexanderufer 3-7, 10117 Berlin
>> > > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > > >>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> > > >>> >>
>> > > >>> >
>> > > >>> >
>> > > >>> > --
>> > > >>> > [image: Aiven] <https://www.aiven.io>
>> > > >>> >
>> > > >>> > *Josep Prat*
>> > > >>> > Open Source Engineering Director, *Aiven*
>> > > >>> > josep.prat@aiven.io   |   +491715557497
>> > > >>> > aiven.io <https://www.aiven.io>   |   <
>> > > >>> https://www.facebook.com/aivencloud>
>> > > >>> >   <https://www.linkedin.com/company/aiven/>   <
>> > > >>> https://twitter.com/aiven_io>
>> > > >>> > *Aiven Deutschland GmbH*
>> > > >>> > Alexanderufer 3-7, 10117 Berlin
>> > > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > > >>> > Amtsgericht Charlottenburg, HRB 209739 B
>> > > >>>
>> > >
>> >

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Greg Harris <gr...@aiven.io.INVALID>]

Hi Josep,

Thanks for the KIP! I think that the motivations for having another
feature release before 4.0 are valid, as it seems that 3.7 is not
suitable as the last 3.x release.
I had a concern about the "Scope" section, in particular the language
about "must haves", and the prevention of further 3.x releases.

As I understand it, the Kafka project uses Time based releases [1], in
which the release cadence has a fixed time between releases, and the
set of features in a release is flexible. This is in contrast to
feature-based releases where the release cadence is flexible, but the
set of features in a release is fixed.
If we are describing particular KIPs as "must-haves" for 3.8, does
that mean that 3.8 is not a time-based release, but a feature-based
release? For example, if KIP-853 was not ready by a deadline in the
release plan, what would happen to the 3.8 release? Would it be
delayed until KIP-853 was ready, or would KIP-853 be postponed until a
3.9 release?
Is KIP-853 sufficiently low-risk that slipping from 3.8 is impossible?
I am not familiar enough with that KIP to answer that question.

Looking back at the discussion on KIP-833 [2], it looks like the "last
3.x version" was interpreted to be an estimation, and the KIP itself
includes "Note: this timeline is very rough and subject to change."
I'll take that to mean that accepting KIP-833 did not count as
explicitly voting that 3.7 would be the last release.
It looks like in the past, we have had loose discussion threads about
bumping to the next major version (ex 1.0 [3] 2.0 [4] and 3.0 [5]) but
I don't see anything labeled VOTE with formal consensus. It looks like
each of the discussions reached informal consensus after the final
minor release reached feature freeze.
If we were to approve this KIP, we would be formally voting that 4.0
is the next release after 3.8. If we were to follow a similar process
today, we would (1) vote down a bump to 4.0 after 3.7, (2) proceed
with a 3.8 release, and (3) after 3.8 feature freeze vote whether to
have 4.0.

There are two separate things to reach consensus about here: Whether
the 3.8 release will exist, and when the 4.0 release will appear. We
can vote for a 3.8 release with or without also agreeing on the timing
for 4.0, and I think that committing to 4.0 now is in violation of our
time-based release policy.
I am whole-heartedly +1 for a 3.8, but I'm hesitant about voting on
4.0 before the feature set of 3.8 is finalized. I think there is a
risk that features that are on-time and eligible for a 3.8 release
could be delayed by some KIPs which are given special treatment.

[1] https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
[2] https://lists.apache.org/thread/90zkqvmmw3y8j6tkgbg3md78m7hs4yn6
[3] https://lists.apache.org/thread/vr1pwbz3q0dpkphb4m4b1snxc4cy8otm
[4] https://lists.apache.org/thread/9vc5q3m6y68czt929tgnk0frcf12oypt
[5] https://lists.apache.org/thread/76dt0s95s73j7rwtccwjf0nh6zkn2sft

Thanks all!
Greg


On Fri, Jan 5, 2024 at 9:25 AM Justine Olshan
<jo...@confluent.io.invalid> wrote:
>
> While I agree we should have this release and should vote on it soon, is it
> worth determining the exact items we need before we vote? Just so we are
> all in agreement?
> There is still some discussion on the road to 4.0 thread that may be worth
> having here.
>
> On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <jo...@aiven.io.invalid>
> wrote:
>
> > Hi Colin,
> > Sorry for being quiet these last days (PTO).
> >
> > I will start the vote thread right away.
> >
> > Best,
> >
> >
> > ---
> > Josep Prat
> > Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > Since this has been open for a few weeks, are there any objections to
> > > starting the vote? What do you think, Josep?
> > >
> > > Since 3.8 is going to be the next release (according to the KIP) we
> > should
> > > really vote this in as soon as possible.
> > >
> > > Also, I created a wiki page about the 3.8 release with a tentative
> > > schedule.
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
> > >
> > > Please let me know if these dates make sense -- they're just proposals
> > > right now.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
> > > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
> > > >> Hey Colin,
> > > >>
> > > >> Some folks were concerned about the lack of automatic unclean leader
> > > >> election. I mentioned that KIP-966 would actually be better with its
> > > >> aggressive recovery option.
> > > >> I think folks were hoping for some availability over durability
> > solution
> > > >> for KRaft, so if we don't do KIP-966 we should provide an alternative
> > > or be
> > > >> able to convince ourselves it is not needed.
> > > >
> > > > Hi Justine,
> > > >
> > > > That's a fair point. We should specify in KIP-1012 that we need to have
> > > > some way to configure the system to automatically do unclean leader
> > > > election. If we run out of time implementing KIP-966, this could be
> > > > something quite simple, like honoring the static
> > > > unclean.leader.election = true configuration.
> > > >
> > > >>
> > > >> I think while many folks decided KIP-853 was a blocker, there were a
> > > lot of
> > > >> other features that many folks were expecting so I don't think we can
> > > say
> > > >> definitively the only must-have is KIP-853 (and hence the discussion
> > > thread
> > > >> here :) )
> > > >>
> > > >> Also as an aside, I filed a ticket to remove ZK from the top of the
> > > >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
> > > >>
> > > >
> > > > Yeah. There is a bunch of docs and quickstart cleanup that we should
> > > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new
> > > > year is a good time to clean things up.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > >> Justine
> > > >>
> > > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > >>
> > > >>> Hi Josep,
> > > >>>
> > > >>> Thanks for the KIP. Based on the discussions we had previously, I
> > agree
> > > >>> that we need a 3.8.
> > > >>>
> > > >>> It would be good to link to KIP-833 in the motivation section, since
> > > this
> > > >>> KIP builds on that one.
> > > >>>
> > > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
> > > >>> general-purpose release that may add some new features. This was
> > > something
> > > >>> that we were on the fence about previously, so it would be good to
> > > clarify
> > > >>> it here.
> > > >>>
> > > >>> On another note. I don't think KIP-966 is a "must-have" for Kafka
> > 3.8,
> > > as
> > > >>> the KIP currently states. I certainly hope that it makes it for 3.8,
> > > but if
> > > >>> it doesn't, it can go into 4.0. It's not needed for migration, so it
> > > could
> > > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really
> > > needs
> > > >>> is "KIP-853: KRaft Controller Membership Changes."
> > > >>>
> > > >>> Along these lines, I think we should drop the language about
> > "strategic
> > > >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper,
> > > and
> > > >>> doesn't need feature parity with it. For example, ZK implemented
> > > >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably
> > > won't
> > > >>> in 3.8). We probably won't add this -- or if we do, it won't be so
> > > that we
> > > >>> can have "parity with ZK." Really the only must-have in 3.8 is
> > > KIP-853, and
> > > >>> we should be clear about that.
> > > >>>
> > > >>> I think we should start issuing a deprecation log message at ERROR
> > > level
> > > >>> when brokers start up in ZK mode. This message could point out that
> > > some
> > > >>> safety mechanisms and new features will not be available in ZK mode,
> > > and
> > > >>> give a link to our documentation about migration.
> > > >>>
> > > >>> We should probably also move the example configurations for kraft
> > from
> > > >>> config/kraft to config. And move the zk ones into config/zk. Or maybe
> > > even
> > > >>> drop the ZK ones altogether, since they're not needed for migration
> > or
> > > >>> upgrade.
> > > >>>
> > > >>> best,
> > > >>> Colin
> > > >>>
> > > >>>
> > > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
> > > >>> > On this note, I'd like to add that I would volunteer to be the
> > > release
> > > >>> > manager of such release 3.8.0.
> > > >>> >
> > > >>> > Best,
> > > >>> >
> > > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io>
> > > wrote:
> > > >>> >
> > > >>> >> Hi all!
> > > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
> > > KIP-1012 to
> > > >>> >> discuss and I'd like to open it up for discussion:
> > > >>> >>
> > > >>>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > >>> >>
> > > >>> >> Let's use this KIP to:
> > > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
> > > >>> >> b) Define which KIPs are the must-haves in regards to KRaft that
> > > should
> > > >>> be
> > > >>> >> included there.
> > > >>> >>
> > > >>> >> Please let me know your feedback and suggestions.
> > > >>> >>
> > > >>> >> Best,
> > > >>> >>
> > > >>> >> --
> > > >>> >> [image: Aiven] <https://www.aiven.io>
> > > >>> >>
> > > >>> >> *Josep Prat*
> > > >>> >> Open Source Engineering Director, *Aiven*
> > > >>> >> josep.prat@aiven.io   |   +491715557497
> > > >>> >> aiven.io <https://www.aiven.io>   |
> > > >>> >> <https://www.facebook.com/aivencloud>
> > > >>> >> <https://www.linkedin.com/company/aiven/>   <
> > > >>> https://twitter.com/aiven_io>
> > > >>> >> *Aiven Deutschland GmbH*
> > > >>> >> Alexanderufer 3-7, 10117 Berlin
> > > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>> >> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>> >>
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > [image: Aiven] <https://www.aiven.io>
> > > >>> >
> > > >>> > *Josep Prat*
> > > >>> > Open Source Engineering Director, *Aiven*
> > > >>> > josep.prat@aiven.io   |   +491715557497
> > > >>> > aiven.io <https://www.aiven.io>   |   <
> > > >>> https://www.facebook.com/aivencloud>
> > > >>> >   <https://www.linkedin.com/company/aiven/>   <
> > > >>> https://twitter.com/aiven_io>
> > > >>> > *Aiven Deutschland GmbH*
> > > >>> > Alexanderufer 3-7, 10117 Berlin
> > > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>> > Amtsgericht Charlottenburg, HRB 209739 B
> > > >>>
> > >
> >

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Justine Olshan <jo...@confluent.io.INVALID>]

While I agree we should have this release and should vote on it soon, is it
worth determining the exact items we need before we vote? Just so we are
all in agreement?
There is still some discussion on the road to 4.0 thread that may be worth
having here.

On Fri, Jan 5, 2024 at 1:25 AM Josep Prat <jo...@aiven.io.invalid>
wrote:

> Hi Colin,
> Sorry for being quiet these last days (PTO).
>
> I will start the vote thread right away.
>
> Best,
>
>
> ---
> Josep Prat
> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:
>
> > Hi all,
> >
> > Since this has been open for a few weeks, are there any objections to
> > starting the vote? What do you think, Josep?
> >
> > Since 3.8 is going to be the next release (according to the KIP) we
> should
> > really vote this in as soon as possible.
> >
> > Also, I created a wiki page about the 3.8 release with a tentative
> > schedule.
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
> >
> > Please let me know if these dates make sense -- they're just proposals
> > right now.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
> > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
> > >> Hey Colin,
> > >>
> > >> Some folks were concerned about the lack of automatic unclean leader
> > >> election. I mentioned that KIP-966 would actually be better with its
> > >> aggressive recovery option.
> > >> I think folks were hoping for some availability over durability
> solution
> > >> for KRaft, so if we don't do KIP-966 we should provide an alternative
> > or be
> > >> able to convince ourselves it is not needed.
> > >
> > > Hi Justine,
> > >
> > > That's a fair point. We should specify in KIP-1012 that we need to have
> > > some way to configure the system to automatically do unclean leader
> > > election. If we run out of time implementing KIP-966, this could be
> > > something quite simple, like honoring the static
> > > unclean.leader.election = true configuration.
> > >
> > >>
> > >> I think while many folks decided KIP-853 was a blocker, there were a
> > lot of
> > >> other features that many folks were expecting so I don't think we can
> > say
> > >> definitively the only must-have is KIP-853 (and hence the discussion
> > thread
> > >> here :) )
> > >>
> > >> Also as an aside, I filed a ticket to remove ZK from the top of the
> > >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
> > >>
> > >
> > > Yeah. There is a bunch of docs and quickstart cleanup that we should
> > > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new
> > > year is a good time to clean things up.
> > >
> > > best,
> > > Colin
> > >
> > >
> > >> Justine
> > >>
> > >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > >>
> > >>> Hi Josep,
> > >>>
> > >>> Thanks for the KIP. Based on the discussions we had previously, I
> agree
> > >>> that we need a 3.8.
> > >>>
> > >>> It would be good to link to KIP-833 in the motivation section, since
> > this
> > >>> KIP builds on that one.
> > >>>
> > >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
> > >>> general-purpose release that may add some new features. This was
> > something
> > >>> that we were on the fence about previously, so it would be good to
> > clarify
> > >>> it here.
> > >>>
> > >>> On another note. I don't think KIP-966 is a "must-have" for Kafka
> 3.8,
> > as
> > >>> the KIP currently states. I certainly hope that it makes it for 3.8,
> > but if
> > >>> it doesn't, it can go into 4.0. It's not needed for migration, so it
> > could
> > >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really
> > needs
> > >>> is "KIP-853: KRaft Controller Membership Changes."
> > >>>
> > >>> Along these lines, I think we should drop the language about
> "strategic
> > >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper,
> > and
> > >>> doesn't need feature parity with it. For example, ZK implemented
> > >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably
> > won't
> > >>> in 3.8). We probably won't add this -- or if we do, it won't be so
> > that we
> > >>> can have "parity with ZK." Really the only must-have in 3.8 is
> > KIP-853, and
> > >>> we should be clear about that.
> > >>>
> > >>> I think we should start issuing a deprecation log message at ERROR
> > level
> > >>> when brokers start up in ZK mode. This message could point out that
> > some
> > >>> safety mechanisms and new features will not be available in ZK mode,
> > and
> > >>> give a link to our documentation about migration.
> > >>>
> > >>> We should probably also move the example configurations for kraft
> from
> > >>> config/kraft to config. And move the zk ones into config/zk. Or maybe
> > even
> > >>> drop the ZK ones altogether, since they're not needed for migration
> or
> > >>> upgrade.
> > >>>
> > >>> best,
> > >>> Colin
> > >>>
> > >>>
> > >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
> > >>> > On this note, I'd like to add that I would volunteer to be the
> > release
> > >>> > manager of such release 3.8.0.
> > >>> >
> > >>> > Best,
> > >>> >
> > >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io>
> > wrote:
> > >>> >
> > >>> >> Hi all!
> > >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
> > KIP-1012 to
> > >>> >> discuss and I'd like to open it up for discussion:
> > >>> >>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > >>> >>
> > >>> >> Let's use this KIP to:
> > >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
> > >>> >> b) Define which KIPs are the must-haves in regards to KRaft that
> > should
> > >>> be
> > >>> >> included there.
> > >>> >>
> > >>> >> Please let me know your feedback and suggestions.
> > >>> >>
> > >>> >> Best,
> > >>> >>
> > >>> >> --
> > >>> >> [image: Aiven] <https://www.aiven.io>
> > >>> >>
> > >>> >> *Josep Prat*
> > >>> >> Open Source Engineering Director, *Aiven*
> > >>> >> josep.prat@aiven.io   |   +491715557497
> > >>> >> aiven.io <https://www.aiven.io>   |
> > >>> >> <https://www.facebook.com/aivencloud>
> > >>> >> <https://www.linkedin.com/company/aiven/>   <
> > >>> https://twitter.com/aiven_io>
> > >>> >> *Aiven Deutschland GmbH*
> > >>> >> Alexanderufer 3-7, 10117 Berlin
> > >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>> >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>> >>
> > >>> >
> > >>> >
> > >>> > --
> > >>> > [image: Aiven] <https://www.aiven.io>
> > >>> >
> > >>> > *Josep Prat*
> > >>> > Open Source Engineering Director, *Aiven*
> > >>> > josep.prat@aiven.io   |   +491715557497
> > >>> > aiven.io <https://www.aiven.io>   |   <
> > >>> https://www.facebook.com/aivencloud>
> > >>> >   <https://www.linkedin.com/company/aiven/>   <
> > >>> https://twitter.com/aiven_io>
> > >>> > *Aiven Deutschland GmbH*
> > >>> > Alexanderufer 3-7, 10117 Berlin
> > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>> > Amtsgericht Charlottenburg, HRB 209739 B
> > >>>
> >
>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Josep Prat <jo...@aiven.io.INVALID>]

Hi Colin,
Sorry for being quiet these last days (PTO).

I will start the vote thread right away.

Best,


---
Josep Prat
Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
+491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Fri, Jan 5, 2024, 00:24 Colin McCabe <cm...@apache.org> wrote:

> Hi all,
>
> Since this has been open for a few weeks, are there any objections to
> starting the vote? What do you think, Josep?
>
> Since 3.8 is going to be the next release (according to the KIP) we should
> really vote this in as soon as possible.
>
> Also, I created a wiki page about the 3.8 release with a tentative
> schedule.
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
>
> Please let me know if these dates make sense -- they're just proposals
> right now.
>
> best,
> Colin
>
>
> On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
> > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
> >> Hey Colin,
> >>
> >> Some folks were concerned about the lack of automatic unclean leader
> >> election. I mentioned that KIP-966 would actually be better with its
> >> aggressive recovery option.
> >> I think folks were hoping for some availability over durability solution
> >> for KRaft, so if we don't do KIP-966 we should provide an alternative
> or be
> >> able to convince ourselves it is not needed.
> >
> > Hi Justine,
> >
> > That's a fair point. We should specify in KIP-1012 that we need to have
> > some way to configure the system to automatically do unclean leader
> > election. If we run out of time implementing KIP-966, this could be
> > something quite simple, like honoring the static
> > unclean.leader.election = true configuration.
> >
> >>
> >> I think while many folks decided KIP-853 was a blocker, there were a
> lot of
> >> other features that many folks were expecting so I don't think we can
> say
> >> definitively the only must-have is KIP-853 (and hence the discussion
> thread
> >> here :) )
> >>
> >> Also as an aside, I filed a ticket to remove ZK from the top of the
> >> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
> >>
> >
> > Yeah. There is a bunch of docs and quickstart cleanup that we should
> > do. I don't think any of it is a blocker for 3.8 or 4.0, but the new
> > year is a good time to clean things up.
> >
> > best,
> > Colin
> >
> >
> >> Justine
> >>
> >> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org>
> wrote:
> >>
> >>> Hi Josep,
> >>>
> >>> Thanks for the KIP. Based on the discussions we had previously, I agree
> >>> that we need a 3.8.
> >>>
> >>> It would be good to link to KIP-833 in the motivation section, since
> this
> >>> KIP builds on that one.
> >>>
> >>> Also, I think we should mention in KIP-1012 that 3.8 will be a
> >>> general-purpose release that may add some new features. This was
> something
> >>> that we were on the fence about previously, so it would be good to
> clarify
> >>> it here.
> >>>
> >>> On another note. I don't think KIP-966 is a "must-have" for Kafka 3.8,
> as
> >>> the KIP currently states. I certainly hope that it makes it for 3.8,
> but if
> >>> it doesn't, it can go into 4.0. It's not needed for migration, so it
> could
> >>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really
> needs
> >>> is "KIP-853: KRaft Controller Membership Changes."
> >>>
> >>> Along these lines, I think we should drop the language about "strategic
> >>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper,
> and
> >>> doesn't need feature parity with it. For example, ZK implemented
> >>> Netty-TcNative OpenSSL Support, but we don't have that (and probably
> won't
> >>> in 3.8). We probably won't add this -- or if we do, it won't be so
> that we
> >>> can have "parity with ZK." Really the only must-have in 3.8 is
> KIP-853, and
> >>> we should be clear about that.
> >>>
> >>> I think we should start issuing a deprecation log message at ERROR
> level
> >>> when brokers start up in ZK mode. This message could point out that
> some
> >>> safety mechanisms and new features will not be available in ZK mode,
> and
> >>> give a link to our documentation about migration.
> >>>
> >>> We should probably also move the example configurations for kraft from
> >>> config/kraft to config. And move the zk ones into config/zk. Or maybe
> even
> >>> drop the ZK ones altogether, since they're not needed for migration or
> >>> upgrade.
> >>>
> >>> best,
> >>> Colin
> >>>
> >>>
> >>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
> >>> > On this note, I'd like to add that I would volunteer to be the
> release
> >>> > manager of such release 3.8.0.
> >>> >
> >>> > Best,
> >>> >
> >>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io>
> wrote:
> >>> >
> >>> >> Hi all!
> >>> >> As agreed on the "Road to Kafka 4.0" email thread, I created
> KIP-1012 to
> >>> >> discuss and I'd like to open it up for discussion:
> >>> >>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> >>> >>
> >>> >> Let's use this KIP to:
> >>> >> a) Leave a papertrail agreement for the need of a 3.8 version
> >>> >> b) Define which KIPs are the must-haves in regards to KRaft that
> should
> >>> be
> >>> >> included there.
> >>> >>
> >>> >> Please let me know your feedback and suggestions.
> >>> >>
> >>> >> Best,
> >>> >>
> >>> >> --
> >>> >> [image: Aiven] <https://www.aiven.io>
> >>> >>
> >>> >> *Josep Prat*
> >>> >> Open Source Engineering Director, *Aiven*
> >>> >> josep.prat@aiven.io   |   +491715557497
> >>> >> aiven.io <https://www.aiven.io>   |
> >>> >> <https://www.facebook.com/aivencloud>
> >>> >> <https://www.linkedin.com/company/aiven/>   <
> >>> https://twitter.com/aiven_io>
> >>> >> *Aiven Deutschland GmbH*
> >>> >> Alexanderufer 3-7, 10117 Berlin
> >>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > [image: Aiven] <https://www.aiven.io>
> >>> >
> >>> > *Josep Prat*
> >>> > Open Source Engineering Director, *Aiven*
> >>> > josep.prat@aiven.io   |   +491715557497
> >>> > aiven.io <https://www.aiven.io>   |   <
> >>> https://www.facebook.com/aivencloud>
> >>> >   <https://www.linkedin.com/company/aiven/>   <
> >>> https://twitter.com/aiven_io>
> >>> > *Aiven Deutschland GmbH*
> >>> > Alexanderufer 3-7, 10117 Berlin
> >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>> > Amtsgericht Charlottenburg, HRB 209739 B
> >>>
>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe <cm...@apache.org>]

Hi all,

Since this has been open for a few weeks, are there any objections to starting the vote? What do you think, Josep?

Since 3.8 is going to be the next release (according to the KIP) we should really vote this in as soon as possible.

Also, I created a wiki page about the 3.8 release with a tentative schedule.
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0

Please let me know if these dates make sense -- they're just proposals right now.

best,
Colin


On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
> On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
>> Hey Colin,
>>
>> Some folks were concerned about the lack of automatic unclean leader
>> election. I mentioned that KIP-966 would actually be better with its
>> aggressive recovery option.
>> I think folks were hoping for some availability over durability solution
>> for KRaft, so if we don't do KIP-966 we should provide an alternative or be
>> able to convince ourselves it is not needed.
>
> Hi Justine,
>
> That's a fair point. We should specify in KIP-1012 that we need to have 
> some way to configure the system to automatically do unclean leader 
> election. If we run out of time implementing KIP-966, this could be 
> something quite simple, like honoring the static 
> unclean.leader.election = true configuration.
>
>>
>> I think while many folks decided KIP-853 was a blocker, there were a lot of
>> other features that many folks were expecting so I don't think we can say
>> definitively the only must-have is KIP-853 (and hence the discussion thread
>> here :) )
>>
>> Also as an aside, I filed a ticket to remove ZK from the top of the
>> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
>>
>
> Yeah. There is a bunch of docs and quickstart cleanup that we should 
> do. I don't think any of it is a blocker for 3.8 or 4.0, but the new 
> year is a good time to clean things up.
>
> best,
> Colin
>
>
>> Justine
>>
>> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org> wrote:
>>
>>> Hi Josep,
>>>
>>> Thanks for the KIP. Based on the discussions we had previously, I agree
>>> that we need a 3.8.
>>>
>>> It would be good to link to KIP-833 in the motivation section, since this
>>> KIP builds on that one.
>>>
>>> Also, I think we should mention in KIP-1012 that 3.8 will be a
>>> general-purpose release that may add some new features. This was something
>>> that we were on the fence about previously, so it would be good to clarify
>>> it here.
>>>
>>> On another note. I don't think KIP-966 is a "must-have" for Kafka 3.8, as
>>> the KIP currently states. I certainly hope that it makes it for 3.8, but if
>>> it doesn't, it can go into 4.0. It's not needed for migration, so it could
>>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really needs
>>> is "KIP-853: KRaft Controller Membership Changes."
>>>
>>> Along these lines, I think we should drop the language about "strategic
>>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper, and
>>> doesn't need feature parity with it. For example, ZK implemented
>>> Netty-TcNative OpenSSL Support, but we don't have that (and probably won't
>>> in 3.8). We probably won't add this -- or if we do, it won't be so that we
>>> can have "parity with ZK." Really the only must-have in 3.8 is KIP-853, and
>>> we should be clear about that.
>>>
>>> I think we should start issuing a deprecation log message at ERROR  level
>>> when brokers start up in ZK mode. This message could point out that some
>>> safety mechanisms and new features will not be available in ZK mode, and
>>> give a link to our documentation about migration.
>>>
>>> We should probably also move the example configurations for kraft from
>>> config/kraft to config. And move the zk ones into config/zk. Or maybe even
>>> drop the ZK ones altogether, since they're not needed for migration or
>>> upgrade.
>>>
>>> best,
>>> Colin
>>>
>>>
>>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
>>> > On this note, I'd like to add that I would volunteer to be the release
>>> > manager of such release 3.8.0.
>>> >
>>> > Best,
>>> >
>>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io> wrote:
>>> >
>>> >> Hi all!
>>> >> As agreed on the "Road to Kafka 4.0" email thread, I created KIP-1012 to
>>> >> discuss and I'd like to open it up for discussion:
>>> >>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>>> >>
>>> >> Let's use this KIP to:
>>> >> a) Leave a papertrail agreement for the need of a 3.8 version
>>> >> b) Define which KIPs are the must-haves in regards to KRaft that should
>>> be
>>> >> included there.
>>> >>
>>> >> Please let me know your feedback and suggestions.
>>> >>
>>> >> Best,
>>> >>
>>> >> --
>>> >> [image: Aiven] <https://www.aiven.io>
>>> >>
>>> >> *Josep Prat*
>>> >> Open Source Engineering Director, *Aiven*
>>> >> josep.prat@aiven.io   |   +491715557497
>>> >> aiven.io <https://www.aiven.io>   |
>>> >> <https://www.facebook.com/aivencloud>
>>> >> <https://www.linkedin.com/company/aiven/>   <
>>> https://twitter.com/aiven_io>
>>> >> *Aiven Deutschland GmbH*
>>> >> Alexanderufer 3-7, 10117 Berlin
>>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> >> Amtsgericht Charlottenburg, HRB 209739 B
>>> >>
>>> >
>>> >
>>> > --
>>> > [image: Aiven] <https://www.aiven.io>
>>> >
>>> > *Josep Prat*
>>> > Open Source Engineering Director, *Aiven*
>>> > josep.prat@aiven.io   |   +491715557497
>>> > aiven.io <https://www.aiven.io>   |   <
>>> https://www.facebook.com/aivencloud>
>>> >   <https://www.linkedin.com/company/aiven/>   <
>>> https://twitter.com/aiven_io>
>>> > *Aiven Deutschland GmbH*
>>> > Alexanderufer 3-7, 10117 Berlin
>>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> > Amtsgericht Charlottenburg, HRB 209739 B
>>>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe <cm...@apache.org>]

On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
> Hey Colin,
>
> Some folks were concerned about the lack of automatic unclean leader
> election. I mentioned that KIP-966 would actually be better with its
> aggressive recovery option.
> I think folks were hoping for some availability over durability solution
> for KRaft, so if we don't do KIP-966 we should provide an alternative or be
> able to convince ourselves it is not needed.

Hi Justine,

That's a fair point. We should specify in KIP-1012 that we need to have some way to configure the system to automatically do unclean leader election. If we run out of time implementing KIP-966, this could be something quite simple, like honoring the static unclean.leader.election = true configuration.

>
> I think while many folks decided KIP-853 was a blocker, there were a lot of
> other features that many folks were expecting so I don't think we can say
> definitively the only must-have is KIP-853 (and hence the discussion thread
> here :) )
>
> Also as an aside, I filed a ticket to remove ZK from the top of the
> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
>

Yeah. There is a bunch of docs and quickstart cleanup that we should do. I don't think any of it is a blocker for 3.8 or 4.0, but the new year is a good time to clean things up.

best,
Colin


> Justine
>
> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org> wrote:
>
>> Hi Josep,
>>
>> Thanks for the KIP. Based on the discussions we had previously, I agree
>> that we need a 3.8.
>>
>> It would be good to link to KIP-833 in the motivation section, since this
>> KIP builds on that one.
>>
>> Also, I think we should mention in KIP-1012 that 3.8 will be a
>> general-purpose release that may add some new features. This was something
>> that we were on the fence about previously, so it would be good to clarify
>> it here.
>>
>> On another note. I don't think KIP-966 is a "must-have" for Kafka 3.8, as
>> the KIP currently states. I certainly hope that it makes it for 3.8, but if
>> it doesn't, it can go into 4.0. It's not needed for migration, so it could
>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really needs
>> is "KIP-853: KRaft Controller Membership Changes."
>>
>> Along these lines, I think we should drop the language about "strategic
>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper, and
>> doesn't need feature parity with it. For example, ZK implemented
>> Netty-TcNative OpenSSL Support, but we don't have that (and probably won't
>> in 3.8). We probably won't add this -- or if we do, it won't be so that we
>> can have "parity with ZK." Really the only must-have in 3.8 is KIP-853, and
>> we should be clear about that.
>>
>> I think we should start issuing a deprecation log message at ERROR  level
>> when brokers start up in ZK mode. This message could point out that some
>> safety mechanisms and new features will not be available in ZK mode, and
>> give a link to our documentation about migration.
>>
>> We should probably also move the example configurations for kraft from
>> config/kraft to config. And move the zk ones into config/zk. Or maybe even
>> drop the ZK ones altogether, since they're not needed for migration or
>> upgrade.
>>
>> best,
>> Colin
>>
>>
>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
>> > On this note, I'd like to add that I would volunteer to be the release
>> > manager of such release 3.8.0.
>> >
>> > Best,
>> >
>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io> wrote:
>> >
>> >> Hi all!
>> >> As agreed on the "Road to Kafka 4.0" email thread, I created KIP-1012 to
>> >> discuss and I'd like to open it up for discussion:
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>> >>
>> >> Let's use this KIP to:
>> >> a) Leave a papertrail agreement for the need of a 3.8 version
>> >> b) Define which KIPs are the must-haves in regards to KRaft that should
>> be
>> >> included there.
>> >>
>> >> Please let me know your feedback and suggestions.
>> >>
>> >> Best,
>> >>
>> >> --
>> >> [image: Aiven] <https://www.aiven.io>
>> >>
>> >> *Josep Prat*
>> >> Open Source Engineering Director, *Aiven*
>> >> josep.prat@aiven.io   |   +491715557497
>> >> aiven.io <https://www.aiven.io>   |
>> >> <https://www.facebook.com/aivencloud>
>> >> <https://www.linkedin.com/company/aiven/>   <
>> https://twitter.com/aiven_io>
>> >> *Aiven Deutschland GmbH*
>> >> Alexanderufer 3-7, 10117 Berlin
>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> >>
>> >
>> >
>> > --
>> > [image: Aiven] <https://www.aiven.io>
>> >
>> > *Josep Prat*
>> > Open Source Engineering Director, *Aiven*
>> > josep.prat@aiven.io   |   +491715557497
>> > aiven.io <https://www.aiven.io>   |   <
>> https://www.facebook.com/aivencloud>
>> >   <https://www.linkedin.com/company/aiven/>   <
>> https://twitter.com/aiven_io>
>> > *Aiven Deutschland GmbH*
>> > Alexanderufer 3-7, 10117 Berlin
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > Amtsgericht Charlottenburg, HRB 209739 B
>>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Justine Olshan <jo...@confluent.io.INVALID>]

Hey Colin,

Some folks were concerned about the lack of automatic unclean leader
election. I mentioned that KIP-966 would actually be better with its
aggressive recovery option.
I think folks were hoping for some availability over durability solution
for KRaft, so if we don't do KIP-966 we should provide an alternative or be
able to convince ourselves it is not needed.

I think while many folks decided KIP-853 was a blocker, there were a lot of
other features that many folks were expecting so I don't think we can say
definitively the only must-have is KIP-853 (and hence the discussion thread
here :) )

Also as an aside, I filed a ticket to remove ZK from the top of the
quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975

Justine

On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe <cm...@apache.org> wrote:

> Hi Josep,
>
> Thanks for the KIP. Based on the discussions we had previously, I agree
> that we need a 3.8.
>
> It would be good to link to KIP-833 in the motivation section, since this
> KIP builds on that one.
>
> Also, I think we should mention in KIP-1012 that 3.8 will be a
> general-purpose release that may add some new features. This was something
> that we were on the fence about previously, so it would be good to clarify
> it here.
>
> On another note. I don't think KIP-966 is a "must-have" for Kafka 3.8, as
> the KIP currently states. I certainly hope that it makes it for 3.8, but if
> it doesn't, it can go into 4.0. It's not needed for migration, so it could
> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really needs
> is "KIP-853: KRaft Controller Membership Changes."
>
> Along these lines, I think we should drop the language about "strategic
> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper, and
> doesn't need feature parity with it. For example, ZK implemented
> Netty-TcNative OpenSSL Support, but we don't have that (and probably won't
> in 3.8). We probably won't add this -- or if we do, it won't be so that we
> can have "parity with ZK." Really the only must-have in 3.8 is KIP-853, and
> we should be clear about that.
>
> I think we should start issuing a deprecation log message at ERROR  level
> when brokers start up in ZK mode. This message could point out that some
> safety mechanisms and new features will not be available in ZK mode, and
> give a link to our documentation about migration.
>
> We should probably also move the example configurations for kraft from
> config/kraft to config. And move the zk ones into config/zk. Or maybe even
> drop the ZK ones altogether, since they're not needed for migration or
> upgrade.
>
> best,
> Colin
>
>
> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
> > On this note, I'd like to add that I would volunteer to be the release
> > manager of such release 3.8.0.
> >
> > Best,
> >
> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io> wrote:
> >
> >> Hi all!
> >> As agreed on the "Road to Kafka 4.0" email thread, I created KIP-1012 to
> >> discuss and I'd like to open it up for discussion:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> >>
> >> Let's use this KIP to:
> >> a) Leave a papertrail agreement for the need of a 3.8 version
> >> b) Define which KIPs are the must-haves in regards to KRaft that should
> be
> >> included there.
> >>
> >> Please let me know your feedback and suggestions.
> >>
> >> Best,
> >>
> >> --
> >> [image: Aiven] <https://www.aiven.io>
> >>
> >> *Josep Prat*
> >> Open Source Engineering Director, *Aiven*
> >> josep.prat@aiven.io   |   +491715557497
> >> aiven.io <https://www.aiven.io>   |
> >> <https://www.facebook.com/aivencloud>
> >> <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> >> *Aiven Deutschland GmbH*
> >> Alexanderufer 3-7, 10117 Berlin
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.prat@aiven.io   |   +491715557497
> > aiven.io <https://www.aiven.io>   |   <
> https://www.facebook.com/aivencloud>
> >   <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe <cm...@apache.org>]

Hi Josep,

Thanks for the KIP. Based on the discussions we had previously, I agree that we need a 3.8.

It would be good to link to KIP-833 in the motivation section, since this KIP builds on that one.

Also, I think we should mention in KIP-1012 that 3.8 will be a general-purpose release that may add some new features. This was something that we were on the fence about previously, so it would be good to clarify it here.

On another note. I don't think KIP-966 is a "must-have" for Kafka 3.8, as the KIP currently states. I certainly hope that it makes it for 3.8, but if it doesn't, it can go into 4.0. It's not needed for migration, so it could just as easily go into 4.0 as 3.8. The only thing that KIP-966 really needs is "KIP-853: KRaft Controller Membership Changes."

Along these lines, I think we should drop the language about "strategic feature parity with Zookeeper." Kafka isn't competing with ZooKeeper, and doesn't need feature parity with it. For example, ZK implemented Netty-TcNative OpenSSL Support, but we don't have that (and probably won't in 3.8). We probably won't add this -- or if we do, it won't be so that we can have "parity with ZK." Really the only must-have in 3.8 is KIP-853, and we should be clear about that.

I think we should start issuing a deprecation log message at ERROR  level when brokers start up in ZK mode. This message could point out that some safety mechanisms and new features will not be available in ZK mode, and give a link to our documentation about migration.

We should probably also move the example configurations for kraft from config/kraft to config. And move the zk ones into config/zk. Or maybe even drop the ZK ones altogether, since they're not needed for migration or upgrade.

best,
Colin


On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
> On this note, I'd like to add that I would volunteer to be the release
> manager of such release 3.8.0.
>
> Best,
>
> On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io> wrote:
>
>> Hi all!
>> As agreed on the "Road to Kafka 4.0" email thread, I created KIP-1012 to
>> discuss and I'd like to open it up for discussion:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>>
>> Let's use this KIP to:
>> a) Leave a papertrail agreement for the need of a 3.8 version
>> b) Define which KIPs are the must-haves in regards to KRaft that should be
>> included there.
>>
>> Please let me know your feedback and suggestions.
>>
>> Best,
>>
>> --
>> [image: Aiven] <https://www.aiven.io>
>>
>> *Josep Prat*
>> Open Source Engineering Director, *Aiven*
>> josep.prat@aiven.io   |   +491715557497
>> aiven.io <https://www.aiven.io>   |
>> <https://www.facebook.com/aivencloud>
>> <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
>> *Aiven Deutschland GmbH*
>> Alexanderufer 3-7, 10117 Berlin
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>
>
> -- 
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
>   <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Josep Prat <jo...@aiven.io.INVALID>]

On this note, I'd like to add that I would volunteer to be the release
manager of such release 3.8.0.

Best,

On Fri, Dec 22, 2023 at 1:31 PM Josep Prat <jo...@aiven.io> wrote:

> Hi all!
> As agreed on the "Road to Kafka 4.0" email thread, I created KIP-1012 to
> discuss and I'd like to open it up for discussion:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>
> Let's use this KIP to:
> a) Leave a papertrail agreement for the need of a 3.8 version
> b) Define which KIPs are the must-haves in regards to KRaft that should be
> included there.
>
> Please let me know your feedback and suggestions.
>
> Best,
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |
> <https://www.facebook.com/aivencloud>
> <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B