You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Mickael Maison <mi...@apache.org> on 2023/05/22 15:30:10 UTC

[VOTE] 3.5.0 RC0

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.5.0. Some of the
major features include:
- KIP-710: Full support for distributed mode in dedicated MirrorMaker
2.0 clusters
- KIP-881: Rack-aware Partition Assignment for Kafka Consumers
- KIP-887: Add ConfigProvider to make use of environment variables
- KIP-889: Versioned State Stores
- KIP-894: Use incrementalAlterConfig for syncing topic configurations
- KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
Kafka Brokers

Release notes for the 3.5.0 release:
https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Friday, May 26, 5pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~mimaison/kafka-3.5.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/

* Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
https://github.com/apache/kafka/releases/tag/3.5.0-rc0

The PR adding the 35 documentation is not merged yet
(https://github.com/apache/kafka-site/pull/513)
* Documentation:
https://kafka.apache.org/35/documentation.html
* Protocol:
https://kafka.apache.org/35/protocol.html

* Successful Jenkins builds for the 3.5 branch:
Unit/integration tests: Jenkins is not detecting the 3.5 branch,
working with INFRA to sort it out:
https://issues.apache.org/jira/browse/INFRA-24577
System tests: The build is still running, I'll send an update once I
have the results

Thanks,
Mickael

Re: [VOTE] 3.5.0 RC0

Posted by Colin McCabe <cm...@apache.org>.

Hi Ismael,

I do personally think they are blockers. But just as importantly, KAFKA-14996 is not related to zk to kraft migration, and is a blocker. So we will need another RC in any case.

best,
Colin

On Wed, May 24, 2023, at 11:50, Ismael Juma wrote:
> Are all these blockers? For example, zk to kraft migration are is still in
> preview - can we fix some of these in 3.5.1?
>
> Ismael
>
> On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org> wrote:
>
>> Hi Mickael,
>>
>> Thanks for putting together this RC. Unfortunately, we've identified
>> several blocker issues in this release candidate.
>>
>> KAFKA-15009: New ACLs are not written to ZK during migration
>> KAFKA-15007: MV is not set correctly in the MetadataPropagator in
>> migration.
>> KAFKA-15004: Topic config changes are not synced during zk to kraft
>> migration (dual-write)
>> KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
>> (dual-write) when partitions are changed for topic
>> KAFKA-14996: The KRaft controller should properly handle overly large user
>> operations
>>
>> We are working on PRs for these issues and will get them in soon, we think!
>>
>> So unfortunately I have to leave a -1 here for RC0. Let's aim for another
>> RC next week.
>>
>> best,
>> Colin
>>
>> On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
>> > Hi David,
>> >
>> > We're already quite a bit behind schedule. If you think these fixes
>> > are really important and can be ready in the next couple of days, I'm
>> > open to backport them and build another release candidate. Let me know
>> > once you've investigated the severity of KAFKA-15010.
>> >
>> > Thanks,
>> > Mickael
>> >
>> >
>> > On Tue, May 23, 2023 at 6:34 PM David Arthur
>> > <da...@confluent.io.invalid> wrote:
>> >>
>> >> Mickael, we have some migration fixes on trunk, is it okay to
>> cherry-pick
>> >> these to 3.5?
>> >>
>> >> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
>> >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
>> >>
>> >> There is another issue KAFKA-15010 that I'm also investigating to
>> determine
>> >> the impact and likelihood of seeing it in practice. This one may be a
>> >> significant migration blocker
>> >>
>> >> Cheers,
>> >> David
>> >>
>> >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
>> mickael.maison@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi Christo,
>> >> >
>> >> > Yes this is expected. This happens when nested fields also accept
>> >> > optional tagged fields. The tables list all fields, so they may
>> >> > include _tagged_fields multiple times.
>> >> > Clearly the layout of this page could be improved, if you have ideas
>> >> > how to describe the protocol in a better way, feel free to share them.
>> >> >
>> >> > Thanks,
>> >> > Mickael
>> >> >
>> >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
>> mickael.maison@gmail.com>
>> >> > wrote:
>> >> > >
>> >> > > Hi Josep,
>> >> > >
>> >> > > Good catch! I opened a PR to fix this:
>> >> > > https://github.com/apache/kafka-site/pull/514
>> >> > >
>> >> > > Thanks,
>> >> > > Mickael
>> >> > >
>> >> > >
>> >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
>> christololov@gmail.com>
>> >> > wrote:
>> >> > > >
>> >> > > > Hey Mickael!
>> >> > > >
>> >> > > > I am giving a +1 (non-binding) for this candidate release.
>> >> > > >
>> >> > > > * Built from the binary tar.gz source with Java 17 and Scala 2.13
>> on
>> >> > Intel
>> >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
>> >> > > > * Ran unit and integration tests on Intel and ARM machines.
>> >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel
>> and ARM
>> >> > > > machines.
>> >> > > >
>> >> > > > Question:
>> >> > > > * I went through https://kafka.apache.org/35/protocol.html and
>> there
>> >> > are
>> >> > > > quite a few repetitive __tagged_fileds fields within the same
>> >> > structures -
>> >> > > > is this expected?
>> >> > > >
>> >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
>> <jo...@aiven.io.invalid>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Hi Mickael,
>> >> > > > > I just wanted to point out that I think the documentation you
>> >> > recently
>> >> > > > > merged on Kafka site regarding the 3.5.0 version has a problem
>> when
>> >> > it
>> >> > > > > states the version number and the sub-menu that links to
>> previous
>> >> > versions.
>> >> > > > > Left a comment here:
>> >> > > > >
>> >> >
>> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
>> >> > > > >
>> >> > > > > Best,
>> >> > > > >
>> >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <josep.prat@aiven.io
>> >
>> >> > wrote:
>> >> > > > >
>> >> > > > > > Hi Mickael,
>> >> > > > > >
>> >> > > > > > I can +1 this candidate. I verified the following:
>> >> > > > > > - Built from source with Java 17 and Scala 2.13
>> >> > > > > > - Signatures and hashes of the artifacts generated
>> >> > > > > > - Navigated through Javadoc including links to JDK classes
>> >> > > > > > - Run the unit tests
>> >> > > > > > - Run integration tests
>> >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
>> >> > > > > >
>> >> > > > > > Best,
>> >> > > > > >
>> >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
>> >> > mimaison@apache.org>
>> >> > > > > > wrote:
>> >> > > > > >
>> >> > > > > >> Hello Kafka users, developers and client-developers,
>> >> > > > > >>
>> >> > > > > >> This is the first candidate for release of Apache Kafka
>> 3.5.0.
>> >> > Some of
>> >> > > > > the
>> >> > > > > >> major features include:
>> >> > > > > >> - KIP-710: Full support for distributed mode in dedicated
>> >> > MirrorMaker
>> >> > > > > >> 2.0 clusters
>> >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
>> Consumers
>> >> > > > > >> - KIP-887: Add ConfigProvider to make use of environment
>> variables
>> >> > > > > >> - KIP-889: Versioned State Stores
>> >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
>> >> > configurations
>> >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to support
>> SCRAM
>> >> > for
>> >> > > > > >> Kafka Brokers
>> >> > > > > >>
>> >> > > > > >> Release notes for the 3.5.0 release:
>> >> > > > > >>
>> >> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>> >> > > > > >>
>> >> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
>> >> > > > > >>
>> >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the
>> release:
>> >> > > > > >> https://kafka.apache.org/KEYS
>> >> > > > > >>
>> >> > > > > >> * Release artifacts to be voted upon (source and binary):
>> >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>> >> > > > > >>
>> >> > > > > >> * Maven artifacts to be voted upon:
>> >> > > > > >>
>> >> >
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> >> > > > > >>
>> >> > > > > >> * Javadoc:
>> >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>> >> > > > > >>
>> >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
>> >> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>> >> > > > > >>
>> >> > > > > >> The PR adding the 35 documentation is not merged yet
>> >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
>> >> > > > > >> * Documentation:
>> >> > > > > >> https://kafka.apache.org/35/documentation.html
>> >> > > > > >> * Protocol:
>> >> > > > > >> https://kafka.apache.org/35/protocol.html
>> >> > > > > >>
>> >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
>> >> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5
>> branch,
>> >> > > > > >> working with INFRA to sort it out:
>> >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
>> >> > > > > >> System tests: The build is still running, I'll send an update
>> >> > once I
>> >> > > > > >> have the results
>> >> > > > > >>
>> >> > > > > >> Thanks,
>> >> > > > > >> Mickael
>> >> > > > > >>
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > --
>> >> > > > > > [image: Aiven] <https://www.aiven.io>
>> >> > > > > >
>> >> > > > > > *Josep Prat*
>> >> > > > > > Open Source Engineering Director, *Aiven*
>> >> > > > > > josep.prat@aiven.io   |   +491715557497
>> >> > > > > > aiven.io <https://www.aiven.io>   |
>> >> > > > > > <https://www.facebook.com/aivencloud>
>> >> > > > > > <https://www.linkedin.com/company/aiven/>   <
>> >> > > > > https://twitter.com/aiven_io>
>> >> > > > > > *Aiven Deutschland GmbH*
>> >> > > > > > Alexanderufer 3-7, 10117 Berlin
>> >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
>> >> > > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > --
>> >> > > > > [image: Aiven] <https://www.aiven.io>
>> >> > > > >
>> >> > > > > *Josep Prat*
>> >> > > > > Open Source Engineering Director, *Aiven*
>> >> > > > > josep.prat@aiven.io   |   +491715557497
>> >> > > > > aiven.io <https://www.aiven.io>   |   <
>> >> > https://www.facebook.com/aivencloud
>> >> > > > > >
>> >> > > > >   <https://www.linkedin.com/company/aiven/>   <
>> >> > > > > https://twitter.com/aiven_io>
>> >> > > > > *Aiven Deutschland GmbH*
>> >> > > > > Alexanderufer 3-7, 10117 Berlin
>> >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
>> >> > > > >
>> >> >
>> >>
>> >>
>> >> --
>> >> -David
>>

Re: [VOTE] 3.5.0 RC0

Posted by David Arthur <da...@confluent.io.INVALID>.

Mickael, all of our migration fixes are in.

Thanks!
David

On Thu, Jun 1, 2023 at 6:10 PM Colin McCabe <cm...@apache.org> wrote:

> Hi Mickael,
>
> Can you start the new RC tomorrow? There's one last PR we'd like to get in.
>
> If we can't get it in by tomorrow then let's go ahead anyway.
>
> Thanks very much,
> Colin
>
>
> On Thu, Jun 1, 2023, at 14:15, Mickael Maison wrote:
> > Hi David,
> >
> > The PR you mentioned is merged now. Can I start working on a new RC or is
> > there more work needed? It seems the associated ticket is still open.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, 31 May 2023, 23:52 Justine Olshan, <jolshan@confluent.io.invalid
> >
> > wrote:
> >
> >> Hey Mickael --
> >> This is done. Thanks!
> >>
> >> On Wed, May 31, 2023 at 11:24 AM Mickael Maison <
> mickael.maison@gmail.com>
> >> wrote:
> >>
> >> > Hi Justine,
> >> >
> >> > Yes you can merge that into 3.5.
> >> >
> >> > Thanks,
> >> > Mickael
> >> >
> >> > On Wed, May 31, 2023 at 7:56 PM Justine Olshan
> >> > <jo...@confluent.io.invalid> wrote:
> >> > >
> >> > > FYI -- I just saw this PR regarding a dependency for ARM. We may
> want
> >> to
> >> > > get this in for 3.5 as well. It should be quick.
> >> > >
> >> > > https://issues.apache.org/jira/browse/KAFKA-15044
> >> > > https://github.com/apache/kafka/pull/13786
> >> > >
> >> > > Justine
> >> > >
> >> > > On Wed, May 31, 2023 at 9:28 AM David Arthur
> >> > > <da...@confluent.io.invalid> wrote:
> >> > >
> >> > > > Mickael,
> >> > > >
> >> > > > Colin has approved my patch for KAFKA-15010, I'm just waiting on a
> >> > build
> >> > > > before merging. I'll go ahead and backport the other fixes that
> need
> >> to
> >> > > > precede this one into 3.5.
> >> > > >
> >> > > > -David
> >> > > >
> >> > > > On Wed, May 31, 2023 at 11:52 AM Mickael Maison <
> >> > mickael.maison@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > The issue mentioned by Greg has been fixed. As soon as the fix
> for
> >> > > > > KAFKA-15010 is merged I'll build another RC.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Mickael
> >> > > > >
> >> > > > > On Tue, May 30, 2023 at 10:33 AM Mickael Maison
> >> > > > > <mi...@gmail.com> wrote:
> >> > > > > >
> >> > > > > > Hi David,
> >> > > > > >
> >> > > > > > Feel free to backport the necessary fixes to 3.5.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Mickael
> >> > > > > >
> >> > > > > > On Tue, May 30, 2023 at 10:32 AM Mickael Maison
> >> > > > > > <mi...@gmail.com> wrote:
> >> > > > > > >
> >> > > > > > > Hi Greg,
> >> > > > > > >
> >> > > > > > > Thanks for the heads up, this indeed looks like something we
> >> > want in
> >> > > > > > > 3.5. I've replied in the PR.
> >> > > > > > >
> >> > > > > > > Mickael
> >> > > > > > >
> >> > > > > > > On Sat, May 27, 2023 at 11:44 PM David Arthur
> >> > > > > > > <da...@confluent.io.invalid> wrote:
> >> > > > > > > >
> >> > > > > > > > Mickael, after looking more closely, I definitely think
> >> > KAFKA-15010
> >> > > > > is a
> >> > > > > > > > blocker. It creates the case where the controller can
> totally
> >> > miss
> >> > > > a
> >> > > > > > > > metadata update and not write it back to ZK. Since things
> >> like
> >> > > > > dynamic
> >> > > > > > > > configs and ACLs are only read from ZK by the ZK brokers,
> we
> >> > could
> >> > > > > have
> >> > > > > > > > significant problems while the brokers are being migrated
> >> (when
> >> > > > some
> >> > > > > are
> >> > > > > > > > KRaft and some are ZK). E.g., ZK brokers could be totally
> >> > unaware
> >> > > > of
> >> > > > > an ACL
> >> > > > > > > > change while the KRaft brokers have it. I have a fix ready
> >> here
> >> > > > > > > > https://github.com/apache/kafka/pull/13758. I think we
> can
> >> > get it
> >> > > > > committed
> >> > > > > > > > soon.
> >> > > > > > > >
> >> > > > > > > > Another blocker is KAFKA-15004 which was just merged to
> >> trunk.
> >> > This
> >> > > > > is
> >> > > > > > > > another dual-write bug where new topic/broker configs will
> >> not
> >> > be
> >> > > > > written
> >> > > > > > > > back to ZK by the controller.
> >> > > > > > > >
> >> > > > > > > > The fix for KAFKA-15010 has a few dependencies on fixes we
> >> made
> >> > > > this
> >> > > > > past
> >> > > > > > > > week, so we'll need to cherry-pick a few commits. The
> changes
> >> > are
> >> > > > > totally
> >> > > > > > > > contained within the migration area of code, so I think
> the
> >> > risk in
> >> > > > > > > > including them is fairly low.
> >> > > > > > > >
> >> > > > > > > > -David
> >> > > > > > > >
> >> > > > > > > > On Thu, May 25, 2023 at 2:15 PM Greg Harris
> >> > > > > <gr...@aiven.io.invalid>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hey all,
> >> > > > > > > > >
> >> > > > > > > > > A contributor just pointed out a small but noticeable
> flaw
> >> > in the
> >> > > > > > > > > implementation of KIP-581
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> >> > > > > > > > > which is planned for this release.
> >> > > > > > > > > Impact: the feature works for root values in a record,
> but
> >> > does
> >> > > > not
> >> > > > > > > > > work for any fields within structs. Fields within
> structs
> >> > will
> >> > > > > > > > > continue to have their previous, backwards-compatible
> >> > behavior.
> >> > > > > > > > > The contributor has submitted a bug-fix PR which reports
> >> the
> >> > > > > problem
> >> > > > > > > > > and does not yet have a merge-able solution, but they
> are
> >> > > > actively
> >> > > > > > > > > responding and interested in having this fixed:
> >> > > > > > > > > https://github.com/apache/kafka/pull/13748
> >> > > > > > > > > The overall fix should be a one-liner + some unit tests.
> >> > While
> >> > > > > this is
> >> > > > > > > > > not a regression, it does make the feature largely
> useless,
> >> > as
> >> > > > the
> >> > > > > > > > > majority of use-cases will be for struct fields.
> >> > > > > > > > >
> >> > > > > > > > > Thanks!
> >> > > > > > > > > Greg Harris
> >> > > > > > > > >
> >> > > > > > > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <
> >> > ismael@juma.me.uk>
> >> > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > I agree the migration should be functional - it wasn't
> >> > obvious
> >> > > > > if the
> >> > > > > > > > > > migration issues are edge cases or not. If they are
> edge
> >> > cases,
> >> > > > > I think
> >> > > > > > > > > > 3.5.1 would be fine given the preview status.
> >> > > > > > > > > >
> >> > > > > > > > > > I understand that a new RC is needed, but that doesn't
> >> > mean we
> >> > > > > should let
> >> > > > > > > > > > everything in. Each change carries some risk. And if
> we
> >> > don't
> >> > > > > agree on
> >> > > > > > > > > the
> >> > > > > > > > > > bar for the migration work, we may be having the same
> >> > > > discussion
> >> > > > > next
> >> > > > > > > > > week.
> >> > > > > > > > > > :)
> >> > > > > > > > > >
> >> > > > > > > > > > Ismael
> >> > > > > > > > > >
> >> > > > > > > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat
> >> > > > > <jo...@aiven.io.invalid>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi there,
> >> > > > > > > > > > > Is the plan described in KIP-833[1] still valid? In
> >> > there it
> >> > > > > states
> >> > > > > > > > > that
> >> > > > > > > > > > > 3.5.0 should aim at deprecation of Zookeeper, so
> >> > > > conceptually,
> >> > > > > the
> >> > > > > > > > > path to
> >> > > > > > > > > > > migrate to Kraft should be somewhat functional (in
> my
> >> > > > > opinion). If we
> >> > > > > > > > > don't
> >> > > > > > > > > > > want to deprecate Zookeeper in 3.5.0, then I share
> >> > Ismael's
> >> > > > > opinion
> >> > > > > > > > > that
> >> > > > > > > > > > > these could be fixed in subsequent patches of 3.5.x.
> >> > Just my
> >> > > > > 5cts.
> >> > > > > > > > > > >
> >> > > > > > > > > > > [1]:
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> >> > > > > > > > > > > Best,
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <
> >> > > > ismael@juma.me.uk>
> >> > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Are all these blockers? For example, zk to kraft
> >> > migration
> >> > > > > are is
> >> > > > > > > > > still
> >> > > > > > > > > > > in
> >> > > > > > > > > > > > preview - can we fix some of these in 3.5.1?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Ismael
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <
> >> > > > > cmccabe@apache.org>
> >> > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi Mickael,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks for putting together this RC.
> Unfortunately,
> >> > we've
> >> > > > > > > > > identified
> >> > > > > > > > > > > > > several blocker issues in this release
> candidate.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > KAFKA-15009: New ACLs are not written to ZK
> during
> >> > > > > migration
> >> > > > > > > > > > > > > KAFKA-15007: MV is not set correctly in the
> >> > > > > MetadataPropagator in
> >> > > > > > > > > > > > > migration.
> >> > > > > > > > > > > > > KAFKA-15004: Topic config changes are not synced
> >> > during
> >> > > > zk
> >> > > > > to kraft
> >> > > > > > > > > > > > > migration (dual-write)
> >> > > > > > > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not
> >> updated
> >> > in
> >> > > > > migration
> >> > > > > > > > > > > > > (dual-write) when partitions are changed for
> topic
> >> > > > > > > > > > > > > KAFKA-14996: The KRaft controller should
> properly
> >> > handle
> >> > > > > overly
> >> > > > > > > > > large
> >> > > > > > > > > > > > user
> >> > > > > > > > > > > > > operations
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > We are working on PRs for these issues and will
> get
> >> > them
> >> > > > > in soon,
> >> > > > > > > > > we
> >> > > > > > > > > > > > think!
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > So unfortunately I have to leave a -1 here for
> RC0.
> >> > Let's
> >> > > > > aim for
> >> > > > > > > > > > > another
> >> > > > > > > > > > > > > RC next week.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > best,
> >> > > > > > > > > > > > > Colin
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison
> >> wrote:
> >> > > > > > > > > > > > > > Hi David,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > We're already quite a bit behind schedule. If
> you
> >> > think
> >> > > > > these
> >> > > > > > > > > fixes
> >> > > > > > > > > > > > > > are really important and can be ready in the
> next
> >> > > > couple
> >> > > > > of
> >> > > > > > > > > days, I'm
> >> > > > > > > > > > > > > > open to backport them and build another
> release
> >> > > > > candidate. Let me
> >> > > > > > > > > > > know
> >> > > > > > > > > > > > > > once you've investigated the severity of
> >> > KAFKA-15010.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > Mickael
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> >> > > > > > > > > > > > > > <da...@confluent.io.invalid> wrote:
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >> Mickael, we have some migration fixes on
> trunk,
> >> > is it
> >> > > > > okay to
> >> > > > > > > > > > > > > cherry-pick
> >> > > > > > > > > > > > > >> these to 3.5?
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion
> in
> >> > > > > > > > > MigrationPropagator
> >> > > > > > > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot
> >> > during
> >> > > > > migration
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >> There is another issue KAFKA-15010 that I'm
> also
> >> > > > > investigating
> >> > > > > > > > > to
> >> > > > > > > > > > > > > determine
> >> > > > > > > > > > > > > >> the impact and likelihood of seeing it in
> >> > practice.
> >> > > > > This one
> >> > > > > > > > > may be
> >> > > > > > > > > > > a
> >> > > > > > > > > > > > > >> significant migration blocker
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >> Cheers,
> >> > > > > > > > > > > > > >> David
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael
> Maison <
> >> > > > > > > > > > > > > mickael.maison@gmail.com>
> >> > > > > > > > > > > > > >> wrote:
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >> > Hi Christo,
> >> > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > >> > Yes this is expected. This happens when
> nested
> >> > > > fields
> >> > > > > also
> >> > > > > > > > > accept
> >> > > > > > > > > > > > > >> > optional tagged fields. The tables list all
> >> > fields,
> >> > > > > so they
> >> > > > > > > > > may
> >> > > > > > > > > > > > > >> > include _tagged_fields multiple times.
> >> > > > > > > > > > > > > >> > Clearly the layout of this page could be
> >> > improved,
> >> > > > if
> >> > > > > you have
> >> > > > > > > > > > > ideas
> >> > > > > > > > > > > > > >> > how to describe the protocol in a better
> way,
> >> > feel
> >> > > > > free to
> >> > > > > > > > > share
> >> > > > > > > > > > > > them.
> >> > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > >> > Thanks,
> >> > > > > > > > > > > > > >> > Mickael
> >> > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael
> >> Maison <
> >> > > > > > > > > > > > > mickael.maison@gmail.com>
> >> > > > > > > > > > > > > >> > wrote:
> >> > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > >> > > Hi Josep,
> >> > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > >> > > Good catch! I opened a PR to fix this:
> >> > > > > > > > > > > > > >> > >
> >> https://github.com/apache/kafka-site/pull/514
> >> > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > >> > > Thanks,
> >> > > > > > > > > > > > > >> > > Mickael
> >> > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo
> >> Lolov
> >> > <
> >> > > > > > > > > > > > > christololov@gmail.com>
> >> > > > > > > > > > > > > >> > wrote:
> >> > > > > > > > > > > > > >> > > >
> >> > > > > > > > > > > > > >> > > > Hey Mickael!
> >> > > > > > > > > > > > > >> > > >
> >> > > > > > > > > > > > > >> > > > I am giving a +1 (non-binding) for this
> >> > > > candidate
> >> > > > > release.
> >> > > > > > > > > > > > > >> > > >
> >> > > > > > > > > > > > > >> > > > * Built from the binary tar.gz source
> with
> >> > Java
> >> > > > > 17 and
> >> > > > > > > > > Scala
> >> > > > > > > > > > > > 2.13
> >> > > > > > > > > > > > > on
> >> > > > > > > > > > > > > >> > Intel
> >> > > > > > > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge)
> >> machines.
> >> > > > > > > > > > > > > >> > > > * Ran unit and integration tests on
> Intel
> >> > and
> >> > > > ARM
> >> > > > > > > > > machines.
> >> > > > > > > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper
> and
> >> > KRaft
> >> > > > > modes on
> >> > > > > > > > > > > Intel
> >> > > > > > > > > > > > > and ARM
> >> > > > > > > > > > > > > >> > > > machines.
> >> > > > > > > > > > > > > >> > > >
> >> > > > > > > > > > > > > >> > > > Question:
> >> > > > > > > > > > > > > >> > > > * I went through
> >> > > > > > > > > https://kafka.apache.org/35/protocol.html
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > > there
> >> > > > > > > > > > > > > >> > are
> >> > > > > > > > > > > > > >> > > > quite a few repetitive __tagged_fileds
> >> > fields
> >> > > > > within the
> >> > > > > > > > > same
> >> > > > > > > > > > > > > >> > structures -
> >> > > > > > > > > > > > > >> > > > is this expected?
> >> > > > > > > > > > > > > >> > > >
> >> > > > > > > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep
> Prat
> >> > > > > > > > > > > > > <jo...@aiven.io.invalid>
> >> > > > > > > > > > > > > >> > > > wrote:
> >> > > > > > > > > > > > > >> > > >
> >> > > > > > > > > > > > > >> > > > > Hi Mickael,
> >> > > > > > > > > > > > > >> > > > > I just wanted to point out that I
> think
> >> > the
> >> > > > > > > > > documentation
> >> > > > > > > > > > > you
> >> > > > > > > > > > > > > >> > recently
> >> > > > > > > > > > > > > >> > > > > merged on Kafka site regarding the
> 3.5.0
> >> > > > > version has a
> >> > > > > > > > > > > problem
> >> > > > > > > > > > > > > when
> >> > > > > > > > > > > > > >> > it
> >> > > > > > > > > > > > > >> > > > > states the version number and the
> >> sub-menu
> >> > > > that
> >> > > > > links to
> >> > > > > > > > > > > > > previous
> >> > > > > > > > > > > > > >> > versions.
> >> > > > > > > > > > > > > >> > > > > Left a comment here:
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> > > > > Best,
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep
> >> > Prat <
> >> > > > > > > > > > > > josep.prat@aiven.io
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >> > wrote:
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> > > > > > Hi Mickael,
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > > > I can +1 this candidate. I verified
> >> the
> >> > > > > following:
> >> > > > > > > > > > > > > >> > > > > > - Built from source with Java 17
> and
> >> > Scala
> >> > > > > 2.13
> >> > > > > > > > > > > > > >> > > > > > - Signatures and hashes of the
> >> artifacts
> >> > > > > generated
> >> > > > > > > > > > > > > >> > > > > > - Navigated through Javadoc
> including
> >> > links
> >> > > > > to JDK
> >> > > > > > > > > classes
> >> > > > > > > > > > > > > >> > > > > > - Run the unit tests
> >> > > > > > > > > > > > > >> > > > > > - Run integration tests
> >> > > > > > > > > > > > > >> > > > > > - Run the quickstart in KRaft and
> >> > Zookeeper
> >> > > > > mode
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > > > Best,
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM
> >> Mickael
> >> > > > > Maison <
> >> > > > > > > > > > > > > >> > mimaison@apache.org>
> >> > > > > > > > > > > > > >> > > > > > wrote:
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > > >> Hello Kafka users, developers and
> >> > > > > client-developers,
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> This is the first candidate for
> >> > release of
> >> > > > > Apache
> >> > > > > > > > > Kafka
> >> > > > > > > > > > > > > 3.5.0.
> >> > > > > > > > > > > > > >> > Some of
> >> > > > > > > > > > > > > >> > > > > the
> >> > > > > > > > > > > > > >> > > > > >> major features include:
> >> > > > > > > > > > > > > >> > > > > >> - KIP-710: Full support for
> >> distributed
> >> > > > mode
> >> > > > > in
> >> > > > > > > > > dedicated
> >> > > > > > > > > > > > > >> > MirrorMaker
> >> > > > > > > > > > > > > >> > > > > >> 2.0 clusters
> >> > > > > > > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition
> >> > Assignment
> >> > > > > for Kafka
> >> > > > > > > > > > > > > Consumers
> >> > > > > > > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to
> make
> >> > use
> >> > > > of
> >> > > > > > > > > environment
> >> > > > > > > > > > > > > variables
> >> > > > > > > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> >> > > > > > > > > > > > > >> > > > > >> - KIP-894: Use
> incrementalAlterConfig
> >> > for
> >> > > > > syncing
> >> > > > > > > > > topic
> >> > > > > > > > > > > > > >> > configurations
> >> > > > > > > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh
> API
> >> > > > > additions to
> >> > > > > > > > > > > support
> >> > > > > > > > > > > > > SCRAM
> >> > > > > > > > > > > > > >> > for
> >> > > > > > > > > > > > > >> > > > > >> Kafka Brokers
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> Release notes for the 3.5.0
> release:
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> >
> >> > > > > > > > > > > >
> >> > > > >
> >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> *** Please download, test and
> vote by
> >> > > > > Friday, May
> >> > > > > > > > > 26, 5pm
> >> > > > > > > > > > > > PT
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP
> keys
> >> > we
> >> > > > use
> >> > > > > to sign
> >> > > > > > > > > the
> >> > > > > > > > > > > > > release:
> >> > > > > > > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> * Release artifacts to be voted
> upon
> >> > > > (source
> >> > > > > and
> >> > > > > > > > > binary):
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> * Maven artifacts to be voted
> upon:
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> * Javadoc:
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > >
> >> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5
> >> > branch) is
> >> > > > > the 3.5.0
> >> > > > > > > > > tag:
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> The PR adding the 35
> documentation is
> >> > not
> >> > > > > merged yet
> >> > > > > > > > > > > > > >> > > > > >> (
> >> > > > > https://github.com/apache/kafka-site/pull/513)
> >> > > > > > > > > > > > > >> > > > > >> * Documentation:
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > https://kafka.apache.org/35/documentation.html
> >> > > > > > > > > > > > > >> > > > > >> * Protocol:
> >> > > > > > > > > > > > > >> > > > > >>
> >> > https://kafka.apache.org/35/protocol.html
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> * Successful Jenkins builds for
> the
> >> 3.5
> >> > > > > branch:
> >> > > > > > > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is
> >> not
> >> > > > > detecting the
> >> > > > > > > > > 3.5
> >> > > > > > > > > > > > > branch,
> >> > > > > > > > > > > > > >> > > > > >> working with INFRA to sort it out:
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > https://issues.apache.org/jira/browse/INFRA-24577
> >> > > > > > > > > > > > > >> > > > > >> System tests: The build is still
> >> > running,
> >> > > > > I'll send
> >> > > > > > > > > an
> >> > > > > > > > > > > > update
> >> > > > > > > > > > > > > >> > once I
> >> > > > > > > > > > > > > >> > > > > >> have the results
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >> Thanks,
> >> > > > > > > > > > > > > >> > > > > >> Mickael
> >> > > > > > > > > > > > > >> > > > > >>
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > > > --
> >> > > > > > > > > > > > > >> > > > > > [image: Aiven] <
> https://www.aiven.io>
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > > > *Josep Prat*
> >> > > > > > > > > > > > > >> > > > > > Open Source Engineering Director,
> >> > *Aiven*
> >> > > > > > > > > > > > > >> > > > > > josep.prat@aiven.io   |
> >>  +491715557497
> >> > > > > > > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>
>  |
> >> > > > > > > > > > > > > >> > > > > > <
> https://www.facebook.com/aivencloud>
> >> > > > > > > > > > > > > >> > > > > > <
> >> > https://www.linkedin.com/company/aiven/>
> >> > > >  <
> >> > > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> >> > > > > > > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> >> > > > > > > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> >> > > > > > > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa &
> >> > Hannu
> >> > > > > Valtonen
> >> > > > > > > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB
> >> 209739 B
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> > > > > --
> >> > > > > > > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io
> >
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> > > > > *Josep Prat*
> >> > > > > > > > > > > > > >> > > > > Open Source Engineering Director,
> >> *Aiven*
> >> > > > > > > > > > > > > >> > > > > josep.prat@aiven.io   |
>  +491715557497
> >> > > > > > > > > > > > > >> > > > > aiven.io <https://www.aiven.io>
>  |   <
> >> > > > > > > > > > > > > >> > https://www.facebook.com/aivencloud
> >> > > > > > > > > > > > > >> > > > > >
> >> > > > > > > > > > > > > >> > > > >   <
> >> > https://www.linkedin.com/company/aiven/>
> >> > > >  <
> >> > > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> >> > > > > > > > > > > > > >> > > > > *Aiven Deutschland GmbH*
> >> > > > > > > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> >> > > > > > > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa &
> >> Hannu
> >> > > > > Valtonen
> >> > > > > > > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB
> 209739 B
> >> > > > > > > > > > > > > >> > > > >
> >> > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > >> --
> >> > > > > > > > > > > > > >> -David
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > > [image: Aiven] <https://www.aiven.io>
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Josep Prat*
> >> > > > > > > > > > > Open Source Engineering Director, *Aiven*
> >> > > > > > > > > > > josep.prat@aiven.io   |   +491715557497
> >> > > > > > > > > > > aiven.io <https://www.aiven.io>   |   <
> >> > > > > > > > > https://www.facebook.com/aivencloud
> >> > > > > > > > > > > >
> >> > > > > > > > > > >   <https://www.linkedin.com/company/aiven/>   <
> >> > > > > > > > > > > https://twitter.com/aiven_io>
> >> > > > > > > > > > > *Aiven Deutschland GmbH*
> >> > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> >> > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > -David
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > -David
> >> > > >
> >> >
> >>
>


-- 
-David

Re: [VOTE] 3.5.0 RC0

Posted by Colin McCabe <cm...@apache.org>.

Hi Mickael,

Can you start the new RC tomorrow? There's one last PR we'd like to get in.

If we can't get it in by tomorrow then let's go ahead anyway.

Thanks very much,
Colin


On Thu, Jun 1, 2023, at 14:15, Mickael Maison wrote:
> Hi David,
>
> The PR you mentioned is merged now. Can I start working on a new RC or is
> there more work needed? It seems the associated ticket is still open.
>
> Thanks,
> Mickael
>
> On Wed, 31 May 2023, 23:52 Justine Olshan, <jo...@confluent.io.invalid>
> wrote:
>
>> Hey Mickael --
>> This is done. Thanks!
>>
>> On Wed, May 31, 2023 at 11:24 AM Mickael Maison <mi...@gmail.com>
>> wrote:
>>
>> > Hi Justine,
>> >
>> > Yes you can merge that into 3.5.
>> >
>> > Thanks,
>> > Mickael
>> >
>> > On Wed, May 31, 2023 at 7:56 PM Justine Olshan
>> > <jo...@confluent.io.invalid> wrote:
>> > >
>> > > FYI -- I just saw this PR regarding a dependency for ARM. We may want
>> to
>> > > get this in for 3.5 as well. It should be quick.
>> > >
>> > > https://issues.apache.org/jira/browse/KAFKA-15044
>> > > https://github.com/apache/kafka/pull/13786
>> > >
>> > > Justine
>> > >
>> > > On Wed, May 31, 2023 at 9:28 AM David Arthur
>> > > <da...@confluent.io.invalid> wrote:
>> > >
>> > > > Mickael,
>> > > >
>> > > > Colin has approved my patch for KAFKA-15010, I'm just waiting on a
>> > build
>> > > > before merging. I'll go ahead and backport the other fixes that need
>> to
>> > > > precede this one into 3.5.
>> > > >
>> > > > -David
>> > > >
>> > > > On Wed, May 31, 2023 at 11:52 AM Mickael Maison <
>> > mickael.maison@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > The issue mentioned by Greg has been fixed. As soon as the fix for
>> > > > > KAFKA-15010 is merged I'll build another RC.
>> > > > >
>> > > > > Thanks,
>> > > > > Mickael
>> > > > >
>> > > > > On Tue, May 30, 2023 at 10:33 AM Mickael Maison
>> > > > > <mi...@gmail.com> wrote:
>> > > > > >
>> > > > > > Hi David,
>> > > > > >
>> > > > > > Feel free to backport the necessary fixes to 3.5.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Mickael
>> > > > > >
>> > > > > > On Tue, May 30, 2023 at 10:32 AM Mickael Maison
>> > > > > > <mi...@gmail.com> wrote:
>> > > > > > >
>> > > > > > > Hi Greg,
>> > > > > > >
>> > > > > > > Thanks for the heads up, this indeed looks like something we
>> > want in
>> > > > > > > 3.5. I've replied in the PR.
>> > > > > > >
>> > > > > > > Mickael
>> > > > > > >
>> > > > > > > On Sat, May 27, 2023 at 11:44 PM David Arthur
>> > > > > > > <da...@confluent.io.invalid> wrote:
>> > > > > > > >
>> > > > > > > > Mickael, after looking more closely, I definitely think
>> > KAFKA-15010
>> > > > > is a
>> > > > > > > > blocker. It creates the case where the controller can totally
>> > miss
>> > > > a
>> > > > > > > > metadata update and not write it back to ZK. Since things
>> like
>> > > > > dynamic
>> > > > > > > > configs and ACLs are only read from ZK by the ZK brokers, we
>> > could
>> > > > > have
>> > > > > > > > significant problems while the brokers are being migrated
>> (when
>> > > > some
>> > > > > are
>> > > > > > > > KRaft and some are ZK). E.g., ZK brokers could be totally
>> > unaware
>> > > > of
>> > > > > an ACL
>> > > > > > > > change while the KRaft brokers have it. I have a fix ready
>> here
>> > > > > > > > https://github.com/apache/kafka/pull/13758. I think we can
>> > get it
>> > > > > committed
>> > > > > > > > soon.
>> > > > > > > >
>> > > > > > > > Another blocker is KAFKA-15004 which was just merged to
>> trunk.
>> > This
>> > > > > is
>> > > > > > > > another dual-write bug where new topic/broker configs will
>> not
>> > be
>> > > > > written
>> > > > > > > > back to ZK by the controller.
>> > > > > > > >
>> > > > > > > > The fix for KAFKA-15010 has a few dependencies on fixes we
>> made
>> > > > this
>> > > > > past
>> > > > > > > > week, so we'll need to cherry-pick a few commits. The changes
>> > are
>> > > > > totally
>> > > > > > > > contained within the migration area of code, so I think the
>> > risk in
>> > > > > > > > including them is fairly low.
>> > > > > > > >
>> > > > > > > > -David
>> > > > > > > >
>> > > > > > > > On Thu, May 25, 2023 at 2:15 PM Greg Harris
>> > > > > <gr...@aiven.io.invalid>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hey all,
>> > > > > > > > >
>> > > > > > > > > A contributor just pointed out a small but noticeable flaw
>> > in the
>> > > > > > > > > implementation of KIP-581
>> > > > > > > > >
>> > > > > > > > >
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
>> > > > > > > > > which is planned for this release.
>> > > > > > > > > Impact: the feature works for root values in a record, but
>> > does
>> > > > not
>> > > > > > > > > work for any fields within structs. Fields within structs
>> > will
>> > > > > > > > > continue to have their previous, backwards-compatible
>> > behavior.
>> > > > > > > > > The contributor has submitted a bug-fix PR which reports
>> the
>> > > > > problem
>> > > > > > > > > and does not yet have a merge-able solution, but they are
>> > > > actively
>> > > > > > > > > responding and interested in having this fixed:
>> > > > > > > > > https://github.com/apache/kafka/pull/13748
>> > > > > > > > > The overall fix should be a one-liner + some unit tests.
>> > While
>> > > > > this is
>> > > > > > > > > not a regression, it does make the feature largely useless,
>> > as
>> > > > the
>> > > > > > > > > majority of use-cases will be for struct fields.
>> > > > > > > > >
>> > > > > > > > > Thanks!
>> > > > > > > > > Greg Harris
>> > > > > > > > >
>> > > > > > > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <
>> > ismael@juma.me.uk>
>> > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > I agree the migration should be functional - it wasn't
>> > obvious
>> > > > > if the
>> > > > > > > > > > migration issues are edge cases or not. If they are edge
>> > cases,
>> > > > > I think
>> > > > > > > > > > 3.5.1 would be fine given the preview status.
>> > > > > > > > > >
>> > > > > > > > > > I understand that a new RC is needed, but that doesn't
>> > mean we
>> > > > > should let
>> > > > > > > > > > everything in. Each change carries some risk. And if we
>> > don't
>> > > > > agree on
>> > > > > > > > > the
>> > > > > > > > > > bar for the migration work, we may be having the same
>> > > > discussion
>> > > > > next
>> > > > > > > > > week.
>> > > > > > > > > > :)
>> > > > > > > > > >
>> > > > > > > > > > Ismael
>> > > > > > > > > >
>> > > > > > > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat
>> > > > > <jo...@aiven.io.invalid>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi there,
>> > > > > > > > > > > Is the plan described in KIP-833[1] still valid? In
>> > there it
>> > > > > states
>> > > > > > > > > that
>> > > > > > > > > > > 3.5.0 should aim at deprecation of Zookeeper, so
>> > > > conceptually,
>> > > > > the
>> > > > > > > > > path to
>> > > > > > > > > > > migrate to Kraft should be somewhat functional (in my
>> > > > > opinion). If we
>> > > > > > > > > don't
>> > > > > > > > > > > want to deprecate Zookeeper in 3.5.0, then I share
>> > Ismael's
>> > > > > opinion
>> > > > > > > > > that
>> > > > > > > > > > > these could be fixed in subsequent patches of 3.5.x.
>> > Just my
>> > > > > 5cts.
>> > > > > > > > > > >
>> > > > > > > > > > > [1]:
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > >
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
>> > > > > > > > > > > Best,
>> > > > > > > > > > >
>> > > > > > > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <
>> > > > ismael@juma.me.uk>
>> > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Are all these blockers? For example, zk to kraft
>> > migration
>> > > > > are is
>> > > > > > > > > still
>> > > > > > > > > > > in
>> > > > > > > > > > > > preview - can we fix some of these in 3.5.1?
>> > > > > > > > > > > >
>> > > > > > > > > > > > Ismael
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <
>> > > > > cmccabe@apache.org>
>> > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi Mickael,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for putting together this RC. Unfortunately,
>> > we've
>> > > > > > > > > identified
>> > > > > > > > > > > > > several blocker issues in this release candidate.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > KAFKA-15009: New ACLs are not written to ZK during
>> > > > > migration
>> > > > > > > > > > > > > KAFKA-15007: MV is not set correctly in the
>> > > > > MetadataPropagator in
>> > > > > > > > > > > > > migration.
>> > > > > > > > > > > > > KAFKA-15004: Topic config changes are not synced
>> > during
>> > > > zk
>> > > > > to kraft
>> > > > > > > > > > > > > migration (dual-write)
>> > > > > > > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not
>> updated
>> > in
>> > > > > migration
>> > > > > > > > > > > > > (dual-write) when partitions are changed for topic
>> > > > > > > > > > > > > KAFKA-14996: The KRaft controller should properly
>> > handle
>> > > > > overly
>> > > > > > > > > large
>> > > > > > > > > > > > user
>> > > > > > > > > > > > > operations
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > We are working on PRs for these issues and will get
>> > them
>> > > > > in soon,
>> > > > > > > > > we
>> > > > > > > > > > > > think!
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > So unfortunately I have to leave a -1 here for RC0.
>> > Let's
>> > > > > aim for
>> > > > > > > > > > > another
>> > > > > > > > > > > > > RC next week.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > best,
>> > > > > > > > > > > > > Colin
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison
>> wrote:
>> > > > > > > > > > > > > > Hi David,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > We're already quite a bit behind schedule. If you
>> > think
>> > > > > these
>> > > > > > > > > fixes
>> > > > > > > > > > > > > > are really important and can be ready in the next
>> > > > couple
>> > > > > of
>> > > > > > > > > days, I'm
>> > > > > > > > > > > > > > open to backport them and build another release
>> > > > > candidate. Let me
>> > > > > > > > > > > know
>> > > > > > > > > > > > > > once you've investigated the severity of
>> > KAFKA-15010.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > Mickael
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
>> > > > > > > > > > > > > > <da...@confluent.io.invalid> wrote:
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> Mickael, we have some migration fixes on trunk,
>> > is it
>> > > > > okay to
>> > > > > > > > > > > > > cherry-pick
>> > > > > > > > > > > > > >> these to 3.5?
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
>> > > > > > > > > MigrationPropagator
>> > > > > > > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot
>> > during
>> > > > > migration
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> There is another issue KAFKA-15010 that I'm also
>> > > > > investigating
>> > > > > > > > > to
>> > > > > > > > > > > > > determine
>> > > > > > > > > > > > > >> the impact and likelihood of seeing it in
>> > practice.
>> > > > > This one
>> > > > > > > > > may be
>> > > > > > > > > > > a
>> > > > > > > > > > > > > >> significant migration blocker
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> Cheers,
>> > > > > > > > > > > > > >> David
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
>> > > > > > > > > > > > > mickael.maison@gmail.com>
>> > > > > > > > > > > > > >> wrote:
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> > Hi Christo,
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > Yes this is expected. This happens when nested
>> > > > fields
>> > > > > also
>> > > > > > > > > accept
>> > > > > > > > > > > > > >> > optional tagged fields. The tables list all
>> > fields,
>> > > > > so they
>> > > > > > > > > may
>> > > > > > > > > > > > > >> > include _tagged_fields multiple times.
>> > > > > > > > > > > > > >> > Clearly the layout of this page could be
>> > improved,
>> > > > if
>> > > > > you have
>> > > > > > > > > > > ideas
>> > > > > > > > > > > > > >> > how to describe the protocol in a better way,
>> > feel
>> > > > > free to
>> > > > > > > > > share
>> > > > > > > > > > > > them.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > Thanks,
>> > > > > > > > > > > > > >> > Mickael
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael
>> Maison <
>> > > > > > > > > > > > > mickael.maison@gmail.com>
>> > > > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > > Hi Josep,
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > > Good catch! I opened a PR to fix this:
>> > > > > > > > > > > > > >> > >
>> https://github.com/apache/kafka-site/pull/514
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > > Thanks,
>> > > > > > > > > > > > > >> > > Mickael
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo
>> Lolov
>> > <
>> > > > > > > > > > > > > christololov@gmail.com>
>> > > > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > Hey Mickael!
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > I am giving a +1 (non-binding) for this
>> > > > candidate
>> > > > > release.
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > * Built from the binary tar.gz source with
>> > Java
>> > > > > 17 and
>> > > > > > > > > Scala
>> > > > > > > > > > > > 2.13
>> > > > > > > > > > > > > on
>> > > > > > > > > > > > > >> > Intel
>> > > > > > > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge)
>> machines.
>> > > > > > > > > > > > > >> > > > * Ran unit and integration tests on Intel
>> > and
>> > > > ARM
>> > > > > > > > > machines.
>> > > > > > > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and
>> > KRaft
>> > > > > modes on
>> > > > > > > > > > > Intel
>> > > > > > > > > > > > > and ARM
>> > > > > > > > > > > > > >> > > > machines.
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > Question:
>> > > > > > > > > > > > > >> > > > * I went through
>> > > > > > > > > https://kafka.apache.org/35/protocol.html
>> > > > > > > > > > > and
>> > > > > > > > > > > > > there
>> > > > > > > > > > > > > >> > are
>> > > > > > > > > > > > > >> > > > quite a few repetitive __tagged_fileds
>> > fields
>> > > > > within the
>> > > > > > > > > same
>> > > > > > > > > > > > > >> > structures -
>> > > > > > > > > > > > > >> > > > is this expected?
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
>> > > > > > > > > > > > > <jo...@aiven.io.invalid>
>> > > > > > > > > > > > > >> > > > wrote:
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > > Hi Mickael,
>> > > > > > > > > > > > > >> > > > > I just wanted to point out that I think
>> > the
>> > > > > > > > > documentation
>> > > > > > > > > > > you
>> > > > > > > > > > > > > >> > recently
>> > > > > > > > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0
>> > > > > version has a
>> > > > > > > > > > > problem
>> > > > > > > > > > > > > when
>> > > > > > > > > > > > > >> > it
>> > > > > > > > > > > > > >> > > > > states the version number and the
>> sub-menu
>> > > > that
>> > > > > links to
>> > > > > > > > > > > > > previous
>> > > > > > > > > > > > > >> > versions.
>> > > > > > > > > > > > > >> > > > > Left a comment here:
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > >
>> > > > >
>> > > >
>> >
>> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > Best,
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep
>> > Prat <
>> > > > > > > > > > > > josep.prat@aiven.io
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > > Hi Mickael,
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > > > I can +1 this candidate. I verified
>> the
>> > > > > following:
>> > > > > > > > > > > > > >> > > > > > - Built from source with Java 17 and
>> > Scala
>> > > > > 2.13
>> > > > > > > > > > > > > >> > > > > > - Signatures and hashes of the
>> artifacts
>> > > > > generated
>> > > > > > > > > > > > > >> > > > > > - Navigated through Javadoc including
>> > links
>> > > > > to JDK
>> > > > > > > > > classes
>> > > > > > > > > > > > > >> > > > > > - Run the unit tests
>> > > > > > > > > > > > > >> > > > > > - Run integration tests
>> > > > > > > > > > > > > >> > > > > > - Run the quickstart in KRaft and
>> > Zookeeper
>> > > > > mode
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > > > Best,
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM
>> Mickael
>> > > > > Maison <
>> > > > > > > > > > > > > >> > mimaison@apache.org>
>> > > > > > > > > > > > > >> > > > > > wrote:
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > > >> Hello Kafka users, developers and
>> > > > > client-developers,
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> This is the first candidate for
>> > release of
>> > > > > Apache
>> > > > > > > > > Kafka
>> > > > > > > > > > > > > 3.5.0.
>> > > > > > > > > > > > > >> > Some of
>> > > > > > > > > > > > > >> > > > > the
>> > > > > > > > > > > > > >> > > > > >> major features include:
>> > > > > > > > > > > > > >> > > > > >> - KIP-710: Full support for
>> distributed
>> > > > mode
>> > > > > in
>> > > > > > > > > dedicated
>> > > > > > > > > > > > > >> > MirrorMaker
>> > > > > > > > > > > > > >> > > > > >> 2.0 clusters
>> > > > > > > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition
>> > Assignment
>> > > > > for Kafka
>> > > > > > > > > > > > > Consumers
>> > > > > > > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make
>> > use
>> > > > of
>> > > > > > > > > environment
>> > > > > > > > > > > > > variables
>> > > > > > > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
>> > > > > > > > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig
>> > for
>> > > > > syncing
>> > > > > > > > > topic
>> > > > > > > > > > > > > >> > configurations
>> > > > > > > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API
>> > > > > additions to
>> > > > > > > > > > > support
>> > > > > > > > > > > > > SCRAM
>> > > > > > > > > > > > > >> > for
>> > > > > > > > > > > > > >> > > > > >> Kafka Brokers
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > >
>> > > > >
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> *** Please download, test and vote by
>> > > > > Friday, May
>> > > > > > > > > 26, 5pm
>> > > > > > > > > > > > PT
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys
>> > we
>> > > > use
>> > > > > to sign
>> > > > > > > > > the
>> > > > > > > > > > > > > release:
>> > > > > > > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> * Release artifacts to be voted upon
>> > > > (source
>> > > > > and
>> > > > > > > > > binary):
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > >
>> > > > > > > > >
>> > > > >
>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> * Javadoc:
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > >
>> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5
>> > branch) is
>> > > > > the 3.5.0
>> > > > > > > > > tag:
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> The PR adding the 35 documentation is
>> > not
>> > > > > merged yet
>> > > > > > > > > > > > > >> > > > > >> (
>> > > > > https://github.com/apache/kafka-site/pull/513)
>> > > > > > > > > > > > > >> > > > > >> * Documentation:
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > https://kafka.apache.org/35/documentation.html
>> > > > > > > > > > > > > >> > > > > >> * Protocol:
>> > > > > > > > > > > > > >> > > > > >>
>> > https://kafka.apache.org/35/protocol.html
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> * Successful Jenkins builds for the
>> 3.5
>> > > > > branch:
>> > > > > > > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is
>> not
>> > > > > detecting the
>> > > > > > > > > 3.5
>> > > > > > > > > > > > > branch,
>> > > > > > > > > > > > > >> > > > > >> working with INFRA to sort it out:
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > https://issues.apache.org/jira/browse/INFRA-24577
>> > > > > > > > > > > > > >> > > > > >> System tests: The build is still
>> > running,
>> > > > > I'll send
>> > > > > > > > > an
>> > > > > > > > > > > > update
>> > > > > > > > > > > > > >> > once I
>> > > > > > > > > > > > > >> > > > > >> have the results
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >> Thanks,
>> > > > > > > > > > > > > >> > > > > >> Mickael
>> > > > > > > > > > > > > >> > > > > >>
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > > > --
>> > > > > > > > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > > > *Josep Prat*
>> > > > > > > > > > > > > >> > > > > > Open Source Engineering Director,
>> > *Aiven*
>> > > > > > > > > > > > > >> > > > > > josep.prat@aiven.io   |
>>  +491715557497
>> > > > > > > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
>> > > > > > > > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
>> > > > > > > > > > > > > >> > > > > > <
>> > https://www.linkedin.com/company/aiven/>
>> > > >  <
>> > > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
>> > > > > > > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
>> > > > > > > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
>> > > > > > > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa &
>> > Hannu
>> > > > > Valtonen
>> > > > > > > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB
>> 209739 B
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > --
>> > > > > > > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > *Josep Prat*
>> > > > > > > > > > > > > >> > > > > Open Source Engineering Director,
>> *Aiven*
>> > > > > > > > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
>> > > > > > > > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
>> > > > > > > > > > > > > >> > https://www.facebook.com/aivencloud
>> > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > >> > > > >   <
>> > https://www.linkedin.com/company/aiven/>
>> > > >  <
>> > > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
>> > > > > > > > > > > > > >> > > > > *Aiven Deutschland GmbH*
>> > > > > > > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
>> > > > > > > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa &
>> Hannu
>> > > > > Valtonen
>> > > > > > > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> --
>> > > > > > > > > > > > > >> -David
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > [image: Aiven] <https://www.aiven.io>
>> > > > > > > > > > >
>> > > > > > > > > > > *Josep Prat*
>> > > > > > > > > > > Open Source Engineering Director, *Aiven*
>> > > > > > > > > > > josep.prat@aiven.io   |   +491715557497
>> > > > > > > > > > > aiven.io <https://www.aiven.io>   |   <
>> > > > > > > > > https://www.facebook.com/aivencloud
>> > > > > > > > > > > >
>> > > > > > > > > > >   <https://www.linkedin.com/company/aiven/>   <
>> > > > > > > > > > > https://twitter.com/aiven_io>
>> > > > > > > > > > > *Aiven Deutschland GmbH*
>> > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
>> > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
>> > > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > -David
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > -David
>> > > >
>> >
>>

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi David,

The PR you mentioned is merged now. Can I start working on a new RC or is
there more work needed? It seems the associated ticket is still open.

Thanks,
Mickael

On Wed, 31 May 2023, 23:52 Justine Olshan, <jo...@confluent.io.invalid>
wrote:

> Hey Mickael --
> This is done. Thanks!
>
> On Wed, May 31, 2023 at 11:24 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Justine,
> >
> > Yes you can merge that into 3.5.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, May 31, 2023 at 7:56 PM Justine Olshan
> > <jo...@confluent.io.invalid> wrote:
> > >
> > > FYI -- I just saw this PR regarding a dependency for ARM. We may want
> to
> > > get this in for 3.5 as well. It should be quick.
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-15044
> > > https://github.com/apache/kafka/pull/13786
> > >
> > > Justine
> > >
> > > On Wed, May 31, 2023 at 9:28 AM David Arthur
> > > <da...@confluent.io.invalid> wrote:
> > >
> > > > Mickael,
> > > >
> > > > Colin has approved my patch for KAFKA-15010, I'm just waiting on a
> > build
> > > > before merging. I'll go ahead and backport the other fixes that need
> to
> > > > precede this one into 3.5.
> > > >
> > > > -David
> > > >
> > > > On Wed, May 31, 2023 at 11:52 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > The issue mentioned by Greg has been fixed. As soon as the fix for
> > > > > KAFKA-15010 is merged I'll build another RC.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Tue, May 30, 2023 at 10:33 AM Mickael Maison
> > > > > <mi...@gmail.com> wrote:
> > > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > Feel free to backport the necessary fixes to 3.5.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Tue, May 30, 2023 at 10:32 AM Mickael Maison
> > > > > > <mi...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Thanks for the heads up, this indeed looks like something we
> > want in
> > > > > > > 3.5. I've replied in the PR.
> > > > > > >
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Sat, May 27, 2023 at 11:44 PM David Arthur
> > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > Mickael, after looking more closely, I definitely think
> > KAFKA-15010
> > > > > is a
> > > > > > > > blocker. It creates the case where the controller can totally
> > miss
> > > > a
> > > > > > > > metadata update and not write it back to ZK. Since things
> like
> > > > > dynamic
> > > > > > > > configs and ACLs are only read from ZK by the ZK brokers, we
> > could
> > > > > have
> > > > > > > > significant problems while the brokers are being migrated
> (when
> > > > some
> > > > > are
> > > > > > > > KRaft and some are ZK). E.g., ZK brokers could be totally
> > unaware
> > > > of
> > > > > an ACL
> > > > > > > > change while the KRaft brokers have it. I have a fix ready
> here
> > > > > > > > https://github.com/apache/kafka/pull/13758. I think we can
> > get it
> > > > > committed
> > > > > > > > soon.
> > > > > > > >
> > > > > > > > Another blocker is KAFKA-15004 which was just merged to
> trunk.
> > This
> > > > > is
> > > > > > > > another dual-write bug where new topic/broker configs will
> not
> > be
> > > > > written
> > > > > > > > back to ZK by the controller.
> > > > > > > >
> > > > > > > > The fix for KAFKA-15010 has a few dependencies on fixes we
> made
> > > > this
> > > > > past
> > > > > > > > week, so we'll need to cherry-pick a few commits. The changes
> > are
> > > > > totally
> > > > > > > > contained within the migration area of code, so I think the
> > risk in
> > > > > > > > including them is fairly low.
> > > > > > > >
> > > > > > > > -David
> > > > > > > >
> > > > > > > > On Thu, May 25, 2023 at 2:15 PM Greg Harris
> > > > > <gr...@aiven.io.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey all,
> > > > > > > > >
> > > > > > > > > A contributor just pointed out a small but noticeable flaw
> > in the
> > > > > > > > > implementation of KIP-581
> > > > > > > > >
> > > > > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > > > > > > > which is planned for this release.
> > > > > > > > > Impact: the feature works for root values in a record, but
> > does
> > > > not
> > > > > > > > > work for any fields within structs. Fields within structs
> > will
> > > > > > > > > continue to have their previous, backwards-compatible
> > behavior.
> > > > > > > > > The contributor has submitted a bug-fix PR which reports
> the
> > > > > problem
> > > > > > > > > and does not yet have a merge-able solution, but they are
> > > > actively
> > > > > > > > > responding and interested in having this fixed:
> > > > > > > > > https://github.com/apache/kafka/pull/13748
> > > > > > > > > The overall fix should be a one-liner + some unit tests.
> > While
> > > > > this is
> > > > > > > > > not a regression, it does make the feature largely useless,
> > as
> > > > the
> > > > > > > > > majority of use-cases will be for struct fields.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Greg Harris
> > > > > > > > >
> > > > > > > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <
> > ismael@juma.me.uk>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I agree the migration should be functional - it wasn't
> > obvious
> > > > > if the
> > > > > > > > > > migration issues are edge cases or not. If they are edge
> > cases,
> > > > > I think
> > > > > > > > > > 3.5.1 would be fine given the preview status.
> > > > > > > > > >
> > > > > > > > > > I understand that a new RC is needed, but that doesn't
> > mean we
> > > > > should let
> > > > > > > > > > everything in. Each change carries some risk. And if we
> > don't
> > > > > agree on
> > > > > > > > > the
> > > > > > > > > > bar for the migration work, we may be having the same
> > > > discussion
> > > > > next
> > > > > > > > > week.
> > > > > > > > > > :)
> > > > > > > > > >
> > > > > > > > > > Ismael
> > > > > > > > > >
> > > > > > > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat
> > > > > <jo...@aiven.io.invalid>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi there,
> > > > > > > > > > > Is the plan described in KIP-833[1] still valid? In
> > there it
> > > > > states
> > > > > > > > > that
> > > > > > > > > > > 3.5.0 should aim at deprecation of Zookeeper, so
> > > > conceptually,
> > > > > the
> > > > > > > > > path to
> > > > > > > > > > > migrate to Kraft should be somewhat functional (in my
> > > > > opinion). If we
> > > > > > > > > don't
> > > > > > > > > > > want to deprecate Zookeeper in 3.5.0, then I share
> > Ismael's
> > > > > opinion
> > > > > > > > > that
> > > > > > > > > > > these could be fixed in subsequent patches of 3.5.x.
> > Just my
> > > > > 5cts.
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <
> > > > ismael@juma.me.uk>
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Are all these blockers? For example, zk to kraft
> > migration
> > > > > are is
> > > > > > > > > still
> > > > > > > > > > > in
> > > > > > > > > > > > preview - can we fix some of these in 3.5.1?
> > > > > > > > > > > >
> > > > > > > > > > > > Ismael
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for putting together this RC. Unfortunately,
> > we've
> > > > > > > > > identified
> > > > > > > > > > > > > several blocker issues in this release candidate.
> > > > > > > > > > > > >
> > > > > > > > > > > > > KAFKA-15009: New ACLs are not written to ZK during
> > > > > migration
> > > > > > > > > > > > > KAFKA-15007: MV is not set correctly in the
> > > > > MetadataPropagator in
> > > > > > > > > > > > > migration.
> > > > > > > > > > > > > KAFKA-15004: Topic config changes are not synced
> > during
> > > > zk
> > > > > to kraft
> > > > > > > > > > > > > migration (dual-write)
> > > > > > > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not
> updated
> > in
> > > > > migration
> > > > > > > > > > > > > (dual-write) when partitions are changed for topic
> > > > > > > > > > > > > KAFKA-14996: The KRaft controller should properly
> > handle
> > > > > overly
> > > > > > > > > large
> > > > > > > > > > > > user
> > > > > > > > > > > > > operations
> > > > > > > > > > > > >
> > > > > > > > > > > > > We are working on PRs for these issues and will get
> > them
> > > > > in soon,
> > > > > > > > > we
> > > > > > > > > > > > think!
> > > > > > > > > > > > >
> > > > > > > > > > > > > So unfortunately I have to leave a -1 here for RC0.
> > Let's
> > > > > aim for
> > > > > > > > > > > another
> > > > > > > > > > > > > RC next week.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison
> wrote:
> > > > > > > > > > > > > > Hi David,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We're already quite a bit behind schedule. If you
> > think
> > > > > these
> > > > > > > > > fixes
> > > > > > > > > > > > > > are really important and can be ready in the next
> > > > couple
> > > > > of
> > > > > > > > > days, I'm
> > > > > > > > > > > > > > open to backport them and build another release
> > > > > candidate. Let me
> > > > > > > > > > > know
> > > > > > > > > > > > > > once you've investigated the severity of
> > KAFKA-15010.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > > > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Mickael, we have some migration fixes on trunk,
> > is it
> > > > > okay to
> > > > > > > > > > > > > cherry-pick
> > > > > > > > > > > > > >> these to 3.5?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > > > > > > > > MigrationPropagator
> > > > > > > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot
> > during
> > > > > migration
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> There is another issue KAFKA-15010 that I'm also
> > > > > investigating
> > > > > > > > > to
> > > > > > > > > > > > > determine
> > > > > > > > > > > > > >> the impact and likelihood of seeing it in
> > practice.
> > > > > This one
> > > > > > > > > may be
> > > > > > > > > > > a
> > > > > > > > > > > > > >> significant migration blocker
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Cheers,
> > > > > > > > > > > > > >> David
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> > Hi Christo,
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Yes this is expected. This happens when nested
> > > > fields
> > > > > also
> > > > > > > > > accept
> > > > > > > > > > > > > >> > optional tagged fields. The tables list all
> > fields,
> > > > > so they
> > > > > > > > > may
> > > > > > > > > > > > > >> > include _tagged_fields multiple times.
> > > > > > > > > > > > > >> > Clearly the layout of this page could be
> > improved,
> > > > if
> > > > > you have
> > > > > > > > > > > ideas
> > > > > > > > > > > > > >> > how to describe the protocol in a better way,
> > feel
> > > > > free to
> > > > > > > > > share
> > > > > > > > > > > > them.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Thanks,
> > > > > > > > > > > > > >> > Mickael
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael
> Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Hi Josep,
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > > > > > > > > >> > >
> https://github.com/apache/kafka-site/pull/514
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Thanks,
> > > > > > > > > > > > > >> > > Mickael
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo
> Lolov
> > <
> > > > > > > > > > > > > christololov@gmail.com>
> > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Hey Mickael!
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > I am giving a +1 (non-binding) for this
> > > > candidate
> > > > > release.
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > * Built from the binary tar.gz source with
> > Java
> > > > > 17 and
> > > > > > > > > Scala
> > > > > > > > > > > > 2.13
> > > > > > > > > > > > > on
> > > > > > > > > > > > > >> > Intel
> > > > > > > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge)
> machines.
> > > > > > > > > > > > > >> > > > * Ran unit and integration tests on Intel
> > and
> > > > ARM
> > > > > > > > > machines.
> > > > > > > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and
> > KRaft
> > > > > modes on
> > > > > > > > > > > Intel
> > > > > > > > > > > > > and ARM
> > > > > > > > > > > > > >> > > > machines.
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Question:
> > > > > > > > > > > > > >> > > > * I went through
> > > > > > > > > https://kafka.apache.org/35/protocol.html
> > > > > > > > > > > and
> > > > > > > > > > > > > there
> > > > > > > > > > > > > >> > are
> > > > > > > > > > > > > >> > > > quite a few repetitive __tagged_fileds
> > fields
> > > > > within the
> > > > > > > > > same
> > > > > > > > > > > > > >> > structures -
> > > > > > > > > > > > > >> > > > is this expected?
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > > > > > > > > <jo...@aiven.io.invalid>
> > > > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > > Hi Mickael,
> > > > > > > > > > > > > >> > > > > I just wanted to point out that I think
> > the
> > > > > > > > > documentation
> > > > > > > > > > > you
> > > > > > > > > > > > > >> > recently
> > > > > > > > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0
> > > > > version has a
> > > > > > > > > > > problem
> > > > > > > > > > > > > when
> > > > > > > > > > > > > >> > it
> > > > > > > > > > > > > >> > > > > states the version number and the
> sub-menu
> > > > that
> > > > > links to
> > > > > > > > > > > > > previous
> > > > > > > > > > > > > >> > versions.
> > > > > > > > > > > > > >> > > > > Left a comment here:
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > > >
> >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > Best,
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep
> > Prat <
> > > > > > > > > > > > josep.prat@aiven.io
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > > Hi Mickael,
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > I can +1 this candidate. I verified
> the
> > > > > following:
> > > > > > > > > > > > > >> > > > > > - Built from source with Java 17 and
> > Scala
> > > > > 2.13
> > > > > > > > > > > > > >> > > > > > - Signatures and hashes of the
> artifacts
> > > > > generated
> > > > > > > > > > > > > >> > > > > > - Navigated through Javadoc including
> > links
> > > > > to JDK
> > > > > > > > > classes
> > > > > > > > > > > > > >> > > > > > - Run the unit tests
> > > > > > > > > > > > > >> > > > > > - Run integration tests
> > > > > > > > > > > > > >> > > > > > - Run the quickstart in KRaft and
> > Zookeeper
> > > > > mode
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > Best,
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM
> Mickael
> > > > > Maison <
> > > > > > > > > > > > > >> > mimaison@apache.org>
> > > > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > >> Hello Kafka users, developers and
> > > > > client-developers,
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> This is the first candidate for
> > release of
> > > > > Apache
> > > > > > > > > Kafka
> > > > > > > > > > > > > 3.5.0.
> > > > > > > > > > > > > >> > Some of
> > > > > > > > > > > > > >> > > > > the
> > > > > > > > > > > > > >> > > > > >> major features include:
> > > > > > > > > > > > > >> > > > > >> - KIP-710: Full support for
> distributed
> > > > mode
> > > > > in
> > > > > > > > > dedicated
> > > > > > > > > > > > > >> > MirrorMaker
> > > > > > > > > > > > > >> > > > > >> 2.0 clusters
> > > > > > > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition
> > Assignment
> > > > > for Kafka
> > > > > > > > > > > > > Consumers
> > > > > > > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make
> > use
> > > > of
> > > > > > > > > environment
> > > > > > > > > > > > > variables
> > > > > > > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > > > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig
> > for
> > > > > syncing
> > > > > > > > > topic
> > > > > > > > > > > > > >> > configurations
> > > > > > > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API
> > > > > additions to
> > > > > > > > > > > support
> > > > > > > > > > > > > SCRAM
> > > > > > > > > > > > > >> > for
> > > > > > > > > > > > > >> > > > > >> Kafka Brokers
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > >
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> *** Please download, test and vote by
> > > > > Friday, May
> > > > > > > > > 26, 5pm
> > > > > > > > > > > > PT
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys
> > we
> > > > use
> > > > > to sign
> > > > > > > > > the
> > > > > > > > > > > > > release:
> > > > > > > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> * Release artifacts to be voted upon
> > > > (source
> > > > > and
> > > > > > > > > binary):
> > > > > > > > > > > > > >> > > > > >>
> > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> * Javadoc:
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > >
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5
> > branch) is
> > > > > the 3.5.0
> > > > > > > > > tag:
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> The PR adding the 35 documentation is
> > not
> > > > > merged yet
> > > > > > > > > > > > > >> > > > > >> (
> > > > > https://github.com/apache/kafka-site/pull/513)
> > > > > > > > > > > > > >> > > > > >> * Documentation:
> > > > > > > > > > > > > >> > > > > >>
> > > > > https://kafka.apache.org/35/documentation.html
> > > > > > > > > > > > > >> > > > > >> * Protocol:
> > > > > > > > > > > > > >> > > > > >>
> > https://kafka.apache.org/35/protocol.html
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> * Successful Jenkins builds for the
> 3.5
> > > > > branch:
> > > > > > > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is
> not
> > > > > detecting the
> > > > > > > > > 3.5
> > > > > > > > > > > > > branch,
> > > > > > > > > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > > > > > > > > >> > > > > >>
> > > > > https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > > > > > > > > >> > > > > >> System tests: The build is still
> > running,
> > > > > I'll send
> > > > > > > > > an
> > > > > > > > > > > > update
> > > > > > > > > > > > > >> > once I
> > > > > > > > > > > > > >> > > > > >> have the results
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >> Thanks,
> > > > > > > > > > > > > >> > > > > >> Mickael
> > > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > --
> > > > > > > > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > *Josep Prat*
> > > > > > > > > > > > > >> > > > > > Open Source Engineering Director,
> > *Aiven*
> > > > > > > > > > > > > >> > > > > > josep.prat@aiven.io   |
>  +491715557497
> > > > > > > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > > > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > > > > > > > > >> > > > > > <
> > https://www.linkedin.com/company/aiven/>
> > > >  <
> > > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa &
> > Hannu
> > > > > Valtonen
> > > > > > > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB
> 209739 B
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > --
> > > > > > > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > *Josep Prat*
> > > > > > > > > > > > > >> > > > > Open Source Engineering Director,
> *Aiven*
> > > > > > > > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > >   <
> > https://www.linkedin.com/company/aiven/>
> > > >  <
> > > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa &
> Hannu
> > > > > Valtonen
> > > > > > > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> --
> > > > > > > > > > > > > >> -David
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > >
> > > > > > > > > > > *Josep Prat*
> > > > > > > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > > https://www.facebook.com/aivencloud
> > > > > > > > > > > >
> > > > > > > > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > > > > https://twitter.com/aiven_io>
> > > > > > > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -David
> > > > >
> > > >
> > > >
> > > > --
> > > > -David
> > > >
> >
>

Re: [VOTE] 3.5.0 RC0

Posted by Justine Olshan <jo...@confluent.io.INVALID>.

Hey Mickael --
This is done. Thanks!

On Wed, May 31, 2023 at 11:24 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Justine,
>
> Yes you can merge that into 3.5.
>
> Thanks,
> Mickael
>
> On Wed, May 31, 2023 at 7:56 PM Justine Olshan
> <jo...@confluent.io.invalid> wrote:
> >
> > FYI -- I just saw this PR regarding a dependency for ARM. We may want to
> > get this in for 3.5 as well. It should be quick.
> >
> > https://issues.apache.org/jira/browse/KAFKA-15044
> > https://github.com/apache/kafka/pull/13786
> >
> > Justine
> >
> > On Wed, May 31, 2023 at 9:28 AM David Arthur
> > <da...@confluent.io.invalid> wrote:
> >
> > > Mickael,
> > >
> > > Colin has approved my patch for KAFKA-15010, I'm just waiting on a
> build
> > > before merging. I'll go ahead and backport the other fixes that need to
> > > precede this one into 3.5.
> > >
> > > -David
> > >
> > > On Wed, May 31, 2023 at 11:52 AM Mickael Maison <
> mickael.maison@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > The issue mentioned by Greg has been fixed. As soon as the fix for
> > > > KAFKA-15010 is merged I'll build another RC.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, May 30, 2023 at 10:33 AM Mickael Maison
> > > > <mi...@gmail.com> wrote:
> > > > >
> > > > > Hi David,
> > > > >
> > > > > Feel free to backport the necessary fixes to 3.5.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Tue, May 30, 2023 at 10:32 AM Mickael Maison
> > > > > <mi...@gmail.com> wrote:
> > > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > Thanks for the heads up, this indeed looks like something we
> want in
> > > > > > 3.5. I've replied in the PR.
> > > > > >
> > > > > > Mickael
> > > > > >
> > > > > > On Sat, May 27, 2023 at 11:44 PM David Arthur
> > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > Mickael, after looking more closely, I definitely think
> KAFKA-15010
> > > > is a
> > > > > > > blocker. It creates the case where the controller can totally
> miss
> > > a
> > > > > > > metadata update and not write it back to ZK. Since things like
> > > > dynamic
> > > > > > > configs and ACLs are only read from ZK by the ZK brokers, we
> could
> > > > have
> > > > > > > significant problems while the brokers are being migrated (when
> > > some
> > > > are
> > > > > > > KRaft and some are ZK). E.g., ZK brokers could be totally
> unaware
> > > of
> > > > an ACL
> > > > > > > change while the KRaft brokers have it. I have a fix ready here
> > > > > > > https://github.com/apache/kafka/pull/13758. I think we can
> get it
> > > > committed
> > > > > > > soon.
> > > > > > >
> > > > > > > Another blocker is KAFKA-15004 which was just merged to trunk.
> This
> > > > is
> > > > > > > another dual-write bug where new topic/broker configs will not
> be
> > > > written
> > > > > > > back to ZK by the controller.
> > > > > > >
> > > > > > > The fix for KAFKA-15010 has a few dependencies on fixes we made
> > > this
> > > > past
> > > > > > > week, so we'll need to cherry-pick a few commits. The changes
> are
> > > > totally
> > > > > > > contained within the migration area of code, so I think the
> risk in
> > > > > > > including them is fairly low.
> > > > > > >
> > > > > > > -David
> > > > > > >
> > > > > > > On Thu, May 25, 2023 at 2:15 PM Greg Harris
> > > > <gr...@aiven.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey all,
> > > > > > > >
> > > > > > > > A contributor just pointed out a small but noticeable flaw
> in the
> > > > > > > > implementation of KIP-581
> > > > > > > >
> > > > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > > > > > > which is planned for this release.
> > > > > > > > Impact: the feature works for root values in a record, but
> does
> > > not
> > > > > > > > work for any fields within structs. Fields within structs
> will
> > > > > > > > continue to have their previous, backwards-compatible
> behavior.
> > > > > > > > The contributor has submitted a bug-fix PR which reports the
> > > > problem
> > > > > > > > and does not yet have a merge-able solution, but they are
> > > actively
> > > > > > > > responding and interested in having this fixed:
> > > > > > > > https://github.com/apache/kafka/pull/13748
> > > > > > > > The overall fix should be a one-liner + some unit tests.
> While
> > > > this is
> > > > > > > > not a regression, it does make the feature largely useless,
> as
> > > the
> > > > > > > > majority of use-cases will be for struct fields.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Greg Harris
> > > > > > > >
> > > > > > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <
> ismael@juma.me.uk>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > I agree the migration should be functional - it wasn't
> obvious
> > > > if the
> > > > > > > > > migration issues are edge cases or not. If they are edge
> cases,
> > > > I think
> > > > > > > > > 3.5.1 would be fine given the preview status.
> > > > > > > > >
> > > > > > > > > I understand that a new RC is needed, but that doesn't
> mean we
> > > > should let
> > > > > > > > > everything in. Each change carries some risk. And if we
> don't
> > > > agree on
> > > > > > > > the
> > > > > > > > > bar for the migration work, we may be having the same
> > > discussion
> > > > next
> > > > > > > > week.
> > > > > > > > > :)
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat
> > > > <jo...@aiven.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi there,
> > > > > > > > > > Is the plan described in KIP-833[1] still valid? In
> there it
> > > > states
> > > > > > > > that
> > > > > > > > > > 3.5.0 should aim at deprecation of Zookeeper, so
> > > conceptually,
> > > > the
> > > > > > > > path to
> > > > > > > > > > migrate to Kraft should be somewhat functional (in my
> > > > opinion). If we
> > > > > > > > don't
> > > > > > > > > > want to deprecate Zookeeper in 3.5.0, then I share
> Ismael's
> > > > opinion
> > > > > > > > that
> > > > > > > > > > these could be fixed in subsequent patches of 3.5.x.
> Just my
> > > > 5cts.
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <
> > > ismael@juma.me.uk>
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Are all these blockers? For example, zk to kraft
> migration
> > > > are is
> > > > > > > > still
> > > > > > > > > > in
> > > > > > > > > > > preview - can we fix some of these in 3.5.1?
> > > > > > > > > > >
> > > > > > > > > > > Ismael
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for putting together this RC. Unfortunately,
> we've
> > > > > > > > identified
> > > > > > > > > > > > several blocker issues in this release candidate.
> > > > > > > > > > > >
> > > > > > > > > > > > KAFKA-15009: New ACLs are not written to ZK during
> > > > migration
> > > > > > > > > > > > KAFKA-15007: MV is not set correctly in the
> > > > MetadataPropagator in
> > > > > > > > > > > > migration.
> > > > > > > > > > > > KAFKA-15004: Topic config changes are not synced
> during
> > > zk
> > > > to kraft
> > > > > > > > > > > > migration (dual-write)
> > > > > > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated
> in
> > > > migration
> > > > > > > > > > > > (dual-write) when partitions are changed for topic
> > > > > > > > > > > > KAFKA-14996: The KRaft controller should properly
> handle
> > > > overly
> > > > > > > > large
> > > > > > > > > > > user
> > > > > > > > > > > > operations
> > > > > > > > > > > >
> > > > > > > > > > > > We are working on PRs for these issues and will get
> them
> > > > in soon,
> > > > > > > > we
> > > > > > > > > > > think!
> > > > > > > > > > > >
> > > > > > > > > > > > So unfortunately I have to leave a -1 here for RC0.
> Let's
> > > > aim for
> > > > > > > > > > another
> > > > > > > > > > > > RC next week.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > > > > > > > > Hi David,
> > > > > > > > > > > > >
> > > > > > > > > > > > > We're already quite a bit behind schedule. If you
> think
> > > > these
> > > > > > > > fixes
> > > > > > > > > > > > > are really important and can be ready in the next
> > > couple
> > > > of
> > > > > > > > days, I'm
> > > > > > > > > > > > > open to backport them and build another release
> > > > candidate. Let me
> > > > > > > > > > know
> > > > > > > > > > > > > once you've investigated the severity of
> KAFKA-15010.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Mickael
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Mickael, we have some migration fixes on trunk,
> is it
> > > > okay to
> > > > > > > > > > > > cherry-pick
> > > > > > > > > > > > >> these to 3.5?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > > > > > > > MigrationPropagator
> > > > > > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot
> during
> > > > migration
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> There is another issue KAFKA-15010 that I'm also
> > > > investigating
> > > > > > > > to
> > > > > > > > > > > > determine
> > > > > > > > > > > > >> the impact and likelihood of seeing it in
> practice.
> > > > This one
> > > > > > > > may be
> > > > > > > > > > a
> > > > > > > > > > > > >> significant migration blocker
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Cheers,
> > > > > > > > > > > > >> David
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi Christo,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Yes this is expected. This happens when nested
> > > fields
> > > > also
> > > > > > > > accept
> > > > > > > > > > > > >> > optional tagged fields. The tables list all
> fields,
> > > > so they
> > > > > > > > may
> > > > > > > > > > > > >> > include _tagged_fields multiple times.
> > > > > > > > > > > > >> > Clearly the layout of this page could be
> improved,
> > > if
> > > > you have
> > > > > > > > > > ideas
> > > > > > > > > > > > >> > how to describe the protocol in a better way,
> feel
> > > > free to
> > > > > > > > share
> > > > > > > > > > > them.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Thanks,
> > > > > > > > > > > > >> > Mickael
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Hi Josep,
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > > > > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Thanks,
> > > > > > > > > > > > >> > > Mickael
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov
> <
> > > > > > > > > > > > christololov@gmail.com>
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Hey Mickael!
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > I am giving a +1 (non-binding) for this
> > > candidate
> > > > release.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > * Built from the binary tar.gz source with
> Java
> > > > 17 and
> > > > > > > > Scala
> > > > > > > > > > > 2.13
> > > > > > > > > > > > on
> > > > > > > > > > > > >> > Intel
> > > > > > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > > > > > > > > >> > > > * Ran unit and integration tests on Intel
> and
> > > ARM
> > > > > > > > machines.
> > > > > > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and
> KRaft
> > > > modes on
> > > > > > > > > > Intel
> > > > > > > > > > > > and ARM
> > > > > > > > > > > > >> > > > machines.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Question:
> > > > > > > > > > > > >> > > > * I went through
> > > > > > > > https://kafka.apache.org/35/protocol.html
> > > > > > > > > > and
> > > > > > > > > > > > there
> > > > > > > > > > > > >> > are
> > > > > > > > > > > > >> > > > quite a few repetitive __tagged_fileds
> fields
> > > > within the
> > > > > > > > same
> > > > > > > > > > > > >> > structures -
> > > > > > > > > > > > >> > > > is this expected?
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > > > > > > > <jo...@aiven.io.invalid>
> > > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > > Hi Mickael,
> > > > > > > > > > > > >> > > > > I just wanted to point out that I think
> the
> > > > > > > > documentation
> > > > > > > > > > you
> > > > > > > > > > > > >> > recently
> > > > > > > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0
> > > > version has a
> > > > > > > > > > problem
> > > > > > > > > > > > when
> > > > > > > > > > > > >> > it
> > > > > > > > > > > > >> > > > > states the version number and the sub-menu
> > > that
> > > > links to
> > > > > > > > > > > > previous
> > > > > > > > > > > > >> > versions.
> > > > > > > > > > > > >> > > > > Left a comment here:
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > >
> > >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Best,
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep
> Prat <
> > > > > > > > > > > josep.prat@aiven.io
> > > > > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > > Hi Mickael,
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > I can +1 this candidate. I verified the
> > > > following:
> > > > > > > > > > > > >> > > > > > - Built from source with Java 17 and
> Scala
> > > > 2.13
> > > > > > > > > > > > >> > > > > > - Signatures and hashes of the artifacts
> > > > generated
> > > > > > > > > > > > >> > > > > > - Navigated through Javadoc including
> links
> > > > to JDK
> > > > > > > > classes
> > > > > > > > > > > > >> > > > > > - Run the unit tests
> > > > > > > > > > > > >> > > > > > - Run integration tests
> > > > > > > > > > > > >> > > > > > - Run the quickstart in KRaft and
> Zookeeper
> > > > mode
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > Best,
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael
> > > > Maison <
> > > > > > > > > > > > >> > mimaison@apache.org>
> > > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > >> Hello Kafka users, developers and
> > > > client-developers,
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> This is the first candidate for
> release of
> > > > Apache
> > > > > > > > Kafka
> > > > > > > > > > > > 3.5.0.
> > > > > > > > > > > > >> > Some of
> > > > > > > > > > > > >> > > > > the
> > > > > > > > > > > > >> > > > > >> major features include:
> > > > > > > > > > > > >> > > > > >> - KIP-710: Full support for distributed
> > > mode
> > > > in
> > > > > > > > dedicated
> > > > > > > > > > > > >> > MirrorMaker
> > > > > > > > > > > > >> > > > > >> 2.0 clusters
> > > > > > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition
> Assignment
> > > > for Kafka
> > > > > > > > > > > > Consumers
> > > > > > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make
> use
> > > of
> > > > > > > > environment
> > > > > > > > > > > > variables
> > > > > > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig
> for
> > > > syncing
> > > > > > > > topic
> > > > > > > > > > > > >> > configurations
> > > > > > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API
> > > > additions to
> > > > > > > > > > support
> > > > > > > > > > > > SCRAM
> > > > > > > > > > > > >> > for
> > > > > > > > > > > > >> > > > > >> Kafka Brokers
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> *** Please download, test and vote by
> > > > Friday, May
> > > > > > > > 26, 5pm
> > > > > > > > > > > PT
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys
> we
> > > use
> > > > to sign
> > > > > > > > the
> > > > > > > > > > > > release:
> > > > > > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> * Release artifacts to be voted upon
> > > (source
> > > > and
> > > > > > > > binary):
> > > > > > > > > > > > >> > > > > >>
> > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > > > > >
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> * Javadoc:
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > >
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5
> branch) is
> > > > the 3.5.0
> > > > > > > > tag:
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> The PR adding the 35 documentation is
> not
> > > > merged yet
> > > > > > > > > > > > >> > > > > >> (
> > > > https://github.com/apache/kafka-site/pull/513)
> > > > > > > > > > > > >> > > > > >> * Documentation:
> > > > > > > > > > > > >> > > > > >>
> > > > https://kafka.apache.org/35/documentation.html
> > > > > > > > > > > > >> > > > > >> * Protocol:
> > > > > > > > > > > > >> > > > > >>
> https://kafka.apache.org/35/protocol.html
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> * Successful Jenkins builds for the 3.5
> > > > branch:
> > > > > > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is not
> > > > detecting the
> > > > > > > > 3.5
> > > > > > > > > > > > branch,
> > > > > > > > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > > > > > > > >> > > > > >>
> > > > https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > > > > > > > >> > > > > >> System tests: The build is still
> running,
> > > > I'll send
> > > > > > > > an
> > > > > > > > > > > update
> > > > > > > > > > > > >> > once I
> > > > > > > > > > > > >> > > > > >> have the results
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >> Thanks,
> > > > > > > > > > > > >> > > > > >> Mickael
> > > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > --
> > > > > > > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > *Josep Prat*
> > > > > > > > > > > > >> > > > > > Open Source Engineering Director,
> *Aiven*
> > > > > > > > > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > > > > > > > >> > > > > > <
> https://www.linkedin.com/company/aiven/>
> > >  <
> > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa &
> Hannu
> > > > Valtonen
> > > > > > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > --
> > > > > > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > *Josep Prat*
> > > > > > > > > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >   <
> https://www.linkedin.com/company/aiven/>
> > >  <
> > > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu
> > > > Valtonen
> > > > > > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >> -David
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > >
> > > > > > > > > > *Josep Prat*
> > > > > > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > https://www.facebook.com/aivencloud
> > > > > > > > > > >
> > > > > > > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > > > https://twitter.com/aiven_io>
> > > > > > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -David
> > > >
> > >
> > >
> > > --
> > > -David
> > >
>

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi Justine,

Yes you can merge that into 3.5.

Thanks,
Mickael

On Wed, May 31, 2023 at 7:56 PM Justine Olshan
<jo...@confluent.io.invalid> wrote:
>
> FYI -- I just saw this PR regarding a dependency for ARM. We may want to
> get this in for 3.5 as well. It should be quick.
>
> https://issues.apache.org/jira/browse/KAFKA-15044
> https://github.com/apache/kafka/pull/13786
>
> Justine
>
> On Wed, May 31, 2023 at 9:28 AM David Arthur
> <da...@confluent.io.invalid> wrote:
>
> > Mickael,
> >
> > Colin has approved my patch for KAFKA-15010, I'm just waiting on a build
> > before merging. I'll go ahead and backport the other fixes that need to
> > precede this one into 3.5.
> >
> > -David
> >
> > On Wed, May 31, 2023 at 11:52 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > The issue mentioned by Greg has been fixed. As soon as the fix for
> > > KAFKA-15010 is merged I'll build another RC.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, May 30, 2023 at 10:33 AM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > > >
> > > > Hi David,
> > > >
> > > > Feel free to backport the necessary fixes to 3.5.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, May 30, 2023 at 10:32 AM Mickael Maison
> > > > <mi...@gmail.com> wrote:
> > > > >
> > > > > Hi Greg,
> > > > >
> > > > > Thanks for the heads up, this indeed looks like something we want in
> > > > > 3.5. I've replied in the PR.
> > > > >
> > > > > Mickael
> > > > >
> > > > > On Sat, May 27, 2023 at 11:44 PM David Arthur
> > > > > <da...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Mickael, after looking more closely, I definitely think KAFKA-15010
> > > is a
> > > > > > blocker. It creates the case where the controller can totally miss
> > a
> > > > > > metadata update and not write it back to ZK. Since things like
> > > dynamic
> > > > > > configs and ACLs are only read from ZK by the ZK brokers, we could
> > > have
> > > > > > significant problems while the brokers are being migrated (when
> > some
> > > are
> > > > > > KRaft and some are ZK). E.g., ZK brokers could be totally unaware
> > of
> > > an ACL
> > > > > > change while the KRaft brokers have it. I have a fix ready here
> > > > > > https://github.com/apache/kafka/pull/13758. I think we can get it
> > > committed
> > > > > > soon.
> > > > > >
> > > > > > Another blocker is KAFKA-15004 which was just merged to trunk. This
> > > is
> > > > > > another dual-write bug where new topic/broker configs will not be
> > > written
> > > > > > back to ZK by the controller.
> > > > > >
> > > > > > The fix for KAFKA-15010 has a few dependencies on fixes we made
> > this
> > > past
> > > > > > week, so we'll need to cherry-pick a few commits. The changes are
> > > totally
> > > > > > contained within the migration area of code, so I think the risk in
> > > > > > including them is fairly low.
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Thu, May 25, 2023 at 2:15 PM Greg Harris
> > > <gr...@aiven.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey all,
> > > > > > >
> > > > > > > A contributor just pointed out a small but noticeable flaw in the
> > > > > > > implementation of KIP-581
> > > > > > >
> > > > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > > > > > which is planned for this release.
> > > > > > > Impact: the feature works for root values in a record, but does
> > not
> > > > > > > work for any fields within structs. Fields within structs will
> > > > > > > continue to have their previous, backwards-compatible behavior.
> > > > > > > The contributor has submitted a bug-fix PR which reports the
> > > problem
> > > > > > > and does not yet have a merge-able solution, but they are
> > actively
> > > > > > > responding and interested in having this fixed:
> > > > > > > https://github.com/apache/kafka/pull/13748
> > > > > > > The overall fix should be a one-liner + some unit tests. While
> > > this is
> > > > > > > not a regression, it does make the feature largely useless, as
> > the
> > > > > > > majority of use-cases will be for struct fields.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Greg Harris
> > > > > > >
> > > > > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > > > > >
> > > > > > > > I agree the migration should be functional - it wasn't obvious
> > > if the
> > > > > > > > migration issues are edge cases or not. If they are edge cases,
> > > I think
> > > > > > > > 3.5.1 would be fine given the preview status.
> > > > > > > >
> > > > > > > > I understand that a new RC is needed, but that doesn't mean we
> > > should let
> > > > > > > > everything in. Each change carries some risk. And if we don't
> > > agree on
> > > > > > > the
> > > > > > > > bar for the migration work, we may be having the same
> > discussion
> > > next
> > > > > > > week.
> > > > > > > > :)
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat
> > > <jo...@aiven.io.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi there,
> > > > > > > > > Is the plan described in KIP-833[1] still valid? In there it
> > > states
> > > > > > > that
> > > > > > > > > 3.5.0 should aim at deprecation of Zookeeper, so
> > conceptually,
> > > the
> > > > > > > path to
> > > > > > > > > migrate to Kraft should be somewhat functional (in my
> > > opinion). If we
> > > > > > > don't
> > > > > > > > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's
> > > opinion
> > > > > > > that
> > > > > > > > > these could be fixed in subsequent patches of 3.5.x. Just my
> > > 5cts.
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <
> > ismael@juma.me.uk>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > Are all these blockers? For example, zk to kraft migration
> > > are is
> > > > > > > still
> > > > > > > > > in
> > > > > > > > > > preview - can we fix some of these in 3.5.1?
> > > > > > > > > >
> > > > > > > > > > Ismael
> > > > > > > > > >
> > > > > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <
> > > cmccabe@apache.org>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for putting together this RC. Unfortunately, we've
> > > > > > > identified
> > > > > > > > > > > several blocker issues in this release candidate.
> > > > > > > > > > >
> > > > > > > > > > > KAFKA-15009: New ACLs are not written to ZK during
> > > migration
> > > > > > > > > > > KAFKA-15007: MV is not set correctly in the
> > > MetadataPropagator in
> > > > > > > > > > > migration.
> > > > > > > > > > > KAFKA-15004: Topic config changes are not synced during
> > zk
> > > to kraft
> > > > > > > > > > > migration (dual-write)
> > > > > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in
> > > migration
> > > > > > > > > > > (dual-write) when partitions are changed for topic
> > > > > > > > > > > KAFKA-14996: The KRaft controller should properly handle
> > > overly
> > > > > > > large
> > > > > > > > > > user
> > > > > > > > > > > operations
> > > > > > > > > > >
> > > > > > > > > > > We are working on PRs for these issues and will get them
> > > in soon,
> > > > > > > we
> > > > > > > > > > think!
> > > > > > > > > > >
> > > > > > > > > > > So unfortunately I have to leave a -1 here for RC0. Let's
> > > aim for
> > > > > > > > > another
> > > > > > > > > > > RC next week.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > > > > > > > Hi David,
> > > > > > > > > > > >
> > > > > > > > > > > > We're already quite a bit behind schedule. If you think
> > > these
> > > > > > > fixes
> > > > > > > > > > > > are really important and can be ready in the next
> > couple
> > > of
> > > > > > > days, I'm
> > > > > > > > > > > > open to backport them and build another release
> > > candidate. Let me
> > > > > > > > > know
> > > > > > > > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Mickael
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> Mickael, we have some migration fixes on trunk, is it
> > > okay to
> > > > > > > > > > > cherry-pick
> > > > > > > > > > > >> these to 3.5?
> > > > > > > > > > > >>
> > > > > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > > > > > > MigrationPropagator
> > > > > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during
> > > migration
> > > > > > > > > > > >>
> > > > > > > > > > > >> There is another issue KAFKA-15010 that I'm also
> > > investigating
> > > > > > > to
> > > > > > > > > > > determine
> > > > > > > > > > > >> the impact and likelihood of seeing it in practice.
> > > This one
> > > > > > > may be
> > > > > > > > > a
> > > > > > > > > > > >> significant migration blocker
> > > > > > > > > > > >>
> > > > > > > > > > > >> Cheers,
> > > > > > > > > > > >> David
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi Christo,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Yes this is expected. This happens when nested
> > fields
> > > also
> > > > > > > accept
> > > > > > > > > > > >> > optional tagged fields. The tables list all fields,
> > > so they
> > > > > > > may
> > > > > > > > > > > >> > include _tagged_fields multiple times.
> > > > > > > > > > > >> > Clearly the layout of this page could be improved,
> > if
> > > you have
> > > > > > > > > ideas
> > > > > > > > > > > >> > how to describe the protocol in a better way, feel
> > > free to
> > > > > > > share
> > > > > > > > > > them.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Thanks,
> > > > > > > > > > > >> > Mickael
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Hi Josep,
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > > > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Thanks,
> > > > > > > > > > > >> > > Mickael
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > > > > > > > > christololov@gmail.com>
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Hey Mickael!
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > I am giving a +1 (non-binding) for this
> > candidate
> > > release.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > * Built from the binary tar.gz source with Java
> > > 17 and
> > > > > > > Scala
> > > > > > > > > > 2.13
> > > > > > > > > > > on
> > > > > > > > > > > >> > Intel
> > > > > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > > > > > > > >> > > > * Ran unit and integration tests on Intel and
> > ARM
> > > > > > > machines.
> > > > > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft
> > > modes on
> > > > > > > > > Intel
> > > > > > > > > > > and ARM
> > > > > > > > > > > >> > > > machines.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Question:
> > > > > > > > > > > >> > > > * I went through
> > > > > > > https://kafka.apache.org/35/protocol.html
> > > > > > > > > and
> > > > > > > > > > > there
> > > > > > > > > > > >> > are
> > > > > > > > > > > >> > > > quite a few repetitive __tagged_fileds fields
> > > within the
> > > > > > > same
> > > > > > > > > > > >> > structures -
> > > > > > > > > > > >> > > > is this expected?
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > > > > > > <jo...@aiven.io.invalid>
> > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > Hi Mickael,
> > > > > > > > > > > >> > > > > I just wanted to point out that I think the
> > > > > > > documentation
> > > > > > > > > you
> > > > > > > > > > > >> > recently
> > > > > > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0
> > > version has a
> > > > > > > > > problem
> > > > > > > > > > > when
> > > > > > > > > > > >> > it
> > > > > > > > > > > >> > > > > states the version number and the sub-menu
> > that
> > > links to
> > > > > > > > > > > previous
> > > > > > > > > > > >> > versions.
> > > > > > > > > > > >> > > > > Left a comment here:
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > >
> > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Best,
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > > > > > > > > josep.prat@aiven.io
> > > > > > > > > > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > Hi Mickael,
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > I can +1 this candidate. I verified the
> > > following:
> > > > > > > > > > > >> > > > > > - Built from source with Java 17 and Scala
> > > 2.13
> > > > > > > > > > > >> > > > > > - Signatures and hashes of the artifacts
> > > generated
> > > > > > > > > > > >> > > > > > - Navigated through Javadoc including links
> > > to JDK
> > > > > > > classes
> > > > > > > > > > > >> > > > > > - Run the unit tests
> > > > > > > > > > > >> > > > > > - Run integration tests
> > > > > > > > > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper
> > > mode
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Best,
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael
> > > Maison <
> > > > > > > > > > > >> > mimaison@apache.org>
> > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > >> Hello Kafka users, developers and
> > > client-developers,
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> This is the first candidate for release of
> > > Apache
> > > > > > > Kafka
> > > > > > > > > > > 3.5.0.
> > > > > > > > > > > >> > Some of
> > > > > > > > > > > >> > > > > the
> > > > > > > > > > > >> > > > > >> major features include:
> > > > > > > > > > > >> > > > > >> - KIP-710: Full support for distributed
> > mode
> > > in
> > > > > > > dedicated
> > > > > > > > > > > >> > MirrorMaker
> > > > > > > > > > > >> > > > > >> 2.0 clusters
> > > > > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment
> > > for Kafka
> > > > > > > > > > > Consumers
> > > > > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use
> > of
> > > > > > > environment
> > > > > > > > > > > variables
> > > > > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for
> > > syncing
> > > > > > > topic
> > > > > > > > > > > >> > configurations
> > > > > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API
> > > additions to
> > > > > > > > > support
> > > > > > > > > > > SCRAM
> > > > > > > > > > > >> > for
> > > > > > > > > > > >> > > > > >> Kafka Brokers
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> >
> > > > > > > > > >
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> *** Please download, test and vote by
> > > Friday, May
> > > > > > > 26, 5pm
> > > > > > > > > > PT
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys we
> > use
> > > to sign
> > > > > > > the
> > > > > > > > > > > release:
> > > > > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> * Release artifacts to be voted upon
> > (source
> > > and
> > > > > > > binary):
> > > > > > > > > > > >> > > > > >>
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> * Javadoc:
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is
> > > the 3.5.0
> > > > > > > tag:
> > > > > > > > > > > >> > > > > >>
> > > > > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> The PR adding the 35 documentation is not
> > > merged yet
> > > > > > > > > > > >> > > > > >> (
> > > https://github.com/apache/kafka-site/pull/513)
> > > > > > > > > > > >> > > > > >> * Documentation:
> > > > > > > > > > > >> > > > > >>
> > > https://kafka.apache.org/35/documentation.html
> > > > > > > > > > > >> > > > > >> * Protocol:
> > > > > > > > > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> * Successful Jenkins builds for the 3.5
> > > branch:
> > > > > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is not
> > > detecting the
> > > > > > > 3.5
> > > > > > > > > > > branch,
> > > > > > > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > > > > > > >> > > > > >>
> > > https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > > > > > > >> > > > > >> System tests: The build is still running,
> > > I'll send
> > > > > > > an
> > > > > > > > > > update
> > > > > > > > > > > >> > once I
> > > > > > > > > > > >> > > > > >> have the results
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >> Thanks,
> > > > > > > > > > > >> > > > > >> Mickael
> > > > > > > > > > > >> > > > > >>
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > --
> > > > > > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > *Josep Prat*
> > > > > > > > > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > > > > > > >> > > > > > <https://www.linkedin.com/company/aiven/>
> >  <
> > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu
> > > Valtonen
> > > > > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > --
> > > > > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > *Josep Prat*
> > > > > > > > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >   <https://www.linkedin.com/company/aiven/>
> >  <
> > > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu
> > > Valtonen
> > > > > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >> -David
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > >
> > > > > > > > > *Josep Prat*
> > > > > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > https://www.facebook.com/aivencloud
> > > > > > > > > >
> > > > > > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > > https://twitter.com/aiven_io>
> > > > > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -David
> > >
> >
> >
> > --
> > -David
> >

Re: [VOTE] 3.5.0 RC0

Posted by Justine Olshan <jo...@confluent.io.INVALID>.

FYI -- I just saw this PR regarding a dependency for ARM. We may want to
get this in for 3.5 as well. It should be quick.

https://issues.apache.org/jira/browse/KAFKA-15044
https://github.com/apache/kafka/pull/13786

Justine

On Wed, May 31, 2023 at 9:28 AM David Arthur
<da...@confluent.io.invalid> wrote:

> Mickael,
>
> Colin has approved my patch for KAFKA-15010, I'm just waiting on a build
> before merging. I'll go ahead and backport the other fixes that need to
> precede this one into 3.5.
>
> -David
>
> On Wed, May 31, 2023 at 11:52 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi,
> >
> > The issue mentioned by Greg has been fixed. As soon as the fix for
> > KAFKA-15010 is merged I'll build another RC.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 30, 2023 at 10:33 AM Mickael Maison
> > <mi...@gmail.com> wrote:
> > >
> > > Hi David,
> > >
> > > Feel free to backport the necessary fixes to 3.5.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, May 30, 2023 at 10:32 AM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > > >
> > > > Hi Greg,
> > > >
> > > > Thanks for the heads up, this indeed looks like something we want in
> > > > 3.5. I've replied in the PR.
> > > >
> > > > Mickael
> > > >
> > > > On Sat, May 27, 2023 at 11:44 PM David Arthur
> > > > <da...@confluent.io.invalid> wrote:
> > > > >
> > > > > Mickael, after looking more closely, I definitely think KAFKA-15010
> > is a
> > > > > blocker. It creates the case where the controller can totally miss
> a
> > > > > metadata update and not write it back to ZK. Since things like
> > dynamic
> > > > > configs and ACLs are only read from ZK by the ZK brokers, we could
> > have
> > > > > significant problems while the brokers are being migrated (when
> some
> > are
> > > > > KRaft and some are ZK). E.g., ZK brokers could be totally unaware
> of
> > an ACL
> > > > > change while the KRaft brokers have it. I have a fix ready here
> > > > > https://github.com/apache/kafka/pull/13758. I think we can get it
> > committed
> > > > > soon.
> > > > >
> > > > > Another blocker is KAFKA-15004 which was just merged to trunk. This
> > is
> > > > > another dual-write bug where new topic/broker configs will not be
> > written
> > > > > back to ZK by the controller.
> > > > >
> > > > > The fix for KAFKA-15010 has a few dependencies on fixes we made
> this
> > past
> > > > > week, so we'll need to cherry-pick a few commits. The changes are
> > totally
> > > > > contained within the migration area of code, so I think the risk in
> > > > > including them is fairly low.
> > > > >
> > > > > -David
> > > > >
> > > > > On Thu, May 25, 2023 at 2:15 PM Greg Harris
> > <gr...@aiven.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > A contributor just pointed out a small but noticeable flaw in the
> > > > > > implementation of KIP-581
> > > > > >
> > > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > > > > which is planned for this release.
> > > > > > Impact: the feature works for root values in a record, but does
> not
> > > > > > work for any fields within structs. Fields within structs will
> > > > > > continue to have their previous, backwards-compatible behavior.
> > > > > > The contributor has submitted a bug-fix PR which reports the
> > problem
> > > > > > and does not yet have a merge-able solution, but they are
> actively
> > > > > > responding and interested in having this fixed:
> > > > > > https://github.com/apache/kafka/pull/13748
> > > > > > The overall fix should be a one-liner + some unit tests. While
> > this is
> > > > > > not a regression, it does make the feature largely useless, as
> the
> > > > > > majority of use-cases will be for struct fields.
> > > > > >
> > > > > > Thanks!
> > > > > > Greg Harris
> > > > > >
> > > > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > > > > >
> > > > > > > I agree the migration should be functional - it wasn't obvious
> > if the
> > > > > > > migration issues are edge cases or not. If they are edge cases,
> > I think
> > > > > > > 3.5.1 would be fine given the preview status.
> > > > > > >
> > > > > > > I understand that a new RC is needed, but that doesn't mean we
> > should let
> > > > > > > everything in. Each change carries some risk. And if we don't
> > agree on
> > > > > > the
> > > > > > > bar for the migration work, we may be having the same
> discussion
> > next
> > > > > > week.
> > > > > > > :)
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat
> > <jo...@aiven.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi there,
> > > > > > > > Is the plan described in KIP-833[1] still valid? In there it
> > states
> > > > > > that
> > > > > > > > 3.5.0 should aim at deprecation of Zookeeper, so
> conceptually,
> > the
> > > > > > path to
> > > > > > > > migrate to Kraft should be somewhat functional (in my
> > opinion). If we
> > > > > > don't
> > > > > > > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's
> > opinion
> > > > > > that
> > > > > > > > these could be fixed in subsequent patches of 3.5.x. Just my
> > 5cts.
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > >
> > > > > > > >
> > > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <
> ismael@juma.me.uk>
> > wrote:
> > > > > > > >
> > > > > > > > > Are all these blockers? For example, zk to kraft migration
> > are is
> > > > > > still
> > > > > > > > in
> > > > > > > > > preview - can we fix some of these in 3.5.1?
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <
> > cmccabe@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > Thanks for putting together this RC. Unfortunately, we've
> > > > > > identified
> > > > > > > > > > several blocker issues in this release candidate.
> > > > > > > > > >
> > > > > > > > > > KAFKA-15009: New ACLs are not written to ZK during
> > migration
> > > > > > > > > > KAFKA-15007: MV is not set correctly in the
> > MetadataPropagator in
> > > > > > > > > > migration.
> > > > > > > > > > KAFKA-15004: Topic config changes are not synced during
> zk
> > to kraft
> > > > > > > > > > migration (dual-write)
> > > > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in
> > migration
> > > > > > > > > > (dual-write) when partitions are changed for topic
> > > > > > > > > > KAFKA-14996: The KRaft controller should properly handle
> > overly
> > > > > > large
> > > > > > > > > user
> > > > > > > > > > operations
> > > > > > > > > >
> > > > > > > > > > We are working on PRs for these issues and will get them
> > in soon,
> > > > > > we
> > > > > > > > > think!
> > > > > > > > > >
> > > > > > > > > > So unfortunately I have to leave a -1 here for RC0. Let's
> > aim for
> > > > > > > > another
> > > > > > > > > > RC next week.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > > > > > > Hi David,
> > > > > > > > > > >
> > > > > > > > > > > We're already quite a bit behind schedule. If you think
> > these
> > > > > > fixes
> > > > > > > > > > > are really important and can be ready in the next
> couple
> > of
> > > > > > days, I'm
> > > > > > > > > > > open to backport them and build another release
> > candidate. Let me
> > > > > > > > know
> > > > > > > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mickael
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> Mickael, we have some migration fixes on trunk, is it
> > okay to
> > > > > > > > > > cherry-pick
> > > > > > > > > > >> these to 3.5?
> > > > > > > > > > >>
> > > > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > > > > > MigrationPropagator
> > > > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during
> > migration
> > > > > > > > > > >>
> > > > > > > > > > >> There is another issue KAFKA-15010 that I'm also
> > investigating
> > > > > > to
> > > > > > > > > > determine
> > > > > > > > > > >> the impact and likelihood of seeing it in practice.
> > This one
> > > > > > may be
> > > > > > > > a
> > > > > > > > > > >> significant migration blocker
> > > > > > > > > > >>
> > > > > > > > > > >> Cheers,
> > > > > > > > > > >> David
> > > > > > > > > > >>
> > > > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Christo,
> > > > > > > > > > >> >
> > > > > > > > > > >> > Yes this is expected. This happens when nested
> fields
> > also
> > > > > > accept
> > > > > > > > > > >> > optional tagged fields. The tables list all fields,
> > so they
> > > > > > may
> > > > > > > > > > >> > include _tagged_fields multiple times.
> > > > > > > > > > >> > Clearly the layout of this page could be improved,
> if
> > you have
> > > > > > > > ideas
> > > > > > > > > > >> > how to describe the protocol in a better way, feel
> > free to
> > > > > > share
> > > > > > > > > them.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Mickael
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Hi Josep,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > > > > >> > > Mickael
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > > > > > > > christololov@gmail.com>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Hey Mickael!
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > I am giving a +1 (non-binding) for this
> candidate
> > release.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > * Built from the binary tar.gz source with Java
> > 17 and
> > > > > > Scala
> > > > > > > > > 2.13
> > > > > > > > > > on
> > > > > > > > > > >> > Intel
> > > > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > > > > > > >> > > > * Ran unit and integration tests on Intel and
> ARM
> > > > > > machines.
> > > > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft
> > modes on
> > > > > > > > Intel
> > > > > > > > > > and ARM
> > > > > > > > > > >> > > > machines.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Question:
> > > > > > > > > > >> > > > * I went through
> > > > > > https://kafka.apache.org/35/protocol.html
> > > > > > > > and
> > > > > > > > > > there
> > > > > > > > > > >> > are
> > > > > > > > > > >> > > > quite a few repetitive __tagged_fileds fields
> > within the
> > > > > > same
> > > > > > > > > > >> > structures -
> > > > > > > > > > >> > > > is this expected?
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > > > > > <jo...@aiven.io.invalid>
> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Hi Mickael,
> > > > > > > > > > >> > > > > I just wanted to point out that I think the
> > > > > > documentation
> > > > > > > > you
> > > > > > > > > > >> > recently
> > > > > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0
> > version has a
> > > > > > > > problem
> > > > > > > > > > when
> > > > > > > > > > >> > it
> > > > > > > > > > >> > > > > states the version number and the sub-menu
> that
> > links to
> > > > > > > > > > previous
> > > > > > > > > > >> > versions.
> > > > > > > > > > >> > > > > Left a comment here:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Best,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > > > > > > > josep.prat@aiven.io
> > > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > Hi Mickael,
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > I can +1 this candidate. I verified the
> > following:
> > > > > > > > > > >> > > > > > - Built from source with Java 17 and Scala
> > 2.13
> > > > > > > > > > >> > > > > > - Signatures and hashes of the artifacts
> > generated
> > > > > > > > > > >> > > > > > - Navigated through Javadoc including links
> > to JDK
> > > > > > classes
> > > > > > > > > > >> > > > > > - Run the unit tests
> > > > > > > > > > >> > > > > > - Run integration tests
> > > > > > > > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper
> > mode
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Best,
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael
> > Maison <
> > > > > > > > > > >> > mimaison@apache.org>
> > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >> Hello Kafka users, developers and
> > client-developers,
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> This is the first candidate for release of
> > Apache
> > > > > > Kafka
> > > > > > > > > > 3.5.0.
> > > > > > > > > > >> > Some of
> > > > > > > > > > >> > > > > the
> > > > > > > > > > >> > > > > >> major features include:
> > > > > > > > > > >> > > > > >> - KIP-710: Full support for distributed
> mode
> > in
> > > > > > dedicated
> > > > > > > > > > >> > MirrorMaker
> > > > > > > > > > >> > > > > >> 2.0 clusters
> > > > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment
> > for Kafka
> > > > > > > > > > Consumers
> > > > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use
> of
> > > > > > environment
> > > > > > > > > > variables
> > > > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for
> > syncing
> > > > > > topic
> > > > > > > > > > >> > configurations
> > > > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API
> > additions to
> > > > > > > > support
> > > > > > > > > > SCRAM
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > > > >> Kafka Brokers
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> >
> > > > > > > > >
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> *** Please download, test and vote by
> > Friday, May
> > > > > > 26, 5pm
> > > > > > > > > PT
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys we
> use
> > to sign
> > > > > > the
> > > > > > > > > > release:
> > > > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> * Release artifacts to be voted upon
> (source
> > and
> > > > > > binary):
> > > > > > > > > > >> > > > > >>
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> >
> > > > > > > > > >
> > > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> * Javadoc:
> > > > > > > > > > >> > > > > >>
> > > > > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is
> > the 3.5.0
> > > > > > tag:
> > > > > > > > > > >> > > > > >>
> > > > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> The PR adding the 35 documentation is not
> > merged yet
> > > > > > > > > > >> > > > > >> (
> > https://github.com/apache/kafka-site/pull/513)
> > > > > > > > > > >> > > > > >> * Documentation:
> > > > > > > > > > >> > > > > >>
> > https://kafka.apache.org/35/documentation.html
> > > > > > > > > > >> > > > > >> * Protocol:
> > > > > > > > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> * Successful Jenkins builds for the 3.5
> > branch:
> > > > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is not
> > detecting the
> > > > > > 3.5
> > > > > > > > > > branch,
> > > > > > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > > > > > >> > > > > >>
> > https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > > > > > >> > > > > >> System tests: The build is still running,
> > I'll send
> > > > > > an
> > > > > > > > > update
> > > > > > > > > > >> > once I
> > > > > > > > > > >> > > > > >> have the results
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >> Thanks,
> > > > > > > > > > >> > > > > >> Mickael
> > > > > > > > > > >> > > > > >>
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > --
> > > > > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > *Josep Prat*
> > > > > > > > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > > > > > >> > > > > > <https://www.linkedin.com/company/aiven/>
>  <
> > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu
> > Valtonen
> > > > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > --
> > > > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > *Josep Prat*
> > > > > > > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >   <https://www.linkedin.com/company/aiven/>
>  <
> > > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu
> > Valtonen
> > > > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> -David
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > >
> > > > > > > > *Josep Prat*
> > > > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > https://www.facebook.com/aivencloud
> > > > > > > > >
> > > > > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > https://twitter.com/aiven_io>
> > > > > > > > *Aiven Deutschland GmbH*
> > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -David
> >
>
>
> --
> -David
>

Re: [VOTE] 3.5.0 RC0

Posted by David Arthur <da...@confluent.io.INVALID>.

Mickael,

Colin has approved my patch for KAFKA-15010, I'm just waiting on a build
before merging. I'll go ahead and backport the other fixes that need to
precede this one into 3.5.

-David

On Wed, May 31, 2023 at 11:52 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi,
>
> The issue mentioned by Greg has been fixed. As soon as the fix for
> KAFKA-15010 is merged I'll build another RC.
>
> Thanks,
> Mickael
>
> On Tue, May 30, 2023 at 10:33 AM Mickael Maison
> <mi...@gmail.com> wrote:
> >
> > Hi David,
> >
> > Feel free to backport the necessary fixes to 3.5.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 30, 2023 at 10:32 AM Mickael Maison
> > <mi...@gmail.com> wrote:
> > >
> > > Hi Greg,
> > >
> > > Thanks for the heads up, this indeed looks like something we want in
> > > 3.5. I've replied in the PR.
> > >
> > > Mickael
> > >
> > > On Sat, May 27, 2023 at 11:44 PM David Arthur
> > > <da...@confluent.io.invalid> wrote:
> > > >
> > > > Mickael, after looking more closely, I definitely think KAFKA-15010
> is a
> > > > blocker. It creates the case where the controller can totally miss a
> > > > metadata update and not write it back to ZK. Since things like
> dynamic
> > > > configs and ACLs are only read from ZK by the ZK brokers, we could
> have
> > > > significant problems while the brokers are being migrated (when some
> are
> > > > KRaft and some are ZK). E.g., ZK brokers could be totally unaware of
> an ACL
> > > > change while the KRaft brokers have it. I have a fix ready here
> > > > https://github.com/apache/kafka/pull/13758. I think we can get it
> committed
> > > > soon.
> > > >
> > > > Another blocker is KAFKA-15004 which was just merged to trunk. This
> is
> > > > another dual-write bug where new topic/broker configs will not be
> written
> > > > back to ZK by the controller.
> > > >
> > > > The fix for KAFKA-15010 has a few dependencies on fixes we made this
> past
> > > > week, so we'll need to cherry-pick a few commits. The changes are
> totally
> > > > contained within the migration area of code, so I think the risk in
> > > > including them is fairly low.
> > > >
> > > > -David
> > > >
> > > > On Thu, May 25, 2023 at 2:15 PM Greg Harris
> <gr...@aiven.io.invalid>
> > > > wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > A contributor just pointed out a small but noticeable flaw in the
> > > > > implementation of KIP-581
> > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > > > which is planned for this release.
> > > > > Impact: the feature works for root values in a record, but does not
> > > > > work for any fields within structs. Fields within structs will
> > > > > continue to have their previous, backwards-compatible behavior.
> > > > > The contributor has submitted a bug-fix PR which reports the
> problem
> > > > > and does not yet have a merge-able solution, but they are actively
> > > > > responding and interested in having this fixed:
> > > > > https://github.com/apache/kafka/pull/13748
> > > > > The overall fix should be a one-liner + some unit tests. While
> this is
> > > > > not a regression, it does make the feature largely useless, as the
> > > > > majority of use-cases will be for struct fields.
> > > > >
> > > > > Thanks!
> > > > > Greg Harris
> > > > >
> > > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> > > > > >
> > > > > > I agree the migration should be functional - it wasn't obvious
> if the
> > > > > > migration issues are edge cases or not. If they are edge cases,
> I think
> > > > > > 3.5.1 would be fine given the preview status.
> > > > > >
> > > > > > I understand that a new RC is needed, but that doesn't mean we
> should let
> > > > > > everything in. Each change carries some risk. And if we don't
> agree on
> > > > > the
> > > > > > bar for the migration work, we may be having the same discussion
> next
> > > > > week.
> > > > > > :)
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat
> <jo...@aiven.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi there,
> > > > > > > Is the plan described in KIP-833[1] still valid? In there it
> states
> > > > > that
> > > > > > > 3.5.0 should aim at deprecation of Zookeeper, so conceptually,
> the
> > > > > path to
> > > > > > > migrate to Kraft should be somewhat functional (in my
> opinion). If we
> > > > > don't
> > > > > > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's
> opinion
> > > > > that
> > > > > > > these could be fixed in subsequent patches of 3.5.x. Just my
> 5cts.
> > > > > > >
> > > > > > > [1]:
> > > > > > >
> > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > > > > Best,
> > > > > > >
> > > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> > > > > > >
> > > > > > > > Are all these blockers? For example, zk to kraft migration
> are is
> > > > > still
> > > > > > > in
> > > > > > > > preview - can we fix some of these in 3.5.1?
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <
> cmccabe@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > Thanks for putting together this RC. Unfortunately, we've
> > > > > identified
> > > > > > > > > several blocker issues in this release candidate.
> > > > > > > > >
> > > > > > > > > KAFKA-15009: New ACLs are not written to ZK during
> migration
> > > > > > > > > KAFKA-15007: MV is not set correctly in the
> MetadataPropagator in
> > > > > > > > > migration.
> > > > > > > > > KAFKA-15004: Topic config changes are not synced during zk
> to kraft
> > > > > > > > > migration (dual-write)
> > > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in
> migration
> > > > > > > > > (dual-write) when partitions are changed for topic
> > > > > > > > > KAFKA-14996: The KRaft controller should properly handle
> overly
> > > > > large
> > > > > > > > user
> > > > > > > > > operations
> > > > > > > > >
> > > > > > > > > We are working on PRs for these issues and will get them
> in soon,
> > > > > we
> > > > > > > > think!
> > > > > > > > >
> > > > > > > > > So unfortunately I have to leave a -1 here for RC0. Let's
> aim for
> > > > > > > another
> > > > > > > > > RC next week.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > > > > > Hi David,
> > > > > > > > > >
> > > > > > > > > > We're already quite a bit behind schedule. If you think
> these
> > > > > fixes
> > > > > > > > > > are really important and can be ready in the next couple
> of
> > > > > days, I'm
> > > > > > > > > > open to backport them and build another release
> candidate. Let me
> > > > > > > know
> > > > > > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > > > >>
> > > > > > > > > >> Mickael, we have some migration fixes on trunk, is it
> okay to
> > > > > > > > > cherry-pick
> > > > > > > > > >> these to 3.5?
> > > > > > > > > >>
> > > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > > > > MigrationPropagator
> > > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during
> migration
> > > > > > > > > >>
> > > > > > > > > >> There is another issue KAFKA-15010 that I'm also
> investigating
> > > > > to
> > > > > > > > > determine
> > > > > > > > > >> the impact and likelihood of seeing it in practice.
> This one
> > > > > may be
> > > > > > > a
> > > > > > > > > >> significant migration blocker
> > > > > > > > > >>
> > > > > > > > > >> Cheers,
> > > > > > > > > >> David
> > > > > > > > > >>
> > > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Christo,
> > > > > > > > > >> >
> > > > > > > > > >> > Yes this is expected. This happens when nested fields
> also
> > > > > accept
> > > > > > > > > >> > optional tagged fields. The tables list all fields,
> so they
> > > > > may
> > > > > > > > > >> > include _tagged_fields multiple times.
> > > > > > > > > >> > Clearly the layout of this page could be improved, if
> you have
> > > > > > > ideas
> > > > > > > > > >> > how to describe the protocol in a better way, feel
> free to
> > > > > share
> > > > > > > > them.
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > Mickael
> > > > > > > > > >> >
> > > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > Hi Josep,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> > > Mickael
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > > > > > > christololov@gmail.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Hey Mickael!
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > I am giving a +1 (non-binding) for this candidate
> release.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > * Built from the binary tar.gz source with Java
> 17 and
> > > > > Scala
> > > > > > > > 2.13
> > > > > > > > > on
> > > > > > > > > >> > Intel
> > > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > > > > > >> > > > * Ran unit and integration tests on Intel and ARM
> > > > > machines.
> > > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft
> modes on
> > > > > > > Intel
> > > > > > > > > and ARM
> > > > > > > > > >> > > > machines.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Question:
> > > > > > > > > >> > > > * I went through
> > > > > https://kafka.apache.org/35/protocol.html
> > > > > > > and
> > > > > > > > > there
> > > > > > > > > >> > are
> > > > > > > > > >> > > > quite a few repetitive __tagged_fileds fields
> within the
> > > > > same
> > > > > > > > > >> > structures -
> > > > > > > > > >> > > > is this expected?
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > > > > <jo...@aiven.io.invalid>
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Hi Mickael,
> > > > > > > > > >> > > > > I just wanted to point out that I think the
> > > > > documentation
> > > > > > > you
> > > > > > > > > >> > recently
> > > > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0
> version has a
> > > > > > > problem
> > > > > > > > > when
> > > > > > > > > >> > it
> > > > > > > > > >> > > > > states the version number and the sub-menu that
> links to
> > > > > > > > > previous
> > > > > > > > > >> > versions.
> > > > > > > > > >> > > > > Left a comment here:
> > > > > > > > > >> > > > >
> > > > > > > > > >> >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Best,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > > > > > > josep.prat@aiven.io
> > > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > Hi Mickael,
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > I can +1 this candidate. I verified the
> following:
> > > > > > > > > >> > > > > > - Built from source with Java 17 and Scala
> 2.13
> > > > > > > > > >> > > > > > - Signatures and hashes of the artifacts
> generated
> > > > > > > > > >> > > > > > - Navigated through Javadoc including links
> to JDK
> > > > > classes
> > > > > > > > > >> > > > > > - Run the unit tests
> > > > > > > > > >> > > > > > - Run integration tests
> > > > > > > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper
> mode
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Best,
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael
> Maison <
> > > > > > > > > >> > mimaison@apache.org>
> > > > > > > > > >> > > > > > wrote:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >> Hello Kafka users, developers and
> client-developers,
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> This is the first candidate for release of
> Apache
> > > > > Kafka
> > > > > > > > > 3.5.0.
> > > > > > > > > >> > Some of
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > >> major features include:
> > > > > > > > > >> > > > > >> - KIP-710: Full support for distributed mode
> in
> > > > > dedicated
> > > > > > > > > >> > MirrorMaker
> > > > > > > > > >> > > > > >> 2.0 clusters
> > > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment
> for Kafka
> > > > > > > > > Consumers
> > > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of
> > > > > environment
> > > > > > > > > variables
> > > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for
> syncing
> > > > > topic
> > > > > > > > > >> > configurations
> > > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API
> additions to
> > > > > > > support
> > > > > > > > > SCRAM
> > > > > > > > > >> > for
> > > > > > > > > >> > > > > >> Kafka Brokers
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> >
> > > > > > > >
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> *** Please download, test and vote by
> Friday, May
> > > > > 26, 5pm
> > > > > > > > PT
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys we use
> to sign
> > > > > the
> > > > > > > > > release:
> > > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> * Release artifacts to be voted upon (source
> and
> > > > > binary):
> > > > > > > > > >> > > > > >>
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> >
> > > > > > > > >
> > > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> * Javadoc:
> > > > > > > > > >> > > > > >>
> > > > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is
> the 3.5.0
> > > > > tag:
> > > > > > > > > >> > > > > >>
> > > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> The PR adding the 35 documentation is not
> merged yet
> > > > > > > > > >> > > > > >> (
> https://github.com/apache/kafka-site/pull/513)
> > > > > > > > > >> > > > > >> * Documentation:
> > > > > > > > > >> > > > > >>
> https://kafka.apache.org/35/documentation.html
> > > > > > > > > >> > > > > >> * Protocol:
> > > > > > > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> * Successful Jenkins builds for the 3.5
> branch:
> > > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is not
> detecting the
> > > > > 3.5
> > > > > > > > > branch,
> > > > > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > > > > >> > > > > >>
> https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > > > > >> > > > > >> System tests: The build is still running,
> I'll send
> > > > > an
> > > > > > > > update
> > > > > > > > > >> > once I
> > > > > > > > > >> > > > > >> have the results
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >> Thanks,
> > > > > > > > > >> > > > > >> Mickael
> > > > > > > > > >> > > > > >>
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > --
> > > > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > *Josep Prat*
> > > > > > > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > > > > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu
> Valtonen
> > > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > --
> > > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > *Josep Prat*
> > > > > > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu
> Valtonen
> > > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > >> > > > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> -David
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > >
> > > > > > > *Josep Prat*
> > > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > https://www.facebook.com/aivencloud
> > > > > > > >
> > > > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > https://twitter.com/aiven_io>
> > > > > > > *Aiven Deutschland GmbH*
> > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -David
>


-- 
-David

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi,

The issue mentioned by Greg has been fixed. As soon as the fix for
KAFKA-15010 is merged I'll build another RC.

Thanks,
Mickael

On Tue, May 30, 2023 at 10:33 AM Mickael Maison
<mi...@gmail.com> wrote:
>
> Hi David,
>
> Feel free to backport the necessary fixes to 3.5.
>
> Thanks,
> Mickael
>
> On Tue, May 30, 2023 at 10:32 AM Mickael Maison
> <mi...@gmail.com> wrote:
> >
> > Hi Greg,
> >
> > Thanks for the heads up, this indeed looks like something we want in
> > 3.5. I've replied in the PR.
> >
> > Mickael
> >
> > On Sat, May 27, 2023 at 11:44 PM David Arthur
> > <da...@confluent.io.invalid> wrote:
> > >
> > > Mickael, after looking more closely, I definitely think KAFKA-15010 is a
> > > blocker. It creates the case where the controller can totally miss a
> > > metadata update and not write it back to ZK. Since things like dynamic
> > > configs and ACLs are only read from ZK by the ZK brokers, we could have
> > > significant problems while the brokers are being migrated (when some are
> > > KRaft and some are ZK). E.g., ZK brokers could be totally unaware of an ACL
> > > change while the KRaft brokers have it. I have a fix ready here
> > > https://github.com/apache/kafka/pull/13758. I think we can get it committed
> > > soon.
> > >
> > > Another blocker is KAFKA-15004 which was just merged to trunk. This is
> > > another dual-write bug where new topic/broker configs will not be written
> > > back to ZK by the controller.
> > >
> > > The fix for KAFKA-15010 has a few dependencies on fixes we made this past
> > > week, so we'll need to cherry-pick a few commits. The changes are totally
> > > contained within the migration area of code, so I think the risk in
> > > including them is fairly low.
> > >
> > > -David
> > >
> > > On Thu, May 25, 2023 at 2:15 PM Greg Harris <gr...@aiven.io.invalid>
> > > wrote:
> > >
> > > > Hey all,
> > > >
> > > > A contributor just pointed out a small but noticeable flaw in the
> > > > implementation of KIP-581
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > > which is planned for this release.
> > > > Impact: the feature works for root values in a record, but does not
> > > > work for any fields within structs. Fields within structs will
> > > > continue to have their previous, backwards-compatible behavior.
> > > > The contributor has submitted a bug-fix PR which reports the problem
> > > > and does not yet have a merge-able solution, but they are actively
> > > > responding and interested in having this fixed:
> > > > https://github.com/apache/kafka/pull/13748
> > > > The overall fix should be a one-liner + some unit tests. While this is
> > > > not a regression, it does make the feature largely useless, as the
> > > > majority of use-cases will be for struct fields.
> > > >
> > > > Thanks!
> > > > Greg Harris
> > > >
> > > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
> > > > >
> > > > > I agree the migration should be functional - it wasn't obvious if the
> > > > > migration issues are edge cases or not. If they are edge cases, I think
> > > > > 3.5.1 would be fine given the preview status.
> > > > >
> > > > > I understand that a new RC is needed, but that doesn't mean we should let
> > > > > everything in. Each change carries some risk. And if we don't agree on
> > > > the
> > > > > bar for the migration work, we may be having the same discussion next
> > > > week.
> > > > > :)
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, May 24, 2023, 12:00 PM Josep Prat <jo...@aiven.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi there,
> > > > > > Is the plan described in KIP-833[1] still valid? In there it states
> > > > that
> > > > > > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the
> > > > path to
> > > > > > migrate to Kraft should be somewhat functional (in my opinion). If we
> > > > don't
> > > > > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion
> > > > that
> > > > > > these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > > > Best,
> > > > > >
> > > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk> wrote:
> > > > > >
> > > > > > > Are all these blockers? For example, zk to kraft migration are is
> > > > still
> > > > > > in
> > > > > > > preview - can we fix some of these in 3.5.1?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > Thanks for putting together this RC. Unfortunately, we've
> > > > identified
> > > > > > > > several blocker issues in this release candidate.
> > > > > > > >
> > > > > > > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > > > > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > > > > > > migration.
> > > > > > > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > > > > > > migration (dual-write)
> > > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > > > > > > (dual-write) when partitions are changed for topic
> > > > > > > > KAFKA-14996: The KRaft controller should properly handle overly
> > > > large
> > > > > > > user
> > > > > > > > operations
> > > > > > > >
> > > > > > > > We are working on PRs for these issues and will get them in soon,
> > > > we
> > > > > > > think!
> > > > > > > >
> > > > > > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> > > > > > another
> > > > > > > > RC next week.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > > > > Hi David,
> > > > > > > > >
> > > > > > > > > We're already quite a bit behind schedule. If you think these
> > > > fixes
> > > > > > > > > are really important and can be ready in the next couple of
> > > > days, I'm
> > > > > > > > > open to backport them and build another release candidate. Let me
> > > > > > know
> > > > > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > > >>
> > > > > > > > >> Mickael, we have some migration fixes on trunk, is it okay to
> > > > > > > > cherry-pick
> > > > > > > > >> these to 3.5?
> > > > > > > > >>
> > > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > > > MigrationPropagator
> > > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > > > > > > > >>
> > > > > > > > >> There is another issue KAFKA-15010 that I'm also investigating
> > > > to
> > > > > > > > determine
> > > > > > > > >> the impact and likelihood of seeing it in practice. This one
> > > > may be
> > > > > > a
> > > > > > > > >> significant migration blocker
> > > > > > > > >>
> > > > > > > > >> Cheers,
> > > > > > > > >> David
> > > > > > > > >>
> > > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Christo,
> > > > > > > > >> >
> > > > > > > > >> > Yes this is expected. This happens when nested fields also
> > > > accept
> > > > > > > > >> > optional tagged fields. The tables list all fields, so they
> > > > may
> > > > > > > > >> > include _tagged_fields multiple times.
> > > > > > > > >> > Clearly the layout of this page could be improved, if you have
> > > > > > ideas
> > > > > > > > >> > how to describe the protocol in a better way, feel free to
> > > > share
> > > > > > > them.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Mickael
> > > > > > > > >> >
> > > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > Hi Josep,
> > > > > > > > >> > >
> > > > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Mickael
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > > > > > christololov@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > Hey Mickael!
> > > > > > > > >> > > >
> > > > > > > > >> > > > I am giving a +1 (non-binding) for this candidate release.
> > > > > > > > >> > > >
> > > > > > > > >> > > > * Built from the binary tar.gz source with Java 17 and
> > > > Scala
> > > > > > > 2.13
> > > > > > > > on
> > > > > > > > >> > Intel
> > > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > > > > >> > > > * Ran unit and integration tests on Intel and ARM
> > > > machines.
> > > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on
> > > > > > Intel
> > > > > > > > and ARM
> > > > > > > > >> > > > machines.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Question:
> > > > > > > > >> > > > * I went through
> > > > https://kafka.apache.org/35/protocol.html
> > > > > > and
> > > > > > > > there
> > > > > > > > >> > are
> > > > > > > > >> > > > quite a few repetitive __tagged_fileds fields within the
> > > > same
> > > > > > > > >> > structures -
> > > > > > > > >> > > > is this expected?
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > > > <jo...@aiven.io.invalid>
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Hi Mickael,
> > > > > > > > >> > > > > I just wanted to point out that I think the
> > > > documentation
> > > > > > you
> > > > > > > > >> > recently
> > > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0 version has a
> > > > > > problem
> > > > > > > > when
> > > > > > > > >> > it
> > > > > > > > >> > > > > states the version number and the sub-menu that links to
> > > > > > > > previous
> > > > > > > > >> > versions.
> > > > > > > > >> > > > > Left a comment here:
> > > > > > > > >> > > > >
> > > > > > > > >> >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Best,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > > > > > josep.prat@aiven.io
> > > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Hi Mickael,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > I can +1 this candidate. I verified the following:
> > > > > > > > >> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > > > >> > > > > > - Signatures and hashes of the artifacts generated
> > > > > > > > >> > > > > > - Navigated through Javadoc including links to JDK
> > > > classes
> > > > > > > > >> > > > > > - Run the unit tests
> > > > > > > > >> > > > > > - Run integration tests
> > > > > > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Best,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > > > > > > > >> > mimaison@apache.org>
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >> Hello Kafka users, developers and client-developers,
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> This is the first candidate for release of Apache
> > > > Kafka
> > > > > > > > 3.5.0.
> > > > > > > > >> > Some of
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > >> major features include:
> > > > > > > > >> > > > > >> - KIP-710: Full support for distributed mode in
> > > > dedicated
> > > > > > > > >> > MirrorMaker
> > > > > > > > >> > > > > >> 2.0 clusters
> > > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> > > > > > > > Consumers
> > > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of
> > > > environment
> > > > > > > > variables
> > > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing
> > > > topic
> > > > > > > > >> > configurations
> > > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to
> > > > > > support
> > > > > > > > SCRAM
> > > > > > > > >> > for
> > > > > > > > >> > > > > >> Kafka Brokers
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > > > >> > > > > >>
> > > > > > > > >> >
> > > > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> *** Please download, test and vote by Friday, May
> > > > 26, 5pm
> > > > > > > PT
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign
> > > > the
> > > > > > > > release:
> > > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> * Release artifacts to be voted upon (source and
> > > > binary):
> > > > > > > > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > > > >> > > > > >>
> > > > > > > > >> >
> > > > > > > >
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> * Javadoc:
> > > > > > > > >> > > > > >>
> > > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0
> > > > tag:
> > > > > > > > >> > > > > >>
> > > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> The PR adding the 35 documentation is not merged yet
> > > > > > > > >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > > > > > >> > > > > >> * Documentation:
> > > > > > > > >> > > > > >> https://kafka.apache.org/35/documentation.html
> > > > > > > > >> > > > > >> * Protocol:
> > > > > > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > > > > > >> > > > > >> Unit/integration tests: Jenkins is not detecting the
> > > > 3.5
> > > > > > > > branch,
> > > > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > > > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > > > >> > > > > >> System tests: The build is still running, I'll send
> > > > an
> > > > > > > update
> > > > > > > > >> > once I
> > > > > > > > >> > > > > >> have the results
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >> Thanks,
> > > > > > > > >> > > > > >> Mickael
> > > > > > > > >> > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > --
> > > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > *Josep Prat*
> > > > > > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > > > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > --
> > > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > *Josep Prat*
> > > > > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > >> > > > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> -David
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > >
> > > > > > *Josep Prat*
> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > aiven.io <https://www.aiven.io>   |   <
> > > > https://www.facebook.com/aivencloud
> > > > > > >
> > > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > https://twitter.com/aiven_io>
> > > > > > *Aiven Deutschland GmbH*
> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > >
> > >
> > >
> > > --
> > > -David

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi David,

Feel free to backport the necessary fixes to 3.5.

Thanks,
Mickael

On Tue, May 30, 2023 at 10:32 AM Mickael Maison
<mi...@gmail.com> wrote:
>
> Hi Greg,
>
> Thanks for the heads up, this indeed looks like something we want in
> 3.5. I've replied in the PR.
>
> Mickael
>
> On Sat, May 27, 2023 at 11:44 PM David Arthur
> <da...@confluent.io.invalid> wrote:
> >
> > Mickael, after looking more closely, I definitely think KAFKA-15010 is a
> > blocker. It creates the case where the controller can totally miss a
> > metadata update and not write it back to ZK. Since things like dynamic
> > configs and ACLs are only read from ZK by the ZK brokers, we could have
> > significant problems while the brokers are being migrated (when some are
> > KRaft and some are ZK). E.g., ZK brokers could be totally unaware of an ACL
> > change while the KRaft brokers have it. I have a fix ready here
> > https://github.com/apache/kafka/pull/13758. I think we can get it committed
> > soon.
> >
> > Another blocker is KAFKA-15004 which was just merged to trunk. This is
> > another dual-write bug where new topic/broker configs will not be written
> > back to ZK by the controller.
> >
> > The fix for KAFKA-15010 has a few dependencies on fixes we made this past
> > week, so we'll need to cherry-pick a few commits. The changes are totally
> > contained within the migration area of code, so I think the risk in
> > including them is fairly low.
> >
> > -David
> >
> > On Thu, May 25, 2023 at 2:15 PM Greg Harris <gr...@aiven.io.invalid>
> > wrote:
> >
> > > Hey all,
> > >
> > > A contributor just pointed out a small but noticeable flaw in the
> > > implementation of KIP-581
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > which is planned for this release.
> > > Impact: the feature works for root values in a record, but does not
> > > work for any fields within structs. Fields within structs will
> > > continue to have their previous, backwards-compatible behavior.
> > > The contributor has submitted a bug-fix PR which reports the problem
> > > and does not yet have a merge-able solution, but they are actively
> > > responding and interested in having this fixed:
> > > https://github.com/apache/kafka/pull/13748
> > > The overall fix should be a one-liner + some unit tests. While this is
> > > not a regression, it does make the feature largely useless, as the
> > > majority of use-cases will be for struct fields.
> > >
> > > Thanks!
> > > Greg Harris
> > >
> > > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
> > > >
> > > > I agree the migration should be functional - it wasn't obvious if the
> > > > migration issues are edge cases or not. If they are edge cases, I think
> > > > 3.5.1 would be fine given the preview status.
> > > >
> > > > I understand that a new RC is needed, but that doesn't mean we should let
> > > > everything in. Each change carries some risk. And if we don't agree on
> > > the
> > > > bar for the migration work, we may be having the same discussion next
> > > week.
> > > > :)
> > > >
> > > > Ismael
> > > >
> > > > On Wed, May 24, 2023, 12:00 PM Josep Prat <jo...@aiven.io.invalid>
> > > > wrote:
> > > >
> > > > > Hi there,
> > > > > Is the plan described in KIP-833[1] still valid? In there it states
> > > that
> > > > > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the
> > > path to
> > > > > migrate to Kraft should be somewhat functional (in my opinion). If we
> > > don't
> > > > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion
> > > that
> > > > > these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > > Best,
> > > > >
> > > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk> wrote:
> > > > >
> > > > > > Are all these blockers? For example, zk to kraft migration are is
> > > still
> > > > > in
> > > > > > preview - can we fix some of these in 3.5.1?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > Thanks for putting together this RC. Unfortunately, we've
> > > identified
> > > > > > > several blocker issues in this release candidate.
> > > > > > >
> > > > > > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > > > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > > > > > migration.
> > > > > > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > > > > > migration (dual-write)
> > > > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > > > > > (dual-write) when partitions are changed for topic
> > > > > > > KAFKA-14996: The KRaft controller should properly handle overly
> > > large
> > > > > > user
> > > > > > > operations
> > > > > > >
> > > > > > > We are working on PRs for these issues and will get them in soon,
> > > we
> > > > > > think!
> > > > > > >
> > > > > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> > > > > another
> > > > > > > RC next week.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > > > Hi David,
> > > > > > > >
> > > > > > > > We're already quite a bit behind schedule. If you think these
> > > fixes
> > > > > > > > are really important and can be ready in the next couple of
> > > days, I'm
> > > > > > > > open to backport them and build another release candidate. Let me
> > > > > know
> > > > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > > >>
> > > > > > > >> Mickael, we have some migration fixes on trunk, is it okay to
> > > > > > > cherry-pick
> > > > > > > >> these to 3.5?
> > > > > > > >>
> > > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > > MigrationPropagator
> > > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > > > > > > >>
> > > > > > > >> There is another issue KAFKA-15010 that I'm also investigating
> > > to
> > > > > > > determine
> > > > > > > >> the impact and likelihood of seeing it in practice. This one
> > > may be
> > > > > a
> > > > > > > >> significant migration blocker
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >> David
> > > > > > > >>
> > > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Christo,
> > > > > > > >> >
> > > > > > > >> > Yes this is expected. This happens when nested fields also
> > > accept
> > > > > > > >> > optional tagged fields. The tables list all fields, so they
> > > may
> > > > > > > >> > include _tagged_fields multiple times.
> > > > > > > >> > Clearly the layout of this page could be improved, if you have
> > > > > ideas
> > > > > > > >> > how to describe the protocol in a better way, feel free to
> > > share
> > > > > > them.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Mickael
> > > > > > > >> >
> > > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > > Hi Josep,
> > > > > > > >> > >
> > > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Mickael
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > > > > christololov@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > Hey Mickael!
> > > > > > > >> > > >
> > > > > > > >> > > > I am giving a +1 (non-binding) for this candidate release.
> > > > > > > >> > > >
> > > > > > > >> > > > * Built from the binary tar.gz source with Java 17 and
> > > Scala
> > > > > > 2.13
> > > > > > > on
> > > > > > > >> > Intel
> > > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > > > >> > > > * Ran unit and integration tests on Intel and ARM
> > > machines.
> > > > > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on
> > > > > Intel
> > > > > > > and ARM
> > > > > > > >> > > > machines.
> > > > > > > >> > > >
> > > > > > > >> > > > Question:
> > > > > > > >> > > > * I went through
> > > https://kafka.apache.org/35/protocol.html
> > > > > and
> > > > > > > there
> > > > > > > >> > are
> > > > > > > >> > > > quite a few repetitive __tagged_fileds fields within the
> > > same
> > > > > > > >> > structures -
> > > > > > > >> > > > is this expected?
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > > <jo...@aiven.io.invalid>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi Mickael,
> > > > > > > >> > > > > I just wanted to point out that I think the
> > > documentation
> > > > > you
> > > > > > > >> > recently
> > > > > > > >> > > > > merged on Kafka site regarding the 3.5.0 version has a
> > > > > problem
> > > > > > > when
> > > > > > > >> > it
> > > > > > > >> > > > > states the version number and the sub-menu that links to
> > > > > > > previous
> > > > > > > >> > versions.
> > > > > > > >> > > > > Left a comment here:
> > > > > > > >> > > > >
> > > > > > > >> >
> > > > > > >
> > > > > >
> > > > >
> > > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > > >> > > > >
> > > > > > > >> > > > > Best,
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > > > > josep.prat@aiven.io
> > > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hi Mickael,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I can +1 this candidate. I verified the following:
> > > > > > > >> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > > >> > > > > > - Signatures and hashes of the artifacts generated
> > > > > > > >> > > > > > - Navigated through Javadoc including links to JDK
> > > classes
> > > > > > > >> > > > > > - Run the unit tests
> > > > > > > >> > > > > > - Run integration tests
> > > > > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Best,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > > > > > > >> > mimaison@apache.org>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> Hello Kafka users, developers and client-developers,
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> This is the first candidate for release of Apache
> > > Kafka
> > > > > > > 3.5.0.
> > > > > > > >> > Some of
> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> major features include:
> > > > > > > >> > > > > >> - KIP-710: Full support for distributed mode in
> > > dedicated
> > > > > > > >> > MirrorMaker
> > > > > > > >> > > > > >> 2.0 clusters
> > > > > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> > > > > > > Consumers
> > > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of
> > > environment
> > > > > > > variables
> > > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing
> > > topic
> > > > > > > >> > configurations
> > > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to
> > > > > support
> > > > > > > SCRAM
> > > > > > > >> > for
> > > > > > > >> > > > > >> Kafka Brokers
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > > >> > > > > >>
> > > > > > > >> >
> > > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> *** Please download, test and vote by Friday, May
> > > 26, 5pm
> > > > > > PT
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign
> > > the
> > > > > > > release:
> > > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> * Release artifacts to be voted upon (source and
> > > binary):
> > > > > > > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > > >> > > > > >>
> > > > > > > >> >
> > > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> * Javadoc:
> > > > > > > >> > > > > >>
> > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0
> > > tag:
> > > > > > > >> > > > > >>
> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> The PR adding the 35 documentation is not merged yet
> > > > > > > >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > > > > >> > > > > >> * Documentation:
> > > > > > > >> > > > > >> https://kafka.apache.org/35/documentation.html
> > > > > > > >> > > > > >> * Protocol:
> > > > > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > > > > >> > > > > >> Unit/integration tests: Jenkins is not detecting the
> > > 3.5
> > > > > > > branch,
> > > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > > >> > > > > >> System tests: The build is still running, I'll send
> > > an
> > > > > > update
> > > > > > > >> > once I
> > > > > > > >> > > > > >> have the results
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> Thanks,
> > > > > > > >> > > > > >> Mickael
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > --
> > > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > *Josep Prat*
> > > > > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > --
> > > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > > >> > > > >
> > > > > > > >> > > > > *Josep Prat*
> > > > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > > >> > > > > >
> > > > > > > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >> > > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> -David
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > [image: Aiven] <https://www.aiven.io>
> > > > >
> > > > > *Josep Prat*
> > > > > Open Source Engineering Director, *Aiven*
> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > aiven.io <https://www.aiven.io>   |   <
> > > https://www.facebook.com/aivencloud
> > > > > >
> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > https://twitter.com/aiven_io>
> > > > > *Aiven Deutschland GmbH*
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > >
> >
> >
> > --
> > -David

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi Greg,

Thanks for the heads up, this indeed looks like something we want in
3.5. I've replied in the PR.

Mickael

On Sat, May 27, 2023 at 11:44 PM David Arthur
<da...@confluent.io.invalid> wrote:
>
> Mickael, after looking more closely, I definitely think KAFKA-15010 is a
> blocker. It creates the case where the controller can totally miss a
> metadata update and not write it back to ZK. Since things like dynamic
> configs and ACLs are only read from ZK by the ZK brokers, we could have
> significant problems while the brokers are being migrated (when some are
> KRaft and some are ZK). E.g., ZK brokers could be totally unaware of an ACL
> change while the KRaft brokers have it. I have a fix ready here
> https://github.com/apache/kafka/pull/13758. I think we can get it committed
> soon.
>
> Another blocker is KAFKA-15004 which was just merged to trunk. This is
> another dual-write bug where new topic/broker configs will not be written
> back to ZK by the controller.
>
> The fix for KAFKA-15010 has a few dependencies on fixes we made this past
> week, so we'll need to cherry-pick a few commits. The changes are totally
> contained within the migration area of code, so I think the risk in
> including them is fairly low.
>
> -David
>
> On Thu, May 25, 2023 at 2:15 PM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hey all,
> >
> > A contributor just pointed out a small but noticeable flaw in the
> > implementation of KIP-581
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > which is planned for this release.
> > Impact: the feature works for root values in a record, but does not
> > work for any fields within structs. Fields within structs will
> > continue to have their previous, backwards-compatible behavior.
> > The contributor has submitted a bug-fix PR which reports the problem
> > and does not yet have a merge-able solution, but they are actively
> > responding and interested in having this fixed:
> > https://github.com/apache/kafka/pull/13748
> > The overall fix should be a one-liner + some unit tests. While this is
> > not a regression, it does make the feature largely useless, as the
> > majority of use-cases will be for struct fields.
> >
> > Thanks!
> > Greg Harris
> >
> > On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > I agree the migration should be functional - it wasn't obvious if the
> > > migration issues are edge cases or not. If they are edge cases, I think
> > > 3.5.1 would be fine given the preview status.
> > >
> > > I understand that a new RC is needed, but that doesn't mean we should let
> > > everything in. Each change carries some risk. And if we don't agree on
> > the
> > > bar for the migration work, we may be having the same discussion next
> > week.
> > > :)
> > >
> > > Ismael
> > >
> > > On Wed, May 24, 2023, 12:00 PM Josep Prat <jo...@aiven.io.invalid>
> > > wrote:
> > >
> > > > Hi there,
> > > > Is the plan described in KIP-833[1] still valid? In there it states
> > that
> > > > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the
> > path to
> > > > migrate to Kraft should be somewhat functional (in my opinion). If we
> > don't
> > > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion
> > that
> > > > these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
> > > >
> > > > [1]:
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > > Best,
> > > >
> > > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk> wrote:
> > > >
> > > > > Are all these blockers? For example, zk to kraft migration are is
> > still
> > > > in
> > > > > preview - can we fix some of these in 3.5.1?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Thanks for putting together this RC. Unfortunately, we've
> > identified
> > > > > > several blocker issues in this release candidate.
> > > > > >
> > > > > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > > > > migration.
> > > > > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > > > > migration (dual-write)
> > > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > > > > (dual-write) when partitions are changed for topic
> > > > > > KAFKA-14996: The KRaft controller should properly handle overly
> > large
> > > > > user
> > > > > > operations
> > > > > >
> > > > > > We are working on PRs for these issues and will get them in soon,
> > we
> > > > > think!
> > > > > >
> > > > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> > > > another
> > > > > > RC next week.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > > Hi David,
> > > > > > >
> > > > > > > We're already quite a bit behind schedule. If you think these
> > fixes
> > > > > > > are really important and can be ready in the next couple of
> > days, I'm
> > > > > > > open to backport them and build another release candidate. Let me
> > > > know
> > > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > > <da...@confluent.io.invalid> wrote:
> > > > > > >>
> > > > > > >> Mickael, we have some migration fixes on trunk, is it okay to
> > > > > > cherry-pick
> > > > > > >> these to 3.5?
> > > > > > >>
> > > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> > MigrationPropagator
> > > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > > > > > >>
> > > > > > >> There is another issue KAFKA-15010 that I'm also investigating
> > to
> > > > > > determine
> > > > > > >> the impact and likelihood of seeing it in practice. This one
> > may be
> > > > a
> > > > > > >> significant migration blocker
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >> David
> > > > > > >>
> > > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Christo,
> > > > > > >> >
> > > > > > >> > Yes this is expected. This happens when nested fields also
> > accept
> > > > > > >> > optional tagged fields. The tables list all fields, so they
> > may
> > > > > > >> > include _tagged_fields multiple times.
> > > > > > >> > Clearly the layout of this page could be improved, if you have
> > > > ideas
> > > > > > >> > how to describe the protocol in a better way, feel free to
> > share
> > > > > them.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Mickael
> > > > > > >> >
> > > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > > mickael.maison@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > > Hi Josep,
> > > > > > >> > >
> > > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Mickael
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > > > christololov@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> > > >
> > > > > > >> > > > Hey Mickael!
> > > > > > >> > > >
> > > > > > >> > > > I am giving a +1 (non-binding) for this candidate release.
> > > > > > >> > > >
> > > > > > >> > > > * Built from the binary tar.gz source with Java 17 and
> > Scala
> > > > > 2.13
> > > > > > on
> > > > > > >> > Intel
> > > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > > >> > > > * Ran unit and integration tests on Intel and ARM
> > machines.
> > > > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on
> > > > Intel
> > > > > > and ARM
> > > > > > >> > > > machines.
> > > > > > >> > > >
> > > > > > >> > > > Question:
> > > > > > >> > > > * I went through
> > https://kafka.apache.org/35/protocol.html
> > > > and
> > > > > > there
> > > > > > >> > are
> > > > > > >> > > > quite a few repetitive __tagged_fileds fields within the
> > same
> > > > > > >> > structures -
> > > > > > >> > > > is this expected?
> > > > > > >> > > >
> > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > > <jo...@aiven.io.invalid>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Mickael,
> > > > > > >> > > > > I just wanted to point out that I think the
> > documentation
> > > > you
> > > > > > >> > recently
> > > > > > >> > > > > merged on Kafka site regarding the 3.5.0 version has a
> > > > problem
> > > > > > when
> > > > > > >> > it
> > > > > > >> > > > > states the version number and the sub-menu that links to
> > > > > > previous
> > > > > > >> > versions.
> > > > > > >> > > > > Left a comment here:
> > > > > > >> > > > >
> > > > > > >> >
> > > > > >
> > > > >
> > > >
> > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > > >> > > > >
> > > > > > >> > > > > Best,
> > > > > > >> > > > >
> > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > > > josep.prat@aiven.io
> > > > > > >
> > > > > > >> > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi Mickael,
> > > > > > >> > > > > >
> > > > > > >> > > > > > I can +1 this candidate. I verified the following:
> > > > > > >> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > >> > > > > > - Signatures and hashes of the artifacts generated
> > > > > > >> > > > > > - Navigated through Javadoc including links to JDK
> > classes
> > > > > > >> > > > > > - Run the unit tests
> > > > > > >> > > > > > - Run integration tests
> > > > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > > > >> > > > > >
> > > > > > >> > > > > > Best,
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > > > > > >> > mimaison@apache.org>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > >> Hello Kafka users, developers and client-developers,
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> This is the first candidate for release of Apache
> > Kafka
> > > > > > 3.5.0.
> > > > > > >> > Some of
> > > > > > >> > > > > the
> > > > > > >> > > > > >> major features include:
> > > > > > >> > > > > >> - KIP-710: Full support for distributed mode in
> > dedicated
> > > > > > >> > MirrorMaker
> > > > > > >> > > > > >> 2.0 clusters
> > > > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> > > > > > Consumers
> > > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of
> > environment
> > > > > > variables
> > > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing
> > topic
> > > > > > >> > configurations
> > > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to
> > > > support
> > > > > > SCRAM
> > > > > > >> > for
> > > > > > >> > > > > >> Kafka Brokers
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > > >> > > > > >>
> > > > > > >> >
> > > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> *** Please download, test and vote by Friday, May
> > 26, 5pm
> > > > > PT
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign
> > the
> > > > > > release:
> > > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> * Release artifacts to be voted upon (source and
> > binary):
> > > > > > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > > >> > > > > >>
> > > > > > >> >
> > > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> * Javadoc:
> > > > > > >> > > > > >>
> > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0
> > tag:
> > > > > > >> > > > > >>
> > https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> The PR adding the 35 documentation is not merged yet
> > > > > > >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > > > >> > > > > >> * Documentation:
> > > > > > >> > > > > >> https://kafka.apache.org/35/documentation.html
> > > > > > >> > > > > >> * Protocol:
> > > > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > > > >> > > > > >> Unit/integration tests: Jenkins is not detecting the
> > 3.5
> > > > > > branch,
> > > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > > > >> > > > > >> System tests: The build is still running, I'll send
> > an
> > > > > update
> > > > > > >> > once I
> > > > > > >> > > > > >> have the results
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> Thanks,
> > > > > > >> > > > > >> Mickael
> > > > > > >> > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > --
> > > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > >> > > > > >
> > > > > > >> > > > > > *Josep Prat*
> > > > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > --
> > > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > > >> > > > >
> > > > > > >> > > > > *Josep Prat*
> > > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > > >> > https://www.facebook.com/aivencloud
> > > > > > >> > > > > >
> > > > > > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >> > > > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> -David
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > [image: Aiven] <https://www.aiven.io>
> > > >
> > > > *Josep Prat*
> > > > Open Source Engineering Director, *Aiven*
> > > > josep.prat@aiven.io   |   +491715557497
> > > > aiven.io <https://www.aiven.io>   |   <
> > https://www.facebook.com/aivencloud
> > > > >
> > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > https://twitter.com/aiven_io>
> > > > *Aiven Deutschland GmbH*
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> >
>
>
> --
> -David

Re: [VOTE] 3.5.0 RC0

Posted by David Arthur <da...@confluent.io.INVALID>.

Mickael, after looking more closely, I definitely think KAFKA-15010 is a
blocker. It creates the case where the controller can totally miss a
metadata update and not write it back to ZK. Since things like dynamic
configs and ACLs are only read from ZK by the ZK brokers, we could have
significant problems while the brokers are being migrated (when some are
KRaft and some are ZK). E.g., ZK brokers could be totally unaware of an ACL
change while the KRaft brokers have it. I have a fix ready here
https://github.com/apache/kafka/pull/13758. I think we can get it committed
soon.

Another blocker is KAFKA-15004 which was just merged to trunk. This is
another dual-write bug where new topic/broker configs will not be written
back to ZK by the controller.

The fix for KAFKA-15010 has a few dependencies on fixes we made this past
week, so we'll need to cherry-pick a few commits. The changes are totally
contained within the migration area of code, so I think the risk in
including them is fairly low.

-David

On Thu, May 25, 2023 at 2:15 PM Greg Harris <gr...@aiven.io.invalid>
wrote:

> Hey all,
>
> A contributor just pointed out a small but noticeable flaw in the
> implementation of KIP-581
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> which is planned for this release.
> Impact: the feature works for root values in a record, but does not
> work for any fields within structs. Fields within structs will
> continue to have their previous, backwards-compatible behavior.
> The contributor has submitted a bug-fix PR which reports the problem
> and does not yet have a merge-able solution, but they are actively
> responding and interested in having this fixed:
> https://github.com/apache/kafka/pull/13748
> The overall fix should be a one-liner + some unit tests. While this is
> not a regression, it does make the feature largely useless, as the
> majority of use-cases will be for struct fields.
>
> Thanks!
> Greg Harris
>
> On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > I agree the migration should be functional - it wasn't obvious if the
> > migration issues are edge cases or not. If they are edge cases, I think
> > 3.5.1 would be fine given the preview status.
> >
> > I understand that a new RC is needed, but that doesn't mean we should let
> > everything in. Each change carries some risk. And if we don't agree on
> the
> > bar for the migration work, we may be having the same discussion next
> week.
> > :)
> >
> > Ismael
> >
> > On Wed, May 24, 2023, 12:00 PM Josep Prat <jo...@aiven.io.invalid>
> > wrote:
> >
> > > Hi there,
> > > Is the plan described in KIP-833[1] still valid? In there it states
> that
> > > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the
> path to
> > > migrate to Kraft should be somewhat functional (in my opinion). If we
> don't
> > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion
> that
> > > these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
> > >
> > > [1]:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > Best,
> > >
> > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Are all these blockers? For example, zk to kraft migration are is
> still
> > > in
> > > > preview - can we fix some of these in 3.5.1?
> > > >
> > > > Ismael
> > > >
> > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for putting together this RC. Unfortunately, we've
> identified
> > > > > several blocker issues in this release candidate.
> > > > >
> > > > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > > > migration.
> > > > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > > > migration (dual-write)
> > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > > > (dual-write) when partitions are changed for topic
> > > > > KAFKA-14996: The KRaft controller should properly handle overly
> large
> > > > user
> > > > > operations
> > > > >
> > > > > We are working on PRs for these issues and will get them in soon,
> we
> > > > think!
> > > > >
> > > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> > > another
> > > > > RC next week.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > Hi David,
> > > > > >
> > > > > > We're already quite a bit behind schedule. If you think these
> fixes
> > > > > > are really important and can be ready in the next couple of
> days, I'm
> > > > > > open to backport them and build another release candidate. Let me
> > > know
> > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > >
> > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > > <da...@confluent.io.invalid> wrote:
> > > > > >>
> > > > > >> Mickael, we have some migration fixes on trunk, is it okay to
> > > > > cherry-pick
> > > > > >> these to 3.5?
> > > > > >>
> > > > > >> KAFKA-15007 Use the correct MetadataVersion in
> MigrationPropagator
> > > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > > > > >>
> > > > > >> There is another issue KAFKA-15010 that I'm also investigating
> to
> > > > > determine
> > > > > >> the impact and likelihood of seeing it in practice. This one
> may be
> > > a
> > > > > >> significant migration blocker
> > > > > >>
> > > > > >> Cheers,
> > > > > >> David
> > > > > >>
> > > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Christo,
> > > > > >> >
> > > > > >> > Yes this is expected. This happens when nested fields also
> accept
> > > > > >> > optional tagged fields. The tables list all fields, so they
> may
> > > > > >> > include _tagged_fields multiple times.
> > > > > >> > Clearly the layout of this page could be improved, if you have
> > > ideas
> > > > > >> > how to describe the protocol in a better way, feel free to
> share
> > > > them.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Mickael
> > > > > >> >
> > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > > Hi Josep,
> > > > > >> > >
> > > > > >> > > Good catch! I opened a PR to fix this:
> > > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Mickael
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > > christololov@gmail.com>
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > Hey Mickael!
> > > > > >> > > >
> > > > > >> > > > I am giving a +1 (non-binding) for this candidate release.
> > > > > >> > > >
> > > > > >> > > > * Built from the binary tar.gz source with Java 17 and
> Scala
> > > > 2.13
> > > > > on
> > > > > >> > Intel
> > > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > > >> > > > * Ran unit and integration tests on Intel and ARM
> machines.
> > > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on
> > > Intel
> > > > > and ARM
> > > > > >> > > > machines.
> > > > > >> > > >
> > > > > >> > > > Question:
> > > > > >> > > > * I went through
> https://kafka.apache.org/35/protocol.html
> > > and
> > > > > there
> > > > > >> > are
> > > > > >> > > > quite a few repetitive __tagged_fileds fields within the
> same
> > > > > >> > structures -
> > > > > >> > > > is this expected?
> > > > > >> > > >
> > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > > <jo...@aiven.io.invalid>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Mickael,
> > > > > >> > > > > I just wanted to point out that I think the
> documentation
> > > you
> > > > > >> > recently
> > > > > >> > > > > merged on Kafka site regarding the 3.5.0 version has a
> > > problem
> > > > > when
> > > > > >> > it
> > > > > >> > > > > states the version number and the sub-menu that links to
> > > > > previous
> > > > > >> > versions.
> > > > > >> > > > > Left a comment here:
> > > > > >> > > > >
> > > > > >> >
> > > > >
> > > >
> > >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > > >> > > > >
> > > > > >> > > > > Best,
> > > > > >> > > > >
> > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > > josep.prat@aiven.io
> > > > > >
> > > > > >> > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Mickael,
> > > > > >> > > > > >
> > > > > >> > > > > > I can +1 this candidate. I verified the following:
> > > > > >> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > >> > > > > > - Signatures and hashes of the artifacts generated
> > > > > >> > > > > > - Navigated through Javadoc including links to JDK
> classes
> > > > > >> > > > > > - Run the unit tests
> > > > > >> > > > > > - Run integration tests
> > > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > > >> > > > > >
> > > > > >> > > > > > Best,
> > > > > >> > > > > >
> > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > > > > >> > mimaison@apache.org>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > >> Hello Kafka users, developers and client-developers,
> > > > > >> > > > > >>
> > > > > >> > > > > >> This is the first candidate for release of Apache
> Kafka
> > > > > 3.5.0.
> > > > > >> > Some of
> > > > > >> > > > > the
> > > > > >> > > > > >> major features include:
> > > > > >> > > > > >> - KIP-710: Full support for distributed mode in
> dedicated
> > > > > >> > MirrorMaker
> > > > > >> > > > > >> 2.0 clusters
> > > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> > > > > Consumers
> > > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of
> environment
> > > > > variables
> > > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing
> topic
> > > > > >> > configurations
> > > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to
> > > support
> > > > > SCRAM
> > > > > >> > for
> > > > > >> > > > > >> Kafka Brokers
> > > > > >> > > > > >>
> > > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > > >> > > > > >>
> > > > > >> >
> > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > >> > > > > >>
> > > > > >> > > > > >> *** Please download, test and vote by Friday, May
> 26, 5pm
> > > > PT
> > > > > >> > > > > >>
> > > > > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign
> the
> > > > > release:
> > > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > > >> > > > > >>
> > > > > >> > > > > >> * Release artifacts to be voted upon (source and
> binary):
> > > > > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > >> > > > > >>
> > > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > > >> > > > > >>
> > > > > >> >
> > > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >> > > > > >>
> > > > > >> > > > > >> * Javadoc:
> > > > > >> > > > > >>
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > >> > > > > >>
> > > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0
> tag:
> > > > > >> > > > > >>
> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > >> > > > > >>
> > > > > >> > > > > >> The PR adding the 35 documentation is not merged yet
> > > > > >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > > >> > > > > >> * Documentation:
> > > > > >> > > > > >> https://kafka.apache.org/35/documentation.html
> > > > > >> > > > > >> * Protocol:
> > > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > >> > > > > >>
> > > > > >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > > >> > > > > >> Unit/integration tests: Jenkins is not detecting the
> 3.5
> > > > > branch,
> > > > > >> > > > > >> working with INFRA to sort it out:
> > > > > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > > >> > > > > >> System tests: The build is still running, I'll send
> an
> > > > update
> > > > > >> > once I
> > > > > >> > > > > >> have the results
> > > > > >> > > > > >>
> > > > > >> > > > > >> Thanks,
> > > > > >> > > > > >> Mickael
> > > > > >> > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > --
> > > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > >> > > > > >
> > > > > >> > > > > > *Josep Prat*
> > > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > --
> > > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > > >> > > > >
> > > > > >> > > > > *Josep Prat*
> > > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > > >> > https://www.facebook.com/aivencloud
> > > > > >> > > > > >
> > > > > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > >> > > > > https://twitter.com/aiven_io>
> > > > > >> > > > > *Aiven Deutschland GmbH*
> > > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >> > > > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> -David
> > > > >
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] <https://www.aiven.io>
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.prat@aiven.io   |   +491715557497
> > > aiven.io <https://www.aiven.io>   |   <
> https://www.facebook.com/aivencloud
> > > >
> > >   <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
>


-- 
-David

Re: [VOTE] 3.5.0 RC0

Posted by Greg Harris <gr...@aiven.io.INVALID>.

Hey all,

A contributor just pointed out a small but noticeable flaw in the
implementation of KIP-581
https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
which is planned for this release.
Impact: the feature works for root values in a record, but does not
work for any fields within structs. Fields within structs will
continue to have their previous, backwards-compatible behavior.
The contributor has submitted a bug-fix PR which reports the problem
and does not yet have a merge-able solution, but they are actively
responding and interested in having this fixed:
https://github.com/apache/kafka/pull/13748
The overall fix should be a one-liner + some unit tests. While this is
not a regression, it does make the feature largely useless, as the
majority of use-cases will be for struct fields.

Thanks!
Greg Harris

On Wed, May 24, 2023 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> I agree the migration should be functional - it wasn't obvious if the
> migration issues are edge cases or not. If they are edge cases, I think
> 3.5.1 would be fine given the preview status.
>
> I understand that a new RC is needed, but that doesn't mean we should let
> everything in. Each change carries some risk. And if we don't agree on the
> bar for the migration work, we may be having the same discussion next week.
> :)
>
> Ismael
>
> On Wed, May 24, 2023, 12:00 PM Josep Prat <jo...@aiven.io.invalid>
> wrote:
>
> > Hi there,
> > Is the plan described in KIP-833[1] still valid? In there it states that
> > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the path to
> > migrate to Kraft should be somewhat functional (in my opinion). If we don't
> > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion that
> > these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
> >
> > [1]:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > Best,
> >
> > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Are all these blockers? For example, zk to kraft migration are is still
> > in
> > > preview - can we fix some of these in 3.5.1?
> > >
> > > Ismael
> > >
> > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for putting together this RC. Unfortunately, we've identified
> > > > several blocker issues in this release candidate.
> > > >
> > > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > > migration.
> > > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > > migration (dual-write)
> > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > > (dual-write) when partitions are changed for topic
> > > > KAFKA-14996: The KRaft controller should properly handle overly large
> > > user
> > > > operations
> > > >
> > > > We are working on PRs for these issues and will get them in soon, we
> > > think!
> > > >
> > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> > another
> > > > RC next week.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > Hi David,
> > > > >
> > > > > We're already quite a bit behind schedule. If you think these fixes
> > > > > are really important and can be ready in the next couple of days, I'm
> > > > > open to backport them and build another release candidate. Let me
> > know
> > > > > once you've investigated the severity of KAFKA-15010.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > > <da...@confluent.io.invalid> wrote:
> > > > >>
> > > > >> Mickael, we have some migration fixes on trunk, is it okay to
> > > > cherry-pick
> > > > >> these to 3.5?
> > > > >>
> > > > >> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
> > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > > > >>
> > > > >> There is another issue KAFKA-15010 that I'm also investigating to
> > > > determine
> > > > >> the impact and likelihood of seeing it in practice. This one may be
> > a
> > > > >> significant migration blocker
> > > > >>
> > > > >> Cheers,
> > > > >> David
> > > > >>
> > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Christo,
> > > > >> >
> > > > >> > Yes this is expected. This happens when nested fields also accept
> > > > >> > optional tagged fields. The tables list all fields, so they may
> > > > >> > include _tagged_fields multiple times.
> > > > >> > Clearly the layout of this page could be improved, if you have
> > ideas
> > > > >> > how to describe the protocol in a better way, feel free to share
> > > them.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Mickael
> > > > >> >
> > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > Hi Josep,
> > > > >> > >
> > > > >> > > Good catch! I opened a PR to fix this:
> > > > >> > > https://github.com/apache/kafka-site/pull/514
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Mickael
> > > > >> > >
> > > > >> > >
> > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > > christololov@gmail.com>
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > Hey Mickael!
> > > > >> > > >
> > > > >> > > > I am giving a +1 (non-binding) for this candidate release.
> > > > >> > > >
> > > > >> > > > * Built from the binary tar.gz source with Java 17 and Scala
> > > 2.13
> > > > on
> > > > >> > Intel
> > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > >> > > > * Ran unit and integration tests on Intel and ARM machines.
> > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on
> > Intel
> > > > and ARM
> > > > >> > > > machines.
> > > > >> > > >
> > > > >> > > > Question:
> > > > >> > > > * I went through https://kafka.apache.org/35/protocol.html
> > and
> > > > there
> > > > >> > are
> > > > >> > > > quite a few repetitive __tagged_fileds fields within the same
> > > > >> > structures -
> > > > >> > > > is this expected?
> > > > >> > > >
> > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > > <jo...@aiven.io.invalid>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Mickael,
> > > > >> > > > > I just wanted to point out that I think the documentation
> > you
> > > > >> > recently
> > > > >> > > > > merged on Kafka site regarding the 3.5.0 version has a
> > problem
> > > > when
> > > > >> > it
> > > > >> > > > > states the version number and the sub-menu that links to
> > > > previous
> > > > >> > versions.
> > > > >> > > > > Left a comment here:
> > > > >> > > > >
> > > > >> >
> > > >
> > >
> > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > >> > > > >
> > > > >> > > > > Best,
> > > > >> > > > >
> > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > > josep.prat@aiven.io
> > > > >
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Mickael,
> > > > >> > > > > >
> > > > >> > > > > > I can +1 this candidate. I verified the following:
> > > > >> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > >> > > > > > - Signatures and hashes of the artifacts generated
> > > > >> > > > > > - Navigated through Javadoc including links to JDK classes
> > > > >> > > > > > - Run the unit tests
> > > > >> > > > > > - Run integration tests
> > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > >> > > > > >
> > > > >> > > > > > Best,
> > > > >> > > > > >
> > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > > > >> > mimaison@apache.org>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > >> Hello Kafka users, developers and client-developers,
> > > > >> > > > > >>
> > > > >> > > > > >> This is the first candidate for release of Apache Kafka
> > > > 3.5.0.
> > > > >> > Some of
> > > > >> > > > > the
> > > > >> > > > > >> major features include:
> > > > >> > > > > >> - KIP-710: Full support for distributed mode in dedicated
> > > > >> > MirrorMaker
> > > > >> > > > > >> 2.0 clusters
> > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> > > > Consumers
> > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of environment
> > > > variables
> > > > >> > > > > >> - KIP-889: Versioned State Stores
> > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
> > > > >> > configurations
> > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to
> > support
> > > > SCRAM
> > > > >> > for
> > > > >> > > > > >> Kafka Brokers
> > > > >> > > > > >>
> > > > >> > > > > >> Release notes for the 3.5.0 release:
> > > > >> > > > > >>
> > > > >> >
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > >> > > > > >>
> > > > >> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm
> > > PT
> > > > >> > > > > >>
> > > > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the
> > > > release:
> > > > >> > > > > >> https://kafka.apache.org/KEYS
> > > > >> > > > > >>
> > > > >> > > > > >> * Release artifacts to be voted upon (source and binary):
> > > > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > >> > > > > >>
> > > > >> > > > > >> * Maven artifacts to be voted upon:
> > > > >> > > > > >>
> > > > >> >
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >> > > > > >>
> > > > >> > > > > >> * Javadoc:
> > > > >> > > > > >>
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > >> > > > > >>
> > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > > >> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > >> > > > > >>
> > > > >> > > > > >> The PR adding the 35 documentation is not merged yet
> > > > >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > >> > > > > >> * Documentation:
> > > > >> > > > > >> https://kafka.apache.org/35/documentation.html
> > > > >> > > > > >> * Protocol:
> > > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > >> > > > > >>
> > > > >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > >> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5
> > > > branch,
> > > > >> > > > > >> working with INFRA to sort it out:
> > > > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > >> > > > > >> System tests: The build is still running, I'll send an
> > > update
> > > > >> > once I
> > > > >> > > > > >> have the results
> > > > >> > > > > >>
> > > > >> > > > > >> Thanks,
> > > > >> > > > > >> Mickael
> > > > >> > > > > >>
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > --
> > > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > >> > > > > >
> > > > >> > > > > > *Josep Prat*
> > > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > >> > > > > https://twitter.com/aiven_io>
> > > > >> > > > > > *Aiven Deutschland GmbH*
> > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > > >> > > > >
> > > > >> > > > > *Josep Prat*
> > > > >> > > > > Open Source Engineering Director, *Aiven*
> > > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > > >> > https://www.facebook.com/aivencloud
> > > > >> > > > > >
> > > > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > >> > > > > https://twitter.com/aiven_io>
> > > > >> > > > > *Aiven Deutschland GmbH*
> > > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >> > > > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> -David
> > > >
> > >
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.prat@aiven.io   |   +491715557497
> > aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud
> > >
> >   <https://www.linkedin.com/company/aiven/>   <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >

Re: [VOTE] 3.5.0 RC0

Posted by Ismael Juma <is...@juma.me.uk>.

I agree the migration should be functional - it wasn't obvious if the
migration issues are edge cases or not. If they are edge cases, I think
3.5.1 would be fine given the preview status.

I understand that a new RC is needed, but that doesn't mean we should let
everything in. Each change carries some risk. And if we don't agree on the
bar for the migration work, we may be having the same discussion next week.
:)

Ismael

On Wed, May 24, 2023, 12:00 PM Josep Prat <jo...@aiven.io.invalid>
wrote:

> Hi there,
> Is the plan described in KIP-833[1] still valid? In there it states that
> 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the path to
> migrate to Kraft should be somewhat functional (in my opinion). If we don't
> want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion that
> these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
>
> [1]:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> Best,
>
> On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Are all these blockers? For example, zk to kraft migration are is still
> in
> > preview - can we fix some of these in 3.5.1?
> >
> > Ismael
> >
> > On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi Mickael,
> > >
> > > Thanks for putting together this RC. Unfortunately, we've identified
> > > several blocker issues in this release candidate.
> > >
> > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > migration.
> > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > migration (dual-write)
> > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > (dual-write) when partitions are changed for topic
> > > KAFKA-14996: The KRaft controller should properly handle overly large
> > user
> > > operations
> > >
> > > We are working on PRs for these issues and will get them in soon, we
> > think!
> > >
> > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> another
> > > RC next week.
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > Hi David,
> > > >
> > > > We're already quite a bit behind schedule. If you think these fixes
> > > > are really important and can be ready in the next couple of days, I'm
> > > > open to backport them and build another release candidate. Let me
> know
> > > > once you've investigated the severity of KAFKA-15010.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > <da...@confluent.io.invalid> wrote:
> > > >>
> > > >> Mickael, we have some migration fixes on trunk, is it okay to
> > > cherry-pick
> > > >> these to 3.5?
> > > >>
> > > >> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
> > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > > >>
> > > >> There is another issue KAFKA-15010 that I'm also investigating to
> > > determine
> > > >> the impact and likelihood of seeing it in practice. This one may be
> a
> > > >> significant migration blocker
> > > >>
> > > >> Cheers,
> > > >> David
> > > >>
> > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > mickael.maison@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Christo,
> > > >> >
> > > >> > Yes this is expected. This happens when nested fields also accept
> > > >> > optional tagged fields. The tables list all fields, so they may
> > > >> > include _tagged_fields multiple times.
> > > >> > Clearly the layout of this page could be improved, if you have
> ideas
> > > >> > how to describe the protocol in a better way, feel free to share
> > them.
> > > >> >
> > > >> > Thanks,
> > > >> > Mickael
> > > >> >
> > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > > mickael.maison@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > Hi Josep,
> > > >> > >
> > > >> > > Good catch! I opened a PR to fix this:
> > > >> > > https://github.com/apache/kafka-site/pull/514
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Mickael
> > > >> > >
> > > >> > >
> > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > > christololov@gmail.com>
> > > >> > wrote:
> > > >> > > >
> > > >> > > > Hey Mickael!
> > > >> > > >
> > > >> > > > I am giving a +1 (non-binding) for this candidate release.
> > > >> > > >
> > > >> > > > * Built from the binary tar.gz source with Java 17 and Scala
> > 2.13
> > > on
> > > >> > Intel
> > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > >> > > > * Ran unit and integration tests on Intel and ARM machines.
> > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on
> Intel
> > > and ARM
> > > >> > > > machines.
> > > >> > > >
> > > >> > > > Question:
> > > >> > > > * I went through https://kafka.apache.org/35/protocol.html
> and
> > > there
> > > >> > are
> > > >> > > > quite a few repetitive __tagged_fileds fields within the same
> > > >> > structures -
> > > >> > > > is this expected?
> > > >> > > >
> > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > > <jo...@aiven.io.invalid>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Mickael,
> > > >> > > > > I just wanted to point out that I think the documentation
> you
> > > >> > recently
> > > >> > > > > merged on Kafka site regarding the 3.5.0 version has a
> problem
> > > when
> > > >> > it
> > > >> > > > > states the version number and the sub-menu that links to
> > > previous
> > > >> > versions.
> > > >> > > > > Left a comment here:
> > > >> > > > >
> > > >> >
> > >
> >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > >> > > > >
> > > >> > > > > Best,
> > > >> > > > >
> > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> > josep.prat@aiven.io
> > > >
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Mickael,
> > > >> > > > > >
> > > >> > > > > > I can +1 this candidate. I verified the following:
> > > >> > > > > > - Built from source with Java 17 and Scala 2.13
> > > >> > > > > > - Signatures and hashes of the artifacts generated
> > > >> > > > > > - Navigated through Javadoc including links to JDK classes
> > > >> > > > > > - Run the unit tests
> > > >> > > > > > - Run integration tests
> > > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > >> > > > > >
> > > >> > > > > > Best,
> > > >> > > > > >
> > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > > >> > mimaison@apache.org>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > >> Hello Kafka users, developers and client-developers,
> > > >> > > > > >>
> > > >> > > > > >> This is the first candidate for release of Apache Kafka
> > > 3.5.0.
> > > >> > Some of
> > > >> > > > > the
> > > >> > > > > >> major features include:
> > > >> > > > > >> - KIP-710: Full support for distributed mode in dedicated
> > > >> > MirrorMaker
> > > >> > > > > >> 2.0 clusters
> > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> > > Consumers
> > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of environment
> > > variables
> > > >> > > > > >> - KIP-889: Versioned State Stores
> > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
> > > >> > configurations
> > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to
> support
> > > SCRAM
> > > >> > for
> > > >> > > > > >> Kafka Brokers
> > > >> > > > > >>
> > > >> > > > > >> Release notes for the 3.5.0 release:
> > > >> > > > > >>
> > > >> >
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > >> > > > > >>
> > > >> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm
> > PT
> > > >> > > > > >>
> > > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the
> > > release:
> > > >> > > > > >> https://kafka.apache.org/KEYS
> > > >> > > > > >>
> > > >> > > > > >> * Release artifacts to be voted upon (source and binary):
> > > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > >> > > > > >>
> > > >> > > > > >> * Maven artifacts to be voted upon:
> > > >> > > > > >>
> > > >> >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >> > > > > >>
> > > >> > > > > >> * Javadoc:
> > > >> > > > > >>
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > >> > > > > >>
> > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > >> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > >> > > > > >>
> > > >> > > > > >> The PR adding the 35 documentation is not merged yet
> > > >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > >> > > > > >> * Documentation:
> > > >> > > > > >> https://kafka.apache.org/35/documentation.html
> > > >> > > > > >> * Protocol:
> > > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > > >> > > > > >>
> > > >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > >> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5
> > > branch,
> > > >> > > > > >> working with INFRA to sort it out:
> > > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > >> > > > > >> System tests: The build is still running, I'll send an
> > update
> > > >> > once I
> > > >> > > > > >> have the results
> > > >> > > > > >>
> > > >> > > > > >> Thanks,
> > > >> > > > > >> Mickael
> > > >> > > > > >>
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > --
> > > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > > >> > > > > >
> > > >> > > > > > *Josep Prat*
> > > >> > > > > > Open Source Engineering Director, *Aiven*
> > > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > > >> > > > > > aiven.io <https://www.aiven.io>   |
> > > >> > > > > > <https://www.facebook.com/aivencloud>
> > > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > >> > > > > https://twitter.com/aiven_io>
> > > >> > > > > > *Aiven Deutschland GmbH*
> > > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > [image: Aiven] <https://www.aiven.io>
> > > >> > > > >
> > > >> > > > > *Josep Prat*
> > > >> > > > > Open Source Engineering Director, *Aiven*
> > > >> > > > > josep.prat@aiven.io   |   +491715557497
> > > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > > >> > https://www.facebook.com/aivencloud
> > > >> > > > > >
> > > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > >> > > > > https://twitter.com/aiven_io>
> > > >> > > > > *Aiven Deutschland GmbH*
> > > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >> > > > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> -David
> > >
> >
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud
> >
>   <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [VOTE] 3.5.0 RC0

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi there,
Is the plan described in KIP-833[1] still valid? In there it states that
3.5.0 should aim at deprecation of Zookeeper, so conceptually, the path to
migrate to Kraft should be somewhat functional (in my opinion). If we don't
want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion that
these could be fixed in subsequent patches of 3.5.x. Just my 5cts.

[1]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
Best,

On Wed, May 24, 2023 at 8:51 PM Ismael Juma <is...@juma.me.uk> wrote:

> Are all these blockers? For example, zk to kraft migration are is still in
> preview - can we fix some of these in 3.5.1?
>
> Ismael
>
> On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Mickael,
> >
> > Thanks for putting together this RC. Unfortunately, we've identified
> > several blocker issues in this release candidate.
> >
> > KAFKA-15009: New ACLs are not written to ZK during migration
> > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > migration.
> > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > migration (dual-write)
> > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > (dual-write) when partitions are changed for topic
> > KAFKA-14996: The KRaft controller should properly handle overly large
> user
> > operations
> >
> > We are working on PRs for these issues and will get them in soon, we
> think!
> >
> > So unfortunately I have to leave a -1 here for RC0. Let's aim for another
> > RC next week.
> >
> > best,
> > Colin
> >
> > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > Hi David,
> > >
> > > We're already quite a bit behind schedule. If you think these fixes
> > > are really important and can be ready in the next couple of days, I'm
> > > open to backport them and build another release candidate. Let me know
> > > once you've investigated the severity of KAFKA-15010.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > <da...@confluent.io.invalid> wrote:
> > >>
> > >> Mickael, we have some migration fixes on trunk, is it okay to
> > cherry-pick
> > >> these to 3.5?
> > >>
> > >> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
> > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > >>
> > >> There is another issue KAFKA-15010 that I'm also investigating to
> > determine
> > >> the impact and likelihood of seeing it in practice. This one may be a
> > >> significant migration blocker
> > >>
> > >> Cheers,
> > >> David
> > >>
> > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Christo,
> > >> >
> > >> > Yes this is expected. This happens when nested fields also accept
> > >> > optional tagged fields. The tables list all fields, so they may
> > >> > include _tagged_fields multiple times.
> > >> > Clearly the layout of this page could be improved, if you have ideas
> > >> > how to describe the protocol in a better way, feel free to share
> them.
> > >> >
> > >> > Thanks,
> > >> > Mickael
> > >> >
> > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> > mickael.maison@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > Hi Josep,
> > >> > >
> > >> > > Good catch! I opened a PR to fix this:
> > >> > > https://github.com/apache/kafka-site/pull/514
> > >> > >
> > >> > > Thanks,
> > >> > > Mickael
> > >> > >
> > >> > >
> > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> > christololov@gmail.com>
> > >> > wrote:
> > >> > > >
> > >> > > > Hey Mickael!
> > >> > > >
> > >> > > > I am giving a +1 (non-binding) for this candidate release.
> > >> > > >
> > >> > > > * Built from the binary tar.gz source with Java 17 and Scala
> 2.13
> > on
> > >> > Intel
> > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > >> > > > * Ran unit and integration tests on Intel and ARM machines.
> > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel
> > and ARM
> > >> > > > machines.
> > >> > > >
> > >> > > > Question:
> > >> > > > * I went through https://kafka.apache.org/35/protocol.html and
> > there
> > >> > are
> > >> > > > quite a few repetitive __tagged_fileds fields within the same
> > >> > structures -
> > >> > > > is this expected?
> > >> > > >
> > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> > <jo...@aiven.io.invalid>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Mickael,
> > >> > > > > I just wanted to point out that I think the documentation you
> > >> > recently
> > >> > > > > merged on Kafka site regarding the 3.5.0 version has a problem
> > when
> > >> > it
> > >> > > > > states the version number and the sub-menu that links to
> > previous
> > >> > versions.
> > >> > > > > Left a comment here:
> > >> > > > >
> > >> >
> >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > >> > > > >
> > >> > > > > Best,
> > >> > > > >
> > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <
> josep.prat@aiven.io
> > >
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Hi Mickael,
> > >> > > > > >
> > >> > > > > > I can +1 this candidate. I verified the following:
> > >> > > > > > - Built from source with Java 17 and Scala 2.13
> > >> > > > > > - Signatures and hashes of the artifacts generated
> > >> > > > > > - Navigated through Javadoc including links to JDK classes
> > >> > > > > > - Run the unit tests
> > >> > > > > > - Run integration tests
> > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > >> > > > > >
> > >> > > > > > Best,
> > >> > > > > >
> > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > >> > mimaison@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > >> Hello Kafka users, developers and client-developers,
> > >> > > > > >>
> > >> > > > > >> This is the first candidate for release of Apache Kafka
> > 3.5.0.
> > >> > Some of
> > >> > > > > the
> > >> > > > > >> major features include:
> > >> > > > > >> - KIP-710: Full support for distributed mode in dedicated
> > >> > MirrorMaker
> > >> > > > > >> 2.0 clusters
> > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> > Consumers
> > >> > > > > >> - KIP-887: Add ConfigProvider to make use of environment
> > variables
> > >> > > > > >> - KIP-889: Versioned State Stores
> > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
> > >> > configurations
> > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to support
> > SCRAM
> > >> > for
> > >> > > > > >> Kafka Brokers
> > >> > > > > >>
> > >> > > > > >> Release notes for the 3.5.0 release:
> > >> > > > > >>
> > >> >
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > >> > > > > >>
> > >> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm
> PT
> > >> > > > > >>
> > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the
> > release:
> > >> > > > > >> https://kafka.apache.org/KEYS
> > >> > > > > >>
> > >> > > > > >> * Release artifacts to be voted upon (source and binary):
> > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > >> > > > > >>
> > >> > > > > >> * Maven artifacts to be voted upon:
> > >> > > > > >>
> > >> >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > > > > >>
> > >> > > > > >> * Javadoc:
> > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > >> > > > > >>
> > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > >> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > >> > > > > >>
> > >> > > > > >> The PR adding the 35 documentation is not merged yet
> > >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > >> > > > > >> * Documentation:
> > >> > > > > >> https://kafka.apache.org/35/documentation.html
> > >> > > > > >> * Protocol:
> > >> > > > > >> https://kafka.apache.org/35/protocol.html
> > >> > > > > >>
> > >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > >> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5
> > branch,
> > >> > > > > >> working with INFRA to sort it out:
> > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > >> > > > > >> System tests: The build is still running, I'll send an
> update
> > >> > once I
> > >> > > > > >> have the results
> > >> > > > > >>
> > >> > > > > >> Thanks,
> > >> > > > > >> Mickael
> > >> > > > > >>
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > --
> > >> > > > > > [image: Aiven] <https://www.aiven.io>
> > >> > > > > >
> > >> > > > > > *Josep Prat*
> > >> > > > > > Open Source Engineering Director, *Aiven*
> > >> > > > > > josep.prat@aiven.io   |   +491715557497
> > >> > > > > > aiven.io <https://www.aiven.io>   |
> > >> > > > > > <https://www.facebook.com/aivencloud>
> > >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > >> > > > > https://twitter.com/aiven_io>
> > >> > > > > > *Aiven Deutschland GmbH*
> > >> > > > > > Alexanderufer 3-7, 10117 Berlin
> > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > [image: Aiven] <https://www.aiven.io>
> > >> > > > >
> > >> > > > > *Josep Prat*
> > >> > > > > Open Source Engineering Director, *Aiven*
> > >> > > > > josep.prat@aiven.io   |   +491715557497
> > >> > > > > aiven.io <https://www.aiven.io>   |   <
> > >> > https://www.facebook.com/aivencloud
> > >> > > > > >
> > >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > >> > > > > https://twitter.com/aiven_io>
> > >> > > > > *Aiven Deutschland GmbH*
> > >> > > > > Alexanderufer 3-7, 10117 Berlin
> > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > >> > > > >
> > >> >
> > >>
> > >>
> > >> --
> > >> -David
> >
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [VOTE] 3.5.0 RC0

Posted by Ismael Juma <is...@juma.me.uk>.

Are all these blockers? For example, zk to kraft migration are is still in
preview - can we fix some of these in 3.5.1?

Ismael

On Wed, May 24, 2023, 10:22 AM Colin McCabe <cm...@apache.org> wrote:

> Hi Mickael,
>
> Thanks for putting together this RC. Unfortunately, we've identified
> several blocker issues in this release candidate.
>
> KAFKA-15009: New ACLs are not written to ZK during migration
> KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> migration.
> KAFKA-15004: Topic config changes are not synced during zk to kraft
> migration (dual-write)
> KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> (dual-write) when partitions are changed for topic
> KAFKA-14996: The KRaft controller should properly handle overly large user
> operations
>
> We are working on PRs for these issues and will get them in soon, we think!
>
> So unfortunately I have to leave a -1 here for RC0. Let's aim for another
> RC next week.
>
> best,
> Colin
>
> On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > Hi David,
> >
> > We're already quite a bit behind schedule. If you think these fixes
> > are really important and can be ready in the next couple of days, I'm
> > open to backport them and build another release candidate. Let me know
> > once you've investigated the severity of KAFKA-15010.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > <da...@confluent.io.invalid> wrote:
> >>
> >> Mickael, we have some migration fixes on trunk, is it okay to
> cherry-pick
> >> these to 3.5?
> >>
> >> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
> >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> >>
> >> There is another issue KAFKA-15010 that I'm also investigating to
> determine
> >> the impact and likelihood of seeing it in practice. This one may be a
> >> significant migration blocker
> >>
> >> Cheers,
> >> David
> >>
> >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> mickael.maison@gmail.com>
> >> wrote:
> >>
> >> > Hi Christo,
> >> >
> >> > Yes this is expected. This happens when nested fields also accept
> >> > optional tagged fields. The tables list all fields, so they may
> >> > include _tagged_fields multiple times.
> >> > Clearly the layout of this page could be improved, if you have ideas
> >> > how to describe the protocol in a better way, feel free to share them.
> >> >
> >> > Thanks,
> >> > Mickael
> >> >
> >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <
> mickael.maison@gmail.com>
> >> > wrote:
> >> > >
> >> > > Hi Josep,
> >> > >
> >> > > Good catch! I opened a PR to fix this:
> >> > > https://github.com/apache/kafka-site/pull/514
> >> > >
> >> > > Thanks,
> >> > > Mickael
> >> > >
> >> > >
> >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <
> christololov@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > Hey Mickael!
> >> > > >
> >> > > > I am giving a +1 (non-binding) for this candidate release.
> >> > > >
> >> > > > * Built from the binary tar.gz source with Java 17 and Scala 2.13
> on
> >> > Intel
> >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> >> > > > * Ran unit and integration tests on Intel and ARM machines.
> >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel
> and ARM
> >> > > > machines.
> >> > > >
> >> > > > Question:
> >> > > > * I went through https://kafka.apache.org/35/protocol.html and
> there
> >> > are
> >> > > > quite a few repetitive __tagged_fileds fields within the same
> >> > structures -
> >> > > > is this expected?
> >> > > >
> >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat
> <jo...@aiven.io.invalid>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Mickael,
> >> > > > > I just wanted to point out that I think the documentation you
> >> > recently
> >> > > > > merged on Kafka site regarding the 3.5.0 version has a problem
> when
> >> > it
> >> > > > > states the version number and the sub-menu that links to
> previous
> >> > versions.
> >> > > > > Left a comment here:
> >> > > > >
> >> >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> >> > > > >
> >> > > > > Best,
> >> > > > >
> >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <josep.prat@aiven.io
> >
> >> > wrote:
> >> > > > >
> >> > > > > > Hi Mickael,
> >> > > > > >
> >> > > > > > I can +1 this candidate. I verified the following:
> >> > > > > > - Built from source with Java 17 and Scala 2.13
> >> > > > > > - Signatures and hashes of the artifacts generated
> >> > > > > > - Navigated through Javadoc including links to JDK classes
> >> > > > > > - Run the unit tests
> >> > > > > > - Run integration tests
> >> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> >> > > > > >
> >> > > > > > Best,
> >> > > > > >
> >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> >> > mimaison@apache.org>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > >> Hello Kafka users, developers and client-developers,
> >> > > > > >>
> >> > > > > >> This is the first candidate for release of Apache Kafka
> 3.5.0.
> >> > Some of
> >> > > > > the
> >> > > > > >> major features include:
> >> > > > > >> - KIP-710: Full support for distributed mode in dedicated
> >> > MirrorMaker
> >> > > > > >> 2.0 clusters
> >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka
> Consumers
> >> > > > > >> - KIP-887: Add ConfigProvider to make use of environment
> variables
> >> > > > > >> - KIP-889: Versioned State Stores
> >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
> >> > configurations
> >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to support
> SCRAM
> >> > for
> >> > > > > >> Kafka Brokers
> >> > > > > >>
> >> > > > > >> Release notes for the 3.5.0 release:
> >> > > > > >>
> >> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> >> > > > > >>
> >> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> >> > > > > >>
> >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the
> release:
> >> > > > > >> https://kafka.apache.org/KEYS
> >> > > > > >>
> >> > > > > >> * Release artifacts to be voted upon (source and binary):
> >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> >> > > > > >>
> >> > > > > >> * Maven artifacts to be voted upon:
> >> > > > > >>
> >> >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >> > > > > >>
> >> > > > > >> * Javadoc:
> >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> >> > > > > >>
> >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> >> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> >> > > > > >>
> >> > > > > >> The PR adding the 35 documentation is not merged yet
> >> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> >> > > > > >> * Documentation:
> >> > > > > >> https://kafka.apache.org/35/documentation.html
> >> > > > > >> * Protocol:
> >> > > > > >> https://kafka.apache.org/35/protocol.html
> >> > > > > >>
> >> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> >> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5
> branch,
> >> > > > > >> working with INFRA to sort it out:
> >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> >> > > > > >> System tests: The build is still running, I'll send an update
> >> > once I
> >> > > > > >> have the results
> >> > > > > >>
> >> > > > > >> Thanks,
> >> > > > > >> Mickael
> >> > > > > >>
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > [image: Aiven] <https://www.aiven.io>
> >> > > > > >
> >> > > > > > *Josep Prat*
> >> > > > > > Open Source Engineering Director, *Aiven*
> >> > > > > > josep.prat@aiven.io   |   +491715557497
> >> > > > > > aiven.io <https://www.aiven.io>   |
> >> > > > > > <https://www.facebook.com/aivencloud>
> >> > > > > > <https://www.linkedin.com/company/aiven/>   <
> >> > > > > https://twitter.com/aiven_io>
> >> > > > > > *Aiven Deutschland GmbH*
> >> > > > > > Alexanderufer 3-7, 10117 Berlin
> >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > [image: Aiven] <https://www.aiven.io>
> >> > > > >
> >> > > > > *Josep Prat*
> >> > > > > Open Source Engineering Director, *Aiven*
> >> > > > > josep.prat@aiven.io   |   +491715557497
> >> > > > > aiven.io <https://www.aiven.io>   |   <
> >> > https://www.facebook.com/aivencloud
> >> > > > > >
> >> > > > >   <https://www.linkedin.com/company/aiven/>   <
> >> > > > > https://twitter.com/aiven_io>
> >> > > > > *Aiven Deutschland GmbH*
> >> > > > > Alexanderufer 3-7, 10117 Berlin
> >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> >> > > > >
> >> >
> >>
> >>
> >> --
> >> -David
>

Re: [VOTE] 3.5.0 RC0

Posted by Colin McCabe <cm...@apache.org>.

Hi Mickael,

Thanks for putting together this RC. Unfortunately, we've identified several blocker issues in this release candidate.

KAFKA-15009: New ACLs are not written to ZK during migration
KAFKA-15007: MV is not set correctly in the MetadataPropagator in migration.
KAFKA-15004: Topic config changes are not synced during zk to kraft migration (dual-write)
KAFKA-15003: TopicIdReplicaAssignment is not updated in migration (dual-write) when partitions are changed for topic
KAFKA-14996: The KRaft controller should properly handle overly large user operations

We are working on PRs for these issues and will get them in soon, we think!

So unfortunately I have to leave a -1 here for RC0. Let's aim for another RC next week.

best,
Colin

On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> Hi David,
>
> We're already quite a bit behind schedule. If you think these fixes
> are really important and can be ready in the next couple of days, I'm
> open to backport them and build another release candidate. Let me know
> once you've investigated the severity of KAFKA-15010.
>
> Thanks,
> Mickael
>
>
> On Tue, May 23, 2023 at 6:34 PM David Arthur
> <da...@confluent.io.invalid> wrote:
>>
>> Mickael, we have some migration fixes on trunk, is it okay to cherry-pick
>> these to 3.5?
>>
>> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
>> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
>>
>> There is another issue KAFKA-15010 that I'm also investigating to determine
>> the impact and likelihood of seeing it in practice. This one may be a
>> significant migration blocker
>>
>> Cheers,
>> David
>>
>> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <mi...@gmail.com>
>> wrote:
>>
>> > Hi Christo,
>> >
>> > Yes this is expected. This happens when nested fields also accept
>> > optional tagged fields. The tables list all fields, so they may
>> > include _tagged_fields multiple times.
>> > Clearly the layout of this page could be improved, if you have ideas
>> > how to describe the protocol in a better way, feel free to share them.
>> >
>> > Thanks,
>> > Mickael
>> >
>> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <mi...@gmail.com>
>> > wrote:
>> > >
>> > > Hi Josep,
>> > >
>> > > Good catch! I opened a PR to fix this:
>> > > https://github.com/apache/kafka-site/pull/514
>> > >
>> > > Thanks,
>> > > Mickael
>> > >
>> > >
>> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <ch...@gmail.com>
>> > wrote:
>> > > >
>> > > > Hey Mickael!
>> > > >
>> > > > I am giving a +1 (non-binding) for this candidate release.
>> > > >
>> > > > * Built from the binary tar.gz source with Java 17 and Scala 2.13 on
>> > Intel
>> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
>> > > > * Ran unit and integration tests on Intel and ARM machines.
>> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
>> > > > machines.
>> > > >
>> > > > Question:
>> > > > * I went through https://kafka.apache.org/35/protocol.html and there
>> > are
>> > > > quite a few repetitive __tagged_fileds fields within the same
>> > structures -
>> > > > is this expected?
>> > > >
>> > > > On Tue, 23 May 2023 at 12:01, Josep Prat <jo...@aiven.io.invalid>
>> > > > wrote:
>> > > >
>> > > > > Hi Mickael,
>> > > > > I just wanted to point out that I think the documentation you
>> > recently
>> > > > > merged on Kafka site regarding the 3.5.0 version has a problem when
>> > it
>> > > > > states the version number and the sub-menu that links to previous
>> > versions.
>> > > > > Left a comment here:
>> > > > >
>> > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
>> > > > >
>> > > > > Best,
>> > > > >
>> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io>
>> > wrote:
>> > > > >
>> > > > > > Hi Mickael,
>> > > > > >
>> > > > > > I can +1 this candidate. I verified the following:
>> > > > > > - Built from source with Java 17 and Scala 2.13
>> > > > > > - Signatures and hashes of the artifacts generated
>> > > > > > - Navigated through Javadoc including links to JDK classes
>> > > > > > - Run the unit tests
>> > > > > > - Run integration tests
>> > > > > > - Run the quickstart in KRaft and Zookeeper mode
>> > > > > >
>> > > > > > Best,
>> > > > > >
>> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
>> > mimaison@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Hello Kafka users, developers and client-developers,
>> > > > > >>
>> > > > > >> This is the first candidate for release of Apache Kafka 3.5.0.
>> > Some of
>> > > > > the
>> > > > > >> major features include:
>> > > > > >> - KIP-710: Full support for distributed mode in dedicated
>> > MirrorMaker
>> > > > > >> 2.0 clusters
>> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
>> > > > > >> - KIP-887: Add ConfigProvider to make use of environment variables
>> > > > > >> - KIP-889: Versioned State Stores
>> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
>> > configurations
>> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
>> > for
>> > > > > >> Kafka Brokers
>> > > > > >>
>> > > > > >> Release notes for the 3.5.0 release:
>> > > > > >>
>> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>> > > > > >>
>> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
>> > > > > >>
>> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > > > >> https://kafka.apache.org/KEYS
>> > > > > >>
>> > > > > >> * Release artifacts to be voted upon (source and binary):
>> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>> > > > > >>
>> > > > > >> * Maven artifacts to be voted upon:
>> > > > > >>
>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > > > > >>
>> > > > > >> * Javadoc:
>> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>> > > > > >>
>> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
>> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>> > > > > >>
>> > > > > >> The PR adding the 35 documentation is not merged yet
>> > > > > >> (https://github.com/apache/kafka-site/pull/513)
>> > > > > >> * Documentation:
>> > > > > >> https://kafka.apache.org/35/documentation.html
>> > > > > >> * Protocol:
>> > > > > >> https://kafka.apache.org/35/protocol.html
>> > > > > >>
>> > > > > >> * Successful Jenkins builds for the 3.5 branch:
>> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
>> > > > > >> working with INFRA to sort it out:
>> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
>> > > > > >> System tests: The build is still running, I'll send an update
>> > once I
>> > > > > >> have the results
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> Mickael
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > [image: Aiven] <https://www.aiven.io>
>> > > > > >
>> > > > > > *Josep Prat*
>> > > > > > Open Source Engineering Director, *Aiven*
>> > > > > > josep.prat@aiven.io   |   +491715557497
>> > > > > > aiven.io <https://www.aiven.io>   |
>> > > > > > <https://www.facebook.com/aivencloud>
>> > > > > > <https://www.linkedin.com/company/aiven/>   <
>> > > > > https://twitter.com/aiven_io>
>> > > > > > *Aiven Deutschland GmbH*
>> > > > > > Alexanderufer 3-7, 10117 Berlin
>> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > [image: Aiven] <https://www.aiven.io>
>> > > > >
>> > > > > *Josep Prat*
>> > > > > Open Source Engineering Director, *Aiven*
>> > > > > josep.prat@aiven.io   |   +491715557497
>> > > > > aiven.io <https://www.aiven.io>   |   <
>> > https://www.facebook.com/aivencloud
>> > > > > >
>> > > > >   <https://www.linkedin.com/company/aiven/>   <
>> > > > > https://twitter.com/aiven_io>
>> > > > > *Aiven Deutschland GmbH*
>> > > > > Alexanderufer 3-7, 10117 Berlin
>> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > > > > Amtsgericht Charlottenburg, HRB 209739 B
>> > > > >
>> >
>>
>>
>> --
>> -David

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi David,

We're already quite a bit behind schedule. If you think these fixes
are really important and can be ready in the next couple of days, I'm
open to backport them and build another release candidate. Let me know
once you've investigated the severity of KAFKA-15010.

Thanks,
Mickael


On Tue, May 23, 2023 at 6:34 PM David Arthur
<da...@confluent.io.invalid> wrote:
>
> Mickael, we have some migration fixes on trunk, is it okay to cherry-pick
> these to 3.5?
>
> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
>
> There is another issue KAFKA-15010 that I'm also investigating to determine
> the impact and likelihood of seeing it in practice. This one may be a
> significant migration blocker
>
> Cheers,
> David
>
> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Christo,
> >
> > Yes this is expected. This happens when nested fields also accept
> > optional tagged fields. The tables list all fields, so they may
> > include _tagged_fields multiple times.
> > Clearly the layout of this page could be improved, if you have ideas
> > how to describe the protocol in a better way, feel free to share them.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > Hi Josep,
> > >
> > > Good catch! I opened a PR to fix this:
> > > https://github.com/apache/kafka-site/pull/514
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <ch...@gmail.com>
> > wrote:
> > > >
> > > > Hey Mickael!
> > > >
> > > > I am giving a +1 (non-binding) for this candidate release.
> > > >
> > > > * Built from the binary tar.gz source with Java 17 and Scala 2.13 on
> > Intel
> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > > * Ran unit and integration tests on Intel and ARM machines.
> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
> > > > machines.
> > > >
> > > > Question:
> > > > * I went through https://kafka.apache.org/35/protocol.html and there
> > are
> > > > quite a few repetitive __tagged_fileds fields within the same
> > structures -
> > > > is this expected?
> > > >
> > > > On Tue, 23 May 2023 at 12:01, Josep Prat <jo...@aiven.io.invalid>
> > > > wrote:
> > > >
> > > > > Hi Mickael,
> > > > > I just wanted to point out that I think the documentation you
> > recently
> > > > > merged on Kafka site regarding the 3.5.0 version has a problem when
> > it
> > > > > states the version number and the sub-menu that links to previous
> > versions.
> > > > > Left a comment here:
> > > > >
> > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > > >
> > > > > Best,
> > > > >
> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io>
> > wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I can +1 this candidate. I verified the following:
> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > - Signatures and hashes of the artifacts generated
> > > > > > - Navigated through Javadoc including links to JDK classes
> > > > > > - Run the unit tests
> > > > > > - Run integration tests
> > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> > mimaison@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > >> Hello Kafka users, developers and client-developers,
> > > > > >>
> > > > > >> This is the first candidate for release of Apache Kafka 3.5.0.
> > Some of
> > > > > the
> > > > > >> major features include:
> > > > > >> - KIP-710: Full support for distributed mode in dedicated
> > MirrorMaker
> > > > > >> 2.0 clusters
> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > > > >> - KIP-887: Add ConfigProvider to make use of environment variables
> > > > > >> - KIP-889: Versioned State Stores
> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
> > configurations
> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
> > for
> > > > > >> Kafka Brokers
> > > > > >>
> > > > > >> Release notes for the 3.5.0 release:
> > > > > >>
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > > >>
> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> > > > > >>
> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > >> https://kafka.apache.org/KEYS
> > > > > >>
> > > > > >> * Release artifacts to be voted upon (source and binary):
> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > > >>
> > > > > >> * Maven artifacts to be voted upon:
> > > > > >>
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >>
> > > > > >> * Javadoc:
> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > > >>
> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > > >>
> > > > > >> The PR adding the 35 documentation is not merged yet
> > > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > > >> * Documentation:
> > > > > >> https://kafka.apache.org/35/documentation.html
> > > > > >> * Protocol:
> > > > > >> https://kafka.apache.org/35/protocol.html
> > > > > >>
> > > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> > > > > >> working with INFRA to sort it out:
> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > > >> System tests: The build is still running, I'll send an update
> > once I
> > > > > >> have the results
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Mickael
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > [image: Aiven] <https://www.aiven.io>
> > > > > >
> > > > > > *Josep Prat*
> > > > > > Open Source Engineering Director, *Aiven*
> > > > > > josep.prat@aiven.io   |   +491715557497
> > > > > > aiven.io <https://www.aiven.io>   |
> > > > > > <https://www.facebook.com/aivencloud>
> > > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > > https://twitter.com/aiven_io>
> > > > > > *Aiven Deutschland GmbH*
> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > [image: Aiven] <https://www.aiven.io>
> > > > >
> > > > > *Josep Prat*
> > > > > Open Source Engineering Director, *Aiven*
> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > aiven.io <https://www.aiven.io>   |   <
> > https://www.facebook.com/aivencloud
> > > > > >
> > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > > https://twitter.com/aiven_io>
> > > > > *Aiven Deutschland GmbH*
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> >
>
>
> --
> -David

Re: [VOTE] 3.5.0 RC0

Posted by David Arthur <da...@confluent.io.INVALID>.

Mickael, we have some migration fixes on trunk, is it okay to cherry-pick
these to 3.5?

KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
KAFKA-15009 Handle new ACLs in KRaft snapshot during migration

There is another issue KAFKA-15010 that I'm also investigating to determine
the impact and likelihood of seeing it in practice. This one may be a
significant migration blocker

Cheers,
David

On Tue, May 23, 2023 at 9:57 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Christo,
>
> Yes this is expected. This happens when nested fields also accept
> optional tagged fields. The tables list all fields, so they may
> include _tagged_fields multiple times.
> Clearly the layout of this page could be improved, if you have ideas
> how to describe the protocol in a better way, feel free to share them.
>
> Thanks,
> Mickael
>
> On Tue, May 23, 2023 at 3:50 PM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Hi Josep,
> >
> > Good catch! I opened a PR to fix this:
> > https://github.com/apache/kafka-site/pull/514
> >
> > Thanks,
> > Mickael
> >
> >
> > On Tue, May 23, 2023 at 3:36 PM Christo Lolov <ch...@gmail.com>
> wrote:
> > >
> > > Hey Mickael!
> > >
> > > I am giving a +1 (non-binding) for this candidate release.
> > >
> > > * Built from the binary tar.gz source with Java 17 and Scala 2.13 on
> Intel
> > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > * Ran unit and integration tests on Intel and ARM machines.
> > > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
> > > machines.
> > >
> > > Question:
> > > * I went through https://kafka.apache.org/35/protocol.html and there
> are
> > > quite a few repetitive __tagged_fileds fields within the same
> structures -
> > > is this expected?
> > >
> > > On Tue, 23 May 2023 at 12:01, Josep Prat <jo...@aiven.io.invalid>
> > > wrote:
> > >
> > > > Hi Mickael,
> > > > I just wanted to point out that I think the documentation you
> recently
> > > > merged on Kafka site regarding the 3.5.0 version has a problem when
> it
> > > > states the version number and the sub-menu that links to previous
> versions.
> > > > Left a comment here:
> > > >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > >
> > > > Best,
> > > >
> > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io>
> wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I can +1 this candidate. I verified the following:
> > > > > - Built from source with Java 17 and Scala 2.13
> > > > > - Signatures and hashes of the artifacts generated
> > > > > - Navigated through Javadoc including links to JDK classes
> > > > > - Run the unit tests
> > > > > - Run integration tests
> > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > >
> > > > > Best,
> > > > >
> > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> mimaison@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Hello Kafka users, developers and client-developers,
> > > > >>
> > > > >> This is the first candidate for release of Apache Kafka 3.5.0.
> Some of
> > > > the
> > > > >> major features include:
> > > > >> - KIP-710: Full support for distributed mode in dedicated
> MirrorMaker
> > > > >> 2.0 clusters
> > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > > >> - KIP-887: Add ConfigProvider to make use of environment variables
> > > > >> - KIP-889: Versioned State Stores
> > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
> configurations
> > > > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
> for
> > > > >> Kafka Brokers
> > > > >>
> > > > >> Release notes for the 3.5.0 release:
> > > > >>
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > >>
> > > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> > > > >>
> > > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > >> https://kafka.apache.org/KEYS
> > > > >>
> > > > >> * Release artifacts to be voted upon (source and binary):
> > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > >>
> > > > >> * Maven artifacts to be voted upon:
> > > > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >>
> > > > >> * Javadoc:
> > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > >>
> > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > >>
> > > > >> The PR adding the 35 documentation is not merged yet
> > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > >> * Documentation:
> > > > >> https://kafka.apache.org/35/documentation.html
> > > > >> * Protocol:
> > > > >> https://kafka.apache.org/35/protocol.html
> > > > >>
> > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> > > > >> working with INFRA to sort it out:
> > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > >> System tests: The build is still running, I'll send an update
> once I
> > > > >> have the results
> > > > >>
> > > > >> Thanks,
> > > > >> Mickael
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > [image: Aiven] <https://www.aiven.io>
> > > > >
> > > > > *Josep Prat*
> > > > > Open Source Engineering Director, *Aiven*
> > > > > josep.prat@aiven.io   |   +491715557497
> > > > > aiven.io <https://www.aiven.io>   |
> > > > > <https://www.facebook.com/aivencloud>
> > > > > <https://www.linkedin.com/company/aiven/>   <
> > > > https://twitter.com/aiven_io>
> > > > > *Aiven Deutschland GmbH*
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > >
> > > >
> > > > --
> > > > [image: Aiven] <https://www.aiven.io>
> > > >
> > > > *Josep Prat*
> > > > Open Source Engineering Director, *Aiven*
> > > > josep.prat@aiven.io   |   +491715557497
> > > > aiven.io <https://www.aiven.io>   |   <
> https://www.facebook.com/aivencloud
> > > > >
> > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > https://twitter.com/aiven_io>
> > > > *Aiven Deutschland GmbH*
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
>


-- 
-David

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi Christo,

Yes this is expected. This happens when nested fields also accept
optional tagged fields. The tables list all fields, so they may
include _tagged_fields multiple times.
Clearly the layout of this page could be improved, if you have ideas
how to describe the protocol in a better way, feel free to share them.

Thanks,
Mickael

On Tue, May 23, 2023 at 3:50 PM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi Josep,
>
> Good catch! I opened a PR to fix this:
> https://github.com/apache/kafka-site/pull/514
>
> Thanks,
> Mickael
>
>
> On Tue, May 23, 2023 at 3:36 PM Christo Lolov <ch...@gmail.com> wrote:
> >
> > Hey Mickael!
> >
> > I am giving a +1 (non-binding) for this candidate release.
> >
> > * Built from the binary tar.gz source with Java 17 and Scala 2.13 on Intel
> > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > * Ran unit and integration tests on Intel and ARM machines.
> > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
> > machines.
> >
> > Question:
> > * I went through https://kafka.apache.org/35/protocol.html and there are
> > quite a few repetitive __tagged_fileds fields within the same structures -
> > is this expected?
> >
> > On Tue, 23 May 2023 at 12:01, Josep Prat <jo...@aiven.io.invalid>
> > wrote:
> >
> > > Hi Mickael,
> > > I just wanted to point out that I think the documentation you recently
> > > merged on Kafka site regarding the 3.5.0 version has a problem when it
> > > states the version number and the sub-menu that links to previous versions.
> > > Left a comment here:
> > > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > >
> > > Best,
> > >
> > > On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > I can +1 this candidate. I verified the following:
> > > > - Built from source with Java 17 and Scala 2.13
> > > > - Signatures and hashes of the artifacts generated
> > > > - Navigated through Javadoc including links to JDK classes
> > > > - Run the unit tests
> > > > - Run integration tests
> > > > - Run the quickstart in KRaft and Zookeeper mode
> > > >
> > > > Best,
> > > >
> > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <mi...@apache.org>
> > > > wrote:
> > > >
> > > >> Hello Kafka users, developers and client-developers,
> > > >>
> > > >> This is the first candidate for release of Apache Kafka 3.5.0. Some of
> > > the
> > > >> major features include:
> > > >> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > > >> 2.0 clusters
> > > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > >> - KIP-887: Add ConfigProvider to make use of environment variables
> > > >> - KIP-889: Versioned State Stores
> > > >> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > > >> Kafka Brokers
> > > >>
> > > >> Release notes for the 3.5.0 release:
> > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > >>
> > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> > > >>
> > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >> https://kafka.apache.org/KEYS
> > > >>
> > > >> * Release artifacts to be voted upon (source and binary):
> > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > >>
> > > >> * Maven artifacts to be voted upon:
> > > >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >>
> > > >> * Javadoc:
> > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > >>
> > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > >>
> > > >> The PR adding the 35 documentation is not merged yet
> > > >> (https://github.com/apache/kafka-site/pull/513)
> > > >> * Documentation:
> > > >> https://kafka.apache.org/35/documentation.html
> > > >> * Protocol:
> > > >> https://kafka.apache.org/35/protocol.html
> > > >>
> > > >> * Successful Jenkins builds for the 3.5 branch:
> > > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> > > >> working with INFRA to sort it out:
> > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > >> System tests: The build is still running, I'll send an update once I
> > > >> have the results
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >
> > > >
> > > > --
> > > > [image: Aiven] <https://www.aiven.io>
> > > >
> > > > *Josep Prat*
> > > > Open Source Engineering Director, *Aiven*
> > > > josep.prat@aiven.io   |   +491715557497
> > > > aiven.io <https://www.aiven.io>   |
> > > > <https://www.facebook.com/aivencloud>
> > > > <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > > *Aiven Deutschland GmbH*
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] <https://www.aiven.io>
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.prat@aiven.io   |   +491715557497
> > > aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud
> > > >
> > >   <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >

Re: [VOTE] 3.5.0 RC0

Posted by Mickael Maison <mi...@gmail.com>.

Hi Josep,

Good catch! I opened a PR to fix this:
https://github.com/apache/kafka-site/pull/514

Thanks,
Mickael


On Tue, May 23, 2023 at 3:36 PM Christo Lolov <ch...@gmail.com> wrote:
>
> Hey Mickael!
>
> I am giving a +1 (non-binding) for this candidate release.
>
> * Built from the binary tar.gz source with Java 17 and Scala 2.13 on Intel
> (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> * Ran unit and integration tests on Intel and ARM machines.
> * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
> machines.
>
> Question:
> * I went through https://kafka.apache.org/35/protocol.html and there are
> quite a few repetitive __tagged_fileds fields within the same structures -
> is this expected?
>
> On Tue, 23 May 2023 at 12:01, Josep Prat <jo...@aiven.io.invalid>
> wrote:
>
> > Hi Mickael,
> > I just wanted to point out that I think the documentation you recently
> > merged on Kafka site regarding the 3.5.0 version has a problem when it
> > states the version number and the sub-menu that links to previous versions.
> > Left a comment here:
> > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> >
> > Best,
> >
> > On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io> wrote:
> >
> > > Hi Mickael,
> > >
> > > I can +1 this candidate. I verified the following:
> > > - Built from source with Java 17 and Scala 2.13
> > > - Signatures and hashes of the artifacts generated
> > > - Navigated through Javadoc including links to JDK classes
> > > - Run the unit tests
> > > - Run integration tests
> > > - Run the quickstart in KRaft and Zookeeper mode
> > >
> > > Best,
> > >
> > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <mi...@apache.org>
> > > wrote:
> > >
> > >> Hello Kafka users, developers and client-developers,
> > >>
> > >> This is the first candidate for release of Apache Kafka 3.5.0. Some of
> > the
> > >> major features include:
> > >> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > >> 2.0 clusters
> > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > >> - KIP-887: Add ConfigProvider to make use of environment variables
> > >> - KIP-889: Versioned State Stores
> > >> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > >> Kafka Brokers
> > >>
> > >> Release notes for the 3.5.0 release:
> > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > >>
> > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> > >>
> > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> https://kafka.apache.org/KEYS
> > >>
> > >> * Release artifacts to be voted upon (source and binary):
> > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > >>
> > >> * Maven artifacts to be voted upon:
> > >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>
> > >> * Javadoc:
> > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > >>
> > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > >>
> > >> The PR adding the 35 documentation is not merged yet
> > >> (https://github.com/apache/kafka-site/pull/513)
> > >> * Documentation:
> > >> https://kafka.apache.org/35/documentation.html
> > >> * Protocol:
> > >> https://kafka.apache.org/35/protocol.html
> > >>
> > >> * Successful Jenkins builds for the 3.5 branch:
> > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> > >> working with INFRA to sort it out:
> > >> https://issues.apache.org/jira/browse/INFRA-24577
> > >> System tests: The build is still running, I'll send an update once I
> > >> have the results
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] <https://www.aiven.io>
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.prat@aiven.io   |   +491715557497
> > > aiven.io <https://www.aiven.io>   |
> > > <https://www.facebook.com/aivencloud>
> > > <https://www.linkedin.com/company/aiven/>   <
> > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.prat@aiven.io   |   +491715557497
> > aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud
> > >
> >   <https://www.linkedin.com/company/aiven/>   <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >

Re: [VOTE] 3.5.0 RC0

Posted by Christo Lolov <ch...@gmail.com>.

Hey Mickael!

I am giving a +1 (non-binding) for this candidate release.

* Built from the binary tar.gz source with Java 17 and Scala 2.13 on Intel
(m5.4xlarge) and ARM (m6g.4xlarge) machines.
* Ran unit and integration tests on Intel and ARM machines.
* Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
machines.

Question:
* I went through https://kafka.apache.org/35/protocol.html and there are
quite a few repetitive __tagged_fileds fields within the same structures -
is this expected?

On Tue, 23 May 2023 at 12:01, Josep Prat <jo...@aiven.io.invalid>
wrote:

> Hi Mickael,
> I just wanted to point out that I think the documentation you recently
> merged on Kafka site regarding the 3.5.0 version has a problem when it
> states the version number and the sub-menu that links to previous versions.
> Left a comment here:
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
>
> Best,
>
> On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io> wrote:
>
> > Hi Mickael,
> >
> > I can +1 this candidate. I verified the following:
> > - Built from source with Java 17 and Scala 2.13
> > - Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> >
> > Best,
> >
> > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <mi...@apache.org>
> > wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the first candidate for release of Apache Kafka 3.5.0. Some of
> the
> >> major features include:
> >> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> >> 2.0 clusters
> >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> >> - KIP-887: Add ConfigProvider to make use of environment variables
> >> - KIP-889: Versioned State Stores
> >> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> >> Kafka Brokers
> >>
> >> Release notes for the 3.5.0 release:
> >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Friday, May 26, 5pm PT
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> https://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc:
> >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> >>
> >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> >>
> >> The PR adding the 35 documentation is not merged yet
> >> (https://github.com/apache/kafka-site/pull/513)
> >> * Documentation:
> >> https://kafka.apache.org/35/documentation.html
> >> * Protocol:
> >> https://kafka.apache.org/35/protocol.html
> >>
> >> * Successful Jenkins builds for the 3.5 branch:
> >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> >> working with INFRA to sort it out:
> >> https://issues.apache.org/jira/browse/INFRA-24577
> >> System tests: The build is still running, I'll send an update once I
> >> have the results
> >>
> >> Thanks,
> >> Mickael
> >>
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.prat@aiven.io   |   +491715557497
> > aiven.io <https://www.aiven.io>   |
> > <https://www.facebook.com/aivencloud>
> > <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud
> >
>   <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [VOTE] 3.5.0 RC0

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi Mickael,
I just wanted to point out that I think the documentation you recently
merged on Kafka site regarding the 3.5.0 version has a problem when it
states the version number and the sub-menu that links to previous versions.
Left a comment here:
https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939

Best,

On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io> wrote:

> Hi Mickael,
>
> I can +1 this candidate. I verified the following:
> - Built from source with Java 17 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
>
> Best,
>
> On Mon, May 22, 2023 at 5:30 PM Mickael Maison <mi...@apache.org>
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for release of Apache Kafka 3.5.0. Some of the
>> major features include:
>> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
>> 2.0 clusters
>> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
>> - KIP-887: Add ConfigProvider to make use of environment variables
>> - KIP-889: Versioned State Stores
>> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
>> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
>> Kafka Brokers
>>
>> Release notes for the 3.5.0 release:
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Friday, May 26, 5pm PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc:
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>>
>> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
>> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>>
>> The PR adding the 35 documentation is not merged yet
>> (https://github.com/apache/kafka-site/pull/513)
>> * Documentation:
>> https://kafka.apache.org/35/documentation.html
>> * Protocol:
>> https://kafka.apache.org/35/protocol.html
>>
>> * Successful Jenkins builds for the 3.5 branch:
>> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
>> working with INFRA to sort it out:
>> https://issues.apache.org/jira/browse/INFRA-24577
>> System tests: The build is still running, I'll send an update once I
>> have the results
>>
>> Thanks,
>> Mickael
>>
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |
> <https://www.facebook.com/aivencloud>
> <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [VOTE] 3.5.0 RC0

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi Mickael,
I just wanted to point out that I think the documentation you recently
merged on Kafka site regarding the 3.5.0 version has a problem when it
states the version number and the sub-menu that links to previous versions.
Left a comment here:
https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939

Best,

On Tue, May 23, 2023 at 9:29 AM Josep Prat <jo...@aiven.io> wrote:

> Hi Mickael,
>
> I can +1 this candidate. I verified the following:
> - Built from source with Java 17 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
>
> Best,
>
> On Mon, May 22, 2023 at 5:30 PM Mickael Maison <mi...@apache.org>
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for release of Apache Kafka 3.5.0. Some of the
>> major features include:
>> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
>> 2.0 clusters
>> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
>> - KIP-887: Add ConfigProvider to make use of environment variables
>> - KIP-889: Versioned State Stores
>> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
>> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
>> Kafka Brokers
>>
>> Release notes for the 3.5.0 release:
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Friday, May 26, 5pm PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc:
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>>
>> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
>> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>>
>> The PR adding the 35 documentation is not merged yet
>> (https://github.com/apache/kafka-site/pull/513)
>> * Documentation:
>> https://kafka.apache.org/35/documentation.html
>> * Protocol:
>> https://kafka.apache.org/35/protocol.html
>>
>> * Successful Jenkins builds for the 3.5 branch:
>> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
>> working with INFRA to sort it out:
>> https://issues.apache.org/jira/browse/INFRA-24577
>> System tests: The build is still running, I'll send an update once I
>> have the results
>>
>> Thanks,
>> Mickael
>>
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |
> <https://www.facebook.com/aivencloud>
> <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [VOTE] 3.5.0 RC0

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi Mickael,

I can +1 this candidate. I verified the following:
- Built from source with Java 17 and Scala 2.13
- Signatures and hashes of the artifacts generated
- Navigated through Javadoc including links to JDK classes
- Run the unit tests
- Run integration tests
- Run the quickstart in KRaft and Zookeeper mode

Best,

On Mon, May 22, 2023 at 5:30 PM Mickael Maison <mi...@apache.org> wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 3.5.0. Some of the
> major features include:
> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> 2.0 clusters
> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> - KIP-887: Add ConfigProvider to make use of environment variables
> - KIP-889: Versioned State Stores
> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> Kafka Brokers
>
> Release notes for the 3.5.0 release:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, May 26, 5pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>
> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>
> The PR adding the 35 documentation is not merged yet
> (https://github.com/apache/kafka-site/pull/513)
> * Documentation:
> https://kafka.apache.org/35/documentation.html
> * Protocol:
> https://kafka.apache.org/35/protocol.html
>
> * Successful Jenkins builds for the 3.5 branch:
> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> working with INFRA to sort it out:
> https://issues.apache.org/jira/browse/INFRA-24577
> System tests: The build is still running, I'll send an update once I
> have the results
>
> Thanks,
> Mickael
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [VOTE] 3.5.0 RC0

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi Mickael,

I can +1 this candidate. I verified the following:
- Built from source with Java 17 and Scala 2.13
- Signatures and hashes of the artifacts generated
- Navigated through Javadoc including links to JDK classes
- Run the unit tests
- Run integration tests
- Run the quickstart in KRaft and Zookeeper mode

Best,

On Mon, May 22, 2023 at 5:30 PM Mickael Maison <mi...@apache.org> wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 3.5.0. Some of the
> major features include:
> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> 2.0 clusters
> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> - KIP-887: Add ConfigProvider to make use of environment variables
> - KIP-889: Versioned State Stores
> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> Kafka Brokers
>
> Release notes for the 3.5.0 release:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, May 26, 5pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>
> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>
> The PR adding the 35 documentation is not merged yet
> (https://github.com/apache/kafka-site/pull/513)
> * Documentation:
> https://kafka.apache.org/35/documentation.html
> * Protocol:
> https://kafka.apache.org/35/protocol.html
>
> * Successful Jenkins builds for the 3.5 branch:
> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> working with INFRA to sort it out:
> https://issues.apache.org/jira/browse/INFRA-24577
> System tests: The build is still running, I'll send an update once I
> have the results
>
> Thanks,
> Mickael
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B