You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Magnus Edenhill <ma...@edenhill.se> on 2021/09/01 14:24:40 UTC
Re: [VOTE] 3.0.0 RC1
Hi Konstantine,
Some findings from running 3.0.0-RC1 with the librdkafka test suite:
* Compaction seems to take slightly longer to kick in when segment sizes
exceed their threshold. (Used to take less than 20 seconds, now takes
20..30 seconds.)
* CreateTopic seems to take slightly longer to propagate through the
cluster,
e.g., before a new topic is available in metadata from other brokers.
* CreateTopics seems to take longer when the Admin request timeout is set,
looks like a plateau at 10 seconds:
https://imgur.com/a/n6y76sj
(This is a 3 broker cluster with identical configs between 2.8 and 3.0.0.)
Nothing critical, but could be an indication of regression so I thought I'd
mention it.
Regards,
Magnus
Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
kkarantasis@apache.org>:
> Small correction to my previous email.
> The actual link for public preview of the 3.0.0 blog post draft is:
>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
>
> (see also the email thread with title: [DISCUSS] Please review the 3.0.0
> blog post)
>
> Best,
> Konstantine
>
> On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second release candidate for Apache Kafka 3.0.0.
> > It corresponds to a major release that includes many new features,
> > including:
> >
> > * The deprecation of support for Java 8 and Scala 2.12.
> > * Kafka Raft support for snapshots of the metadata topic and
> > other improvements in the self-managed quorum.
> > * Deprecation of message formats v0 and v1.
> > * Stronger delivery guarantees for the Kafka producer enabled by default.
> > * Optimizations in OffsetFetch and FindCoordinator requests.
> > * More flexible Mirror Maker 2 configuration and deprecation of
> > Mirror Maker 1.
> > * Ability to restart a connector's tasks on a single call in Kafka
> Connect.
> > * Connector log contexts and connector client overrides are now enabled
> > by default.
> > * Enhanced semantics for timestamp synchronization in Kafka Streams.
> > * Revamped public API for Stream's TaskId.
> > * Default serde becomes null in Kafka Streams and several
> > other configuration changes.
> >
> > You may read and review a more detailed list of changes in the 3.0.0 blog
> > post draft here:
> >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> >
> > Release notes for the 3.0.0 release:
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Wednesday, September 8, 2021 ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/30/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/30/protocol.html
> >
> > * Successful Jenkins builds for the 3.0 branch:
> > Unit/integration tests:
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > (only few flaky failures)
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> >
> > /**************************************
> >
> > Thanks,
> > Konstantine
> >
>
Re: [Suspected Spam] Re: [VOTE] 3.0.0 RC1
Posted by Tom Bentley <tb...@redhat.com>.
Hi Konstantine,
I just noticed that the new public DescribeConsumerGroupsResult constructor
refers to the non-public API KafkaFutureImpl. If the constructor has to be
public I think it should use the public KafkaFuture. TBH I don't know if
we'd consider a change to the signature after 3.0 to be a compatibility
break given that KafkaFutureImpl is non-public, so people shouldn't be
using it. I opened https://issues.apache.org/jira/browse/KAFKA-13276 and
tagged it as a 3.0 blocker, but if you disagree feel free to remove. PR to
remove use a KafkaFutureImpl from all *Result class constructors available:
https://github.com/apache/kafka/pull/11301
Kind regards,
Tom
On Fri, Sep 3, 2021 at 5:10 PM Gary Russell <gr...@vmware.com> wrote:
> This [1] is a blocker for me.
>
> Also, the documentation link [2] returns a 404.
>
> [1]: https://issues.apache.org/jira/browse/KAFKA-13262
> [2]: https://kafka.apache.org/30/documentation.html
> ________________________________
> From: Ron Dagostino <rn...@gmail.com>
> Sent: Thursday, September 2, 2021 9:16 PM
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Cc: Users <us...@kafka.apache.org>; kafka-clients <
> kafka-clients@googlegroups.com>
> Subject: [Suspected Spam] Re: [VOTE] 3.0.0 RC1
>
> Hi Konstantine. I have opened a probable blocker ticket
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-13270&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DmGTJeScZXxP5WoszUhdqsJodj%2FCiHuFPXhSwkedR%2BI%3D&reserved=0.
> I will work on a PR
> shortly. The description on that ticket is as follows:
>
> The implementation of
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FZOOKEEPER-3593&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5LV9W5CWAQD%2BnnBIcRXXsLUjhN6%2BqQ8ZjSNEZOlprOg%3D&reserved=0
> in
> ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
> client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
> problem if Kafka tries to retrieve a large amount of data across many
> znodes – in such a case the ZooKeeper client will repeatedly emit a message
> of the form "java.io.IOException: Packet len <####> is out of range" and
> the Kafka broker will never connect to ZooKeeper and fail to make progress
> on the startup sequence. We can avoid the potential for this issue to occur
> by explicitly setting the value to 4MB whenever we create a new ZooKeeper
> client as long as no explicit value has been set via the `jute.maxbuffer`
> system property.
>
> Ron
>
> On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Magnus,
> >
> > Please could you share the machine and network specs?
> >
> > How much CPU, RAM is available on each node?
> >
> > What JDK, JRE version are you using?
> >
> > What are your broker and client configuration values? Please could you
> > share this info if possible?
> >
> > Thanks.
> >
> >
> >
> > On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> > wrote:
> >
> > > Hi Konstantine,
> > >
> > > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> > >
> > > * Compaction seems to take slightly longer to kick in when segment
> sizes
> > > exceed their threshold. (Used to take less than 20 seconds, now takes
> > > 20..30 seconds.)
> > >
> > > * CreateTopic seems to take slightly longer to propagate through the
> > > cluster,
> > > e.g., before a new topic is available in metadata from other brokers.
> > >
> > > * CreateTopics seems to take longer when the Admin request timeout is
> > set,
> > > looks like a plateau at 10 seconds:
> > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fimgur.com%2Fa%2Fn6y76sj&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1wARQVJ%2BycVR1GLla70EbZ9o4hPNpmey4wpGcWfzHvI%3D&reserved=0
> > >
> > > (This is a 3 broker cluster with identical configs between 2.8 and
> > 3.0.0.)
> > >
> > > Nothing critical, but could be an indication of regression so I thought
> > I'd
> > > mention it.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > kkarantasis@apache.org>:
> > >
> > > > Small correction to my previous email.
> > > > The actual link for public preview of the 3.0.0 blog post draft is:
> > > >
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.apache.org%2Fpreview%2Fkafka%2F%3FpreviewEntry%3Dwhat-s-new-in-apache6&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2F12fmUzsupiMAZy%2FZktR56j6F7njNiUVqNMo6LG6jvQ%3D&reserved=0
> > > >
> > > > (see also the email thread with title: [DISCUSS] Please review the
> > 3.0.0
> > > > blog post)
> > > >
> > > > Best,
> > > > Konstantine
> > > >
> > > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > kkarantasis@apache.org> wrote:
> > > >
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > > It corresponds to a major release that includes many new features,
> > > > > including:
> > > > >
> > > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > > other improvements in the self-managed quorum.
> > > > > * Deprecation of message formats v0 and v1.
> > > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > > default.
> > > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > > Mirror Maker 1.
> > > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > > Connect.
> > > > > * Connector log contexts and connector client overrides are now
> > enabled
> > > > > by default.
> > > > > * Enhanced semantics for timestamp synchronization in Kafka
> Streams.
> > > > > * Revamped public API for Stream's TaskId.
> > > > > * Default serde becomes null in Kafka Streams and several
> > > > > other configuration changes.
> > > > >
> > > > > You may read and review a more detailed list of changes in the
> 3.0.0
> > > blog
> > > > > post draft here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.apache.org%2Froller-ui%2Fauthoring%2Fpreview%2Fkafka%2F%3FpreviewEntry%3Dwhat-s-new-in-apache6&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=2FkRhhFa25hPsFpsmmdykpBwJ8gyhjtMprYcDJdYAig%3D&reserved=0
> > > > >
> > > > > Release notes for the 3.0.0 release:
> > > > >
> > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~kkarantasis%2Fkafka-3.0.0-rc1%2FRELEASE_NOTES.html&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cxER6I5H49zHgOlNtBTazzWynt5tiM81o6XvIrezVPo%3D&reserved=0
> > > > >
> > > > > *** Please download, test and vote by Wednesday, September 8, 2021
> > ***
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYS&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xCODs9z81Ja7zFMJgUey%2BGFce38yD9djSxkV99FxDMg%3D&reserved=0
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~kkarantasis%2Fkafka-3.0.0-rc1%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=WL0A4vvZ8jT5CYopAAGCoXWB9pfaItxEPSCEptA283M%3D&reserved=0
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fkafka%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Lsc0Uz9GcEo9mYHp7OCj2Ii6bMGlcwsA%2BRE2esl%2FKKw%3D&reserved=0
> > > > >
> > > > > * Javadoc:
> > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~kkarantasis%2Fkafka-3.0.0-rc1%2Fjavadoc%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Tfys%2Fk8rmqZ8gUaQbYvlwU5Lh69%2F4QDs%2FW3bXt4YJZY%3D&reserved=0
> > > > >
> > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Freleases%2Ftag%2F3.0.0-rc1&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Tisu%2FNQ6EiI7tgWrhXZVfQhJqUqgPnyJtse27ZldG1o%3D&reserved=0
> > > > >
> > > > > * Documentation:
> > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2F30%2Fdocumentation.html&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9qtwMRr48c2CWYbZYeODw4yenaexqfhzo7Cq8d85LFw%3D&reserved=0
> > > > >
> > > > > * Protocol:
> > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2F30%2Fprotocol.html&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FFFwlpeIVyMIYcIulLKjRevvxfzsZu1Hp64XqYrdSBo%3D&reserved=0
> > > > >
> > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > Unit/integration tests:
> > > > >
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci-builds.apache.org%2Fblue%2Forganizations%2Fjenkins%2FKafka%252Fkafka%2Fdetail%2F3.0%2F121%2Fpipeline%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PCQ3NJC0HDoJ4e74qy7gMCrG4t3R%2FAZCgB%2F9YEYvnOs%3D&reserved=0
> > > > > (only few flaky failures)
> > > > > System tests:
> > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjenkins.confluent.io%2Fjob%2Fsystem-test-kafka%2Fjob%2F3.0%2F57%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=G%2FIJVIpMEzaPZmUhOXKFjPz2MwSOl3BKvr2dPrtHvdk%3D&reserved=0
> > > > >
> > > > > /**************************************
> > > > >
> > > > > Thanks,
> > > > > Konstantine
> > > > >
> > > >
> > >
> >
>
Re: [Suspected Spam] Re: [VOTE] 3.0.0 RC1
Posted by Gary Russell <gr...@vmware.com>.
This [1] is a blocker for me.
Also, the documentation link [2] returns a 404.
[1]: https://issues.apache.org/jira/browse/KAFKA-13262
[2]: https://kafka.apache.org/30/documentation.html
________________________________
From: Ron Dagostino <rn...@gmail.com>
Sent: Thursday, September 2, 2021 9:16 PM
To: dev@kafka.apache.org <de...@kafka.apache.org>
Cc: Users <us...@kafka.apache.org>; kafka-clients <ka...@googlegroups.com>
Subject: [Suspected Spam] Re: [VOTE] 3.0.0 RC1
Hi Konstantine. I have opened a probable blocker ticket
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-13270&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DmGTJeScZXxP5WoszUhdqsJodj%2FCiHuFPXhSwkedR%2BI%3D&reserved=0. I will work on a PR
shortly. The description on that ticket is as follows:
The implementation of https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FZOOKEEPER-3593&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5LV9W5CWAQD%2BnnBIcRXXsLUjhN6%2BqQ8ZjSNEZOlprOg%3D&reserved=0 in
ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
problem if Kafka tries to retrieve a large amount of data across many
znodes – in such a case the ZooKeeper client will repeatedly emit a message
of the form "java.io.IOException: Packet len <####> is out of range" and
the Kafka broker will never connect to ZooKeeper and fail to make progress
on the startup sequence. We can avoid the potential for this issue to occur
by explicitly setting the value to 4MB whenever we create a new ZooKeeper
client as long as no explicit value has been set via the `jute.maxbuffer`
system property.
Ron
On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo <is...@gmail.com> wrote:
> Magnus,
>
> Please could you share the machine and network specs?
>
> How much CPU, RAM is available on each node?
>
> What JDK, JRE version are you using?
>
> What are your broker and client configuration values? Please could you
> share this info if possible?
>
> Thanks.
>
>
>
> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > Hi Konstantine,
> >
> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> >
> > * Compaction seems to take slightly longer to kick in when segment sizes
> > exceed their threshold. (Used to take less than 20 seconds, now takes
> > 20..30 seconds.)
> >
> > * CreateTopic seems to take slightly longer to propagate through the
> > cluster,
> > e.g., before a new topic is available in metadata from other brokers.
> >
> > * CreateTopics seems to take longer when the Admin request timeout is
> set,
> > looks like a plateau at 10 seconds:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fimgur.com%2Fa%2Fn6y76sj&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1wARQVJ%2BycVR1GLla70EbZ9o4hPNpmey4wpGcWfzHvI%3D&reserved=0
> >
> > (This is a 3 broker cluster with identical configs between 2.8 and
> 3.0.0.)
> >
> > Nothing critical, but could be an indication of regression so I thought
> I'd
> > mention it.
> >
> > Regards,
> > Magnus
> >
> >
> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > kkarantasis@apache.org>:
> >
> > > Small correction to my previous email.
> > > The actual link for public preview of the 3.0.0 blog post draft is:
> > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.apache.org%2Fpreview%2Fkafka%2F%3FpreviewEntry%3Dwhat-s-new-in-apache6&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2F12fmUzsupiMAZy%2FZktR56j6F7njNiUVqNMo6LG6jvQ%3D&reserved=0
> > >
> > > (see also the email thread with title: [DISCUSS] Please review the
> 3.0.0
> > > blog post)
> > >
> > > Best,
> > > Konstantine
> > >
> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > It corresponds to a major release that includes many new features,
> > > > including:
> > > >
> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > other improvements in the self-managed quorum.
> > > > * Deprecation of message formats v0 and v1.
> > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > default.
> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > Mirror Maker 1.
> > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > Connect.
> > > > * Connector log contexts and connector client overrides are now
> enabled
> > > > by default.
> > > > * Enhanced semantics for timestamp synchronization in Kafka Streams.
> > > > * Revamped public API for Stream's TaskId.
> > > > * Default serde becomes null in Kafka Streams and several
> > > > other configuration changes.
> > > >
> > > > You may read and review a more detailed list of changes in the 3.0.0
> > blog
> > > > post draft here:
> > > >
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.apache.org%2Froller-ui%2Fauthoring%2Fpreview%2Fkafka%2F%3FpreviewEntry%3Dwhat-s-new-in-apache6&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=2FkRhhFa25hPsFpsmmdykpBwJ8gyhjtMprYcDJdYAig%3D&reserved=0
> > > >
> > > > Release notes for the 3.0.0 release:
> > > >
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~kkarantasis%2Fkafka-3.0.0-rc1%2FRELEASE_NOTES.html&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cxER6I5H49zHgOlNtBTazzWynt5tiM81o6XvIrezVPo%3D&reserved=0
> > > >
> > > > *** Please download, test and vote by Wednesday, September 8, 2021
> ***
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYS&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xCODs9z81Ja7zFMJgUey%2BGFce38yD9djSxkV99FxDMg%3D&reserved=0
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~kkarantasis%2Fkafka-3.0.0-rc1%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=WL0A4vvZ8jT5CYopAAGCoXWB9pfaItxEPSCEptA283M%3D&reserved=0
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fkafka%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Lsc0Uz9GcEo9mYHp7OCj2Ii6bMGlcwsA%2BRE2esl%2FKKw%3D&reserved=0
> > > >
> > > > * Javadoc:
> > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~kkarantasis%2Fkafka-3.0.0-rc1%2Fjavadoc%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Tfys%2Fk8rmqZ8gUaQbYvlwU5Lh69%2F4QDs%2FW3bXt4YJZY%3D&reserved=0
> > > >
> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Freleases%2Ftag%2F3.0.0-rc1&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Tisu%2FNQ6EiI7tgWrhXZVfQhJqUqgPnyJtse27ZldG1o%3D&reserved=0
> > > >
> > > > * Documentation:
> > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2F30%2Fdocumentation.html&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9qtwMRr48c2CWYbZYeODw4yenaexqfhzo7Cq8d85LFw%3D&reserved=0
> > > >
> > > > * Protocol:
> > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2F30%2Fprotocol.html&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FFFwlpeIVyMIYcIulLKjRevvxfzsZu1Hp64XqYrdSBo%3D&reserved=0
> > > >
> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > Unit/integration tests:
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci-builds.apache.org%2Fblue%2Forganizations%2Fjenkins%2FKafka%252Fkafka%2Fdetail%2F3.0%2F121%2Fpipeline%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PCQ3NJC0HDoJ4e74qy7gMCrG4t3R%2FAZCgB%2F9YEYvnOs%3D&reserved=0
> > > > (only few flaky failures)
> > > > System tests:
> > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjenkins.confluent.io%2Fjob%2Fsystem-test-kafka%2Fjob%2F3.0%2F57%2F&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287415477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=G%2FIJVIpMEzaPZmUhOXKFjPz2MwSOl3BKvr2dPrtHvdk%3D&reserved=0
> > > >
> > > > /**************************************
> > > >
> > > > Thanks,
> > > > Konstantine
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Hi David,
Thanks for discovering the issue reported in KAFKA-13266. I'm tentatively
approving it if we can review the PR this week, since we are going to have
a new RC due to KAFKA-13270 anyways. This is not a change of a few lines of
code though and while most of it seems to be shuffling around existing code
and has additional test coverage, would you mind commenting on the risk of
this change at this point? I'll be checking the PR reviews as well. Again,
for now, this is approved for 3.0.0 unless we decide otherwise this week.
Konstantine
On Fri, Sep 3, 2021 at 3:45 PM David Jacot <dj...@confluent.io.invalid>
wrote:
> Hi Konstantine,
>
> I'd like to raise a potential blocker:
> https://issues.apache.org/jira/browse/KAFKA-13266.
>
> When this bug is hit, the partition remains failed until the broker is
> restated or
> the partition is updated (e.g. its epoch is bumped). It only affects the
> KRaft mode.
>
> The PR is ready: https://issues.apache.org/jira/browse/KAFKA-13266.
>
> Best,
> David
>
> On Fri, Sep 3, 2021 at 3:17 AM Ron Dagostino <rn...@gmail.com> wrote:
>
> > Hi Konstantine. I have opened a probable blocker ticket
> > https://issues.apache.org/jira/browse/KAFKA-13270. I will work on a PR
> > shortly. The description on that ticket is as follows:
> >
> > The implementation of
> https://issues.apache.org/jira/browse/ZOOKEEPER-3593
> > in
> > ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
> > client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
> > problem if Kafka tries to retrieve a large amount of data across many
> > znodes – in such a case the ZooKeeper client will repeatedly emit a
> message
> > of the form "java.io.IOException: Packet len <####> is out of range" and
> > the Kafka broker will never connect to ZooKeeper and fail to make
> progress
> > on the startup sequence. We can avoid the potential for this issue to
> occur
> > by explicitly setting the value to 4MB whenever we create a new ZooKeeper
> > client as long as no explicit value has been set via the `jute.maxbuffer`
> > system property.
> >
> > Ron
> >
> > On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo <is...@gmail.com> wrote:
> >
> > > Magnus,
> > >
> > > Please could you share the machine and network specs?
> > >
> > > How much CPU, RAM is available on each node?
> > >
> > > What JDK, JRE version are you using?
> > >
> > > What are your broker and client configuration values? Please could you
> > > share this info if possible?
> > >
> > > Thanks.
> > >
> > >
> > >
> > > On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> > > wrote:
> > >
> > > > Hi Konstantine,
> > > >
> > > > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> > > >
> > > > * Compaction seems to take slightly longer to kick in when segment
> > sizes
> > > > exceed their threshold. (Used to take less than 20 seconds, now
> takes
> > > > 20..30 seconds.)
> > > >
> > > > * CreateTopic seems to take slightly longer to propagate through the
> > > > cluster,
> > > > e.g., before a new topic is available in metadata from other
> brokers.
> > > >
> > > > * CreateTopics seems to take longer when the Admin request timeout is
> > > set,
> > > > looks like a plateau at 10 seconds:
> > > > https://imgur.com/a/n6y76sj
> > > >
> > > > (This is a 3 broker cluster with identical configs between 2.8 and
> > > 3.0.0.)
> > > >
> > > > Nothing critical, but could be an indication of regression so I
> thought
> > > I'd
> > > > mention it.
> > > >
> > > > Regards,
> > > > Magnus
> > > >
> > > >
> > > > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > > kkarantasis@apache.org>:
> > > >
> > > > > Small correction to my previous email.
> > > > > The actual link for public preview of the 3.0.0 blog post draft is:
> > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > >
> > > > > (see also the email thread with title: [DISCUSS] Please review the
> > > 3.0.0
> > > > > blog post)
> > > > >
> > > > > Best,
> > > > > Konstantine
> > > > >
> > > > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > > kkarantasis@apache.org> wrote:
> > > > >
> > > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > > > It corresponds to a major release that includes many new
> features,
> > > > > > including:
> > > > > >
> > > > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > > > other improvements in the self-managed quorum.
> > > > > > * Deprecation of message formats v0 and v1.
> > > > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > > > default.
> > > > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > > > Mirror Maker 1.
> > > > > > * Ability to restart a connector's tasks on a single call in
> Kafka
> > > > > Connect.
> > > > > > * Connector log contexts and connector client overrides are now
> > > enabled
> > > > > > by default.
> > > > > > * Enhanced semantics for timestamp synchronization in Kafka
> > Streams.
> > > > > > * Revamped public API for Stream's TaskId.
> > > > > > * Default serde becomes null in Kafka Streams and several
> > > > > > other configuration changes.
> > > > > >
> > > > > > You may read and review a more detailed list of changes in the
> > 3.0.0
> > > > blog
> > > > > > post draft here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > > >
> > > > > > Release notes for the 3.0.0 release:
> > > > > >
> > > >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by Wednesday, September 8,
> 2021
> > > ***
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > > > >
> > > > > > * Documentation:
> > > > > > https://kafka.apache.org/30/documentation.html
> > > > > >
> > > > > > * Protocol:
> > > > > > https://kafka.apache.org/30/protocol.html
> > > > > >
> > > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > > Unit/integration tests:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > > > (only few flaky failures)
> > > > > > System tests:
> > > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > > > >
> > > > > > /**************************************
> > > > > >
> > > > > > Thanks,
> > > > > > Konstantine
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Konstantine,
I'd like to raise a potential blocker:
https://issues.apache.org/jira/browse/KAFKA-13266.
When this bug is hit, the partition remains failed until the broker is
restated or
the partition is updated (e.g. its epoch is bumped). It only affects the
KRaft mode.
The PR is ready: https://issues.apache.org/jira/browse/KAFKA-13266.
Best,
David
On Fri, Sep 3, 2021 at 3:17 AM Ron Dagostino <rn...@gmail.com> wrote:
> Hi Konstantine. I have opened a probable blocker ticket
> https://issues.apache.org/jira/browse/KAFKA-13270. I will work on a PR
> shortly. The description on that ticket is as follows:
>
> The implementation of https://issues.apache.org/jira/browse/ZOOKEEPER-3593
> in
> ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
> client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
> problem if Kafka tries to retrieve a large amount of data across many
> znodes – in such a case the ZooKeeper client will repeatedly emit a message
> of the form "java.io.IOException: Packet len <####> is out of range" and
> the Kafka broker will never connect to ZooKeeper and fail to make progress
> on the startup sequence. We can avoid the potential for this issue to occur
> by explicitly setting the value to 4MB whenever we create a new ZooKeeper
> client as long as no explicit value has been set via the `jute.maxbuffer`
> system property.
>
> Ron
>
> On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Magnus,
> >
> > Please could you share the machine and network specs?
> >
> > How much CPU, RAM is available on each node?
> >
> > What JDK, JRE version are you using?
> >
> > What are your broker and client configuration values? Please could you
> > share this info if possible?
> >
> > Thanks.
> >
> >
> >
> > On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> > wrote:
> >
> > > Hi Konstantine,
> > >
> > > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> > >
> > > * Compaction seems to take slightly longer to kick in when segment
> sizes
> > > exceed their threshold. (Used to take less than 20 seconds, now takes
> > > 20..30 seconds.)
> > >
> > > * CreateTopic seems to take slightly longer to propagate through the
> > > cluster,
> > > e.g., before a new topic is available in metadata from other brokers.
> > >
> > > * CreateTopics seems to take longer when the Admin request timeout is
> > set,
> > > looks like a plateau at 10 seconds:
> > > https://imgur.com/a/n6y76sj
> > >
> > > (This is a 3 broker cluster with identical configs between 2.8 and
> > 3.0.0.)
> > >
> > > Nothing critical, but could be an indication of regression so I thought
> > I'd
> > > mention it.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > kkarantasis@apache.org>:
> > >
> > > > Small correction to my previous email.
> > > > The actual link for public preview of the 3.0.0 blog post draft is:
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >
> > > > (see also the email thread with title: [DISCUSS] Please review the
> > 3.0.0
> > > > blog post)
> > > >
> > > > Best,
> > > > Konstantine
> > > >
> > > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > kkarantasis@apache.org> wrote:
> > > >
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > > It corresponds to a major release that includes many new features,
> > > > > including:
> > > > >
> > > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > > other improvements in the self-managed quorum.
> > > > > * Deprecation of message formats v0 and v1.
> > > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > > default.
> > > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > > Mirror Maker 1.
> > > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > > Connect.
> > > > > * Connector log contexts and connector client overrides are now
> > enabled
> > > > > by default.
> > > > > * Enhanced semantics for timestamp synchronization in Kafka
> Streams.
> > > > > * Revamped public API for Stream's TaskId.
> > > > > * Default serde becomes null in Kafka Streams and several
> > > > > other configuration changes.
> > > > >
> > > > > You may read and review a more detailed list of changes in the
> 3.0.0
> > > blog
> > > > > post draft here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > >
> > > > > Release notes for the 3.0.0 release:
> > > > >
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Wednesday, September 8, 2021
> > ***
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/30/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/30/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > Unit/integration tests:
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > > (only few flaky failures)
> > > > > System tests:
> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > > >
> > > > > /**************************************
> > > > >
> > > > > Thanks,
> > > > > Konstantine
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Hi Ron,
Thanks for reporting this issue. KAFKA-13270 is an obvious regression from
previous versions, so the fix - that I now see has been merged already - is
approved for inclusion in 3.0.0 and I'll add it to the next release
candidate that we'll need to generate for 3.0.0.
Konstantine
On Fri, Sep 3, 2021 at 4:24 AM Ron Dagostino <rn...@gmail.com> wrote:
> Hi Konstantine. I have opened a probable blocker ticket
> https://issues.apache.org/jira/browse/KAFKA-13270. I will work on a PR
> shortly. The description on that ticket is as follows:
>
> The implementation of https://issues.apache.org/jira/browse/ZOOKEEPER-3593
> in
> ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
> client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
> problem if Kafka tries to retrieve a large amount of data across many
> znodes – in such a case the ZooKeeper client will repeatedly emit a message
> of the form "java.io.IOException: Packet len <####> is out of range" and
> the Kafka broker will never connect to ZooKeeper and fail to make progress
> on the startup sequence. We can avoid the potential for this issue to occur
> by explicitly setting the value to 4MB whenever we create a new ZooKeeper
> client as long as no explicit value has been set via the `jute.maxbuffer`
> system property.
>
> Ron
>
> On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Magnus,
> >
> > Please could you share the machine and network specs?
> >
> > How much CPU, RAM is available on each node?
> >
> > What JDK, JRE version are you using?
> >
> > What are your broker and client configuration values? Please could you
> > share this info if possible?
> >
> > Thanks.
> >
> >
> >
> > On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> > wrote:
> >
> > > Hi Konstantine,
> > >
> > > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> > >
> > > * Compaction seems to take slightly longer to kick in when segment
> sizes
> > > exceed their threshold. (Used to take less than 20 seconds, now takes
> > > 20..30 seconds.)
> > >
> > > * CreateTopic seems to take slightly longer to propagate through the
> > > cluster,
> > > e.g., before a new topic is available in metadata from other brokers.
> > >
> > > * CreateTopics seems to take longer when the Admin request timeout is
> > set,
> > > looks like a plateau at 10 seconds:
> > > https://imgur.com/a/n6y76sj
> > >
> > > (This is a 3 broker cluster with identical configs between 2.8 and
> > 3.0.0.)
> > >
> > > Nothing critical, but could be an indication of regression so I thought
> > I'd
> > > mention it.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > kkarantasis@apache.org>:
> > >
> > > > Small correction to my previous email.
> > > > The actual link for public preview of the 3.0.0 blog post draft is:
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >
> > > > (see also the email thread with title: [DISCUSS] Please review the
> > 3.0.0
> > > > blog post)
> > > >
> > > > Best,
> > > > Konstantine
> > > >
> > > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > kkarantasis@apache.org> wrote:
> > > >
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > > It corresponds to a major release that includes many new features,
> > > > > including:
> > > > >
> > > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > > other improvements in the self-managed quorum.
> > > > > * Deprecation of message formats v0 and v1.
> > > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > > default.
> > > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > > Mirror Maker 1.
> > > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > > Connect.
> > > > > * Connector log contexts and connector client overrides are now
> > enabled
> > > > > by default.
> > > > > * Enhanced semantics for timestamp synchronization in Kafka
> Streams.
> > > > > * Revamped public API for Stream's TaskId.
> > > > > * Default serde becomes null in Kafka Streams and several
> > > > > other configuration changes.
> > > > >
> > > > > You may read and review a more detailed list of changes in the
> 3.0.0
> > > blog
> > > > > post draft here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > >
> > > > > Release notes for the 3.0.0 release:
> > > > >
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Wednesday, September 8, 2021
> > ***
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/30/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/30/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > Unit/integration tests:
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > > (only few flaky failures)
> > > > > System tests:
> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > > >
> > > > > /**************************************
> > > > >
> > > > > Thanks,
> > > > > Konstantine
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Konstantine. I have opened a probable blocker ticket
https://issues.apache.org/jira/browse/KAFKA-13270. I will work on a PR
shortly. The description on that ticket is as follows:
The implementation of https://issues.apache.org/jira/browse/ZOOKEEPER-3593 in
ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
problem if Kafka tries to retrieve a large amount of data across many
znodes – in such a case the ZooKeeper client will repeatedly emit a message
of the form "java.io.IOException: Packet len <####> is out of range" and
the Kafka broker will never connect to ZooKeeper and fail to make progress
on the startup sequence. We can avoid the potential for this issue to occur
by explicitly setting the value to 4MB whenever we create a new ZooKeeper
client as long as no explicit value has been set via the `jute.maxbuffer`
system property.
Ron
On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo <is...@gmail.com> wrote:
> Magnus,
>
> Please could you share the machine and network specs?
>
> How much CPU, RAM is available on each node?
>
> What JDK, JRE version are you using?
>
> What are your broker and client configuration values? Please could you
> share this info if possible?
>
> Thanks.
>
>
>
> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > Hi Konstantine,
> >
> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> >
> > * Compaction seems to take slightly longer to kick in when segment sizes
> > exceed their threshold. (Used to take less than 20 seconds, now takes
> > 20..30 seconds.)
> >
> > * CreateTopic seems to take slightly longer to propagate through the
> > cluster,
> > e.g., before a new topic is available in metadata from other brokers.
> >
> > * CreateTopics seems to take longer when the Admin request timeout is
> set,
> > looks like a plateau at 10 seconds:
> > https://imgur.com/a/n6y76sj
> >
> > (This is a 3 broker cluster with identical configs between 2.8 and
> 3.0.0.)
> >
> > Nothing critical, but could be an indication of regression so I thought
> I'd
> > mention it.
> >
> > Regards,
> > Magnus
> >
> >
> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > kkarantasis@apache.org>:
> >
> > > Small correction to my previous email.
> > > The actual link for public preview of the 3.0.0 blog post draft is:
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >
> > > (see also the email thread with title: [DISCUSS] Please review the
> 3.0.0
> > > blog post)
> > >
> > > Best,
> > > Konstantine
> > >
> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > It corresponds to a major release that includes many new features,
> > > > including:
> > > >
> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > other improvements in the self-managed quorum.
> > > > * Deprecation of message formats v0 and v1.
> > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > default.
> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > Mirror Maker 1.
> > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > Connect.
> > > > * Connector log contexts and connector client overrides are now
> enabled
> > > > by default.
> > > > * Enhanced semantics for timestamp synchronization in Kafka Streams.
> > > > * Revamped public API for Stream's TaskId.
> > > > * Default serde becomes null in Kafka Streams and several
> > > > other configuration changes.
> > > >
> > > > You may read and review a more detailed list of changes in the 3.0.0
> > blog
> > > > post draft here:
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >
> > > > Release notes for the 3.0.0 release:
> > > >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Wednesday, September 8, 2021
> ***
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/30/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/30/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > Unit/integration tests:
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > (only few flaky failures)
> > > > System tests:
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > >
> > > > /**************************************
> > > >
> > > > Thanks,
> > > > Konstantine
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Regarding the documentation link which I forgot to answer above, docs are
currently not deployed during preview. The link is generated in this
templated format while running the release scripts.
Gary, for now, you may find the docs for AK 3.0 in:
https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/ as archives in:
https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/kafka_2.12-3.0.0-site-docs.tgz
and
https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/kafka_2.13-3.0.0-site-docs.tgz
Konstantine
On Tue, Sep 7, 2021 at 11:46 AM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
>
> Hi Gary,
>
> Regarding KAFKA-13262, this might need a more detailed and strong
> justification to be considered as a blocker at this point in the release.
> But we are already moving towards RC2 because of the blockers mentioned
> above, so I'd be in favor of a PR that would revert the change that made
> this method final if we could get one this week. I'm not sure why the
> MockConsumer::close method was marked as final in
> https://github.com/apache/kafka/pull/10438. The change is not included in
> the description. Maybe it wasn't obvious that this class is used outside
> this project or maybe we thought we should discourage such use.
>
> Gary, is this the only change you are referring to?
> Ismael, I see you authored the PR originally. Any thoughts on the
> suggested change?
>
> Konstantine
>
> On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
>> Hi Magnus,
>>
>> Thanks for reporting the numbers.
>>
>> I agree that what you observe warrants further investigation, not so much
>> because these are clear performance regressions but because they might be
>> indications of underlying issues, as you noted. However, from a major
>> release standpoint I'd say that such preliminary evidence probably falls
>> within the range of acceptable differences in the presence of major changes
>> and can be optimized over time in subsequent releases.
>>
>> Having said that, I'd take Israel's comment one step further and say that
>> in order to be able to give more definitive answers it'd be good to have a
>> description of the setup that is reproducible. Briefly, that would include
>> both the benchmark setup (specs etc) as well as the benchmark suite and the
>> way to run it (links to benchmark code including configurations and
>> description of how to run and reproduce).
>>
>> Small note that the attached plot is missing units on the x-axis (y-axis
>> I assume is milliseconds from your description).
>>
>> Konstantine
>>
>>
>> On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com> wrote:
>>
>>> Magnus,
>>>
>>> Please could you share the machine and network specs?
>>>
>>> How much CPU, RAM is available on each node?
>>>
>>> What JDK, JRE version are you using?
>>>
>>> What are your broker and client configuration values? Please could you
>>> share this info if possible?
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
>>> wrote:
>>>
>>> > Hi Konstantine,
>>> >
>>> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
>>> >
>>> > * Compaction seems to take slightly longer to kick in when segment
>>> sizes
>>> > exceed their threshold. (Used to take less than 20 seconds, now takes
>>> > 20..30 seconds.)
>>> >
>>> > * CreateTopic seems to take slightly longer to propagate through the
>>> > cluster,
>>> > e.g., before a new topic is available in metadata from other brokers.
>>> >
>>> > * CreateTopics seems to take longer when the Admin request timeout is
>>> set,
>>> > looks like a plateau at 10 seconds:
>>> > https://imgur.com/a/n6y76sj
>>> >
>>> > (This is a 3 broker cluster with identical configs between 2.8 and
>>> 3.0.0.)
>>> >
>>> > Nothing critical, but could be an indication of regression so I
>>> thought I'd
>>> > mention it.
>>> >
>>> > Regards,
>>> > Magnus
>>> >
>>> >
>>> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
>>> > kkarantasis@apache.org>:
>>> >
>>> > > Small correction to my previous email.
>>> > > The actual link for public preview of the 3.0.0 blog post draft is:
>>> > >
>>> > >
>>> >
>>> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
>>> > >
>>> > > (see also the email thread with title: [DISCUSS] Please review the
>>> 3.0.0
>>> > > blog post)
>>> > >
>>> > > Best,
>>> > > Konstantine
>>> > >
>>> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
>>> > > kkarantasis@apache.org> wrote:
>>> > >
>>> > > >
>>> > > > Hello Kafka users, developers and client-developers,
>>> > > >
>>> > > > This is the second release candidate for Apache Kafka 3.0.0.
>>> > > > It corresponds to a major release that includes many new features,
>>> > > > including:
>>> > > >
>>> > > > * The deprecation of support for Java 8 and Scala 2.12.
>>> > > > * Kafka Raft support for snapshots of the metadata topic and
>>> > > > other improvements in the self-managed quorum.
>>> > > > * Deprecation of message formats v0 and v1.
>>> > > > * Stronger delivery guarantees for the Kafka producer enabled by
>>> > default.
>>> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
>>> > > > * More flexible Mirror Maker 2 configuration and deprecation of
>>> > > > Mirror Maker 1.
>>> > > > * Ability to restart a connector's tasks on a single call in Kafka
>>> > > Connect.
>>> > > > * Connector log contexts and connector client overrides are now
>>> enabled
>>> > > > by default.
>>> > > > * Enhanced semantics for timestamp synchronization in Kafka
>>> Streams.
>>> > > > * Revamped public API for Stream's TaskId.
>>> > > > * Default serde becomes null in Kafka Streams and several
>>> > > > other configuration changes.
>>> > > >
>>> > > > You may read and review a more detailed list of changes in the
>>> 3.0.0
>>> > blog
>>> > > > post draft here:
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
>>> > > >
>>> > > > Release notes for the 3.0.0 release:
>>> > > >
>>> >
>>> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
>>> > > >
>>> > > > *** Please download, test and vote by Wednesday, September 8, 2021
>>> ***
>>> > > >
>>> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
>>> > > > https://kafka.apache.org/KEYS
>>> > > >
>>> > > > * Release artifacts to be voted upon (source and binary):
>>> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
>>> > > >
>>> > > > * Maven artifacts to be voted upon:
>>> > > >
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> > > >
>>> > > > * Javadoc:
>>> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
>>> > > >
>>> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
>>> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
>>> > > >
>>> > > > * Documentation:
>>> > > > https://kafka.apache.org/30/documentation.html
>>> > > >
>>> > > > * Protocol:
>>> > > > https://kafka.apache.org/30/protocol.html
>>> > > >
>>> > > > * Successful Jenkins builds for the 3.0 branch:
>>> > > > Unit/integration tests:
>>> > > >
>>> > >
>>> >
>>> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
>>> > > > (only few flaky failures)
>>> > > > System tests:
>>> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
>>> > > >
>>> > > > /**************************************
>>> > > >
>>> > > > Thanks,
>>> > > > Konstantine
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Hi Tom,
It'd be good to avoid a breaking change in a subsequent release that is
related to the modernization of KafkaFuture. Avoiding to use a reference to
an non-public implementation class in the public interfaces seems to help
us with that, as you observed. I'm approving KAFKA-13276 to be included in
RC2 along with the other blockers.
Thanks,
Konstantine
On Tue, Sep 7, 2021 at 11:46 AM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
>
> Hi Gary,
>
> Regarding KAFKA-13262, this might need a more detailed and strong
> justification to be considered as a blocker at this point in the release.
> But we are already moving towards RC2 because of the blockers mentioned
> above, so I'd be in favor of a PR that would revert the change that made
> this method final if we could get one this week. I'm not sure why the
> MockConsumer::close method was marked as final in
> https://github.com/apache/kafka/pull/10438. The change is not included in
> the description. Maybe it wasn't obvious that this class is used outside
> this project or maybe we thought we should discourage such use.
>
> Gary, is this the only change you are referring to?
> Ismael, I see you authored the PR originally. Any thoughts on the
> suggested change?
>
> Konstantine
>
> On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
>> Hi Magnus,
>>
>> Thanks for reporting the numbers.
>>
>> I agree that what you observe warrants further investigation, not so much
>> because these are clear performance regressions but because they might be
>> indications of underlying issues, as you noted. However, from a major
>> release standpoint I'd say that such preliminary evidence probably falls
>> within the range of acceptable differences in the presence of major changes
>> and can be optimized over time in subsequent releases.
>>
>> Having said that, I'd take Israel's comment one step further and say that
>> in order to be able to give more definitive answers it'd be good to have a
>> description of the setup that is reproducible. Briefly, that would include
>> both the benchmark setup (specs etc) as well as the benchmark suite and the
>> way to run it (links to benchmark code including configurations and
>> description of how to run and reproduce).
>>
>> Small note that the attached plot is missing units on the x-axis (y-axis
>> I assume is milliseconds from your description).
>>
>> Konstantine
>>
>>
>> On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com> wrote:
>>
>>> Magnus,
>>>
>>> Please could you share the machine and network specs?
>>>
>>> How much CPU, RAM is available on each node?
>>>
>>> What JDK, JRE version are you using?
>>>
>>> What are your broker and client configuration values? Please could you
>>> share this info if possible?
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
>>> wrote:
>>>
>>> > Hi Konstantine,
>>> >
>>> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
>>> >
>>> > * Compaction seems to take slightly longer to kick in when segment
>>> sizes
>>> > exceed their threshold. (Used to take less than 20 seconds, now takes
>>> > 20..30 seconds.)
>>> >
>>> > * CreateTopic seems to take slightly longer to propagate through the
>>> > cluster,
>>> > e.g., before a new topic is available in metadata from other brokers.
>>> >
>>> > * CreateTopics seems to take longer when the Admin request timeout is
>>> set,
>>> > looks like a plateau at 10 seconds:
>>> > https://imgur.com/a/n6y76sj
>>> >
>>> > (This is a 3 broker cluster with identical configs between 2.8 and
>>> 3.0.0.)
>>> >
>>> > Nothing critical, but could be an indication of regression so I
>>> thought I'd
>>> > mention it.
>>> >
>>> > Regards,
>>> > Magnus
>>> >
>>> >
>>> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
>>> > kkarantasis@apache.org>:
>>> >
>>> > > Small correction to my previous email.
>>> > > The actual link for public preview of the 3.0.0 blog post draft is:
>>> > >
>>> > >
>>> >
>>> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
>>> > >
>>> > > (see also the email thread with title: [DISCUSS] Please review the
>>> 3.0.0
>>> > > blog post)
>>> > >
>>> > > Best,
>>> > > Konstantine
>>> > >
>>> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
>>> > > kkarantasis@apache.org> wrote:
>>> > >
>>> > > >
>>> > > > Hello Kafka users, developers and client-developers,
>>> > > >
>>> > > > This is the second release candidate for Apache Kafka 3.0.0.
>>> > > > It corresponds to a major release that includes many new features,
>>> > > > including:
>>> > > >
>>> > > > * The deprecation of support for Java 8 and Scala 2.12.
>>> > > > * Kafka Raft support for snapshots of the metadata topic and
>>> > > > other improvements in the self-managed quorum.
>>> > > > * Deprecation of message formats v0 and v1.
>>> > > > * Stronger delivery guarantees for the Kafka producer enabled by
>>> > default.
>>> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
>>> > > > * More flexible Mirror Maker 2 configuration and deprecation of
>>> > > > Mirror Maker 1.
>>> > > > * Ability to restart a connector's tasks on a single call in Kafka
>>> > > Connect.
>>> > > > * Connector log contexts and connector client overrides are now
>>> enabled
>>> > > > by default.
>>> > > > * Enhanced semantics for timestamp synchronization in Kafka
>>> Streams.
>>> > > > * Revamped public API for Stream's TaskId.
>>> > > > * Default serde becomes null in Kafka Streams and several
>>> > > > other configuration changes.
>>> > > >
>>> > > > You may read and review a more detailed list of changes in the
>>> 3.0.0
>>> > blog
>>> > > > post draft here:
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
>>> > > >
>>> > > > Release notes for the 3.0.0 release:
>>> > > >
>>> >
>>> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
>>> > > >
>>> > > > *** Please download, test and vote by Wednesday, September 8, 2021
>>> ***
>>> > > >
>>> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
>>> > > > https://kafka.apache.org/KEYS
>>> > > >
>>> > > > * Release artifacts to be voted upon (source and binary):
>>> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
>>> > > >
>>> > > > * Maven artifacts to be voted upon:
>>> > > >
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> > > >
>>> > > > * Javadoc:
>>> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
>>> > > >
>>> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
>>> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
>>> > > >
>>> > > > * Documentation:
>>> > > > https://kafka.apache.org/30/documentation.html
>>> > > >
>>> > > > * Protocol:
>>> > > > https://kafka.apache.org/30/protocol.html
>>> > > >
>>> > > > * Successful Jenkins builds for the 3.0 branch:
>>> > > > Unit/integration tests:
>>> > > >
>>> > >
>>> >
>>> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
>>> > > > (only few flaky failures)
>>> > > > System tests:
>>> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
>>> > > >
>>> > > > /**************************************
>>> > > >
>>> > > > Thanks,
>>> > > > Konstantine
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Thanks for clarifying Ismael.
In that case, your proposed change sounds good to include in RC2 as well.
Konstantine
On Tue, Sep 7, 2021 at 4:27 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Konstantine,
>
> I will remove the final modifier for now.
>
> I added it because the removal of the deprecated `close` overload could
> lead to weird behavior if the no-args `close` was overridden (the
> implementation of the no-arg `close` delegated to the removed `close`, but
> that's no longer the case). However, I didn't realize `MockConsumer` was a
> public class (which is a mistake imo since we tweak its behavior regularly
> and sometimes add methods without a KIP).
>
> Ismael
>
> On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> > Hi Gary,
> >
> > Regarding KAFKA-13262, this might need a more detailed and strong
> > justification to be considered as a blocker at this point in the release.
> > But we are already moving towards RC2 because of the blockers mentioned
> > above, so I'd be in favor of a PR that would revert the change that made
> > this method final if we could get one this week. I'm not sure why the
> > MockConsumer::close method was marked as final in
> > https://github.com/apache/kafka/pull/10438. The change is not included
> in
> > the description. Maybe it wasn't obvious that this class is used outside
> > this project or maybe we thought we should discourage such use.
> >
> > Gary, is this the only change you are referring to?
> > Ismael, I see you authored the PR originally. Any thoughts on the
> suggested
> > change?
> >
> > Konstantine
> >
> > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > > Hi Magnus,
> > >
> > > Thanks for reporting the numbers.
> > >
> > > I agree that what you observe warrants further investigation, not so
> much
> > > because these are clear performance regressions but because they might
> be
> > > indications of underlying issues, as you noted. However, from a major
> > > release standpoint I'd say that such preliminary evidence probably
> falls
> > > within the range of acceptable differences in the presence of major
> > changes
> > > and can be optimized over time in subsequent releases.
> > >
> > > Having said that, I'd take Israel's comment one step further and say
> that
> > > in order to be able to give more definitive answers it'd be good to
> have
> > a
> > > description of the setup that is reproducible. Briefly, that would
> > include
> > > both the benchmark setup (specs etc) as well as the benchmark suite and
> > the
> > > way to run it (links to benchmark code including configurations and
> > > description of how to run and reproduce).
> > >
> > > Small note that the attached plot is missing units on the x-axis
> (y-axis
> > I
> > > assume is milliseconds from your description).
> > >
> > > Konstantine
> > >
> > >
> > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com>
> > wrote:
> > >
> > >> Magnus,
> > >>
> > >> Please could you share the machine and network specs?
> > >>
> > >> How much CPU, RAM is available on each node?
> > >>
> > >> What JDK, JRE version are you using?
> > >>
> > >> What are your broker and client configuration values? Please could you
> > >> share this info if possible?
> > >>
> > >> Thanks.
> > >>
> > >>
> > >>
> > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> > >> wrote:
> > >>
> > >> > Hi Konstantine,
> > >> >
> > >> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> > >> >
> > >> > * Compaction seems to take slightly longer to kick in when segment
> > sizes
> > >> > exceed their threshold. (Used to take less than 20 seconds, now
> > takes
> > >> > 20..30 seconds.)
> > >> >
> > >> > * CreateTopic seems to take slightly longer to propagate through the
> > >> > cluster,
> > >> > e.g., before a new topic is available in metadata from other
> > brokers.
> > >> >
> > >> > * CreateTopics seems to take longer when the Admin request timeout
> is
> > >> set,
> > >> > looks like a plateau at 10 seconds:
> > >> > https://imgur.com/a/n6y76sj
> > >> >
> > >> > (This is a 3 broker cluster with identical configs between 2.8 and
> > >> 3.0.0.)
> > >> >
> > >> > Nothing critical, but could be an indication of regression so I
> > thought
> > >> I'd
> > >> > mention it.
> > >> >
> > >> > Regards,
> > >> > Magnus
> > >> >
> > >> >
> > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > >> > kkarantasis@apache.org>:
> > >> >
> > >> > > Small correction to my previous email.
> > >> > > The actual link for public preview of the 3.0.0 blog post draft
> is:
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >> > >
> > >> > > (see also the email thread with title: [DISCUSS] Please review the
> > >> 3.0.0
> > >> > > blog post)
> > >> > >
> > >> > > Best,
> > >> > > Konstantine
> > >> > >
> > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > >> > > kkarantasis@apache.org> wrote:
> > >> > >
> > >> > > >
> > >> > > > Hello Kafka users, developers and client-developers,
> > >> > > >
> > >> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > >> > > > It corresponds to a major release that includes many new
> features,
> > >> > > > including:
> > >> > > >
> > >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > >> > > > * Kafka Raft support for snapshots of the metadata topic and
> > >> > > > other improvements in the self-managed quorum.
> > >> > > > * Deprecation of message formats v0 and v1.
> > >> > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > >> > default.
> > >> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > >> > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > >> > > > Mirror Maker 1.
> > >> > > > * Ability to restart a connector's tasks on a single call in
> Kafka
> > >> > > Connect.
> > >> > > > * Connector log contexts and connector client overrides are now
> > >> enabled
> > >> > > > by default.
> > >> > > > * Enhanced semantics for timestamp synchronization in Kafka
> > Streams.
> > >> > > > * Revamped public API for Stream's TaskId.
> > >> > > > * Default serde becomes null in Kafka Streams and several
> > >> > > > other configuration changes.
> > >> > > >
> > >> > > > You may read and review a more detailed list of changes in the
> > 3.0.0
> > >> > blog
> > >> > > > post draft here:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >> > > >
> > >> > > > Release notes for the 3.0.0 release:
> > >> > > >
> > >> >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > >> > > >
> > >> > > > *** Please download, test and vote by Wednesday, September 8,
> 2021
> > >> ***
> > >> > > >
> > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > >> > > > https://kafka.apache.org/KEYS
> > >> > > >
> > >> > > > * Release artifacts to be voted upon (source and binary):
> > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > >> > > >
> > >> > > > * Maven artifacts to be voted upon:
> > >> > > >
> > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > > >
> > >> > > > * Javadoc:
> > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > >> > > >
> > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > >> > > >
> > >> > > > * Documentation:
> > >> > > > https://kafka.apache.org/30/documentation.html
> > >> > > >
> > >> > > > * Protocol:
> > >> > > > https://kafka.apache.org/30/protocol.html
> > >> > > >
> > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > >> > > > Unit/integration tests:
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > >> > > > (only few flaky failures)
> > >> > > > System tests:
> > >> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > >> > > >
> > >> > > > /**************************************
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Konstantine
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <k....@gmail.com>.
Thanks David.
Also, somehow I missed Rajini's email on KAFKA-13277. In any case, thanks
Rajini and Colin for fixing this blocker too.
Konstantine
On Wed, Sep 8, 2021 at 7:46 PM David Jacot <dj...@confluent.io.invalid>
wrote:
> Hi Konstantine,
>
> FYI, we just merged https://issues.apache.org/jira/browse/KAFKA-13266 to
> trunk and to 3.0.
>
> Best,
> David
>
> On Tue, Sep 7, 2021 at 11:25 PM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> > Thanks for raising this Colin.
> >
> > https://issues.apache.org/jira/browse/KAFKA-13277
> >
> > is a one line fix with a test now. The suggestion to include it in RC2
> > makes sense to me as well. I’ll make sure it’s in.
> >
> > Konstantine
> >
> > On Tue, Sep 7, 2021 at 11:42 PM Colin McCabe <co...@cmccabe.xyz> wrote:
> >
> > > Hi Konstantine,
> > >
> > > Given that we are making a new RC, I would suggest that we merge
> > > "KAFKA-13277; Fix size calculation for tagged string fields in message
> > > generator" to 3.0. What do you think?
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Sep 7, 2021, at 13:20, Konstantine Karantasis wrote:
> > > > Thanks David,
> > > >
> > > > Assuming we'll have the PRs you mention merged soon, I'll include
> them
> > in
> > > > RC2 for 3.0.0 given their low risk and the fact that we need to
> > generate
> > > a
> > > > new RC anyways.
> > > >
> > > > Konstantine
> > > >
> > > > On Tue, Sep 7, 2021 at 4:43 PM David Jacot
> <djacot@confluent.io.invalid
> > >
> > > > wrote:
> > > >
> > > > > Hi Konstantine,
> > > > >
> > > > > I would like to propose https://github.com/apache/kafka/pull/11300
> > as
> > > a
> > > > > blocker
> > > > > as well. The PR fixes KAFKA-13258/13259/13260. There are all very
> > > small but
> > > > > annoying issues. The PR is trivial.
> > > > >
> > > > > Regarding KAFKA-13266, the fix is quite simple, so low risk in my
> > > opinion.
> > > > > Jason
> > > > > will review it soon.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > > >
> > > > > > Hi Konstantine,
> > > > > >
> > > > > > I will remove the final modifier for now.
> > > > > >
> > > > > > I added it because the removal of the deprecated `close`
> overload
> > > could
> > > > > > lead to weird behavior if the no-args `close` was overridden (the
> > > > > > implementation of the no-arg `close` delegated to the removed
> > > `close`,
> > > > > but
> > > > > > that's no longer the case). However, I didn't realize
> > `MockConsumer`
> > > was
> > > > > a
> > > > > > public class (which is a mistake imo since we tweak its behavior
> > > > > regularly
> > > > > > and sometimes add methods without a KIP).
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> > > > > > kkarantasis@apache.org> wrote:
> > > > > >
> > > > > > > Hi Gary,
> > > > > > >
> > > > > > > Regarding KAFKA-13262, this might need a more detailed and
> strong
> > > > > > > justification to be considered as a blocker at this point in
> the
> > > > > release.
> > > > > > > But we are already moving towards RC2 because of the blockers
> > > mentioned
> > > > > > > above, so I'd be in favor of a PR that would revert the change
> > that
> > > > > made
> > > > > > > this method final if we could get one this week. I'm not sure
> why
> > > the
> > > > > > > MockConsumer::close method was marked as final in
> > > > > > > https://github.com/apache/kafka/pull/10438. The change is not
> > > included
> > > > > > in
> > > > > > > the description. Maybe it wasn't obvious that this class is
> used
> > > > > outside
> > > > > > > this project or maybe we thought we should discourage such use.
> > > > > > >
> > > > > > > Gary, is this the only change you are referring to?
> > > > > > > Ismael, I see you authored the PR originally. Any thoughts on
> the
> > > > > > suggested
> > > > > > > change?
> > > > > > >
> > > > > > > Konstantine
> > > > > > >
> > > > > > > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > > > > > > kkarantasis@apache.org> wrote:
> > > > > > >
> > > > > > > > Hi Magnus,
> > > > > > > >
> > > > > > > > Thanks for reporting the numbers.
> > > > > > > >
> > > > > > > > I agree that what you observe warrants further investigation,
> > > not so
> > > > > > much
> > > > > > > > because these are clear performance regressions but because
> > they
> > > > > might
> > > > > > be
> > > > > > > > indications of underlying issues, as you noted. However,
> from a
> > > major
> > > > > > > > release standpoint I'd say that such preliminary evidence
> > > probably
> > > > > > falls
> > > > > > > > within the range of acceptable differences in the presence of
> > > major
> > > > > > > changes
> > > > > > > > and can be optimized over time in subsequent releases.
> > > > > > > >
> > > > > > > > Having said that, I'd take Israel's comment one step further
> > and
> > > say
> > > > > > that
> > > > > > > > in order to be able to give more definitive answers it'd be
> > good
> > > to
> > > > > > have
> > > > > > > a
> > > > > > > > description of the setup that is reproducible. Briefly, that
> > > would
> > > > > > > include
> > > > > > > > both the benchmark setup (specs etc) as well as the benchmark
> > > suite
> > > > > and
> > > > > > > the
> > > > > > > > way to run it (links to benchmark code including
> configurations
> > > and
> > > > > > > > description of how to run and reproduce).
> > > > > > > >
> > > > > > > > Small note that the attached plot is missing units on the
> > x-axis
> > > > > > (y-axis
> > > > > > > I
> > > > > > > > assume is milliseconds from your description).
> > > > > > > >
> > > > > > > > Konstantine
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <
> > > israelekpo@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Magnus,
> > > > > > > >>
> > > > > > > >> Please could you share the machine and network specs?
> > > > > > > >>
> > > > > > > >> How much CPU, RAM is available on each node?
> > > > > > > >>
> > > > > > > >> What JDK, JRE version are you using?
> > > > > > > >>
> > > > > > > >> What are your broker and client configuration values? Please
> > > could
> > > > > you
> > > > > > > >> share this info if possible?
> > > > > > > >>
> > > > > > > >> Thanks.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <
> > > magnus@edenhill.se
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Konstantine,
> > > > > > > >> >
> > > > > > > >> > Some findings from running 3.0.0-RC1 with the librdkafka
> > test
> > > > > suite:
> > > > > > > >> >
> > > > > > > >> > * Compaction seems to take slightly longer to kick in when
> > > segment
> > > > > > > sizes
> > > > > > > >> > exceed their threshold. (Used to take less than 20
> > seconds,
> > > now
> > > > > > > takes
> > > > > > > >> > 20..30 seconds.)
> > > > > > > >> >
> > > > > > > >> > * CreateTopic seems to take slightly longer to propagate
> > > through
> > > > > the
> > > > > > > >> > cluster,
> > > > > > > >> > e.g., before a new topic is available in metadata from
> > other
> > > > > > > brokers.
> > > > > > > >> >
> > > > > > > >> > * CreateTopics seems to take longer when the Admin request
> > > timeout
> > > > > > is
> > > > > > > >> set,
> > > > > > > >> > looks like a plateau at 10 seconds:
> > > > > > > >> > https://imgur.com/a/n6y76sj
> > > > > > > >> >
> > > > > > > >> > (This is a 3 broker cluster with identical configs between
> > > 2.8 and
> > > > > > > >> 3.0.0.)
> > > > > > > >> >
> > > > > > > >> > Nothing critical, but could be an indication of regression
> > so
> > > I
> > > > > > > thought
> > > > > > > >> I'd
> > > > > > > >> > mention it.
> > > > > > > >> >
> > > > > > > >> > Regards,
> > > > > > > >> > Magnus
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine
> Karantasis <
> > > > > > > >> > kkarantasis@apache.org>:
> > > > > > > >> >
> > > > > > > >> > > Small correction to my previous email.
> > > > > > > >> > > The actual link for public preview of the 3.0.0 blog
> post
> > > draft
> > > > > > is:
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > > > > >> > >
> > > > > > > >> > > (see also the email thread with title: [DISCUSS] Please
> > > review
> > > > > the
> > > > > > > >> 3.0.0
> > > > > > > >> > > blog post)
> > > > > > > >> > >
> > > > > > > >> > > Best,
> > > > > > > >> > > Konstantine
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > > > > >> > > kkarantasis@apache.org> wrote:
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > > Hello Kafka users, developers and client-developers,
> > > > > > > >> > > >
> > > > > > > >> > > > This is the second release candidate for Apache Kafka
> > > 3.0.0.
> > > > > > > >> > > > It corresponds to a major release that includes many
> new
> > > > > > features,
> > > > > > > >> > > > including:
> > > > > > > >> > > >
> > > > > > > >> > > > * The deprecation of support for Java 8 and Scala
> 2.12.
> > > > > > > >> > > > * Kafka Raft support for snapshots of the metadata
> topic
> > > and
> > > > > > > >> > > > other improvements in the self-managed quorum.
> > > > > > > >> > > > * Deprecation of message formats v0 and v1.
> > > > > > > >> > > > * Stronger delivery guarantees for the Kafka producer
> > > enabled
> > > > > by
> > > > > > > >> > default.
> > > > > > > >> > > > * Optimizations in OffsetFetch and FindCoordinator
> > > requests.
> > > > > > > >> > > > * More flexible Mirror Maker 2 configuration and
> > > deprecation
> > > > > of
> > > > > > > >> > > > Mirror Maker 1.
> > > > > > > >> > > > * Ability to restart a connector's tasks on a single
> > call
> > > in
> > > > > > Kafka
> > > > > > > >> > > Connect.
> > > > > > > >> > > > * Connector log contexts and connector client
> overrides
> > > are
> > > > > now
> > > > > > > >> enabled
> > > > > > > >> > > > by default.
> > > > > > > >> > > > * Enhanced semantics for timestamp synchronization in
> > > Kafka
> > > > > > > Streams.
> > > > > > > >> > > > * Revamped public API for Stream's TaskId.
> > > > > > > >> > > > * Default serde becomes null in Kafka Streams and
> > several
> > > > > > > >> > > > other configuration changes.
> > > > > > > >> > > >
> > > > > > > >> > > > You may read and review a more detailed list of
> changes
> > > in the
> > > > > > > 3.0.0
> > > > > > > >> > blog
> > > > > > > >> > > > post draft here:
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > > > > >> > > >
> > > > > > > >> > > > Release notes for the 3.0.0 release:
> > > > > > > >> > > >
> > > > > > > >> >
> > > > > > >
> > > > >
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > > > > >> > > >
> > > > > > > >> > > > *** Please download, test and vote by Wednesday,
> > > September 8,
> > > > > > 2021
> > > > > > > >> ***
> > > > > > > >> > > >
> > > > > > > >> > > > Kafka's KEYS file containing PGP keys we use to sign
> the
> > > > > > release:
> > > > > > > >> > > > https://kafka.apache.org/KEYS
> > > > > > > >> > > >
> > > > > > > >> > > > * Release artifacts to be voted upon (source and
> > binary):
> > > > > > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > > > > > >> > > >
> > > > > > > >> > > > * Maven artifacts to be voted upon:
> > > > > > > >> > > >
> > > > > > > >>
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > >> > > >
> > > > > > > >> > > > * Javadoc:
> > > > > > > >> > > >
> > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > > > > > >> > > >
> > > > > > > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0
> > tag:
> > > > > > > >> > > >
> https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > > > > > >> > > >
> > > > > > > >> > > > * Documentation:
> > > > > > > >> > > > https://kafka.apache.org/30/documentation.html
> > > > > > > >> > > >
> > > > > > > >> > > > * Protocol:
> > > > > > > >> > > > https://kafka.apache.org/30/protocol.html
> > > > > > > >> > > >
> > > > > > > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > > > >> > > > Unit/integration tests:
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > > > > >> > > > (only few flaky failures)
> > > > > > > >> > > > System tests:
> > > > > > > >> > > >
> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > > > > > >> > > >
> > > > > > > >> > > > /**************************************
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > Konstantine
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Konstantine,
FYI, we just merged https://issues.apache.org/jira/browse/KAFKA-13266 to
trunk and to 3.0.
Best,
David
On Tue, Sep 7, 2021 at 11:25 PM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
> Thanks for raising this Colin.
>
> https://issues.apache.org/jira/browse/KAFKA-13277
>
> is a one line fix with a test now. The suggestion to include it in RC2
> makes sense to me as well. I’ll make sure it’s in.
>
> Konstantine
>
> On Tue, Sep 7, 2021 at 11:42 PM Colin McCabe <co...@cmccabe.xyz> wrote:
>
> > Hi Konstantine,
> >
> > Given that we are making a new RC, I would suggest that we merge
> > "KAFKA-13277; Fix size calculation for tagged string fields in message
> > generator" to 3.0. What do you think?
> >
> > best,
> > Colin
> >
> >
> > On Tue, Sep 7, 2021, at 13:20, Konstantine Karantasis wrote:
> > > Thanks David,
> > >
> > > Assuming we'll have the PRs you mention merged soon, I'll include them
> in
> > > RC2 for 3.0.0 given their low risk and the fact that we need to
> generate
> > a
> > > new RC anyways.
> > >
> > > Konstantine
> > >
> > > On Tue, Sep 7, 2021 at 4:43 PM David Jacot <djacot@confluent.io.invalid
> >
> > > wrote:
> > >
> > > > Hi Konstantine,
> > > >
> > > > I would like to propose https://github.com/apache/kafka/pull/11300
> as
> > a
> > > > blocker
> > > > as well. The PR fixes KAFKA-13258/13259/13260. There are all very
> > small but
> > > > annoying issues. The PR is trivial.
> > > >
> > > > Regarding KAFKA-13266, the fix is quite simple, so low risk in my
> > opinion.
> > > > Jason
> > > > will review it soon.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> > > >
> > > > > Hi Konstantine,
> > > > >
> > > > > I will remove the final modifier for now.
> > > > >
> > > > > I added it because the removal of the deprecated `close` overload
> > could
> > > > > lead to weird behavior if the no-args `close` was overridden (the
> > > > > implementation of the no-arg `close` delegated to the removed
> > `close`,
> > > > but
> > > > > that's no longer the case). However, I didn't realize
> `MockConsumer`
> > was
> > > > a
> > > > > public class (which is a mistake imo since we tweak its behavior
> > > > regularly
> > > > > and sometimes add methods without a KIP).
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> > > > > kkarantasis@apache.org> wrote:
> > > > >
> > > > > > Hi Gary,
> > > > > >
> > > > > > Regarding KAFKA-13262, this might need a more detailed and strong
> > > > > > justification to be considered as a blocker at this point in the
> > > > release.
> > > > > > But we are already moving towards RC2 because of the blockers
> > mentioned
> > > > > > above, so I'd be in favor of a PR that would revert the change
> that
> > > > made
> > > > > > this method final if we could get one this week. I'm not sure why
> > the
> > > > > > MockConsumer::close method was marked as final in
> > > > > > https://github.com/apache/kafka/pull/10438. The change is not
> > included
> > > > > in
> > > > > > the description. Maybe it wasn't obvious that this class is used
> > > > outside
> > > > > > this project or maybe we thought we should discourage such use.
> > > > > >
> > > > > > Gary, is this the only change you are referring to?
> > > > > > Ismael, I see you authored the PR originally. Any thoughts on the
> > > > > suggested
> > > > > > change?
> > > > > >
> > > > > > Konstantine
> > > > > >
> > > > > > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > > > > > kkarantasis@apache.org> wrote:
> > > > > >
> > > > > > > Hi Magnus,
> > > > > > >
> > > > > > > Thanks for reporting the numbers.
> > > > > > >
> > > > > > > I agree that what you observe warrants further investigation,
> > not so
> > > > > much
> > > > > > > because these are clear performance regressions but because
> they
> > > > might
> > > > > be
> > > > > > > indications of underlying issues, as you noted. However, from a
> > major
> > > > > > > release standpoint I'd say that such preliminary evidence
> > probably
> > > > > falls
> > > > > > > within the range of acceptable differences in the presence of
> > major
> > > > > > changes
> > > > > > > and can be optimized over time in subsequent releases.
> > > > > > >
> > > > > > > Having said that, I'd take Israel's comment one step further
> and
> > say
> > > > > that
> > > > > > > in order to be able to give more definitive answers it'd be
> good
> > to
> > > > > have
> > > > > > a
> > > > > > > description of the setup that is reproducible. Briefly, that
> > would
> > > > > > include
> > > > > > > both the benchmark setup (specs etc) as well as the benchmark
> > suite
> > > > and
> > > > > > the
> > > > > > > way to run it (links to benchmark code including configurations
> > and
> > > > > > > description of how to run and reproduce).
> > > > > > >
> > > > > > > Small note that the attached plot is missing units on the
> x-axis
> > > > > (y-axis
> > > > > > I
> > > > > > > assume is milliseconds from your description).
> > > > > > >
> > > > > > > Konstantine
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <
> > israelekpo@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Magnus,
> > > > > > >>
> > > > > > >> Please could you share the machine and network specs?
> > > > > > >>
> > > > > > >> How much CPU, RAM is available on each node?
> > > > > > >>
> > > > > > >> What JDK, JRE version are you using?
> > > > > > >>
> > > > > > >> What are your broker and client configuration values? Please
> > could
> > > > you
> > > > > > >> share this info if possible?
> > > > > > >>
> > > > > > >> Thanks.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <
> > magnus@edenhill.se
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Konstantine,
> > > > > > >> >
> > > > > > >> > Some findings from running 3.0.0-RC1 with the librdkafka
> test
> > > > suite:
> > > > > > >> >
> > > > > > >> > * Compaction seems to take slightly longer to kick in when
> > segment
> > > > > > sizes
> > > > > > >> > exceed their threshold. (Used to take less than 20
> seconds,
> > now
> > > > > > takes
> > > > > > >> > 20..30 seconds.)
> > > > > > >> >
> > > > > > >> > * CreateTopic seems to take slightly longer to propagate
> > through
> > > > the
> > > > > > >> > cluster,
> > > > > > >> > e.g., before a new topic is available in metadata from
> other
> > > > > > brokers.
> > > > > > >> >
> > > > > > >> > * CreateTopics seems to take longer when the Admin request
> > timeout
> > > > > is
> > > > > > >> set,
> > > > > > >> > looks like a plateau at 10 seconds:
> > > > > > >> > https://imgur.com/a/n6y76sj
> > > > > > >> >
> > > > > > >> > (This is a 3 broker cluster with identical configs between
> > 2.8 and
> > > > > > >> 3.0.0.)
> > > > > > >> >
> > > > > > >> > Nothing critical, but could be an indication of regression
> so
> > I
> > > > > > thought
> > > > > > >> I'd
> > > > > > >> > mention it.
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> > Magnus
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > > > > >> > kkarantasis@apache.org>:
> > > > > > >> >
> > > > > > >> > > Small correction to my previous email.
> > > > > > >> > > The actual link for public preview of the 3.0.0 blog post
> > draft
> > > > > is:
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > > > >> > >
> > > > > > >> > > (see also the email thread with title: [DISCUSS] Please
> > review
> > > > the
> > > > > > >> 3.0.0
> > > > > > >> > > blog post)
> > > > > > >> > >
> > > > > > >> > > Best,
> > > > > > >> > > Konstantine
> > > > > > >> > >
> > > > > > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > > > >> > > kkarantasis@apache.org> wrote:
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > Hello Kafka users, developers and client-developers,
> > > > > > >> > > >
> > > > > > >> > > > This is the second release candidate for Apache Kafka
> > 3.0.0.
> > > > > > >> > > > It corresponds to a major release that includes many new
> > > > > features,
> > > > > > >> > > > including:
> > > > > > >> > > >
> > > > > > >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > > > >> > > > * Kafka Raft support for snapshots of the metadata topic
> > and
> > > > > > >> > > > other improvements in the self-managed quorum.
> > > > > > >> > > > * Deprecation of message formats v0 and v1.
> > > > > > >> > > > * Stronger delivery guarantees for the Kafka producer
> > enabled
> > > > by
> > > > > > >> > default.
> > > > > > >> > > > * Optimizations in OffsetFetch and FindCoordinator
> > requests.
> > > > > > >> > > > * More flexible Mirror Maker 2 configuration and
> > deprecation
> > > > of
> > > > > > >> > > > Mirror Maker 1.
> > > > > > >> > > > * Ability to restart a connector's tasks on a single
> call
> > in
> > > > > Kafka
> > > > > > >> > > Connect.
> > > > > > >> > > > * Connector log contexts and connector client overrides
> > are
> > > > now
> > > > > > >> enabled
> > > > > > >> > > > by default.
> > > > > > >> > > > * Enhanced semantics for timestamp synchronization in
> > Kafka
> > > > > > Streams.
> > > > > > >> > > > * Revamped public API for Stream's TaskId.
> > > > > > >> > > > * Default serde becomes null in Kafka Streams and
> several
> > > > > > >> > > > other configuration changes.
> > > > > > >> > > >
> > > > > > >> > > > You may read and review a more detailed list of changes
> > in the
> > > > > > 3.0.0
> > > > > > >> > blog
> > > > > > >> > > > post draft here:
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > > > >> > > >
> > > > > > >> > > > Release notes for the 3.0.0 release:
> > > > > > >> > > >
> > > > > > >> >
> > > > > >
> > > >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > > > >> > > >
> > > > > > >> > > > *** Please download, test and vote by Wednesday,
> > September 8,
> > > > > 2021
> > > > > > >> ***
> > > > > > >> > > >
> > > > > > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > > > release:
> > > > > > >> > > > https://kafka.apache.org/KEYS
> > > > > > >> > > >
> > > > > > >> > > > * Release artifacts to be voted upon (source and
> binary):
> > > > > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > > > > >> > > >
> > > > > > >> > > > * Maven artifacts to be voted upon:
> > > > > > >> > > >
> > > > > > >>
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > >> > > >
> > > > > > >> > > > * Javadoc:
> > > > > > >> > > >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > > > > >> > > >
> > > > > > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0
> tag:
> > > > > > >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > > > > >> > > >
> > > > > > >> > > > * Documentation:
> > > > > > >> > > > https://kafka.apache.org/30/documentation.html
> > > > > > >> > > >
> > > > > > >> > > > * Protocol:
> > > > > > >> > > > https://kafka.apache.org/30/protocol.html
> > > > > > >> > > >
> > > > > > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > > >> > > > Unit/integration tests:
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > > > >> > > > (only few flaky failures)
> > > > > > >> > > > System tests:
> > > > > > >> > > >
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > > > > >> > > >
> > > > > > >> > > > /**************************************
> > > > > > >> > > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > Konstantine
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Thanks for raising this Colin.
https://issues.apache.org/jira/browse/KAFKA-13277
is a one line fix with a test now. The suggestion to include it in RC2
makes sense to me as well. I’ll make sure it’s in.
Konstantine
On Tue, Sep 7, 2021 at 11:42 PM Colin McCabe <co...@cmccabe.xyz> wrote:
> Hi Konstantine,
>
> Given that we are making a new RC, I would suggest that we merge
> "KAFKA-13277; Fix size calculation for tagged string fields in message
> generator" to 3.0. What do you think?
>
> best,
> Colin
>
>
> On Tue, Sep 7, 2021, at 13:20, Konstantine Karantasis wrote:
> > Thanks David,
> >
> > Assuming we'll have the PRs you mention merged soon, I'll include them in
> > RC2 for 3.0.0 given their low risk and the fact that we need to generate
> a
> > new RC anyways.
> >
> > Konstantine
> >
> > On Tue, Sep 7, 2021 at 4:43 PM David Jacot <dj...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Konstantine,
> > >
> > > I would like to propose https://github.com/apache/kafka/pull/11300 as
> a
> > > blocker
> > > as well. The PR fixes KAFKA-13258/13259/13260. There are all very
> small but
> > > annoying issues. The PR is trivial.
> > >
> > > Regarding KAFKA-13266, the fix is quite simple, so low risk in my
> opinion.
> > > Jason
> > > will review it soon.
> > >
> > > Best,
> > > David
> > >
> > > On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Konstantine,
> > > >
> > > > I will remove the final modifier for now.
> > > >
> > > > I added it because the removal of the deprecated `close` overload
> could
> > > > lead to weird behavior if the no-args `close` was overridden (the
> > > > implementation of the no-arg `close` delegated to the removed
> `close`,
> > > but
> > > > that's no longer the case). However, I didn't realize `MockConsumer`
> was
> > > a
> > > > public class (which is a mistake imo since we tweak its behavior
> > > regularly
> > > > and sometimes add methods without a KIP).
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> > > > kkarantasis@apache.org> wrote:
> > > >
> > > > > Hi Gary,
> > > > >
> > > > > Regarding KAFKA-13262, this might need a more detailed and strong
> > > > > justification to be considered as a blocker at this point in the
> > > release.
> > > > > But we are already moving towards RC2 because of the blockers
> mentioned
> > > > > above, so I'd be in favor of a PR that would revert the change that
> > > made
> > > > > this method final if we could get one this week. I'm not sure why
> the
> > > > > MockConsumer::close method was marked as final in
> > > > > https://github.com/apache/kafka/pull/10438. The change is not
> included
> > > > in
> > > > > the description. Maybe it wasn't obvious that this class is used
> > > outside
> > > > > this project or maybe we thought we should discourage such use.
> > > > >
> > > > > Gary, is this the only change you are referring to?
> > > > > Ismael, I see you authored the PR originally. Any thoughts on the
> > > > suggested
> > > > > change?
> > > > >
> > > > > Konstantine
> > > > >
> > > > > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > > > > kkarantasis@apache.org> wrote:
> > > > >
> > > > > > Hi Magnus,
> > > > > >
> > > > > > Thanks for reporting the numbers.
> > > > > >
> > > > > > I agree that what you observe warrants further investigation,
> not so
> > > > much
> > > > > > because these are clear performance regressions but because they
> > > might
> > > > be
> > > > > > indications of underlying issues, as you noted. However, from a
> major
> > > > > > release standpoint I'd say that such preliminary evidence
> probably
> > > > falls
> > > > > > within the range of acceptable differences in the presence of
> major
> > > > > changes
> > > > > > and can be optimized over time in subsequent releases.
> > > > > >
> > > > > > Having said that, I'd take Israel's comment one step further and
> say
> > > > that
> > > > > > in order to be able to give more definitive answers it'd be good
> to
> > > > have
> > > > > a
> > > > > > description of the setup that is reproducible. Briefly, that
> would
> > > > > include
> > > > > > both the benchmark setup (specs etc) as well as the benchmark
> suite
> > > and
> > > > > the
> > > > > > way to run it (links to benchmark code including configurations
> and
> > > > > > description of how to run and reproduce).
> > > > > >
> > > > > > Small note that the attached plot is missing units on the x-axis
> > > > (y-axis
> > > > > I
> > > > > > assume is milliseconds from your description).
> > > > > >
> > > > > > Konstantine
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <
> israelekpo@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Magnus,
> > > > > >>
> > > > > >> Please could you share the machine and network specs?
> > > > > >>
> > > > > >> How much CPU, RAM is available on each node?
> > > > > >>
> > > > > >> What JDK, JRE version are you using?
> > > > > >>
> > > > > >> What are your broker and client configuration values? Please
> could
> > > you
> > > > > >> share this info if possible?
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <
> magnus@edenhill.se
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Konstantine,
> > > > > >> >
> > > > > >> > Some findings from running 3.0.0-RC1 with the librdkafka test
> > > suite:
> > > > > >> >
> > > > > >> > * Compaction seems to take slightly longer to kick in when
> segment
> > > > > sizes
> > > > > >> > exceed their threshold. (Used to take less than 20 seconds,
> now
> > > > > takes
> > > > > >> > 20..30 seconds.)
> > > > > >> >
> > > > > >> > * CreateTopic seems to take slightly longer to propagate
> through
> > > the
> > > > > >> > cluster,
> > > > > >> > e.g., before a new topic is available in metadata from other
> > > > > brokers.
> > > > > >> >
> > > > > >> > * CreateTopics seems to take longer when the Admin request
> timeout
> > > > is
> > > > > >> set,
> > > > > >> > looks like a plateau at 10 seconds:
> > > > > >> > https://imgur.com/a/n6y76sj
> > > > > >> >
> > > > > >> > (This is a 3 broker cluster with identical configs between
> 2.8 and
> > > > > >> 3.0.0.)
> > > > > >> >
> > > > > >> > Nothing critical, but could be an indication of regression so
> I
> > > > > thought
> > > > > >> I'd
> > > > > >> > mention it.
> > > > > >> >
> > > > > >> > Regards,
> > > > > >> > Magnus
> > > > > >> >
> > > > > >> >
> > > > > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > > > >> > kkarantasis@apache.org>:
> > > > > >> >
> > > > > >> > > Small correction to my previous email.
> > > > > >> > > The actual link for public preview of the 3.0.0 blog post
> draft
> > > > is:
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > > >> > >
> > > > > >> > > (see also the email thread with title: [DISCUSS] Please
> review
> > > the
> > > > > >> 3.0.0
> > > > > >> > > blog post)
> > > > > >> > >
> > > > > >> > > Best,
> > > > > >> > > Konstantine
> > > > > >> > >
> > > > > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > > >> > > kkarantasis@apache.org> wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > Hello Kafka users, developers and client-developers,
> > > > > >> > > >
> > > > > >> > > > This is the second release candidate for Apache Kafka
> 3.0.0.
> > > > > >> > > > It corresponds to a major release that includes many new
> > > > features,
> > > > > >> > > > including:
> > > > > >> > > >
> > > > > >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > > >> > > > * Kafka Raft support for snapshots of the metadata topic
> and
> > > > > >> > > > other improvements in the self-managed quorum.
> > > > > >> > > > * Deprecation of message formats v0 and v1.
> > > > > >> > > > * Stronger delivery guarantees for the Kafka producer
> enabled
> > > by
> > > > > >> > default.
> > > > > >> > > > * Optimizations in OffsetFetch and FindCoordinator
> requests.
> > > > > >> > > > * More flexible Mirror Maker 2 configuration and
> deprecation
> > > of
> > > > > >> > > > Mirror Maker 1.
> > > > > >> > > > * Ability to restart a connector's tasks on a single call
> in
> > > > Kafka
> > > > > >> > > Connect.
> > > > > >> > > > * Connector log contexts and connector client overrides
> are
> > > now
> > > > > >> enabled
> > > > > >> > > > by default.
> > > > > >> > > > * Enhanced semantics for timestamp synchronization in
> Kafka
> > > > > Streams.
> > > > > >> > > > * Revamped public API for Stream's TaskId.
> > > > > >> > > > * Default serde becomes null in Kafka Streams and several
> > > > > >> > > > other configuration changes.
> > > > > >> > > >
> > > > > >> > > > You may read and review a more detailed list of changes
> in the
> > > > > 3.0.0
> > > > > >> > blog
> > > > > >> > > > post draft here:
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > > >> > > >
> > > > > >> > > > Release notes for the 3.0.0 release:
> > > > > >> > > >
> > > > > >> >
> > > > >
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > > >> > > >
> > > > > >> > > > *** Please download, test and vote by Wednesday,
> September 8,
> > > > 2021
> > > > > >> ***
> > > > > >> > > >
> > > > > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > > release:
> > > > > >> > > > https://kafka.apache.org/KEYS
> > > > > >> > > >
> > > > > >> > > > * Release artifacts to be voted upon (source and binary):
> > > > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > > > >> > > >
> > > > > >> > > > * Maven artifacts to be voted upon:
> > > > > >> > > >
> > > > > >>
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >> > > >
> > > > > >> > > > * Javadoc:
> > > > > >> > > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > > > >> > > >
> > > > > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > > >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > > > >> > > >
> > > > > >> > > > * Documentation:
> > > > > >> > > > https://kafka.apache.org/30/documentation.html
> > > > > >> > > >
> > > > > >> > > > * Protocol:
> > > > > >> > > > https://kafka.apache.org/30/protocol.html
> > > > > >> > > >
> > > > > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > >> > > > Unit/integration tests:
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > > >> > > > (only few flaky failures)
> > > > > >> > > > System tests:
> > > > > >> > > >
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > > > >> > > >
> > > > > >> > > > /**************************************
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Konstantine
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Colin McCabe <co...@cmccabe.xyz>.
Hi Konstantine,
Given that we are making a new RC, I would suggest that we merge "KAFKA-13277; Fix size calculation for tagged string fields in message generator" to 3.0. What do you think?
best,
Colin
On Tue, Sep 7, 2021, at 13:20, Konstantine Karantasis wrote:
> Thanks David,
>
> Assuming we'll have the PRs you mention merged soon, I'll include them in
> RC2 for 3.0.0 given their low risk and the fact that we need to generate a
> new RC anyways.
>
> Konstantine
>
> On Tue, Sep 7, 2021 at 4:43 PM David Jacot <dj...@confluent.io.invalid>
> wrote:
>
> > Hi Konstantine,
> >
> > I would like to propose https://github.com/apache/kafka/pull/11300 as a
> > blocker
> > as well. The PR fixes KAFKA-13258/13259/13260. There are all very small but
> > annoying issues. The PR is trivial.
> >
> > Regarding KAFKA-13266, the fix is quite simple, so low risk in my opinion.
> > Jason
> > will review it soon.
> >
> > Best,
> > David
> >
> > On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Konstantine,
> > >
> > > I will remove the final modifier for now.
> > >
> > > I added it because the removal of the deprecated `close` overload could
> > > lead to weird behavior if the no-args `close` was overridden (the
> > > implementation of the no-arg `close` delegated to the removed `close`,
> > but
> > > that's no longer the case). However, I didn't realize `MockConsumer` was
> > a
> > > public class (which is a mistake imo since we tweak its behavior
> > regularly
> > > and sometimes add methods without a KIP).
> > >
> > > Ismael
> > >
> > > On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > > Hi Gary,
> > > >
> > > > Regarding KAFKA-13262, this might need a more detailed and strong
> > > > justification to be considered as a blocker at this point in the
> > release.
> > > > But we are already moving towards RC2 because of the blockers mentioned
> > > > above, so I'd be in favor of a PR that would revert the change that
> > made
> > > > this method final if we could get one this week. I'm not sure why the
> > > > MockConsumer::close method was marked as final in
> > > > https://github.com/apache/kafka/pull/10438. The change is not included
> > > in
> > > > the description. Maybe it wasn't obvious that this class is used
> > outside
> > > > this project or maybe we thought we should discourage such use.
> > > >
> > > > Gary, is this the only change you are referring to?
> > > > Ismael, I see you authored the PR originally. Any thoughts on the
> > > suggested
> > > > change?
> > > >
> > > > Konstantine
> > > >
> > > > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > > > kkarantasis@apache.org> wrote:
> > > >
> > > > > Hi Magnus,
> > > > >
> > > > > Thanks for reporting the numbers.
> > > > >
> > > > > I agree that what you observe warrants further investigation, not so
> > > much
> > > > > because these are clear performance regressions but because they
> > might
> > > be
> > > > > indications of underlying issues, as you noted. However, from a major
> > > > > release standpoint I'd say that such preliminary evidence probably
> > > falls
> > > > > within the range of acceptable differences in the presence of major
> > > > changes
> > > > > and can be optimized over time in subsequent releases.
> > > > >
> > > > > Having said that, I'd take Israel's comment one step further and say
> > > that
> > > > > in order to be able to give more definitive answers it'd be good to
> > > have
> > > > a
> > > > > description of the setup that is reproducible. Briefly, that would
> > > > include
> > > > > both the benchmark setup (specs etc) as well as the benchmark suite
> > and
> > > > the
> > > > > way to run it (links to benchmark code including configurations and
> > > > > description of how to run and reproduce).
> > > > >
> > > > > Small note that the attached plot is missing units on the x-axis
> > > (y-axis
> > > > I
> > > > > assume is milliseconds from your description).
> > > > >
> > > > > Konstantine
> > > > >
> > > > >
> > > > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com>
> > > > wrote:
> > > > >
> > > > >> Magnus,
> > > > >>
> > > > >> Please could you share the machine and network specs?
> > > > >>
> > > > >> How much CPU, RAM is available on each node?
> > > > >>
> > > > >> What JDK, JRE version are you using?
> > > > >>
> > > > >> What are your broker and client configuration values? Please could
> > you
> > > > >> share this info if possible?
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <magnus@edenhill.se
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Konstantine,
> > > > >> >
> > > > >> > Some findings from running 3.0.0-RC1 with the librdkafka test
> > suite:
> > > > >> >
> > > > >> > * Compaction seems to take slightly longer to kick in when segment
> > > > sizes
> > > > >> > exceed their threshold. (Used to take less than 20 seconds, now
> > > > takes
> > > > >> > 20..30 seconds.)
> > > > >> >
> > > > >> > * CreateTopic seems to take slightly longer to propagate through
> > the
> > > > >> > cluster,
> > > > >> > e.g., before a new topic is available in metadata from other
> > > > brokers.
> > > > >> >
> > > > >> > * CreateTopics seems to take longer when the Admin request timeout
> > > is
> > > > >> set,
> > > > >> > looks like a plateau at 10 seconds:
> > > > >> > https://imgur.com/a/n6y76sj
> > > > >> >
> > > > >> > (This is a 3 broker cluster with identical configs between 2.8 and
> > > > >> 3.0.0.)
> > > > >> >
> > > > >> > Nothing critical, but could be an indication of regression so I
> > > > thought
> > > > >> I'd
> > > > >> > mention it.
> > > > >> >
> > > > >> > Regards,
> > > > >> > Magnus
> > > > >> >
> > > > >> >
> > > > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > > >> > kkarantasis@apache.org>:
> > > > >> >
> > > > >> > > Small correction to my previous email.
> > > > >> > > The actual link for public preview of the 3.0.0 blog post draft
> > > is:
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > >> > >
> > > > >> > > (see also the email thread with title: [DISCUSS] Please review
> > the
> > > > >> 3.0.0
> > > > >> > > blog post)
> > > > >> > >
> > > > >> > > Best,
> > > > >> > > Konstantine
> > > > >> > >
> > > > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > >> > > kkarantasis@apache.org> wrote:
> > > > >> > >
> > > > >> > > >
> > > > >> > > > Hello Kafka users, developers and client-developers,
> > > > >> > > >
> > > > >> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > >> > > > It corresponds to a major release that includes many new
> > > features,
> > > > >> > > > including:
> > > > >> > > >
> > > > >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > >> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > >> > > > other improvements in the self-managed quorum.
> > > > >> > > > * Deprecation of message formats v0 and v1.
> > > > >> > > > * Stronger delivery guarantees for the Kafka producer enabled
> > by
> > > > >> > default.
> > > > >> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > >> > > > * More flexible Mirror Maker 2 configuration and deprecation
> > of
> > > > >> > > > Mirror Maker 1.
> > > > >> > > > * Ability to restart a connector's tasks on a single call in
> > > Kafka
> > > > >> > > Connect.
> > > > >> > > > * Connector log contexts and connector client overrides are
> > now
> > > > >> enabled
> > > > >> > > > by default.
> > > > >> > > > * Enhanced semantics for timestamp synchronization in Kafka
> > > > Streams.
> > > > >> > > > * Revamped public API for Stream's TaskId.
> > > > >> > > > * Default serde becomes null in Kafka Streams and several
> > > > >> > > > other configuration changes.
> > > > >> > > >
> > > > >> > > > You may read and review a more detailed list of changes in the
> > > > 3.0.0
> > > > >> > blog
> > > > >> > > > post draft here:
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> > https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > >> > > >
> > > > >> > > > Release notes for the 3.0.0 release:
> > > > >> > > >
> > > > >> >
> > > >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > >> > > >
> > > > >> > > > *** Please download, test and vote by Wednesday, September 8,
> > > 2021
> > > > >> ***
> > > > >> > > >
> > > > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > release:
> > > > >> > > > https://kafka.apache.org/KEYS
> > > > >> > > >
> > > > >> > > > * Release artifacts to be voted upon (source and binary):
> > > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > > >> > > >
> > > > >> > > > * Maven artifacts to be voted upon:
> > > > >> > > >
> > > > >>
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >> > > >
> > > > >> > > > * Javadoc:
> > > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > > >> > > >
> > > > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > > >> > > >
> > > > >> > > > * Documentation:
> > > > >> > > > https://kafka.apache.org/30/documentation.html
> > > > >> > > >
> > > > >> > > > * Protocol:
> > > > >> > > > https://kafka.apache.org/30/protocol.html
> > > > >> > > >
> > > > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > >> > > > Unit/integration tests:
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> > https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > >> > > > (only few flaky failures)
> > > > >> > > > System tests:
> > > > >> > > >
> > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > > >> > > >
> > > > >> > > > /**************************************
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Konstantine
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Thanks David,
Assuming we'll have the PRs you mention merged soon, I'll include them in
RC2 for 3.0.0 given their low risk and the fact that we need to generate a
new RC anyways.
Konstantine
On Tue, Sep 7, 2021 at 4:43 PM David Jacot <dj...@confluent.io.invalid>
wrote:
> Hi Konstantine,
>
> I would like to propose https://github.com/apache/kafka/pull/11300 as a
> blocker
> as well. The PR fixes KAFKA-13258/13259/13260. There are all very small but
> annoying issues. The PR is trivial.
>
> Regarding KAFKA-13266, the fix is quite simple, so low risk in my opinion.
> Jason
> will review it soon.
>
> Best,
> David
>
> On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Konstantine,
> >
> > I will remove the final modifier for now.
> >
> > I added it because the removal of the deprecated `close` overload could
> > lead to weird behavior if the no-args `close` was overridden (the
> > implementation of the no-arg `close` delegated to the removed `close`,
> but
> > that's no longer the case). However, I didn't realize `MockConsumer` was
> a
> > public class (which is a mistake imo since we tweak its behavior
> regularly
> > and sometimes add methods without a KIP).
> >
> > Ismael
> >
> > On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > > Hi Gary,
> > >
> > > Regarding KAFKA-13262, this might need a more detailed and strong
> > > justification to be considered as a blocker at this point in the
> release.
> > > But we are already moving towards RC2 because of the blockers mentioned
> > > above, so I'd be in favor of a PR that would revert the change that
> made
> > > this method final if we could get one this week. I'm not sure why the
> > > MockConsumer::close method was marked as final in
> > > https://github.com/apache/kafka/pull/10438. The change is not included
> > in
> > > the description. Maybe it wasn't obvious that this class is used
> outside
> > > this project or maybe we thought we should discourage such use.
> > >
> > > Gary, is this the only change you are referring to?
> > > Ismael, I see you authored the PR originally. Any thoughts on the
> > suggested
> > > change?
> > >
> > > Konstantine
> > >
> > > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > > Hi Magnus,
> > > >
> > > > Thanks for reporting the numbers.
> > > >
> > > > I agree that what you observe warrants further investigation, not so
> > much
> > > > because these are clear performance regressions but because they
> might
> > be
> > > > indications of underlying issues, as you noted. However, from a major
> > > > release standpoint I'd say that such preliminary evidence probably
> > falls
> > > > within the range of acceptable differences in the presence of major
> > > changes
> > > > and can be optimized over time in subsequent releases.
> > > >
> > > > Having said that, I'd take Israel's comment one step further and say
> > that
> > > > in order to be able to give more definitive answers it'd be good to
> > have
> > > a
> > > > description of the setup that is reproducible. Briefly, that would
> > > include
> > > > both the benchmark setup (specs etc) as well as the benchmark suite
> and
> > > the
> > > > way to run it (links to benchmark code including configurations and
> > > > description of how to run and reproduce).
> > > >
> > > > Small note that the attached plot is missing units on the x-axis
> > (y-axis
> > > I
> > > > assume is milliseconds from your description).
> > > >
> > > > Konstantine
> > > >
> > > >
> > > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com>
> > > wrote:
> > > >
> > > >> Magnus,
> > > >>
> > > >> Please could you share the machine and network specs?
> > > >>
> > > >> How much CPU, RAM is available on each node?
> > > >>
> > > >> What JDK, JRE version are you using?
> > > >>
> > > >> What are your broker and client configuration values? Please could
> you
> > > >> share this info if possible?
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <magnus@edenhill.se
> >
> > > >> wrote:
> > > >>
> > > >> > Hi Konstantine,
> > > >> >
> > > >> > Some findings from running 3.0.0-RC1 with the librdkafka test
> suite:
> > > >> >
> > > >> > * Compaction seems to take slightly longer to kick in when segment
> > > sizes
> > > >> > exceed their threshold. (Used to take less than 20 seconds, now
> > > takes
> > > >> > 20..30 seconds.)
> > > >> >
> > > >> > * CreateTopic seems to take slightly longer to propagate through
> the
> > > >> > cluster,
> > > >> > e.g., before a new topic is available in metadata from other
> > > brokers.
> > > >> >
> > > >> > * CreateTopics seems to take longer when the Admin request timeout
> > is
> > > >> set,
> > > >> > looks like a plateau at 10 seconds:
> > > >> > https://imgur.com/a/n6y76sj
> > > >> >
> > > >> > (This is a 3 broker cluster with identical configs between 2.8 and
> > > >> 3.0.0.)
> > > >> >
> > > >> > Nothing critical, but could be an indication of regression so I
> > > thought
> > > >> I'd
> > > >> > mention it.
> > > >> >
> > > >> > Regards,
> > > >> > Magnus
> > > >> >
> > > >> >
> > > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > >> > kkarantasis@apache.org>:
> > > >> >
> > > >> > > Small correction to my previous email.
> > > >> > > The actual link for public preview of the 3.0.0 blog post draft
> > is:
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >> > >
> > > >> > > (see also the email thread with title: [DISCUSS] Please review
> the
> > > >> 3.0.0
> > > >> > > blog post)
> > > >> > >
> > > >> > > Best,
> > > >> > > Konstantine
> > > >> > >
> > > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > >> > > kkarantasis@apache.org> wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Hello Kafka users, developers and client-developers,
> > > >> > > >
> > > >> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > >> > > > It corresponds to a major release that includes many new
> > features,
> > > >> > > > including:
> > > >> > > >
> > > >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > >> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > >> > > > other improvements in the self-managed quorum.
> > > >> > > > * Deprecation of message formats v0 and v1.
> > > >> > > > * Stronger delivery guarantees for the Kafka producer enabled
> by
> > > >> > default.
> > > >> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > >> > > > * More flexible Mirror Maker 2 configuration and deprecation
> of
> > > >> > > > Mirror Maker 1.
> > > >> > > > * Ability to restart a connector's tasks on a single call in
> > Kafka
> > > >> > > Connect.
> > > >> > > > * Connector log contexts and connector client overrides are
> now
> > > >> enabled
> > > >> > > > by default.
> > > >> > > > * Enhanced semantics for timestamp synchronization in Kafka
> > > Streams.
> > > >> > > > * Revamped public API for Stream's TaskId.
> > > >> > > > * Default serde becomes null in Kafka Streams and several
> > > >> > > > other configuration changes.
> > > >> > > >
> > > >> > > > You may read and review a more detailed list of changes in the
> > > 3.0.0
> > > >> > blog
> > > >> > > > post draft here:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >> > > >
> > > >> > > > Release notes for the 3.0.0 release:
> > > >> > > >
> > > >> >
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > >> > > >
> > > >> > > > *** Please download, test and vote by Wednesday, September 8,
> > 2021
> > > >> ***
> > > >> > > >
> > > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> > release:
> > > >> > > > https://kafka.apache.org/KEYS
> > > >> > > >
> > > >> > > > * Release artifacts to be voted upon (source and binary):
> > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > >> > > >
> > > >> > > > * Maven artifacts to be voted upon:
> > > >> > > >
> > > >>
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >> > > >
> > > >> > > > * Javadoc:
> > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > >> > > >
> > > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > >> > > >
> > > >> > > > * Documentation:
> > > >> > > > https://kafka.apache.org/30/documentation.html
> > > >> > > >
> > > >> > > > * Protocol:
> > > >> > > > https://kafka.apache.org/30/protocol.html
> > > >> > > >
> > > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > > >> > > > Unit/integration tests:
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > >> > > > (only few flaky failures)
> > > >> > > > System tests:
> > > >> > > >
> https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > >> > > >
> > > >> > > > /**************************************
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Konstantine
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Konstantine,
I would also like to include
https://issues.apache.org/jira/browse/KAFKA-13277 in 3.0 as a blocker. It
is a small fix, will submit PR today.
Thank you,
Rajini
On Tue, Sep 7, 2021 at 2:43 PM David Jacot <dj...@confluent.io.invalid>
wrote:
> Hi Konstantine,
>
> I would like to propose https://github.com/apache/kafka/pull/11300 as a
> blocker
> as well. The PR fixes KAFKA-13258/13259/13260. There are all very small but
> annoying issues. The PR is trivial.
>
> Regarding KAFKA-13266, the fix is quite simple, so low risk in my opinion.
> Jason
> will review it soon.
>
> Best,
> David
>
> On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Konstantine,
> >
> > I will remove the final modifier for now.
> >
> > I added it because the removal of the deprecated `close` overload could
> > lead to weird behavior if the no-args `close` was overridden (the
> > implementation of the no-arg `close` delegated to the removed `close`,
> but
> > that's no longer the case). However, I didn't realize `MockConsumer` was
> a
> > public class (which is a mistake imo since we tweak its behavior
> regularly
> > and sometimes add methods without a KIP).
> >
> > Ismael
> >
> > On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > > Hi Gary,
> > >
> > > Regarding KAFKA-13262, this might need a more detailed and strong
> > > justification to be considered as a blocker at this point in the
> release.
> > > But we are already moving towards RC2 because of the blockers mentioned
> > > above, so I'd be in favor of a PR that would revert the change that
> made
> > > this method final if we could get one this week. I'm not sure why the
> > > MockConsumer::close method was marked as final in
> > > https://github.com/apache/kafka/pull/10438. The change is not included
> > in
> > > the description. Maybe it wasn't obvious that this class is used
> outside
> > > this project or maybe we thought we should discourage such use.
> > >
> > > Gary, is this the only change you are referring to?
> > > Ismael, I see you authored the PR originally. Any thoughts on the
> > suggested
> > > change?
> > >
> > > Konstantine
> > >
> > > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > > Hi Magnus,
> > > >
> > > > Thanks for reporting the numbers.
> > > >
> > > > I agree that what you observe warrants further investigation, not so
> > much
> > > > because these are clear performance regressions but because they
> might
> > be
> > > > indications of underlying issues, as you noted. However, from a major
> > > > release standpoint I'd say that such preliminary evidence probably
> > falls
> > > > within the range of acceptable differences in the presence of major
> > > changes
> > > > and can be optimized over time in subsequent releases.
> > > >
> > > > Having said that, I'd take Israel's comment one step further and say
> > that
> > > > in order to be able to give more definitive answers it'd be good to
> > have
> > > a
> > > > description of the setup that is reproducible. Briefly, that would
> > > include
> > > > both the benchmark setup (specs etc) as well as the benchmark suite
> and
> > > the
> > > > way to run it (links to benchmark code including configurations and
> > > > description of how to run and reproduce).
> > > >
> > > > Small note that the attached plot is missing units on the x-axis
> > (y-axis
> > > I
> > > > assume is milliseconds from your description).
> > > >
> > > > Konstantine
> > > >
> > > >
> > > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com>
> > > wrote:
> > > >
> > > >> Magnus,
> > > >>
> > > >> Please could you share the machine and network specs?
> > > >>
> > > >> How much CPU, RAM is available on each node?
> > > >>
> > > >> What JDK, JRE version are you using?
> > > >>
> > > >> What are your broker and client configuration values? Please could
> you
> > > >> share this info if possible?
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <magnus@edenhill.se
> >
> > > >> wrote:
> > > >>
> > > >> > Hi Konstantine,
> > > >> >
> > > >> > Some findings from running 3.0.0-RC1 with the librdkafka test
> suite:
> > > >> >
> > > >> > * Compaction seems to take slightly longer to kick in when segment
> > > sizes
> > > >> > exceed their threshold. (Used to take less than 20 seconds, now
> > > takes
> > > >> > 20..30 seconds.)
> > > >> >
> > > >> > * CreateTopic seems to take slightly longer to propagate through
> the
> > > >> > cluster,
> > > >> > e.g., before a new topic is available in metadata from other
> > > brokers.
> > > >> >
> > > >> > * CreateTopics seems to take longer when the Admin request timeout
> > is
> > > >> set,
> > > >> > looks like a plateau at 10 seconds:
> > > >> > https://imgur.com/a/n6y76sj
> > > >> >
> > > >> > (This is a 3 broker cluster with identical configs between 2.8 and
> > > >> 3.0.0.)
> > > >> >
> > > >> > Nothing critical, but could be an indication of regression so I
> > > thought
> > > >> I'd
> > > >> > mention it.
> > > >> >
> > > >> > Regards,
> > > >> > Magnus
> > > >> >
> > > >> >
> > > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > >> > kkarantasis@apache.org>:
> > > >> >
> > > >> > > Small correction to my previous email.
> > > >> > > The actual link for public preview of the 3.0.0 blog post draft
> > is:
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >> > >
> > > >> > > (see also the email thread with title: [DISCUSS] Please review
> the
> > > >> 3.0.0
> > > >> > > blog post)
> > > >> > >
> > > >> > > Best,
> > > >> > > Konstantine
> > > >> > >
> > > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > >> > > kkarantasis@apache.org> wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Hello Kafka users, developers and client-developers,
> > > >> > > >
> > > >> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > >> > > > It corresponds to a major release that includes many new
> > features,
> > > >> > > > including:
> > > >> > > >
> > > >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > >> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > >> > > > other improvements in the self-managed quorum.
> > > >> > > > * Deprecation of message formats v0 and v1.
> > > >> > > > * Stronger delivery guarantees for the Kafka producer enabled
> by
> > > >> > default.
> > > >> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > >> > > > * More flexible Mirror Maker 2 configuration and deprecation
> of
> > > >> > > > Mirror Maker 1.
> > > >> > > > * Ability to restart a connector's tasks on a single call in
> > Kafka
> > > >> > > Connect.
> > > >> > > > * Connector log contexts and connector client overrides are
> now
> > > >> enabled
> > > >> > > > by default.
> > > >> > > > * Enhanced semantics for timestamp synchronization in Kafka
> > > Streams.
> > > >> > > > * Revamped public API for Stream's TaskId.
> > > >> > > > * Default serde becomes null in Kafka Streams and several
> > > >> > > > other configuration changes.
> > > >> > > >
> > > >> > > > You may read and review a more detailed list of changes in the
> > > 3.0.0
> > > >> > blog
> > > >> > > > post draft here:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >> > > >
> > > >> > > > Release notes for the 3.0.0 release:
> > > >> > > >
> > > >> >
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > >> > > >
> > > >> > > > *** Please download, test and vote by Wednesday, September 8,
> > 2021
> > > >> ***
> > > >> > > >
> > > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> > release:
> > > >> > > > https://kafka.apache.org/KEYS
> > > >> > > >
> > > >> > > > * Release artifacts to be voted upon (source and binary):
> > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > >> > > >
> > > >> > > > * Maven artifacts to be voted upon:
> > > >> > > >
> > > >>
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >> > > >
> > > >> > > > * Javadoc:
> > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > >> > > >
> > > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > >> > > >
> > > >> > > > * Documentation:
> > > >> > > > https://kafka.apache.org/30/documentation.html
> > > >> > > >
> > > >> > > > * Protocol:
> > > >> > > > https://kafka.apache.org/30/protocol.html
> > > >> > > >
> > > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > > >> > > > Unit/integration tests:
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > >> > > > (only few flaky failures)
> > > >> > > > System tests:
> > > >> > > >
> https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > >> > > >
> > > >> > > > /**************************************
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Konstantine
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Konstantine,
I would like to propose https://github.com/apache/kafka/pull/11300 as a
blocker
as well. The PR fixes KAFKA-13258/13259/13260. There are all very small but
annoying issues. The PR is trivial.
Regarding KAFKA-13266, the fix is quite simple, so low risk in my opinion.
Jason
will review it soon.
Best,
David
On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Konstantine,
>
> I will remove the final modifier for now.
>
> I added it because the removal of the deprecated `close` overload could
> lead to weird behavior if the no-args `close` was overridden (the
> implementation of the no-arg `close` delegated to the removed `close`, but
> that's no longer the case). However, I didn't realize `MockConsumer` was a
> public class (which is a mistake imo since we tweak its behavior regularly
> and sometimes add methods without a KIP).
>
> Ismael
>
> On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> > Hi Gary,
> >
> > Regarding KAFKA-13262, this might need a more detailed and strong
> > justification to be considered as a blocker at this point in the release.
> > But we are already moving towards RC2 because of the blockers mentioned
> > above, so I'd be in favor of a PR that would revert the change that made
> > this method final if we could get one this week. I'm not sure why the
> > MockConsumer::close method was marked as final in
> > https://github.com/apache/kafka/pull/10438. The change is not included
> in
> > the description. Maybe it wasn't obvious that this class is used outside
> > this project or maybe we thought we should discourage such use.
> >
> > Gary, is this the only change you are referring to?
> > Ismael, I see you authored the PR originally. Any thoughts on the
> suggested
> > change?
> >
> > Konstantine
> >
> > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > > Hi Magnus,
> > >
> > > Thanks for reporting the numbers.
> > >
> > > I agree that what you observe warrants further investigation, not so
> much
> > > because these are clear performance regressions but because they might
> be
> > > indications of underlying issues, as you noted. However, from a major
> > > release standpoint I'd say that such preliminary evidence probably
> falls
> > > within the range of acceptable differences in the presence of major
> > changes
> > > and can be optimized over time in subsequent releases.
> > >
> > > Having said that, I'd take Israel's comment one step further and say
> that
> > > in order to be able to give more definitive answers it'd be good to
> have
> > a
> > > description of the setup that is reproducible. Briefly, that would
> > include
> > > both the benchmark setup (specs etc) as well as the benchmark suite and
> > the
> > > way to run it (links to benchmark code including configurations and
> > > description of how to run and reproduce).
> > >
> > > Small note that the attached plot is missing units on the x-axis
> (y-axis
> > I
> > > assume is milliseconds from your description).
> > >
> > > Konstantine
> > >
> > >
> > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com>
> > wrote:
> > >
> > >> Magnus,
> > >>
> > >> Please could you share the machine and network specs?
> > >>
> > >> How much CPU, RAM is available on each node?
> > >>
> > >> What JDK, JRE version are you using?
> > >>
> > >> What are your broker and client configuration values? Please could you
> > >> share this info if possible?
> > >>
> > >> Thanks.
> > >>
> > >>
> > >>
> > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> > >> wrote:
> > >>
> > >> > Hi Konstantine,
> > >> >
> > >> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> > >> >
> > >> > * Compaction seems to take slightly longer to kick in when segment
> > sizes
> > >> > exceed their threshold. (Used to take less than 20 seconds, now
> > takes
> > >> > 20..30 seconds.)
> > >> >
> > >> > * CreateTopic seems to take slightly longer to propagate through the
> > >> > cluster,
> > >> > e.g., before a new topic is available in metadata from other
> > brokers.
> > >> >
> > >> > * CreateTopics seems to take longer when the Admin request timeout
> is
> > >> set,
> > >> > looks like a plateau at 10 seconds:
> > >> > https://imgur.com/a/n6y76sj
> > >> >
> > >> > (This is a 3 broker cluster with identical configs between 2.8 and
> > >> 3.0.0.)
> > >> >
> > >> > Nothing critical, but could be an indication of regression so I
> > thought
> > >> I'd
> > >> > mention it.
> > >> >
> > >> > Regards,
> > >> > Magnus
> > >> >
> > >> >
> > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > >> > kkarantasis@apache.org>:
> > >> >
> > >> > > Small correction to my previous email.
> > >> > > The actual link for public preview of the 3.0.0 blog post draft
> is:
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >> > >
> > >> > > (see also the email thread with title: [DISCUSS] Please review the
> > >> 3.0.0
> > >> > > blog post)
> > >> > >
> > >> > > Best,
> > >> > > Konstantine
> > >> > >
> > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > >> > > kkarantasis@apache.org> wrote:
> > >> > >
> > >> > > >
> > >> > > > Hello Kafka users, developers and client-developers,
> > >> > > >
> > >> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > >> > > > It corresponds to a major release that includes many new
> features,
> > >> > > > including:
> > >> > > >
> > >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > >> > > > * Kafka Raft support for snapshots of the metadata topic and
> > >> > > > other improvements in the self-managed quorum.
> > >> > > > * Deprecation of message formats v0 and v1.
> > >> > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > >> > default.
> > >> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > >> > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > >> > > > Mirror Maker 1.
> > >> > > > * Ability to restart a connector's tasks on a single call in
> Kafka
> > >> > > Connect.
> > >> > > > * Connector log contexts and connector client overrides are now
> > >> enabled
> > >> > > > by default.
> > >> > > > * Enhanced semantics for timestamp synchronization in Kafka
> > Streams.
> > >> > > > * Revamped public API for Stream's TaskId.
> > >> > > > * Default serde becomes null in Kafka Streams and several
> > >> > > > other configuration changes.
> > >> > > >
> > >> > > > You may read and review a more detailed list of changes in the
> > 3.0.0
> > >> > blog
> > >> > > > post draft here:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >> > > >
> > >> > > > Release notes for the 3.0.0 release:
> > >> > > >
> > >> >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > >> > > >
> > >> > > > *** Please download, test and vote by Wednesday, September 8,
> 2021
> > >> ***
> > >> > > >
> > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > >> > > > https://kafka.apache.org/KEYS
> > >> > > >
> > >> > > > * Release artifacts to be voted upon (source and binary):
> > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > >> > > >
> > >> > > > * Maven artifacts to be voted upon:
> > >> > > >
> > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > > >
> > >> > > > * Javadoc:
> > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > >> > > >
> > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > >> > > >
> > >> > > > * Documentation:
> > >> > > > https://kafka.apache.org/30/documentation.html
> > >> > > >
> > >> > > > * Protocol:
> > >> > > > https://kafka.apache.org/30/protocol.html
> > >> > > >
> > >> > > > * Successful Jenkins builds for the 3.0 branch:
> > >> > > > Unit/integration tests:
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > >> > > > (only few flaky failures)
> > >> > > > System tests:
> > >> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > >> > > >
> > >> > > > /**************************************
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Konstantine
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Konstantine,
I will remove the final modifier for now.
I added it because the removal of the deprecated `close` overload could
lead to weird behavior if the no-args `close` was overridden (the
implementation of the no-arg `close` delegated to the removed `close`, but
that's no longer the case). However, I didn't realize `MockConsumer` was a
public class (which is a mistake imo since we tweak its behavior regularly
and sometimes add methods without a KIP).
Ismael
On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
> Hi Gary,
>
> Regarding KAFKA-13262, this might need a more detailed and strong
> justification to be considered as a blocker at this point in the release.
> But we are already moving towards RC2 because of the blockers mentioned
> above, so I'd be in favor of a PR that would revert the change that made
> this method final if we could get one this week. I'm not sure why the
> MockConsumer::close method was marked as final in
> https://github.com/apache/kafka/pull/10438. The change is not included in
> the description. Maybe it wasn't obvious that this class is used outside
> this project or maybe we thought we should discourage such use.
>
> Gary, is this the only change you are referring to?
> Ismael, I see you authored the PR originally. Any thoughts on the suggested
> change?
>
> Konstantine
>
> On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> > Hi Magnus,
> >
> > Thanks for reporting the numbers.
> >
> > I agree that what you observe warrants further investigation, not so much
> > because these are clear performance regressions but because they might be
> > indications of underlying issues, as you noted. However, from a major
> > release standpoint I'd say that such preliminary evidence probably falls
> > within the range of acceptable differences in the presence of major
> changes
> > and can be optimized over time in subsequent releases.
> >
> > Having said that, I'd take Israel's comment one step further and say that
> > in order to be able to give more definitive answers it'd be good to have
> a
> > description of the setup that is reproducible. Briefly, that would
> include
> > both the benchmark setup (specs etc) as well as the benchmark suite and
> the
> > way to run it (links to benchmark code including configurations and
> > description of how to run and reproduce).
> >
> > Small note that the attached plot is missing units on the x-axis (y-axis
> I
> > assume is milliseconds from your description).
> >
> > Konstantine
> >
> >
> > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com>
> wrote:
> >
> >> Magnus,
> >>
> >> Please could you share the machine and network specs?
> >>
> >> How much CPU, RAM is available on each node?
> >>
> >> What JDK, JRE version are you using?
> >>
> >> What are your broker and client configuration values? Please could you
> >> share this info if possible?
> >>
> >> Thanks.
> >>
> >>
> >>
> >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> >> wrote:
> >>
> >> > Hi Konstantine,
> >> >
> >> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> >> >
> >> > * Compaction seems to take slightly longer to kick in when segment
> sizes
> >> > exceed their threshold. (Used to take less than 20 seconds, now
> takes
> >> > 20..30 seconds.)
> >> >
> >> > * CreateTopic seems to take slightly longer to propagate through the
> >> > cluster,
> >> > e.g., before a new topic is available in metadata from other
> brokers.
> >> >
> >> > * CreateTopics seems to take longer when the Admin request timeout is
> >> set,
> >> > looks like a plateau at 10 seconds:
> >> > https://imgur.com/a/n6y76sj
> >> >
> >> > (This is a 3 broker cluster with identical configs between 2.8 and
> >> 3.0.0.)
> >> >
> >> > Nothing critical, but could be an indication of regression so I
> thought
> >> I'd
> >> > mention it.
> >> >
> >> > Regards,
> >> > Magnus
> >> >
> >> >
> >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> >> > kkarantasis@apache.org>:
> >> >
> >> > > Small correction to my previous email.
> >> > > The actual link for public preview of the 3.0.0 blog post draft is:
> >> > >
> >> > >
> >> >
> >>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> >> > >
> >> > > (see also the email thread with title: [DISCUSS] Please review the
> >> 3.0.0
> >> > > blog post)
> >> > >
> >> > > Best,
> >> > > Konstantine
> >> > >
> >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> >> > > kkarantasis@apache.org> wrote:
> >> > >
> >> > > >
> >> > > > Hello Kafka users, developers and client-developers,
> >> > > >
> >> > > > This is the second release candidate for Apache Kafka 3.0.0.
> >> > > > It corresponds to a major release that includes many new features,
> >> > > > including:
> >> > > >
> >> > > > * The deprecation of support for Java 8 and Scala 2.12.
> >> > > > * Kafka Raft support for snapshots of the metadata topic and
> >> > > > other improvements in the self-managed quorum.
> >> > > > * Deprecation of message formats v0 and v1.
> >> > > > * Stronger delivery guarantees for the Kafka producer enabled by
> >> > default.
> >> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> >> > > > * More flexible Mirror Maker 2 configuration and deprecation of
> >> > > > Mirror Maker 1.
> >> > > > * Ability to restart a connector's tasks on a single call in Kafka
> >> > > Connect.
> >> > > > * Connector log contexts and connector client overrides are now
> >> enabled
> >> > > > by default.
> >> > > > * Enhanced semantics for timestamp synchronization in Kafka
> Streams.
> >> > > > * Revamped public API for Stream's TaskId.
> >> > > > * Default serde becomes null in Kafka Streams and several
> >> > > > other configuration changes.
> >> > > >
> >> > > > You may read and review a more detailed list of changes in the
> 3.0.0
> >> > blog
> >> > > > post draft here:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> >> > > >
> >> > > > Release notes for the 3.0.0 release:
> >> > > >
> >> >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> >> > > >
> >> > > > *** Please download, test and vote by Wednesday, September 8, 2021
> >> ***
> >> > > >
> >> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> >> > > > https://kafka.apache.org/KEYS
> >> > > >
> >> > > > * Release artifacts to be voted upon (source and binary):
> >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> >> > > >
> >> > > > * Maven artifacts to be voted upon:
> >> > > >
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >> > > >
> >> > > > * Javadoc:
> >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> >> > > >
> >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> >> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> >> > > >
> >> > > > * Documentation:
> >> > > > https://kafka.apache.org/30/documentation.html
> >> > > >
> >> > > > * Protocol:
> >> > > > https://kafka.apache.org/30/protocol.html
> >> > > >
> >> > > > * Successful Jenkins builds for the 3.0 branch:
> >> > > > Unit/integration tests:
> >> > > >
> >> > >
> >> >
> >>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> >> > > > (only few flaky failures)
> >> > > > System tests:
> >> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> >> > > >
> >> > > > /**************************************
> >> > > >
> >> > > > Thanks,
> >> > > > Konstantine
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Hi Gary,
Regarding KAFKA-13262, this might need a more detailed and strong
justification to be considered as a blocker at this point in the release.
But we are already moving towards RC2 because of the blockers mentioned
above, so I'd be in favor of a PR that would revert the change that made
this method final if we could get one this week. I'm not sure why the
MockConsumer::close method was marked as final in
https://github.com/apache/kafka/pull/10438. The change is not included in
the description. Maybe it wasn't obvious that this class is used outside
this project or maybe we thought we should discourage such use.
Gary, is this the only change you are referring to?
Ismael, I see you authored the PR originally. Any thoughts on the suggested
change?
Konstantine
On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
> Hi Magnus,
>
> Thanks for reporting the numbers.
>
> I agree that what you observe warrants further investigation, not so much
> because these are clear performance regressions but because they might be
> indications of underlying issues, as you noted. However, from a major
> release standpoint I'd say that such preliminary evidence probably falls
> within the range of acceptable differences in the presence of major changes
> and can be optimized over time in subsequent releases.
>
> Having said that, I'd take Israel's comment one step further and say that
> in order to be able to give more definitive answers it'd be good to have a
> description of the setup that is reproducible. Briefly, that would include
> both the benchmark setup (specs etc) as well as the benchmark suite and the
> way to run it (links to benchmark code including configurations and
> description of how to run and reproduce).
>
> Small note that the attached plot is missing units on the x-axis (y-axis I
> assume is milliseconds from your description).
>
> Konstantine
>
>
> On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com> wrote:
>
>> Magnus,
>>
>> Please could you share the machine and network specs?
>>
>> How much CPU, RAM is available on each node?
>>
>> What JDK, JRE version are you using?
>>
>> What are your broker and client configuration values? Please could you
>> share this info if possible?
>>
>> Thanks.
>>
>>
>>
>> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
>> wrote:
>>
>> > Hi Konstantine,
>> >
>> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
>> >
>> > * Compaction seems to take slightly longer to kick in when segment sizes
>> > exceed their threshold. (Used to take less than 20 seconds, now takes
>> > 20..30 seconds.)
>> >
>> > * CreateTopic seems to take slightly longer to propagate through the
>> > cluster,
>> > e.g., before a new topic is available in metadata from other brokers.
>> >
>> > * CreateTopics seems to take longer when the Admin request timeout is
>> set,
>> > looks like a plateau at 10 seconds:
>> > https://imgur.com/a/n6y76sj
>> >
>> > (This is a 3 broker cluster with identical configs between 2.8 and
>> 3.0.0.)
>> >
>> > Nothing critical, but could be an indication of regression so I thought
>> I'd
>> > mention it.
>> >
>> > Regards,
>> > Magnus
>> >
>> >
>> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
>> > kkarantasis@apache.org>:
>> >
>> > > Small correction to my previous email.
>> > > The actual link for public preview of the 3.0.0 blog post draft is:
>> > >
>> > >
>> >
>> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
>> > >
>> > > (see also the email thread with title: [DISCUSS] Please review the
>> 3.0.0
>> > > blog post)
>> > >
>> > > Best,
>> > > Konstantine
>> > >
>> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
>> > > kkarantasis@apache.org> wrote:
>> > >
>> > > >
>> > > > Hello Kafka users, developers and client-developers,
>> > > >
>> > > > This is the second release candidate for Apache Kafka 3.0.0.
>> > > > It corresponds to a major release that includes many new features,
>> > > > including:
>> > > >
>> > > > * The deprecation of support for Java 8 and Scala 2.12.
>> > > > * Kafka Raft support for snapshots of the metadata topic and
>> > > > other improvements in the self-managed quorum.
>> > > > * Deprecation of message formats v0 and v1.
>> > > > * Stronger delivery guarantees for the Kafka producer enabled by
>> > default.
>> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
>> > > > * More flexible Mirror Maker 2 configuration and deprecation of
>> > > > Mirror Maker 1.
>> > > > * Ability to restart a connector's tasks on a single call in Kafka
>> > > Connect.
>> > > > * Connector log contexts and connector client overrides are now
>> enabled
>> > > > by default.
>> > > > * Enhanced semantics for timestamp synchronization in Kafka Streams.
>> > > > * Revamped public API for Stream's TaskId.
>> > > > * Default serde becomes null in Kafka Streams and several
>> > > > other configuration changes.
>> > > >
>> > > > You may read and review a more detailed list of changes in the 3.0.0
>> > blog
>> > > > post draft here:
>> > > >
>> > > >
>> > >
>> >
>> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
>> > > >
>> > > > Release notes for the 3.0.0 release:
>> > > >
>> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
>> > > >
>> > > > *** Please download, test and vote by Wednesday, September 8, 2021
>> ***
>> > > >
>> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > > https://kafka.apache.org/KEYS
>> > > >
>> > > > * Release artifacts to be voted upon (source and binary):
>> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
>> > > >
>> > > > * Maven artifacts to be voted upon:
>> > > >
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > > >
>> > > > * Javadoc:
>> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
>> > > >
>> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
>> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
>> > > >
>> > > > * Documentation:
>> > > > https://kafka.apache.org/30/documentation.html
>> > > >
>> > > > * Protocol:
>> > > > https://kafka.apache.org/30/protocol.html
>> > > >
>> > > > * Successful Jenkins builds for the 3.0 branch:
>> > > > Unit/integration tests:
>> > > >
>> > >
>> >
>> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
>> > > > (only few flaky failures)
>> > > > System tests:
>> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
>> > > >
>> > > > /**************************************
>> > > >
>> > > > Thanks,
>> > > > Konstantine
>> > > >
>> > >
>> >
>>
>
Re: [VOTE] 3.0.0 RC1
Posted by Konstantine Karantasis <kk...@apache.org>.
Hi Magnus,
Thanks for reporting the numbers.
I agree that what you observe warrants further investigation, not so much
because these are clear performance regressions but because they might be
indications of underlying issues, as you noted. However, from a major
release standpoint I'd say that such preliminary evidence probably falls
within the range of acceptable differences in the presence of major changes
and can be optimized over time in subsequent releases.
Having said that, I'd take Israel's comment one step further and say that
in order to be able to give more definitive answers it'd be good to have a
description of the setup that is reproducible. Briefly, that would include
both the benchmark setup (specs etc) as well as the benchmark suite and the
way to run it (links to benchmark code including configurations and
description of how to run and reproduce).
Small note that the attached plot is missing units on the x-axis (y-axis I
assume is milliseconds from your description).
Konstantine
On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo <is...@gmail.com> wrote:
> Magnus,
>
> Please could you share the machine and network specs?
>
> How much CPU, RAM is available on each node?
>
> What JDK, JRE version are you using?
>
> What are your broker and client configuration values? Please could you
> share this info if possible?
>
> Thanks.
>
>
>
> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > Hi Konstantine,
> >
> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> >
> > * Compaction seems to take slightly longer to kick in when segment sizes
> > exceed their threshold. (Used to take less than 20 seconds, now takes
> > 20..30 seconds.)
> >
> > * CreateTopic seems to take slightly longer to propagate through the
> > cluster,
> > e.g., before a new topic is available in metadata from other brokers.
> >
> > * CreateTopics seems to take longer when the Admin request timeout is
> set,
> > looks like a plateau at 10 seconds:
> > https://imgur.com/a/n6y76sj
> >
> > (This is a 3 broker cluster with identical configs between 2.8 and
> 3.0.0.)
> >
> > Nothing critical, but could be an indication of regression so I thought
> I'd
> > mention it.
> >
> > Regards,
> > Magnus
> >
> >
> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > kkarantasis@apache.org>:
> >
> > > Small correction to my previous email.
> > > The actual link for public preview of the 3.0.0 blog post draft is:
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >
> > > (see also the email thread with title: [DISCUSS] Please review the
> 3.0.0
> > > blog post)
> > >
> > > Best,
> > > Konstantine
> > >
> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > It corresponds to a major release that includes many new features,
> > > > including:
> > > >
> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > other improvements in the self-managed quorum.
> > > > * Deprecation of message formats v0 and v1.
> > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > default.
> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > Mirror Maker 1.
> > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > Connect.
> > > > * Connector log contexts and connector client overrides are now
> enabled
> > > > by default.
> > > > * Enhanced semantics for timestamp synchronization in Kafka Streams.
> > > > * Revamped public API for Stream's TaskId.
> > > > * Default serde becomes null in Kafka Streams and several
> > > > other configuration changes.
> > > >
> > > > You may read and review a more detailed list of changes in the 3.0.0
> > blog
> > > > post draft here:
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >
> > > > Release notes for the 3.0.0 release:
> > > >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Wednesday, September 8, 2021
> ***
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/30/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/30/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > Unit/integration tests:
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > (only few flaky failures)
> > > > System tests:
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > >
> > > > /**************************************
> > > >
> > > > Thanks,
> > > > Konstantine
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Konstantine. I have opened a probable blocker ticket
https://issues.apache.org/jira/browse/KAFKA-13270. I will work on a PR
shortly. The description on that ticket is as follows:
The implementation of https://issues.apache.org/jira/browse/ZOOKEEPER-3593 in
ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
problem if Kafka tries to retrieve a large amount of data across many
znodes – in such a case the ZooKeeper client will repeatedly emit a message
of the form "java.io.IOException: Packet len <####> is out of range" and
the Kafka broker will never connect to ZooKeeper and fail to make progress
on the startup sequence. We can avoid the potential for this issue to occur
by explicitly setting the value to 4MB whenever we create a new ZooKeeper
client as long as no explicit value has been set via the `jute.maxbuffer`
system property.
Ron
On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo <is...@gmail.com> wrote:
> Magnus,
>
> Please could you share the machine and network specs?
>
> How much CPU, RAM is available on each node?
>
> What JDK, JRE version are you using?
>
> What are your broker and client configuration values? Please could you
> share this info if possible?
>
> Thanks.
>
>
>
> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > Hi Konstantine,
> >
> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> >
> > * Compaction seems to take slightly longer to kick in when segment sizes
> > exceed their threshold. (Used to take less than 20 seconds, now takes
> > 20..30 seconds.)
> >
> > * CreateTopic seems to take slightly longer to propagate through the
> > cluster,
> > e.g., before a new topic is available in metadata from other brokers.
> >
> > * CreateTopics seems to take longer when the Admin request timeout is
> set,
> > looks like a plateau at 10 seconds:
> > https://imgur.com/a/n6y76sj
> >
> > (This is a 3 broker cluster with identical configs between 2.8 and
> 3.0.0.)
> >
> > Nothing critical, but could be an indication of regression so I thought
> I'd
> > mention it.
> >
> > Regards,
> > Magnus
> >
> >
> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > kkarantasis@apache.org>:
> >
> > > Small correction to my previous email.
> > > The actual link for public preview of the 3.0.0 blog post draft is:
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >
> > > (see also the email thread with title: [DISCUSS] Please review the
> 3.0.0
> > > blog post)
> > >
> > > Best,
> > > Konstantine
> > >
> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > It corresponds to a major release that includes many new features,
> > > > including:
> > > >
> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > other improvements in the self-managed quorum.
> > > > * Deprecation of message formats v0 and v1.
> > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > default.
> > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > Mirror Maker 1.
> > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > Connect.
> > > > * Connector log contexts and connector client overrides are now
> enabled
> > > > by default.
> > > > * Enhanced semantics for timestamp synchronization in Kafka Streams.
> > > > * Revamped public API for Stream's TaskId.
> > > > * Default serde becomes null in Kafka Streams and several
> > > > other configuration changes.
> > > >
> > > > You may read and review a more detailed list of changes in the 3.0.0
> > blog
> > > > post draft here:
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >
> > > > Release notes for the 3.0.0 release:
> > > >
> > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Wednesday, September 8, 2021
> ***
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/30/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/30/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.0 branch:
> > > > Unit/integration tests:
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > > (only few flaky failures)
> > > > System tests:
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > > >
> > > > /**************************************
> > > >
> > > > Thanks,
> > > > Konstantine
> > > >
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Israel Ekpo <is...@gmail.com>.
Magnus,
Please could you share the machine and network specs?
How much CPU, RAM is available on each node?
What JDK, JRE version are you using?
What are your broker and client configuration values? Please could you
share this info if possible?
Thanks.
On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se> wrote:
> Hi Konstantine,
>
> Some findings from running 3.0.0-RC1 with the librdkafka test suite:
>
> * Compaction seems to take slightly longer to kick in when segment sizes
> exceed their threshold. (Used to take less than 20 seconds, now takes
> 20..30 seconds.)
>
> * CreateTopic seems to take slightly longer to propagate through the
> cluster,
> e.g., before a new topic is available in metadata from other brokers.
>
> * CreateTopics seems to take longer when the Admin request timeout is set,
> looks like a plateau at 10 seconds:
> https://imgur.com/a/n6y76sj
>
> (This is a 3 broker cluster with identical configs between 2.8 and 3.0.0.)
>
> Nothing critical, but could be an indication of regression so I thought I'd
> mention it.
>
> Regards,
> Magnus
>
>
> Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> kkarantasis@apache.org>:
>
> > Small correction to my previous email.
> > The actual link for public preview of the 3.0.0 blog post draft is:
> >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> >
> > (see also the email thread with title: [DISCUSS] Please review the 3.0.0
> > blog post)
> >
> > Best,
> > Konstantine
> >
> > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second release candidate for Apache Kafka 3.0.0.
> > > It corresponds to a major release that includes many new features,
> > > including:
> > >
> > > * The deprecation of support for Java 8 and Scala 2.12.
> > > * Kafka Raft support for snapshots of the metadata topic and
> > > other improvements in the self-managed quorum.
> > > * Deprecation of message formats v0 and v1.
> > > * Stronger delivery guarantees for the Kafka producer enabled by
> default.
> > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > Mirror Maker 1.
> > > * Ability to restart a connector's tasks on a single call in Kafka
> > Connect.
> > > * Connector log contexts and connector client overrides are now enabled
> > > by default.
> > > * Enhanced semantics for timestamp synchronization in Kafka Streams.
> > > * Revamped public API for Stream's TaskId.
> > > * Default serde becomes null in Kafka Streams and several
> > > other configuration changes.
> > >
> > > You may read and review a more detailed list of changes in the 3.0.0
> blog
> > > post draft here:
> > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >
> > > Release notes for the 3.0.0 release:
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Wednesday, September 8, 2021 ***
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/30/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/30/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.0 branch:
> > > Unit/integration tests:
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > (only few flaky failures)
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > >
> > > /**************************************
> > >
> > > Thanks,
> > > Konstantine
> > >
> >
>
Re: [VOTE] 3.0.0 RC1
Posted by Israel Ekpo <is...@gmail.com>.
Magnus,
Please could you share the machine and network specs?
How much CPU, RAM is available on each node?
What JDK, JRE version are you using?
What are your broker and client configuration values? Please could you
share this info if possible?
Thanks.
On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill <ma...@edenhill.se> wrote:
> Hi Konstantine,
>
> Some findings from running 3.0.0-RC1 with the librdkafka test suite:
>
> * Compaction seems to take slightly longer to kick in when segment sizes
> exceed their threshold. (Used to take less than 20 seconds, now takes
> 20..30 seconds.)
>
> * CreateTopic seems to take slightly longer to propagate through the
> cluster,
> e.g., before a new topic is available in metadata from other brokers.
>
> * CreateTopics seems to take longer when the Admin request timeout is set,
> looks like a plateau at 10 seconds:
> https://imgur.com/a/n6y76sj
>
> (This is a 3 broker cluster with identical configs between 2.8 and 3.0.0.)
>
> Nothing critical, but could be an indication of regression so I thought I'd
> mention it.
>
> Regards,
> Magnus
>
>
> Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> kkarantasis@apache.org>:
>
> > Small correction to my previous email.
> > The actual link for public preview of the 3.0.0 blog post draft is:
> >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> >
> > (see also the email thread with title: [DISCUSS] Please review the 3.0.0
> > blog post)
> >
> > Best,
> > Konstantine
> >
> > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second release candidate for Apache Kafka 3.0.0.
> > > It corresponds to a major release that includes many new features,
> > > including:
> > >
> > > * The deprecation of support for Java 8 and Scala 2.12.
> > > * Kafka Raft support for snapshots of the metadata topic and
> > > other improvements in the self-managed quorum.
> > > * Deprecation of message formats v0 and v1.
> > > * Stronger delivery guarantees for the Kafka producer enabled by
> default.
> > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > Mirror Maker 1.
> > > * Ability to restart a connector's tasks on a single call in Kafka
> > Connect.
> > > * Connector log contexts and connector client overrides are now enabled
> > > by default.
> > > * Enhanced semantics for timestamp synchronization in Kafka Streams.
> > > * Revamped public API for Stream's TaskId.
> > > * Default serde becomes null in Kafka Streams and several
> > > other configuration changes.
> > >
> > > You may read and review a more detailed list of changes in the 3.0.0
> blog
> > > post draft here:
> > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > >
> > > Release notes for the 3.0.0 release:
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Wednesday, September 8, 2021 ***
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/30/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/30/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.0 branch:
> > > Unit/integration tests:
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/
> > > (only few flaky failures)
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/
> > >
> > > /**************************************
> > >
> > > Thanks,
> > > Konstantine
> > >
> >
>