You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Stanislav Kozlovski <st...@confluent.io.INVALID> on 2024/01/11 18:33:19 UTC

[VOTE] 3.7.0 RC2

Hello Kafka users, developers, and client-developers,

This is the first candidate for release of Apache Kafka 3.7.0.

Note it's named "RC2" because I had a few "failed" RCs that I had
cut/uploaded but ultimately had to scrap prior to announcing due to new
blockers arriving before I could even announce them.

Further - I haven't yet been able to set up the system tests successfully.
And the integration/unit tests do have a few failures that I have to spend
time triaging. I would appreciate any help in case anyone notices any tests
failing that they're subject matters experts in. Expect me to follow up in
a day or two with more detailed analysis.

Major changes include:
- Early Access to KIP-848 - the next generation of the consumer rebalance
protocol
- KIP-858: Adding JBOD support to KRaft
- KIP-714: Observability into Client metrics via a standardized interface

Check more information in the WIP blog post:
https://github.com/apache/kafka-site/pull/578

Release notes for the 3.7.0 release:
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Thursday, January 18, 9am PT ***

Usually these deadlines tend to be 2-3 days, but due to this being the
first RC and the tests not having ran yet, I am giving it a bit more time.

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/

* Docker release artifact to be voted upon:
apache/kafka:3.7.0-rc2

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/

* Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
https://github.com/apache/kafka/releases/tag/3.7.0-rc2

* Documentation:
https://kafka.apache.org/37/documentation.html

* Protocol:
https://kafka.apache.org/37/protocol.html

* Successful Jenkins builds for the 3.7 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
There are failing tests here. I have to follow up with triaging some of the
failures and figuring out if they're actual problems or simply flakes.

System tests: https://jenkins.confluent.io/job/system-test-kafka/job/3.7/

No successful system test runs yet. I am working on getting the job to run.

* Successful Docker Image Github Actions Pipeline for 3.7 branch:
Attached are the scan_report and report_jvm output files from the Docker
Build run:
https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673

And the final docker image build job - Docker Build Test Pipeline:
https://github.com/apache/kafka/actions/runs/7486178277

The image is apache/kafka:3.7.0-rc2 -
https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore

/**************************************

Thanks,
Stanislav Kozlovski

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
Hi Kirk,

Given we are going to have to roll a new RC anyway, and the change is so
simple - might as well get it in!

On Mon, Jan 15, 2024 at 8:26 PM Kirk True <ki...@kirktrue.pro> wrote:

> Hi Stanislav,
>
> On Sun, Jan 14, 2024, at 1:17 PM, Stanislav Kozlovski wrote:
> > Hey Kirk and Chris,
> >
> > Unless I'm missing something - KAFKALESS-16029 is simply a bad log due to
> > improper closing. And the PR description implies this has been present
> > since 3.5. While annoying, I don't see a strong reason for this to block
> > the release.
>
> I would imagine that it would result in concerned users reporting the
> issue.
>
> I took another look, and the code that causes the issue was indeed changed
> in 3.7. It is easily reproducible.
>
> The PR is ready for review: https://github.com/apache/kafka/pull/15186
>
> Thanks,
> Kirk



-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Kirk True <ki...@kirktrue.pro>.
Hi Stanislav,

On Sun, Jan 14, 2024, at 1:17 PM, Stanislav Kozlovski wrote:
> Hey Kirk and Chris,
> 
> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due to
> improper closing. And the PR description implies this has been present
> since 3.5. While annoying, I don't see a strong reason for this to block
> the release.

I would imagine that it would result in concerned users reporting the issue.

I took another look, and the code that causes the issue was indeed changed in 3.7. It is easily reproducible.

The PR is ready for review: https://github.com/apache/kafka/pull/15186

Thanks,
Kirk

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
I wanted to circle back and confirm the integration tests + system tests,
plus give an overall update regarding status.

The integration tests have a fair amount of flakes. I ran and inspected 3
consecutive builds (57
<https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/57/>, 58
<https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/>, 59
<https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/59/>), then
cross-checked each run's failures via a script of mine to see any
consistent failures.

Three tests proved very flaky. Two are related to KIP-848 running under
KRaft. The third one is a Trogdor test. All 3 tests pass locally, hence I
deem them not blockers for the release. Especially since KIP-848 is in
early access, I am not particularly concerned with a flaky test. I opened
three JIRAs to track them:
- https://issues.apache.org/jira/browse/KAFKA-16134
- https://issues.apache.org/jira/browse/KAFKA-16135
- https://issues.apache.org/jira/browse/KAFKA-16136

As for the system tests, I again ran 2 consecutive builds (1
<https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1705045120--apache--3.7--d071cceffc/2024-01-11--001./2024-01-11--001./report.html>,
2
<https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1705113592--apache--3.7--d071cceffc/2024-01-12--001./2024-01-12--001./report.html>)
and I found 4 tests that exhibit consecutive failures.
- The whole analysis: https://hackmd.io/@hOneAGCrSmKSpL8VF-1HWQ/HyRgRJmta

The failing tests:
StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
QuotaTest - https://issues.apache.org/jira/browse/KAFKA-16138
ZookeeperMigrationTest - https://issues.apache.org/jira/browse/KAFKA-16140

I am reaching out to subject matter experts regarding the failures.

Thanks to everyone who contributed in testing the release. Here is a
general update regarding known blockers that were recently found:

We are treating https://issues.apache.org/jira/browse/KAFKA-16131 and
https://issues.apache.org/jira/browse/KAFKA-16101 as blockers.

https://issues.apache.org/jira/browse/KAFKA-16132 is a potential other
issue that will likely be treated as a blocker

Best,
Stanislav

On Mon, Jan 15, 2024 at 12:04 PM Jakub Scholz <ja...@scholz.cz> wrote:

> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
> blocker bug because it *
> *> will generate huge amount of logspam. I guess we didn't find it in junit
> tests *
> *> since logspam doesn't fail the automated tests. But certainly it's not
> suitable *
> *> for production. Did you file a JIRA yet?*
>
> Hi Colin,
>
> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>
> Thanks & Regards
> Jakub
>
> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for making the first RC. The fact that it's titled RC2 is messing
> > with my mind a bit. I hope this doesn't make people think that we're
> > farther along than we are, heh.
> >
> > On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > > *> Nice catch! It does seem like we should have gated this behind the
> > > metadata> version as KIP-858 implies. Is the cluster configured with
> > > multiple log> dirs? What is the impact of the error messages?*
> > >
> > > I did not observe any obvious impact. I was able to send and receive
> > > messages as normally. But to be honest, I have no idea what else
> > > this might impact, so I did not try anything special.
> > >
> > > I think everyone upgrading an existing KRaft cluster will go through
> this
> > > stage (running Kafka 3.7 with an older metadata version for at least a
> > > while). So even if it is just a logged exception without any other
> > impact I
> > > wonder if it might scare users from upgrading. But I leave it to others
> > to
> > > decide if this is a blocker or not.
> > >
> >
> > Hi Jakub,
> >
> > Thanks for trying the RC. I think what you found is a blocker bug because
> > it will generate huge amount of logspam. I guess we didn't find it in
> junit
> > tests since logspam doesn't fail the automated tests. But certainly it's
> > not suitable for production. Did you file a JIRA yet?
> >
> > > On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > > <st...@confluent.io.invalid> wrote:
> > >
> > >> Hey Luke,
> > >>
> > >> This is an interesting problem. Given the fact that the KIP for
> having a
> > >> 3.8 release passed, I think it weights the scale towards not calling
> > this a
> > >> blocker and expecting it to be solved in 3.7.1.
> > >>
> > >> It is unfortunate that it would not seem safe to migrate to KRaft in
> > 3.7.0
> > >> (given the inability to rollback safely), but if that's true - the
> same
> > >> case would apply for 3.6.0. So in any case users w\ould be expected to
> > use a
> > >> patch release for this.
> >
> > Hi Luke,
> >
> > Thanks for testing rollback. I think this is a case where the
> > documentation is wrong. The intention was to for the steps to basically
> be:
> >
> > 1. roll all the brokers into zk mode, but with migration enabled
> > 2. take down the kraft quorum
> > 3. rmr /controller, allowing a hybrid broker to take over.
> > 4. roll all the brokers into zk mode without migration enabled (if
> desired)
> >
> > With these steps, there isn't really unavailability since a ZK controller
> > can be elected quickly after the kraft quorum is gone.
> >
> > >> Further, since we will have a 3.8 release - it is
> > >> likely we will ultimately recommend users upgrade from that version
> > given
> > >> its aim is to have strategic KRaft feature parity with ZK.
> > >> That being said, I am not 100% on this. Let me know whether you think
> > this
> > >> should block the release, Luke. I am also tagging Colin and David to
> > weigh
> > >> in with their opinions, as they worked on the migration logic.
> >
> > The rollback docs are new in 3.7 so the fact that they're wrong is a
> clear
> > blocker, I think. But easy to fix, I believe. I will create a PR.
> >
> > best,
> > Colin
> >
> > >>
> > >> Hey Kirk and Chris,
> > >>
> > >> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due
> > to
> > >> improper closing. And the PR description implies this has been present
> > >> since 3.5. While annoying, I don't see a strong reason for this to
> block
> > >> the release.
> > >>
> > >> Hey Jakub,
> > >>
> > >> Nice catch! It does seem like we should have gated this behind the
> > metadata
> > >> version as KIP-858 implies. Is the cluster configured with multiple
> log
> > >> dirs? What is the impact of the error messages?
> > >>
> > >> Tagging Igor (the author of the KIP) to weigh in.
> > >>
> > >> Best,
> > >> Stanislav
> > >>
> > >> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I was trying the RC2 and run into the following issue ... when I run
> > >> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> metadata
> > >> > version, I seem to be getting repeated errors like this in the
> > controller
> > >> > logs:
> > >> >
> > >> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > >> assignReplicasToDirs:
> > >> > event failed with UnsupportedVersionException in 15 microseconds.
> > >> > (org.apache.kafka.controller.QuorumController)
> > >> > [quorum-controller-0-event-handler]
> > >> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
> > error
> > >> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> > >> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > >> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > >> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > >> > partitions=[PartitionData(partitionIndex=2),
> > >> > PartitionData(partitionIndex=1)]),
> > >> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > >> > partitions=[PartitionData(partitionIndex=0)])])]) with context
> > >> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> > >> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > clientAddress=/
> > >> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > >> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> > >> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > >> > softwareVersion=3.7.0), fromPrivilegedListener=false,
> > >> >
> > >> >
> > >>
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > >> > ])
> > >> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> > >> > java.util.concurrent.CompletionException:
> > >> > org.apache.kafka.common.errors.UnsupportedVersionException:
> Directory
> > >> > assignment is not supported yet.
> > >> >
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > >> >  at java.base/java.lang.Thread.run(Thread.java:840)
> > >> >
> > >> > Caused by:
> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >> > Directory assignment is not supported yet.
> > >> >
> > >> > Is that expected? I guess with the metadata version set to 3.6-IV2,
> it
> > >> > makes sense that the request is not supported. But shouldn't then
> the
> > >> > request not be sent at all by the brokers? (I did not opened a JIRA
> > for
> > >> it,
> > >> > but I can open one if you agree this is not expected)
> > >> >
> > >> > Thanks & Regards
> > >> > Jakub
> > >> >
> > >> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> wrote:
> > >> >
> > >> > > Hi Stanislav,
> > >> > >
> > >> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
> > you
> > >> > > missed it.
> > >> > > cross-posting here:
> > >> > >
> > >> > > There is a bug KAFKA-16101
> > >> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> that
> > >> > "Kafka
> > >> > > cluster will be unavailable during KRaft migration rollback".
> > >> > > The impact for this issue is that if brokers try to rollback to ZK
> > mode
> > >> > > during KRaft migration process, there will be a period of time the
> > >> > cluster
> > >> > > is unavailable.
> > >> > > Since ZK migrating to KRaft feature is a production ready
> feature, I
> > >> > think
> > >> > > this should be addressed soon.
> > >> > > Do you think this is a blocker for v3.7.0?
> > >> > >
> > >> > > Thanks.
> > >> > > Luke
> > >> > >
> > >> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > fearthecellos@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks, Kirk!
> > >> > > >
> > >> > > > @Stanislav--do you believe that this warrants a new RC?
> > >> > > >
> > >> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> wrote:
> > >> > > >
> > >> > > > > Hi Chris/Stanislav,
> > >> > > > >
> > >> > > > > I'm working on the 'Unable to find FetchSessionHandler' log
> > problem
> > >> > > > > (KAFKA-16029) and have put out a draft PR (
> > >> > > > > https://github.com/apache/kafka/pull/15186). I will use the
> > >> > quickstart
> > >> > > > > approach as a second means to reproduce/verify while I wait
> for
> > the
> > >> > > PR's
> > >> > > > > Jenkins job to finish.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Kirk
> > >> > > > >
> > >> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > >> > > > > > Hi Stanislav,
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks for running this release!
> > >> > > > > >
> > >> > > > > > To verify, I:
> > >> > > > > > - Built from source using Java 11 with both:
> > >> > > > > > - - the 3.7.0-rc2 tag on GitHub
> > >> > > > > > - - the kafka-3.7.0-src.tgz artifact from
> > >> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> > > > > > - Checked signatures and checksums
> > >> > > > > > - Ran the quickstart using both:
> > >> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> > >> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> with
> > >> > > Java
> > >> > > > > 11
> > >> > > > > > and Scala 13 in KRaft mode
> > >> > > > > > - - Our shiny new broker Docker image,
> apache/kafka:3.7.0-rc2
> > >> > > > > > - Ran all unit tests
> > >> > > > > > - Ran all integration tests for Connect and MM2
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > I found two minor areas for concern:
> > >> > > > > >
> > >> > > > > > 1. (Possibly a blocker)
> > >> > > > > > When running the quickstart, I noticed this ERROR-level log
> > >> message
> > >> > > > being
> > >> > > > > > emitted frequently (not not every time) when I killed my
> > console
> > >> > > > consumer
> > >> > > > > > via ctrl-C:
> > >> > > > > >
> > >> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> > >> > > clientId=console-consumer,
> > >> > > > > > groupId=console-consumer-74388] Unable to find
> > >> FetchSessionHandler
> > >> > > for
> > >> > > > > node
> > >> > > > > > 1. Ignoring fetch response
> > >> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >> > > > > >
> > >> > > > > > I see that this error message is already reported in
> > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think
> we
> > >> > should
> > >> > > > > > prioritize fixing it for this release. I know it's probably
> > >> benign
> > >> > > but
> > >> > > > > it's
> > >> > > > > > really not a good look for us when basic operations log
> error
> > >> > > messages,
> > >> > > > > and
> > >> > > > > > it may give new users some headaches.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > 2. (Probably not a blocker)
> > >> > > > > > The following unit tests failed the first time around, and
> > all of
> > >> > > them
> > >> > > > > > passed the second time I ran them:
> > >> > > > > >
> > >> > > > > > - (clients)
> > >> > > > >
> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > >> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > >> > > > > > - (clients)
> Tls13SelectorTest.testConnectionsByClientMetric()
> > >> > > > > > - (connect)
> > >> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > >> > > > (I
> > >> > > > > > thought I fixed this one! 🤬🤬)
> > >> > > > > > - (core)
> > ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks again for your work on this release, and
> > congratulations
> > >> to
> > >> > > > Kafka
> > >> > > > > > Streams for having zero flaky unit tests during my
> > >> > > highly-experimental
> > >> > > > > > single laptop run!
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Cheers,
> > >> > > > > >
> > >> > > > > > Chris
> > >> > > > > >
> > >> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > >> > > > > > <st...@confluent.io.invalid> wrote:
> > >> > > > > >
> > >> > > > > > > Hello Kafka users, developers, and client-developers,
> > >> > > > > > >
> > >> > > > > > > This is the first candidate for release of Apache Kafka
> > 3.7.0.
> > >> > > > > > >
> > >> > > > > > > Note it's named "RC2" because I had a few "failed" RCs
> that
> > I
> > >> had
> > >> > > > > > > cut/uploaded but ultimately had to scrap prior to
> announcing
> > >> due
> > >> > to
> > >> > > > new
> > >> > > > > > > blockers arriving before I could even announce them.
> > >> > > > > > >
> > >> > > > > > > Further - I haven't yet been able to set up the system
> tests
> > >> > > > > successfully.
> > >> > > > > > > And the integration/unit tests do have a few failures
> that I
> > >> have
> > >> > > to
> > >> > > > > spend
> > >> > > > > > > time triaging. I would appreciate any help in case anyone
> > >> notices
> > >> > > any
> > >> > > > > tests
> > >> > > > > > > failing that they're subject matters experts in. Expect me
> > to
> > >> > > follow
> > >> > > > > up in
> > >> > > > > > > a day or two with more detailed analysis.
> > >> > > > > > >
> > >> > > > > > > Major changes include:
> > >> > > > > > > - Early Access to KIP-848 - the next generation of the
> > consumer
> > >> > > > > rebalance
> > >> > > > > > > protocol
> > >> > > > > > > - KIP-858: Adding JBOD support to KRaft
> > >> > > > > > > - KIP-714: Observability into Client metrics via a
> > standardized
> > >> > > > > interface
> > >> > > > > > >
> > >> > > > > > > Check more information in the WIP blog post:
> > >> > > > > > > https://github.com/apache/kafka-site/pull/578
> > >> > > > > > >
> > >> > > > > > > Release notes for the 3.7.0 release:
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >> > > > > > >
> > >> > > > > > > *** Please download, test and vote by Thursday, January
> 18,
> > 9am
> > >> > PT
> > >> > > > ***
> > >> > > > > > >
> > >> > > > > > > Usually these deadlines tend to be 2-3 days, but due to
> this
> > >> > being
> > >> > > > the
> > >> > > > > > > first RC and the tests not having ran yet, I am giving it
> a
> > bit
> > >> > > more
> > >> > > > > time.
> > >> > > > > > >
> > >> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > >> release:
> > >> > > > > > > https://kafka.apache.org/KEYS
> > >> > > > > > >
> > >> > > > > > > * Release artifacts to be voted upon (source and binary):
> > >> > > > > > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> > > > > > >
> > >> > > > > > > * Docker release artifact to be voted upon:
> > >> > > > > > > apache/kafka:3.7.0-rc2
> > >> > > > > > >
> > >> > > > > > > * Maven artifacts to be voted upon:
> > >> > > > > > >
> > >> > > >
> > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > > > > > >
> > >> > > > > > > * Javadoc:
> > >> > > > > > >
> > >> > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >> > > > > > >
> > >> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > >> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >> > > > > > >
> > >> > > > > > > * Documentation:
> > >> > > > > > > https://kafka.apache.org/37/documentation.html
> > >> > > > > > >
> > >> > > > > > > * Protocol:
> > >> > > > > > > https://kafka.apache.org/37/protocol.html
> > >> > > > > > >
> > >> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> > >> > > > > > > Unit/integration tests:
> > >> > > > > > >
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > >> > > > > > > There are failing tests here. I have to follow up with
> > triaging
> > >> > > some
> > >> > > > of
> > >> > > > > > > the failures and figuring out if they're actual problems
> or
> > >> > simply
> > >> > > > > flakes.
> > >> > > > > > >
> > >> > > > > > > System tests:
> > >> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >> > > > > > >
> > >> > > > > > > No successful system test runs yet. I am working on
> getting
> > the
> > >> > job
> > >> > > > to
> > >> > > > > run.
> > >> > > > > > >
> > >> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> > >> branch:
> > >> > > > > > > Attached are the scan_report and report_jvm output files
> > from
> > >> the
> > >> > > > > Docker
> > >> > > > > > > Build run:
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >> > > > > > >
> > >> > > > > > > And the final docker image build job - Docker Build Test
> > >> > Pipeline:
> > >> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> > >> > > > > > >
> > >> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >> > > > > > >
> > >> > > > > > > /**************************************
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Stanislav Kozlovski
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Best,
> > >> Stanislav
> > >>
> >
>


-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
I wanted to circle back and confirm the integration tests + system tests,
plus give an overall update regarding status.

The integration tests have a fair amount of flakes. I ran and inspected 3
consecutive builds (57
<https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/57/>, 58
<https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/>, 59
<https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/59/>), then
cross-checked each run's failures via a script of mine to see any
consistent failures.

Three tests proved very flaky. Two are related to KIP-848 running under
KRaft. The third one is a Trogdor test. All 3 tests pass locally, hence I
deem them not blockers for the release. Especially since KIP-848 is in
early access, I am not particularly concerned with a flaky test. I opened
three JIRAs to track them:
- https://issues.apache.org/jira/browse/KAFKA-16134
- https://issues.apache.org/jira/browse/KAFKA-16135
- https://issues.apache.org/jira/browse/KAFKA-16136

As for the system tests, I again ran 2 consecutive builds (1
<https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1705045120--apache--3.7--d071cceffc/2024-01-11--001./2024-01-11--001./report.html>,
2
<https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1705113592--apache--3.7--d071cceffc/2024-01-12--001./2024-01-12--001./report.html>)
and I found 4 tests that exhibit consecutive failures.
- The whole analysis: https://hackmd.io/@hOneAGCrSmKSpL8VF-1HWQ/HyRgRJmta

The failing tests:
StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
QuotaTest - https://issues.apache.org/jira/browse/KAFKA-16138
ZookeeperMigrationTest - https://issues.apache.org/jira/browse/KAFKA-16140

I am reaching out to subject matter experts regarding the failures.

Thanks to everyone who contributed in testing the release. Here is a
general update regarding known blockers that were recently found:

We are treating https://issues.apache.org/jira/browse/KAFKA-16131 and
https://issues.apache.org/jira/browse/KAFKA-16101 as blockers.

https://issues.apache.org/jira/browse/KAFKA-16132 is a potential other
issue that will likely be treated as a blocker

Best,
Stanislav

On Mon, Jan 15, 2024 at 12:04 PM Jakub Scholz <ja...@scholz.cz> wrote:

> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
> blocker bug because it *
> *> will generate huge amount of logspam. I guess we didn't find it in junit
> tests *
> *> since logspam doesn't fail the automated tests. But certainly it's not
> suitable *
> *> for production. Did you file a JIRA yet?*
>
> Hi Colin,
>
> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>
> Thanks & Regards
> Jakub
>
> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for making the first RC. The fact that it's titled RC2 is messing
> > with my mind a bit. I hope this doesn't make people think that we're
> > farther along than we are, heh.
> >
> > On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > > *> Nice catch! It does seem like we should have gated this behind the
> > > metadata> version as KIP-858 implies. Is the cluster configured with
> > > multiple log> dirs? What is the impact of the error messages?*
> > >
> > > I did not observe any obvious impact. I was able to send and receive
> > > messages as normally. But to be honest, I have no idea what else
> > > this might impact, so I did not try anything special.
> > >
> > > I think everyone upgrading an existing KRaft cluster will go through
> this
> > > stage (running Kafka 3.7 with an older metadata version for at least a
> > > while). So even if it is just a logged exception without any other
> > impact I
> > > wonder if it might scare users from upgrading. But I leave it to others
> > to
> > > decide if this is a blocker or not.
> > >
> >
> > Hi Jakub,
> >
> > Thanks for trying the RC. I think what you found is a blocker bug because
> > it will generate huge amount of logspam. I guess we didn't find it in
> junit
> > tests since logspam doesn't fail the automated tests. But certainly it's
> > not suitable for production. Did you file a JIRA yet?
> >
> > > On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > > <st...@confluent.io.invalid> wrote:
> > >
> > >> Hey Luke,
> > >>
> > >> This is an interesting problem. Given the fact that the KIP for
> having a
> > >> 3.8 release passed, I think it weights the scale towards not calling
> > this a
> > >> blocker and expecting it to be solved in 3.7.1.
> > >>
> > >> It is unfortunate that it would not seem safe to migrate to KRaft in
> > 3.7.0
> > >> (given the inability to rollback safely), but if that's true - the
> same
> > >> case would apply for 3.6.0. So in any case users w\ould be expected to
> > use a
> > >> patch release for this.
> >
> > Hi Luke,
> >
> > Thanks for testing rollback. I think this is a case where the
> > documentation is wrong. The intention was to for the steps to basically
> be:
> >
> > 1. roll all the brokers into zk mode, but with migration enabled
> > 2. take down the kraft quorum
> > 3. rmr /controller, allowing a hybrid broker to take over.
> > 4. roll all the brokers into zk mode without migration enabled (if
> desired)
> >
> > With these steps, there isn't really unavailability since a ZK controller
> > can be elected quickly after the kraft quorum is gone.
> >
> > >> Further, since we will have a 3.8 release - it is
> > >> likely we will ultimately recommend users upgrade from that version
> > given
> > >> its aim is to have strategic KRaft feature parity with ZK.
> > >> That being said, I am not 100% on this. Let me know whether you think
> > this
> > >> should block the release, Luke. I am also tagging Colin and David to
> > weigh
> > >> in with their opinions, as they worked on the migration logic.
> >
> > The rollback docs are new in 3.7 so the fact that they're wrong is a
> clear
> > blocker, I think. But easy to fix, I believe. I will create a PR.
> >
> > best,
> > Colin
> >
> > >>
> > >> Hey Kirk and Chris,
> > >>
> > >> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due
> > to
> > >> improper closing. And the PR description implies this has been present
> > >> since 3.5. While annoying, I don't see a strong reason for this to
> block
> > >> the release.
> > >>
> > >> Hey Jakub,
> > >>
> > >> Nice catch! It does seem like we should have gated this behind the
> > metadata
> > >> version as KIP-858 implies. Is the cluster configured with multiple
> log
> > >> dirs? What is the impact of the error messages?
> > >>
> > >> Tagging Igor (the author of the KIP) to weigh in.
> > >>
> > >> Best,
> > >> Stanislav
> > >>
> > >> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I was trying the RC2 and run into the following issue ... when I run
> > >> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> metadata
> > >> > version, I seem to be getting repeated errors like this in the
> > controller
> > >> > logs:
> > >> >
> > >> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > >> assignReplicasToDirs:
> > >> > event failed with UnsupportedVersionException in 15 microseconds.
> > >> > (org.apache.kafka.controller.QuorumController)
> > >> > [quorum-controller-0-event-handler]
> > >> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
> > error
> > >> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> > >> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > >> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > >> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > >> > partitions=[PartitionData(partitionIndex=2),
> > >> > PartitionData(partitionIndex=1)]),
> > >> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > >> > partitions=[PartitionData(partitionIndex=0)])])]) with context
> > >> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> > >> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > clientAddress=/
> > >> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > >> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> > >> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > >> > softwareVersion=3.7.0), fromPrivilegedListener=false,
> > >> >
> > >> >
> > >>
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > >> > ])
> > >> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> > >> > java.util.concurrent.CompletionException:
> > >> > org.apache.kafka.common.errors.UnsupportedVersionException:
> Directory
> > >> > assignment is not supported yet.
> > >> >
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > >> >  at java.base/java.lang.Thread.run(Thread.java:840)
> > >> >
> > >> > Caused by:
> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >> > Directory assignment is not supported yet.
> > >> >
> > >> > Is that expected? I guess with the metadata version set to 3.6-IV2,
> it
> > >> > makes sense that the request is not supported. But shouldn't then
> the
> > >> > request not be sent at all by the brokers? (I did not opened a JIRA
> > for
> > >> it,
> > >> > but I can open one if you agree this is not expected)
> > >> >
> > >> > Thanks & Regards
> > >> > Jakub
> > >> >
> > >> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> wrote:
> > >> >
> > >> > > Hi Stanislav,
> > >> > >
> > >> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
> > you
> > >> > > missed it.
> > >> > > cross-posting here:
> > >> > >
> > >> > > There is a bug KAFKA-16101
> > >> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> that
> > >> > "Kafka
> > >> > > cluster will be unavailable during KRaft migration rollback".
> > >> > > The impact for this issue is that if brokers try to rollback to ZK
> > mode
> > >> > > during KRaft migration process, there will be a period of time the
> > >> > cluster
> > >> > > is unavailable.
> > >> > > Since ZK migrating to KRaft feature is a production ready
> feature, I
> > >> > think
> > >> > > this should be addressed soon.
> > >> > > Do you think this is a blocker for v3.7.0?
> > >> > >
> > >> > > Thanks.
> > >> > > Luke
> > >> > >
> > >> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > fearthecellos@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks, Kirk!
> > >> > > >
> > >> > > > @Stanislav--do you believe that this warrants a new RC?
> > >> > > >
> > >> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> wrote:
> > >> > > >
> > >> > > > > Hi Chris/Stanislav,
> > >> > > > >
> > >> > > > > I'm working on the 'Unable to find FetchSessionHandler' log
> > problem
> > >> > > > > (KAFKA-16029) and have put out a draft PR (
> > >> > > > > https://github.com/apache/kafka/pull/15186). I will use the
> > >> > quickstart
> > >> > > > > approach as a second means to reproduce/verify while I wait
> for
> > the
> > >> > > PR's
> > >> > > > > Jenkins job to finish.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Kirk
> > >> > > > >
> > >> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > >> > > > > > Hi Stanislav,
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks for running this release!
> > >> > > > > >
> > >> > > > > > To verify, I:
> > >> > > > > > - Built from source using Java 11 with both:
> > >> > > > > > - - the 3.7.0-rc2 tag on GitHub
> > >> > > > > > - - the kafka-3.7.0-src.tgz artifact from
> > >> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> > > > > > - Checked signatures and checksums
> > >> > > > > > - Ran the quickstart using both:
> > >> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> > >> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> with
> > >> > > Java
> > >> > > > > 11
> > >> > > > > > and Scala 13 in KRaft mode
> > >> > > > > > - - Our shiny new broker Docker image,
> apache/kafka:3.7.0-rc2
> > >> > > > > > - Ran all unit tests
> > >> > > > > > - Ran all integration tests for Connect and MM2
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > I found two minor areas for concern:
> > >> > > > > >
> > >> > > > > > 1. (Possibly a blocker)
> > >> > > > > > When running the quickstart, I noticed this ERROR-level log
> > >> message
> > >> > > > being
> > >> > > > > > emitted frequently (not not every time) when I killed my
> > console
> > >> > > > consumer
> > >> > > > > > via ctrl-C:
> > >> > > > > >
> > >> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> > >> > > clientId=console-consumer,
> > >> > > > > > groupId=console-consumer-74388] Unable to find
> > >> FetchSessionHandler
> > >> > > for
> > >> > > > > node
> > >> > > > > > 1. Ignoring fetch response
> > >> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >> > > > > >
> > >> > > > > > I see that this error message is already reported in
> > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think
> we
> > >> > should
> > >> > > > > > prioritize fixing it for this release. I know it's probably
> > >> benign
> > >> > > but
> > >> > > > > it's
> > >> > > > > > really not a good look for us when basic operations log
> error
> > >> > > messages,
> > >> > > > > and
> > >> > > > > > it may give new users some headaches.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > 2. (Probably not a blocker)
> > >> > > > > > The following unit tests failed the first time around, and
> > all of
> > >> > > them
> > >> > > > > > passed the second time I ran them:
> > >> > > > > >
> > >> > > > > > - (clients)
> > >> > > > >
> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > >> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > >> > > > > > - (clients)
> Tls13SelectorTest.testConnectionsByClientMetric()
> > >> > > > > > - (connect)
> > >> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > >> > > > (I
> > >> > > > > > thought I fixed this one! 🤬🤬)
> > >> > > > > > - (core)
> > ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks again for your work on this release, and
> > congratulations
> > >> to
> > >> > > > Kafka
> > >> > > > > > Streams for having zero flaky unit tests during my
> > >> > > highly-experimental
> > >> > > > > > single laptop run!
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Cheers,
> > >> > > > > >
> > >> > > > > > Chris
> > >> > > > > >
> > >> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > >> > > > > > <st...@confluent.io.invalid> wrote:
> > >> > > > > >
> > >> > > > > > > Hello Kafka users, developers, and client-developers,
> > >> > > > > > >
> > >> > > > > > > This is the first candidate for release of Apache Kafka
> > 3.7.0.
> > >> > > > > > >
> > >> > > > > > > Note it's named "RC2" because I had a few "failed" RCs
> that
> > I
> > >> had
> > >> > > > > > > cut/uploaded but ultimately had to scrap prior to
> announcing
> > >> due
> > >> > to
> > >> > > > new
> > >> > > > > > > blockers arriving before I could even announce them.
> > >> > > > > > >
> > >> > > > > > > Further - I haven't yet been able to set up the system
> tests
> > >> > > > > successfully.
> > >> > > > > > > And the integration/unit tests do have a few failures
> that I
> > >> have
> > >> > > to
> > >> > > > > spend
> > >> > > > > > > time triaging. I would appreciate any help in case anyone
> > >> notices
> > >> > > any
> > >> > > > > tests
> > >> > > > > > > failing that they're subject matters experts in. Expect me
> > to
> > >> > > follow
> > >> > > > > up in
> > >> > > > > > > a day or two with more detailed analysis.
> > >> > > > > > >
> > >> > > > > > > Major changes include:
> > >> > > > > > > - Early Access to KIP-848 - the next generation of the
> > consumer
> > >> > > > > rebalance
> > >> > > > > > > protocol
> > >> > > > > > > - KIP-858: Adding JBOD support to KRaft
> > >> > > > > > > - KIP-714: Observability into Client metrics via a
> > standardized
> > >> > > > > interface
> > >> > > > > > >
> > >> > > > > > > Check more information in the WIP blog post:
> > >> > > > > > > https://github.com/apache/kafka-site/pull/578
> > >> > > > > > >
> > >> > > > > > > Release notes for the 3.7.0 release:
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >> > > > > > >
> > >> > > > > > > *** Please download, test and vote by Thursday, January
> 18,
> > 9am
> > >> > PT
> > >> > > > ***
> > >> > > > > > >
> > >> > > > > > > Usually these deadlines tend to be 2-3 days, but due to
> this
> > >> > being
> > >> > > > the
> > >> > > > > > > first RC and the tests not having ran yet, I am giving it
> a
> > bit
> > >> > > more
> > >> > > > > time.
> > >> > > > > > >
> > >> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > >> release:
> > >> > > > > > > https://kafka.apache.org/KEYS
> > >> > > > > > >
> > >> > > > > > > * Release artifacts to be voted upon (source and binary):
> > >> > > > > > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> > > > > > >
> > >> > > > > > > * Docker release artifact to be voted upon:
> > >> > > > > > > apache/kafka:3.7.0-rc2
> > >> > > > > > >
> > >> > > > > > > * Maven artifacts to be voted upon:
> > >> > > > > > >
> > >> > > >
> > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > > > > > >
> > >> > > > > > > * Javadoc:
> > >> > > > > > >
> > >> > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >> > > > > > >
> > >> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > >> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >> > > > > > >
> > >> > > > > > > * Documentation:
> > >> > > > > > > https://kafka.apache.org/37/documentation.html
> > >> > > > > > >
> > >> > > > > > > * Protocol:
> > >> > > > > > > https://kafka.apache.org/37/protocol.html
> > >> > > > > > >
> > >> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> > >> > > > > > > Unit/integration tests:
> > >> > > > > > >
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > >> > > > > > > There are failing tests here. I have to follow up with
> > triaging
> > >> > > some
> > >> > > > of
> > >> > > > > > > the failures and figuring out if they're actual problems
> or
> > >> > simply
> > >> > > > > flakes.
> > >> > > > > > >
> > >> > > > > > > System tests:
> > >> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >> > > > > > >
> > >> > > > > > > No successful system test runs yet. I am working on
> getting
> > the
> > >> > job
> > >> > > > to
> > >> > > > > run.
> > >> > > > > > >
> > >> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> > >> branch:
> > >> > > > > > > Attached are the scan_report and report_jvm output files
> > from
> > >> the
> > >> > > > > Docker
> > >> > > > > > > Build run:
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >> > > > > > >
> > >> > > > > > > And the final docker image build job - Docker Build Test
> > >> > Pipeline:
> > >> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> > >> > > > > > >
> > >> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >> > > > > > >
> > >> > > > > > > /**************************************
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Stanislav Kozlovski
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Best,
> > >> Stanislav
> > >>
> >
>


-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
Thanks Mickael, sounds good.

KAFKA-160195 and KAFKA-16157 were both merged!

I was made aware of one final blocker, this time for streams - KAFKA-16221.
Matthias was prompt with a short hotfix PR:
https://github.com/apache/kafka/pull/15315

After that goes into 3.7, I think I will be free to build the next RC.
Great work!

On Fri, Feb 2, 2024 at 6:43 PM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Stanislav,
>
> I merged https://github.com/apache/kafka/pull/15308 in trunk. I let
> you cherry-pick it to 3.7.
>
> I think fixing the absolute show stoppers and calling JBOD support in
> KRaft early access in 3.7.0 is probably the right call. Even without
> the bugs we found, there's still quite a few JBOD follow up work to do
> (KAFKA-16061) + system tests and documentation updates.
>
> Thanks,
> Mickael
>
> On Fri, Feb 2, 2024 at 4:49 PM Stanislav Kozlovski
> <st...@confluent.io.invalid> wrote:
> >
> > Thanks for the work everybody. Providing a status update at the end of
> the
> > week:
> >
> > - docs change explaining migration
> > <https://github.com/apache/kafka/pull/15193> was merged
> > - the blocker KAFKA-16162 <https://github.com/apache/kafka/pull/15270>
> was
> > merged
> > - the blocker KAFKA-14616 <https://github.com/apache/kafka/pull/15230>
> was
> > merged
> > - a small blocker problem with the shadow jar plugin
> > <https://github.com/apache/kafka/pull/15308>
> > - the blockers KAFKALESS-16157 & KAFKALESS-16195 aren't merged
> > - the good-to-have KAFKA-16082 isn't merged
> >
> > I think we should prioritize merging KAFKALESS-16195 and *call JBOD EA*.
> I
> > question whether we may find more blocker bugs in the next RC.
> > The release is late by approximately a month so far, so I do want to
> scope
> > down aggressively to meet the time-based goal.
> >
> > Best,
> > Stanislav
> >
> > On Mon, Jan 29, 2024 at 5:46 PM Omnia Ibrahim <o....@gmail.com>
> > wrote:
> >
> > > Hi Stan and Gaurav,
> > > Just to clarify some points mentioned here before
> > >  KAFKA-14616: I raised a year ago so it's not related to JBOD work. It
> is
> > > rather a blocker bug for KRAFT in general. The PR from Colin should fix
> > > this. Am not sure if it is a blocker for 3.7 per-say as it was a major
> bug
> > > since 3.3 and got missed from all other releases.
> > >
> > > Regarding the JBOD's work:
> > > KAFKA-16082:  Is not a blocker for 3.7 instead it's nice fix. The pr
> > > https://github.com/apache/kafka/pull/15136 is quite a small one and
> was
> > > approved by Proven and I but it is waiting for a committer's approval.
> > > KAFKA-16162: This is a blocker for 3.7.  Same it’s a small pr
> > > https://github.com/apache/kafka/pull/15270 and it is approved Proven
> and
> > > I and the PR is waiting for committer's approval.
> > > KAFKA-16157: This is a blocker for 3.7. There is one small suggestion
> for
> > > the pr https://github.com/apache/kafka/pull/15263 but I don't think
> any
> > > of the current feedback is blocking the pr from getting approved.
> Assuming
> > > we get a committer's approval on it.
> > > KAFKA-16195:  Same it's a blocker but it has approval from Proven and I
> > > and we are waiting for committer's approval on the pr
> > > https://github.com/apache/kafka/pull/15262.
> > >
> > > If we can’t get a committer approval for KAFKA-16162, KAFKA-16157 and
> > > KAFKA-16195  in time for 3.7 then we can mark JBOD as early release
> > > assuming we merge at least KAFKA-16195.
> > >
> > > Regards,
> > > Omnia
> > >
> > > > On 26 Jan 2024, at 15:39, kafka@gnarula.com wrote:
> > > >
> > > > Apologies, I duplicated KAFKA-16157 twice in my previous message. I
> > > intended to mention KAFKA-16195
> > > > with the PR at https://github.com/apache/kafka/pull/15262 as the
> second
> > > JIRA.
> > > >
> > > > Thanks,
> > > > Gaurav
> > > >
> > > >> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
> > > >>
> > > >> Hi Stan,
> > > >>
> > > >> I wanted to share some updates about the bugs you shared earlier.
> > > >>
> > > >> - KAFKA-14616: I've reviewed and tested the PR from Colin and have
> > > observed
> > > >> the fix works as intended.
> > > >> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the
> > > proposed fix. I've
> > > >> therefore raised https://github.com/apache/kafka/pull/15270
> following
> > > a discussion with Luke in JIRA.
> > > >> - KAFKA-16082: I don't think this is marked as a blocker anymore.
> I'm
> > > awaiting
> > > >> feedback/reviews at https://github.com/apache/kafka/pull/15136
> > > >>
> > > >> In addition to the above, there are 2 JIRAs I'd like to bring
> > > everyone's attention to:
> > > >>
> > > >> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a
> > > blocker. I've raised
> > > >> https://github.com/apache/kafka/pull/15263 and am awaiting reviews
> on
> > > it.
> > > >> - KAFKA-16157: I raised this yesterday and have addressed feedback
> from
> > > Luke. This should
> > > >> hopefully get merged soon.
> > > >>
> > > >> Regards,
> > > >> Gaurav
> > > >>
> > > >>
> > > >>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
> > > >>>
> > > >>> Hi Stanislav,
> > > >>>
> > > >>> Thanks for bringing these JIRAs/PRs up.
> > > >>>
> > > >>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this
> week
> > > and I hope to have some feedback
> > > >>> by Friday. I gather the latter JIRA is marked as a WIP by Proven
> and
> > > he's away. I'll try to build on his work in the meantime.
> > > >>>
> > > >>> As for KAFKA-16082, we haven't been able to deduce a data loss
> > > scenario. There's a PR open
> > > >>> by me for promoting an abandoned future replica with approvals from
> > > Omnia and Proven,
> > > >>> so I'd appreciate a committer reviewing it.
> > > >>>
> > > >>> Regards,
> > > >>> Gaurav
> > > >>>
> > > >>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <
> stanislav@confluent.io.INVALID>
> > > wrote:
> > > >>>>
> > > >>>> Hey all, I figured I'd give an update about what known blockers we
> > > have
> > > >>>> right now:
> > > >>>>
> > > >>>> - KAFKA-16101: KRaft migration rollback documentation is
> incorrect -
> > > >>>> https://github.com/apache/kafka/pull/15193; This need not block
> RC
> > > >>>> creation, but we need the docs updated so that people can test
> > > properly
> > > >>>> - KAFKA-14616: Topic recreation with offline broker causes
> permanent
> > > URPs -
> > > >>>> https://github.com/apache/kafka/pull/15230 ; I am of the
> > > understanding that
> > > >>>> this is blocking JBOD for 3.7
> > > >>>> - KAFKA-16162: New created topics are unavailable after upgrading
> to
> > > 3.7 -
> > > >>>> a strict blocker with an open PR
> > > https://github.com/apache/kafka/pull/15232
> > > >>>> - although I understand Proveen is out of office
> > > >>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader
> partition -
> > > I am
> > > >>>> hearing mixed opinions on whether this is a blocker (
> > > >>>> https://github.com/apache/kafka/pull/15136)
> > > >>>>
> > > >>>> Given that there are 3 JBOD blocker bugs, and I am not confident
> they
> > > will
> > > >>>> all be merged this week - I am on the edge of voting to revert
> JBOD
> > > from
> > > >>>> this release, or mark it early access.
> > > >>>>
> > > >>>> By all accounts, it seems that if we keep with JBOD the release
> will
> > > have
> > > >>>> to spill into February, which is a month extra from the time-based
> > > release
> > > >>>> plan we had of start of January.
> > > >>>>
> > > >>>> Can I ask others for an opinion?
> > > >>>>
> > > >>>> Best,
> > > >>>> Stan
> > > >>>>
> > > >>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com>
> wrote:
> > > >>>>
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> I think I've found another blocker issue: KAFKA-16162
> > > >>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
> > > >>>>> The impact is after upgrading to 3.7.0, any new created
> > > topics/partitions
> > > >>>>> will be unavailable.
> > > >>>>> I've put my findings in the JIRA.
> > > >>>>>
> > > >>>>> Thanks.
> > > >>>>> Luke
> > > >>>>>
> > > >>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <
> mjsax@apache.org>
> > > wrote:
> > > >>>>>
> > > >>>>>> Stan, thanks for driving this all forward! Excellent job.
> > > >>>>>>
> > > >>>>>> About
> > > >>>>>>
> > > >>>>>>> StreamsStandbyTask -
> > > https://issues.apache.org/jira/browse/KAFKA-16141
> > > >>>>>>> StreamsUpgradeTest -
> > > https://issues.apache.org/jira/browse/KAFKA-16139
> > > >>>>>>
> > > >>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be
> > > fixed
> > > >>>>>> now in trunk and 3.7 (and actually also in 3.6...)
> > > >>>>>>
> > > >>>>>> For `StreamsStandbyTask` the failing test exposes a regression
> bug,
> > > so
> > > >>>>>> it's a blocker. I updated the ticket accordingly. We already
> have an
> > > >>>>>> open PR that reverts the code introducing the regression.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> -Matthias
> > > >>>>>>
> > > >>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
> > > >>>>>>> We have another blocking issue for the RC :
> > > >>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is
> > > similar
> > > >>>>>> to
> > > >>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new
> issue
> > > >>>>> however
> > > >>>>>>> can lead to the new topic having partitions that a producer
> cannot
> > > >>>>> write
> > > >>>>>> to.
> > > >>>>>>>
> > > >>>>>>> --Proven
> > > >>>>>>>
> > > >>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> > > >>>>>> pprovenzano@confluent.io>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
> > > >>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is
> > > building
> > > >>>>> now.
> > > >>>>>>>> --Proven
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <jakub@scholz.cz
> >
> > > wrote:
> > > >>>>>>>>
> > > >>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you
> found
> > > is a
> > > >>>>>>>>> blocker bug because it *
> > > >>>>>>>>> *> will generate huge amount of logspam. I guess we didn't
> find
> > > it in
> > > >>>>>>>>> junit
> > > >>>>>>>>> tests *
> > > >>>>>>>>> *> since logspam doesn't fail the automated tests. But
> certainly
> > > it's
> > > >>>>>> not
> > > >>>>>>>>> suitable *
> > > >>>>>>>>> *> for production. Did you file a JIRA yet?*
> > > >>>>>>>>>
> > > >>>>>>>>> Hi Colin,
> > > >>>>>>>>>
> > > >>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks & Regards
> > > >>>>>>>>> Jakub
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <
> cmccabe@apache.org
> > > >
> > > >>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Stanislav,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for making the first RC. The fact that it's titled
> RC2 is
> > > >>>>>> messing
> > > >>>>>>>>>> with my mind a bit. I hope this doesn't make people think
> that
> > > we're
> > > >>>>>>>>>> farther along than we are, heh.
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > > >>>>>>>>>>> *> Nice catch! It does seem like we should have gated this
> > > behind
> > > >>>>> the
> > > >>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster
> configured
> > > >>>>> with
> > > >>>>>>>>>>> multiple log> dirs? What is the impact of the error
> messages?*
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I did not observe any obvious impact. I was able to send
> and
> > > >>>>> receive
> > > >>>>>>>>>>> messages as normally. But to be honest, I have no idea what
> > > else
> > > >>>>>>>>>>> this might impact, so I did not try anything special.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will
> go
> > > >>>>> through
> > > >>>>>>>>> this
> > > >>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version
> for at
> > > >>>>> least
> > > >>>>>> a
> > > >>>>>>>>>>> while). So even if it is just a logged exception without
> any
> > > other
> > > >>>>>>>>>> impact I
> > > >>>>>>>>>>> wonder if it might scare users from upgrading. But I leave
> it
> > > to
> > > >>>>>>>>> others
> > > >>>>>>>>>> to
> > > >>>>>>>>>>> decide if this is a blocker or not.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hi Jakub,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for trying the RC. I think what you found is a
> blocker
> > > bug
> > > >>>>>>>>> because
> > > >>>>>>>>>> it will generate huge amount of logspam. I guess we didn't
> find
> > > it
> > > >>>>> in
> > > >>>>>>>>> junit
> > > >>>>>>>>>> tests since logspam doesn't fail the automated tests. But
> > > certainly
> > > >>>>>> it's
> > > >>>>>>>>>> not suitable for production. Did you file a JIRA yet?
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > > >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hey Luke,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> This is an interesting problem. Given the fact that the
> KIP
> > > for
> > > >>>>>>>>> having a
> > > >>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards
> not
> > > >>>>> calling
> > > >>>>>>>>>> this a
> > > >>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate
> to
> > > KRaft
> > > >>>>> in
> > > >>>>>>>>>> 3.7.0
> > > >>>>>>>>>>>> (given the inability to rollback safely), but if that's
> true
> > > - the
> > > >>>>>>>>> same
> > > >>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
> > > >>>>> expected
> > > >>>>>>>>> to
> > > >>>>>>>>>> use a
> > > >>>>>>>>>>>> patch release for this.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hi Luke,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for testing rollback. I think this is a case where
> the
> > > >>>>>>>>>> documentation is wrong. The intention was to for the steps
> to
> > > >>>>>> basically
> > > >>>>>>>>> be:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. roll all the brokers into zk mode, but with migration
> enabled
> > > >>>>>>>>>> 2. take down the kraft quorum
> > > >>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
> > > >>>>>>>>>> 4. roll all the brokers into zk mode without migration
> enabled
> > > (if
> > > >>>>>>>>> desired)
> > > >>>>>>>>>>
> > > >>>>>>>>>> With these steps, there isn't really unavailability since a
> ZK
> > > >>>>>>>>> controller
> > > >>>>>>>>>> can be elected quickly after the kraft quorum is gone.
> > > >>>>>>>>>>
> > > >>>>>>>>>>>> Further, since we will have a 3.8 release - it is
> > > >>>>>>>>>>>> likely we will ultimately recommend users upgrade from
> that
> > > >>>>> version
> > > >>>>>>>>>> given
> > > >>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
> > > >>>>>>>>>>>> That being said, I am not 100% on this. Let me know
> whether
> > > you
> > > >>>>>> think
> > > >>>>>>>>>> this
> > > >>>>>>>>>>>> should block the release, Luke. I am also tagging Colin
> and
> > > David
> > > >>>>> to
> > > >>>>>>>>>> weigh
> > > >>>>>>>>>>>> in with their opinions, as they worked on the migration
> logic.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're
> wrong
> > > is a
> > > >>>>>>>>> clear
> > > >>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create
> a
> > > PR.
> > > >>>>>>>>>>
> > > >>>>>>>>>> best,
> > > >>>>>>>>>> Colin
> > > >>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hey Kirk and Chris,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a
> > > bad log
> > > >>>>>>>>> due
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> improper closing. And the PR description implies this has
> been
> > > >>>>>>>>> present
> > > >>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for
> > > this to
> > > >>>>>>>>> block
> > > >>>>>>>>>>>> the release.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hey Jakub,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Nice catch! It does seem like we should have gated this
> > > behind the
> > > >>>>>>>>>> metadata
> > > >>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
> > > >>>>> multiple
> > > >>>>>>>>> log
> > > >>>>>>>>>>>> dirs? What is the impact of the error messages?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Stanislav
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <
> jakub@scholz.cz
> > > >
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I was trying the RC2 and run into the following issue ...
> > > when I
> > > >>>>>>>>> run
> > > >>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to
> 3.6-IV2
> > > >>>>>>>>> metadata
> > > >>>>>>>>>>>>> version, I seem to be getting repeated errors like this
> in
> > > the
> > > >>>>>>>>>> controller
> > > >>>>>>>>>>>>> logs:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > > >>>>>>>>>>>> assignReplicasToDirs:
> > > >>>>>>>>>>>>> event failed with UnsupportedVersionException in 15
> > > microseconds.
> > > >>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
> > > >>>>>>>>>>>>> [quorum-controller-0-event-handler]
> > > >>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
> > > >>>>> Unexpected
> > > >>>>>>>>>> error
> > > >>>>>>>>>>>>> handling request
> > > RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > > >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> > > headerVersion=2)
> > > >>>>> --
> > > >>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000,
> brokerEpoch=5,
> > > >>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > > >>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > > >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
> > > >>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
> > > >>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > > >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with
> > > context
> > > >>>>>>>>>>>>>
> > > >>>>>
> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > > >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> > > headerVersion=2),
> > > >>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > > >>>>>>>>>> clientAddress=/
> > > >>>>>>>>>>>>> 172.16.14.217,
> > > principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > > >>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
> > > >>>>> securityProtocol=SSL,
> > > >>>>>>>>>>>>>
> > > >>>>>
> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > > >>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > > >>>>>>>>>>>>> ])
> > > >>>>>>>>>>>>> (kafka.server.ControllerApis)
> > > [quorum-controller-0-event-handler]
> > > >>>>>>>>>>>>> java.util.concurrent.CompletionException:
> > > >>>>>>>>>>>>>
> org.apache.kafka.common.errors.UnsupportedVersionException:
> > > >>>>>>>>> Directory
> > > >>>>>>>>>>>>> assignment is not supported yet.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > > >>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Caused by:
> > > >>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > > >>>>>>>>>>>>> Directory assignment is not supported yet.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Is that expected? I guess with the metadata version set
> to
> > > >>>>>>>>> 3.6-IV2, it
> > > >>>>>>>>>>>>> makes sense that the request is not supported. But
> shouldn't
> > > then
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>> request not be sent at all by the brokers? (I did not
> opened
> > > a
> > > >>>>> JIRA
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>> it,
> > > >>>>>>>>>>>>> but I can open one if you agree this is not expected)
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks & Regards
> > > >>>>>>>>>>>>> Jakub
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <
> showuon@gmail.com
> > > >
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi Stanislav,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread,
> but
> > > >>>>> maybe
> > > >>>>>>>>>> you
> > > >>>>>>>>>>>>>> missed it.
> > > >>>>>>>>>>>>>> cross-posting here:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> There is a bug KAFKA-16101
> > > >>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101>
> > > reporting
> > > >>>>>>>>> that
> > > >>>>>>>>>>>>> "Kafka
> > > >>>>>>>>>>>>>> cluster will be unavailable during KRaft migration
> > > rollback".
> > > >>>>>>>>>>>>>> The impact for this issue is that if brokers try to
> > > rollback to
> > > >>>>>>>>> ZK
> > > >>>>>>>>>> mode
> > > >>>>>>>>>>>>>> during KRaft migration process, there will be a period
> of
> > > time
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>> cluster
> > > >>>>>>>>>>>>>> is unavailable.
> > > >>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production
> ready
> > > >>>>>>>>> feature, I
> > > >>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>> this should be addressed soon.
> > > >>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks.
> > > >>>>>>>>>>>>>> Luke
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > > >>>>>>>>>> fearthecellos@gmail.com
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks, Kirk!
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <
> kirk@kirktrue.pro>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi Chris/Stanislav,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'm working on the 'Unable to find
> FetchSessionHandler'
> > > log
> > > >>>>>>>>>> problem
> > > >>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> > > >>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will
> use
> > > the
> > > >>>>>>>>>>>>> quickstart
> > > >>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I
> > > wait
> > > >>>>>>>>> for
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> PR's
> > > >>>>>>>>>>>>>>>> Jenkins job to finish.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>> Kirk
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton
> wrote:
> > > >>>>>>>>>>>>>>>>> Hi Stanislav,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for running this release!
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> To verify, I:
> > > >>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
> > > >>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> > > >>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > >>>>>>>>>>>>>>>>> - Checked signatures and checksums
> > > >>>>>>>>>>>>>>>>> - Ran the quickstart using both:
> > > >>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > >>>>>>>>>>>> with
> > > >>>>>>>>>>>>>> Java
> > > >>>>>>>>>>>>>>>> 11
> > > >>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
> > > >>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
> > > >>>>>>>>> apache/kafka:3.7.0-rc2
> > > >>>>>>>>>>>>>>>>> - Ran all unit tests
> > > >>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I found two minor areas for concern:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
> > > >>>>>>>>>>>>>>>>> When running the quickstart, I noticed this
> ERROR-level
> > > log
> > > >>>>>>>>>>>> message
> > > >>>>>>>>>>>>>>> being
> > > >>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I
> killed my
> > > >>>>>>>>>> console
> > > >>>>>>>>>>>>>>> consumer
> > > >>>>>>>>>>>>>>>>> via ctrl-C:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> > > >>>>>>>>>>>>>> clientId=console-consumer,
> > > >>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> > > >>>>>>>>>>>> FetchSessionHandler
> > > >>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> node
> > > >>>>>>>>>>>>>>>>> 1. Ignoring fetch response
> > > >>>>>>>>>>>>>>>>>
> > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I see that this error message is already reported in
> > > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> > > >>>>>>>>> think we
> > > >>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's
> > > probably
> > > >>>>>>>>>>>> benign
> > > >>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>> it's
> > > >>>>>>>>>>>>>>>>> really not a good look for us when basic operations
> log
> > > >>>>>>>>> error
> > > >>>>>>>>>>>>>> messages,
> > > >>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> it may give new users some headaches.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
> > > >>>>>>>>>>>>>>>>> The following unit tests failed the first time
> around,
> > > and
> > > >>>>>>>>>> all of
> > > >>>>>>>>>>>>>> them
> > > >>>>>>>>>>>>>>>>> passed the second time I ran them:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> - (clients)
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>
> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > > >>>>>>>>>>>>>>>>> - (clients)
> SelectorTest.testConnectionsByClientMetric()
> > > >>>>>>>>>>>>>>>>> - (clients)
> > > >>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
> > > >>>>>>>>>>>>>>>>> - (connect)
> > > >>>>>>>>>>>>>>
> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > > >>>>>>>>>>>>>>> (I
> > > >>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
> > > >>>>>>>>>>>>>>>>> - (core)
> > > >>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
> > > >>>>>>>>>> congratulations
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> Kafka
> > > >>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
> > > >>>>>>>>>>>>>> highly-experimental
> > > >>>>>>>>>>>>>>>>> single laptop run!
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Chris
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > > >>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and
> client-developers,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache
> Kafka
> > > >>>>>>>>>> 3.7.0.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed"
> RCs
> > > >>>>>>>>> that
> > > >>>>>>>>>> I
> > > >>>>>>>>>>>> had
> > > >>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> > > >>>>>>>>> announcing
> > > >>>>>>>>>>>> due
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the
> system
> > > >>>>>>>>> tests
> > > >>>>>>>>>>>>>>>> successfully.
> > > >>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few
> failures
> > > >>>>>>>>> that I
> > > >>>>>>>>>>>> have
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> spend
> > > >>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case
> > > anyone
> > > >>>>>>>>>>>> notices
> > > >>>>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>>> tests
> > > >>>>>>>>>>>>>>>>>> failing that they're subject matters experts in.
> Expect
> > > >>>>>>>>> me
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>> follow
> > > >>>>>>>>>>>>>>>> up in
> > > >>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Major changes include:
> > > >>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of
> the
> > > >>>>>>>>>> consumer
> > > >>>>>>>>>>>>>>>> rebalance
> > > >>>>>>>>>>>>>>>>>> protocol
> > > >>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> > > >>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> > > >>>>>>>>>> standardized
> > > >>>>>>>>>>>>>>>> interface
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
> > > >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday,
> January
> > > >>>>>>>>> 18,
> > > >>>>>>>>>> 9am
> > > >>>>>>>>>>>>> PT
> > > >>>>>>>>>>>>>>> ***
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but
> due to
> > > >>>>>>>>> this
> > > >>>>>>>>>>>>> being
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am
> giving
> > > >>>>>>>>> it a
> > > >>>>>>>>>> bit
> > > >>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>> time.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to
> sign the
> > > >>>>>>>>>>>> release:
> > > >>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and
> > > binary):
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
> > > >>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Javadoc:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0
> > > tag:
> > > >>>>>>>>>>>>>>>>>>
> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Documentation:
> > > >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Protocol:
> > > >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> > > >>>>>>>>>>>>>>>>>> Unit/integration tests:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > > >>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up
> with
> > > >>>>>>>>>> triaging
> > > >>>>>>>>>>>>>> some
> > > >>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual
> problems
> > > >>>>>>>>> or
> > > >>>>>>>>>>>>> simply
> > > >>>>>>>>>>>>>>>> flakes.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> System tests:
> > > >>>>>>>>>>>>>>>>
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
> > > >>>>>>>>> getting
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> job
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> run.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline
> for
> > > 3.7
> > > >>>>>>>>>>>> branch:
> > > >>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output
> files
> > > >>>>>>>>>> from
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> Docker
> > > >>>>>>>>>>>>>>>>>> Build run:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > >
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build
> Test
> > > >>>>>>>>>>>>> Pipeline:
> > > >>>>>>>>>>>>>>>>>>
> https://github.com/apache/kafka/actions/runs/7486178277
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>
> > >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> /**************************************
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>> Stanislav Kozlovski
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> --
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Stanislav
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Best,
> > > >>>> Stanislav
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
> > --
> > Best,
> > Stanislav
>
>

-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Mickael Maison <mi...@gmail.com>.
Hi Stanislav,

I merged https://github.com/apache/kafka/pull/15308 in trunk. I let
you cherry-pick it to 3.7.

I think fixing the absolute show stoppers and calling JBOD support in
KRaft early access in 3.7.0 is probably the right call. Even without
the bugs we found, there's still quite a few JBOD follow up work to do
(KAFKA-16061) + system tests and documentation updates.

Thanks,
Mickael

On Fri, Feb 2, 2024 at 4:49 PM Stanislav Kozlovski
<st...@confluent.io.invalid> wrote:
>
> Thanks for the work everybody. Providing a status update at the end of the
> week:
>
> - docs change explaining migration
> <https://github.com/apache/kafka/pull/15193> was merged
> - the blocker KAFKA-16162 <https://github.com/apache/kafka/pull/15270> was
> merged
> - the blocker KAFKA-14616 <https://github.com/apache/kafka/pull/15230> was
> merged
> - a small blocker problem with the shadow jar plugin
> <https://github.com/apache/kafka/pull/15308>
> - the blockers KAFKALESS-16157 & KAFKALESS-16195 aren't merged
> - the good-to-have KAFKA-16082 isn't merged
>
> I think we should prioritize merging KAFKALESS-16195 and *call JBOD EA*. I
> question whether we may find more blocker bugs in the next RC.
> The release is late by approximately a month so far, so I do want to scope
> down aggressively to meet the time-based goal.
>
> Best,
> Stanislav
>
> On Mon, Jan 29, 2024 at 5:46 PM Omnia Ibrahim <o....@gmail.com>
> wrote:
>
> > Hi Stan and Gaurav,
> > Just to clarify some points mentioned here before
> >  KAFKA-14616: I raised a year ago so it's not related to JBOD work. It is
> > rather a blocker bug for KRAFT in general. The PR from Colin should fix
> > this. Am not sure if it is a blocker for 3.7 per-say as it was a major bug
> > since 3.3 and got missed from all other releases.
> >
> > Regarding the JBOD's work:
> > KAFKA-16082:  Is not a blocker for 3.7 instead it's nice fix. The pr
> > https://github.com/apache/kafka/pull/15136 is quite a small one and was
> > approved by Proven and I but it is waiting for a committer's approval.
> > KAFKA-16162: This is a blocker for 3.7.  Same it’s a small pr
> > https://github.com/apache/kafka/pull/15270 and it is approved Proven and
> > I and the PR is waiting for committer's approval.
> > KAFKA-16157: This is a blocker for 3.7. There is one small suggestion for
> > the pr https://github.com/apache/kafka/pull/15263 but I don't think any
> > of the current feedback is blocking the pr from getting approved. Assuming
> > we get a committer's approval on it.
> > KAFKA-16195:  Same it's a blocker but it has approval from Proven and I
> > and we are waiting for committer's approval on the pr
> > https://github.com/apache/kafka/pull/15262.
> >
> > If we can’t get a committer approval for KAFKA-16162, KAFKA-16157 and
> > KAFKA-16195  in time for 3.7 then we can mark JBOD as early release
> > assuming we merge at least KAFKA-16195.
> >
> > Regards,
> > Omnia
> >
> > > On 26 Jan 2024, at 15:39, kafka@gnarula.com wrote:
> > >
> > > Apologies, I duplicated KAFKA-16157 twice in my previous message. I
> > intended to mention KAFKA-16195
> > > with the PR at https://github.com/apache/kafka/pull/15262 as the second
> > JIRA.
> > >
> > > Thanks,
> > > Gaurav
> > >
> > >> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
> > >>
> > >> Hi Stan,
> > >>
> > >> I wanted to share some updates about the bugs you shared earlier.
> > >>
> > >> - KAFKA-14616: I've reviewed and tested the PR from Colin and have
> > observed
> > >> the fix works as intended.
> > >> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the
> > proposed fix. I've
> > >> therefore raised https://github.com/apache/kafka/pull/15270 following
> > a discussion with Luke in JIRA.
> > >> - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm
> > awaiting
> > >> feedback/reviews at https://github.com/apache/kafka/pull/15136
> > >>
> > >> In addition to the above, there are 2 JIRAs I'd like to bring
> > everyone's attention to:
> > >>
> > >> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a
> > blocker. I've raised
> > >> https://github.com/apache/kafka/pull/15263 and am awaiting reviews on
> > it.
> > >> - KAFKA-16157: I raised this yesterday and have addressed feedback from
> > Luke. This should
> > >> hopefully get merged soon.
> > >>
> > >> Regards,
> > >> Gaurav
> > >>
> > >>
> > >>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
> > >>>
> > >>> Hi Stanislav,
> > >>>
> > >>> Thanks for bringing these JIRAs/PRs up.
> > >>>
> > >>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week
> > and I hope to have some feedback
> > >>> by Friday. I gather the latter JIRA is marked as a WIP by Proven and
> > he's away. I'll try to build on his work in the meantime.
> > >>>
> > >>> As for KAFKA-16082, we haven't been able to deduce a data loss
> > scenario. There's a PR open
> > >>> by me for promoting an abandoned future replica with approvals from
> > Omnia and Proven,
> > >>> so I'd appreciate a committer reviewing it.
> > >>>
> > >>> Regards,
> > >>> Gaurav
> > >>>
> > >>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID>
> > wrote:
> > >>>>
> > >>>> Hey all, I figured I'd give an update about what known blockers we
> > have
> > >>>> right now:
> > >>>>
> > >>>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
> > >>>> https://github.com/apache/kafka/pull/15193; This need not block RC
> > >>>> creation, but we need the docs updated so that people can test
> > properly
> > >>>> - KAFKA-14616: Topic recreation with offline broker causes permanent
> > URPs -
> > >>>> https://github.com/apache/kafka/pull/15230 ; I am of the
> > understanding that
> > >>>> this is blocking JBOD for 3.7
> > >>>> - KAFKA-16162: New created topics are unavailable after upgrading to
> > 3.7 -
> > >>>> a strict blocker with an open PR
> > https://github.com/apache/kafka/pull/15232
> > >>>> - although I understand Proveen is out of office
> > >>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition -
> > I am
> > >>>> hearing mixed opinions on whether this is a blocker (
> > >>>> https://github.com/apache/kafka/pull/15136)
> > >>>>
> > >>>> Given that there are 3 JBOD blocker bugs, and I am not confident they
> > will
> > >>>> all be merged this week - I am on the edge of voting to revert JBOD
> > from
> > >>>> this release, or mark it early access.
> > >>>>
> > >>>> By all accounts, it seems that if we keep with JBOD the release will
> > have
> > >>>> to spill into February, which is a month extra from the time-based
> > release
> > >>>> plan we had of start of January.
> > >>>>
> > >>>> Can I ask others for an opinion?
> > >>>>
> > >>>> Best,
> > >>>> Stan
> > >>>>
> > >>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> I think I've found another blocker issue: KAFKA-16162
> > >>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
> > >>>>> The impact is after upgrading to 3.7.0, any new created
> > topics/partitions
> > >>>>> will be unavailable.
> > >>>>> I've put my findings in the JIRA.
> > >>>>>
> > >>>>> Thanks.
> > >>>>> Luke
> > >>>>>
> > >>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > >>>>>
> > >>>>>> Stan, thanks for driving this all forward! Excellent job.
> > >>>>>>
> > >>>>>> About
> > >>>>>>
> > >>>>>>> StreamsStandbyTask -
> > https://issues.apache.org/jira/browse/KAFKA-16141
> > >>>>>>> StreamsUpgradeTest -
> > https://issues.apache.org/jira/browse/KAFKA-16139
> > >>>>>>
> > >>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be
> > fixed
> > >>>>>> now in trunk and 3.7 (and actually also in 3.6...)
> > >>>>>>
> > >>>>>> For `StreamsStandbyTask` the failing test exposes a regression bug,
> > so
> > >>>>>> it's a blocker. I updated the ticket accordingly. We already have an
> > >>>>>> open PR that reverts the code introducing the regression.
> > >>>>>>
> > >>>>>>
> > >>>>>> -Matthias
> > >>>>>>
> > >>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
> > >>>>>>> We have another blocking issue for the RC :
> > >>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is
> > similar
> > >>>>>> to
> > >>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
> > >>>>> however
> > >>>>>>> can lead to the new topic having partitions that a producer cannot
> > >>>>> write
> > >>>>>> to.
> > >>>>>>>
> > >>>>>>> --Proven
> > >>>>>>>
> > >>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> > >>>>>> pprovenzano@confluent.io>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
> > >>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is
> > building
> > >>>>> now.
> > >>>>>>>> --Proven
> > >>>>>>>>
> > >>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz>
> > wrote:
> > >>>>>>>>
> > >>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found
> > is a
> > >>>>>>>>> blocker bug because it *
> > >>>>>>>>> *> will generate huge amount of logspam. I guess we didn't find
> > it in
> > >>>>>>>>> junit
> > >>>>>>>>> tests *
> > >>>>>>>>> *> since logspam doesn't fail the automated tests. But certainly
> > it's
> > >>>>>> not
> > >>>>>>>>> suitable *
> > >>>>>>>>> *> for production. Did you file a JIRA yet?*
> > >>>>>>>>>
> > >>>>>>>>> Hi Colin,
> > >>>>>>>>>
> > >>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> > >>>>>>>>>
> > >>>>>>>>> Thanks & Regards
> > >>>>>>>>> Jakub
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cmccabe@apache.org
> > >
> > >>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Stanislav,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
> > >>>>>> messing
> > >>>>>>>>>> with my mind a bit. I hope this doesn't make people think that
> > we're
> > >>>>>>>>>> farther along than we are, heh.
> > >>>>>>>>>>
> > >>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > >>>>>>>>>>> *> Nice catch! It does seem like we should have gated this
> > behind
> > >>>>> the
> > >>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
> > >>>>> with
> > >>>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
> > >>>>>>>>>>>
> > >>>>>>>>>>> I did not observe any obvious impact. I was able to send and
> > >>>>> receive
> > >>>>>>>>>>> messages as normally. But to be honest, I have no idea what
> > else
> > >>>>>>>>>>> this might impact, so I did not try anything special.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
> > >>>>> through
> > >>>>>>>>> this
> > >>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
> > >>>>> least
> > >>>>>> a
> > >>>>>>>>>>> while). So even if it is just a logged exception without any
> > other
> > >>>>>>>>>> impact I
> > >>>>>>>>>>> wonder if it might scare users from upgrading. But I leave it
> > to
> > >>>>>>>>> others
> > >>>>>>>>>> to
> > >>>>>>>>>>> decide if this is a blocker or not.
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Hi Jakub,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for trying the RC. I think what you found is a blocker
> > bug
> > >>>>>>>>> because
> > >>>>>>>>>> it will generate huge amount of logspam. I guess we didn't find
> > it
> > >>>>> in
> > >>>>>>>>> junit
> > >>>>>>>>>> tests since logspam doesn't fail the automated tests. But
> > certainly
> > >>>>>> it's
> > >>>>>>>>>> not suitable for production. Did you file a JIRA yet?
> > >>>>>>>>>>
> > >>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hey Luke,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This is an interesting problem. Given the fact that the KIP
> > for
> > >>>>>>>>> having a
> > >>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
> > >>>>> calling
> > >>>>>>>>>> this a
> > >>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to
> > KRaft
> > >>>>> in
> > >>>>>>>>>> 3.7.0
> > >>>>>>>>>>>> (given the inability to rollback safely), but if that's true
> > - the
> > >>>>>>>>> same
> > >>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
> > >>>>> expected
> > >>>>>>>>> to
> > >>>>>>>>>> use a
> > >>>>>>>>>>>> patch release for this.
> > >>>>>>>>>>
> > >>>>>>>>>> Hi Luke,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for testing rollback. I think this is a case where the
> > >>>>>>>>>> documentation is wrong. The intention was to for the steps to
> > >>>>>> basically
> > >>>>>>>>> be:
> > >>>>>>>>>>
> > >>>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
> > >>>>>>>>>> 2. take down the kraft quorum
> > >>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
> > >>>>>>>>>> 4. roll all the brokers into zk mode without migration enabled
> > (if
> > >>>>>>>>> desired)
> > >>>>>>>>>>
> > >>>>>>>>>> With these steps, there isn't really unavailability since a ZK
> > >>>>>>>>> controller
> > >>>>>>>>>> can be elected quickly after the kraft quorum is gone.
> > >>>>>>>>>>
> > >>>>>>>>>>>> Further, since we will have a 3.8 release - it is
> > >>>>>>>>>>>> likely we will ultimately recommend users upgrade from that
> > >>>>> version
> > >>>>>>>>>> given
> > >>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
> > >>>>>>>>>>>> That being said, I am not 100% on this. Let me know whether
> > you
> > >>>>>> think
> > >>>>>>>>>> this
> > >>>>>>>>>>>> should block the release, Luke. I am also tagging Colin and
> > David
> > >>>>> to
> > >>>>>>>>>> weigh
> > >>>>>>>>>>>> in with their opinions, as they worked on the migration logic.
> > >>>>>>>>>>
> > >>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong
> > is a
> > >>>>>>>>> clear
> > >>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a
> > PR.
> > >>>>>>>>>>
> > >>>>>>>>>> best,
> > >>>>>>>>>> Colin
> > >>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hey Kirk and Chris,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a
> > bad log
> > >>>>>>>>> due
> > >>>>>>>>>> to
> > >>>>>>>>>>>> improper closing. And the PR description implies this has been
> > >>>>>>>>> present
> > >>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for
> > this to
> > >>>>>>>>> block
> > >>>>>>>>>>>> the release.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hey Jakub,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Nice catch! It does seem like we should have gated this
> > behind the
> > >>>>>>>>>> metadata
> > >>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
> > >>>>> multiple
> > >>>>>>>>> log
> > >>>>>>>>>>>> dirs? What is the impact of the error messages?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Stanislav
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <jakub@scholz.cz
> > >
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I was trying the RC2 and run into the following issue ...
> > when I
> > >>>>>>>>> run
> > >>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> > >>>>>>>>> metadata
> > >>>>>>>>>>>>> version, I seem to be getting repeated errors like this in
> > the
> > >>>>>>>>>> controller
> > >>>>>>>>>>>>> logs:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > >>>>>>>>>>>> assignReplicasToDirs:
> > >>>>>>>>>>>>> event failed with UnsupportedVersionException in 15
> > microseconds.
> > >>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
> > >>>>>>>>>>>>> [quorum-controller-0-event-handler]
> > >>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
> > >>>>> Unexpected
> > >>>>>>>>>> error
> > >>>>>>>>>>>>> handling request
> > RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> > headerVersion=2)
> > >>>>> --
> > >>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > >>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > >>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
> > >>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
> > >>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with
> > context
> > >>>>>>>>>>>>>
> > >>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> > headerVersion=2),
> > >>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > >>>>>>>>>> clientAddress=/
> > >>>>>>>>>>>>> 172.16.14.217,
> > principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > >>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
> > >>>>> securityProtocol=SSL,
> > >>>>>>>>>>>>>
> > >>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > >>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > >>>>>>>>>>>>> ])
> > >>>>>>>>>>>>> (kafka.server.ControllerApis)
> > [quorum-controller-0-event-handler]
> > >>>>>>>>>>>>> java.util.concurrent.CompletionException:
> > >>>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >>>>>>>>> Directory
> > >>>>>>>>>>>>> assignment is not supported yet.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > >>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Caused by:
> > >>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >>>>>>>>>>>>> Directory assignment is not supported yet.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Is that expected? I guess with the metadata version set to
> > >>>>>>>>> 3.6-IV2, it
> > >>>>>>>>>>>>> makes sense that the request is not supported. But shouldn't
> > then
> > >>>>>>>>> the
> > >>>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened
> > a
> > >>>>> JIRA
> > >>>>>>>>>> for
> > >>>>>>>>>>>> it,
> > >>>>>>>>>>>>> but I can open one if you agree this is not expected)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks & Regards
> > >>>>>>>>>>>>> Jakub
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <showuon@gmail.com
> > >
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi Stanislav,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
> > >>>>> maybe
> > >>>>>>>>>> you
> > >>>>>>>>>>>>>> missed it.
> > >>>>>>>>>>>>>> cross-posting here:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> There is a bug KAFKA-16101
> > >>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101>
> > reporting
> > >>>>>>>>> that
> > >>>>>>>>>>>>> "Kafka
> > >>>>>>>>>>>>>> cluster will be unavailable during KRaft migration
> > rollback".
> > >>>>>>>>>>>>>> The impact for this issue is that if brokers try to
> > rollback to
> > >>>>>>>>> ZK
> > >>>>>>>>>> mode
> > >>>>>>>>>>>>>> during KRaft migration process, there will be a period of
> > time
> > >>>>>>>>> the
> > >>>>>>>>>>>>> cluster
> > >>>>>>>>>>>>>> is unavailable.
> > >>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
> > >>>>>>>>> feature, I
> > >>>>>>>>>>>>> think
> > >>>>>>>>>>>>>> this should be addressed soon.
> > >>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>>>> Luke
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > >>>>>>>>>> fearthecellos@gmail.com
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks, Kirk!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Chris/Stanislav,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler'
> > log
> > >>>>>>>>>> problem
> > >>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> > >>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use
> > the
> > >>>>>>>>>>>>> quickstart
> > >>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I
> > wait
> > >>>>>>>>> for
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> PR's
> > >>>>>>>>>>>>>>>> Jenkins job to finish.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>> Kirk
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > >>>>>>>>>>>>>>>>> Hi Stanislav,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for running this release!
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> To verify, I:
> > >>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
> > >>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> > >>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>>>>>>>>>>>>> - Checked signatures and checksums
> > >>>>>>>>>>>>>>>>> - Ran the quickstart using both:
> > >>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>>>>>>>> with
> > >>>>>>>>>>>>>> Java
> > >>>>>>>>>>>>>>>> 11
> > >>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
> > >>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
> > >>>>>>>>> apache/kafka:3.7.0-rc2
> > >>>>>>>>>>>>>>>>> - Ran all unit tests
> > >>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I found two minor areas for concern:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
> > >>>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level
> > log
> > >>>>>>>>>>>> message
> > >>>>>>>>>>>>>>> being
> > >>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
> > >>>>>>>>>> console
> > >>>>>>>>>>>>>>> consumer
> > >>>>>>>>>>>>>>>>> via ctrl-C:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> > >>>>>>>>>>>>>> clientId=console-consumer,
> > >>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> > >>>>>>>>>>>> FetchSessionHandler
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> node
> > >>>>>>>>>>>>>>>>> 1. Ignoring fetch response
> > >>>>>>>>>>>>>>>>>
> > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I see that this error message is already reported in
> > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> > >>>>>>>>> think we
> > >>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's
> > probably
> > >>>>>>>>>>>> benign
> > >>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>>> really not a good look for us when basic operations log
> > >>>>>>>>> error
> > >>>>>>>>>>>>>> messages,
> > >>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> it may give new users some headaches.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
> > >>>>>>>>>>>>>>>>> The following unit tests failed the first time around,
> > and
> > >>>>>>>>>> all of
> > >>>>>>>>>>>>>> them
> > >>>>>>>>>>>>>>>>> passed the second time I ran them:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> - (clients)
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > >>>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
> > >>>>>>>>>>>>>>>>> - (clients)
> > >>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
> > >>>>>>>>>>>>>>>>> - (connect)
> > >>>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > >>>>>>>>>>>>>>> (I
> > >>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
> > >>>>>>>>>>>>>>>>> - (core)
> > >>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
> > >>>>>>>>>> congratulations
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> Kafka
> > >>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
> > >>>>>>>>>>>>>> highly-experimental
> > >>>>>>>>>>>>>>>>> single laptop run!
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Chris
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > >>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
> > >>>>>>>>>> 3.7.0.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
> > >>>>>>>>> that
> > >>>>>>>>>> I
> > >>>>>>>>>>>> had
> > >>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> > >>>>>>>>> announcing
> > >>>>>>>>>>>> due
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
> > >>>>>>>>> tests
> > >>>>>>>>>>>>>>>> successfully.
> > >>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
> > >>>>>>>>> that I
> > >>>>>>>>>>>> have
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> spend
> > >>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case
> > anyone
> > >>>>>>>>>>>> notices
> > >>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>> tests
> > >>>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
> > >>>>>>>>> me
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>> follow
> > >>>>>>>>>>>>>>>> up in
> > >>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Major changes include:
> > >>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
> > >>>>>>>>>> consumer
> > >>>>>>>>>>>>>>>> rebalance
> > >>>>>>>>>>>>>>>>>> protocol
> > >>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> > >>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> > >>>>>>>>>> standardized
> > >>>>>>>>>>>>>>>> interface
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
> > >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
> > >>>>>>>>> 18,
> > >>>>>>>>>> 9am
> > >>>>>>>>>>>>> PT
> > >>>>>>>>>>>>>>> ***
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
> > >>>>>>>>> this
> > >>>>>>>>>>>>> being
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
> > >>>>>>>>> it a
> > >>>>>>>>>> bit
> > >>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
> > >>>>>>>>>>>> release:
> > >>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and
> > binary):
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
> > >>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Javadoc:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0
> > tag:
> > >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Documentation:
> > >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Protocol:
> > >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> > >>>>>>>>>>>>>>>>>> Unit/integration tests:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > >>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
> > >>>>>>>>>> triaging
> > >>>>>>>>>>>>>> some
> > >>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
> > >>>>>>>>> or
> > >>>>>>>>>>>>> simply
> > >>>>>>>>>>>>>>>> flakes.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> System tests:
> > >>>>>>>>>>>>>>>>
> > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
> > >>>>>>>>> getting
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> job
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> run.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for
> > 3.7
> > >>>>>>>>>>>> branch:
> > >>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
> > >>>>>>>>>> from
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> Docker
> > >>>>>>>>>>>>>>>>>> Build run:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
> > >>>>>>>>>>>>> Pipeline:
> > >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> /**************************************
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>> Stanislav Kozlovski
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Stanislav
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best,
> > >>>> Stanislav
> > >>>
> > >>
> > >
> >
> >
>
> --
> Best,
> Stanislav


Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
Thanks for the work everybody. Providing a status update at the end of the
week:

- docs change explaining migration
<https://github.com/apache/kafka/pull/15193> was merged
- the blocker KAFKA-16162 <https://github.com/apache/kafka/pull/15270> was
merged
- the blocker KAFKA-14616 <https://github.com/apache/kafka/pull/15230> was
merged
- a small blocker problem with the shadow jar plugin
<https://github.com/apache/kafka/pull/15308>
- the blockers KAFKALESS-16157 & KAFKALESS-16195 aren't merged
- the good-to-have KAFKA-16082 isn't merged

I think we should prioritize merging KAFKALESS-16195 and *call JBOD EA*. I
question whether we may find more blocker bugs in the next RC.
The release is late by approximately a month so far, so I do want to scope
down aggressively to meet the time-based goal.

Best,
Stanislav

On Mon, Jan 29, 2024 at 5:46 PM Omnia Ibrahim <o....@gmail.com>
wrote:

> Hi Stan and Gaurav,
> Just to clarify some points mentioned here before
>  KAFKA-14616: I raised a year ago so it's not related to JBOD work. It is
> rather a blocker bug for KRAFT in general. The PR from Colin should fix
> this. Am not sure if it is a blocker for 3.7 per-say as it was a major bug
> since 3.3 and got missed from all other releases.
>
> Regarding the JBOD's work:
> KAFKA-16082:  Is not a blocker for 3.7 instead it's nice fix. The pr
> https://github.com/apache/kafka/pull/15136 is quite a small one and was
> approved by Proven and I but it is waiting for a committer's approval.
> KAFKA-16162: This is a blocker for 3.7.  Same it’s a small pr
> https://github.com/apache/kafka/pull/15270 and it is approved Proven and
> I and the PR is waiting for committer's approval.
> KAFKA-16157: This is a blocker for 3.7. There is one small suggestion for
> the pr https://github.com/apache/kafka/pull/15263 but I don't think any
> of the current feedback is blocking the pr from getting approved. Assuming
> we get a committer's approval on it.
> KAFKA-16195:  Same it's a blocker but it has approval from Proven and I
> and we are waiting for committer's approval on the pr
> https://github.com/apache/kafka/pull/15262.
>
> If we can’t get a committer approval for KAFKA-16162, KAFKA-16157 and
> KAFKA-16195  in time for 3.7 then we can mark JBOD as early release
> assuming we merge at least KAFKA-16195.
>
> Regards,
> Omnia
>
> > On 26 Jan 2024, at 15:39, kafka@gnarula.com wrote:
> >
> > Apologies, I duplicated KAFKA-16157 twice in my previous message. I
> intended to mention KAFKA-16195
> > with the PR at https://github.com/apache/kafka/pull/15262 as the second
> JIRA.
> >
> > Thanks,
> > Gaurav
> >
> >> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
> >>
> >> Hi Stan,
> >>
> >> I wanted to share some updates about the bugs you shared earlier.
> >>
> >> - KAFKA-14616: I've reviewed and tested the PR from Colin and have
> observed
> >> the fix works as intended.
> >> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the
> proposed fix. I've
> >> therefore raised https://github.com/apache/kafka/pull/15270 following
> a discussion with Luke in JIRA.
> >> - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm
> awaiting
> >> feedback/reviews at https://github.com/apache/kafka/pull/15136
> >>
> >> In addition to the above, there are 2 JIRAs I'd like to bring
> everyone's attention to:
> >>
> >> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a
> blocker. I've raised
> >> https://github.com/apache/kafka/pull/15263 and am awaiting reviews on
> it.
> >> - KAFKA-16157: I raised this yesterday and have addressed feedback from
> Luke. This should
> >> hopefully get merged soon.
> >>
> >> Regards,
> >> Gaurav
> >>
> >>
> >>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
> >>>
> >>> Hi Stanislav,
> >>>
> >>> Thanks for bringing these JIRAs/PRs up.
> >>>
> >>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week
> and I hope to have some feedback
> >>> by Friday. I gather the latter JIRA is marked as a WIP by Proven and
> he's away. I'll try to build on his work in the meantime.
> >>>
> >>> As for KAFKA-16082, we haven't been able to deduce a data loss
> scenario. There's a PR open
> >>> by me for promoting an abandoned future replica with approvals from
> Omnia and Proven,
> >>> so I'd appreciate a committer reviewing it.
> >>>
> >>> Regards,
> >>> Gaurav
> >>>
> >>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID>
> wrote:
> >>>>
> >>>> Hey all, I figured I'd give an update about what known blockers we
> have
> >>>> right now:
> >>>>
> >>>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
> >>>> https://github.com/apache/kafka/pull/15193; This need not block RC
> >>>> creation, but we need the docs updated so that people can test
> properly
> >>>> - KAFKA-14616: Topic recreation with offline broker causes permanent
> URPs -
> >>>> https://github.com/apache/kafka/pull/15230 ; I am of the
> understanding that
> >>>> this is blocking JBOD for 3.7
> >>>> - KAFKA-16162: New created topics are unavailable after upgrading to
> 3.7 -
> >>>> a strict blocker with an open PR
> https://github.com/apache/kafka/pull/15232
> >>>> - although I understand Proveen is out of office
> >>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition -
> I am
> >>>> hearing mixed opinions on whether this is a blocker (
> >>>> https://github.com/apache/kafka/pull/15136)
> >>>>
> >>>> Given that there are 3 JBOD blocker bugs, and I am not confident they
> will
> >>>> all be merged this week - I am on the edge of voting to revert JBOD
> from
> >>>> this release, or mark it early access.
> >>>>
> >>>> By all accounts, it seems that if we keep with JBOD the release will
> have
> >>>> to spill into February, which is a month extra from the time-based
> release
> >>>> plan we had of start of January.
> >>>>
> >>>> Can I ask others for an opinion?
> >>>>
> >>>> Best,
> >>>> Stan
> >>>>
> >>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> I think I've found another blocker issue: KAFKA-16162
> >>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
> >>>>> The impact is after upgrading to 3.7.0, any new created
> topics/partitions
> >>>>> will be unavailable.
> >>>>> I've put my findings in the JIRA.
> >>>>>
> >>>>> Thanks.
> >>>>> Luke
> >>>>>
> >>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >>>>>
> >>>>>> Stan, thanks for driving this all forward! Excellent job.
> >>>>>>
> >>>>>> About
> >>>>>>
> >>>>>>> StreamsStandbyTask -
> https://issues.apache.org/jira/browse/KAFKA-16141
> >>>>>>> StreamsUpgradeTest -
> https://issues.apache.org/jira/browse/KAFKA-16139
> >>>>>>
> >>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be
> fixed
> >>>>>> now in trunk and 3.7 (and actually also in 3.6...)
> >>>>>>
> >>>>>> For `StreamsStandbyTask` the failing test exposes a regression bug,
> so
> >>>>>> it's a blocker. I updated the ticket accordingly. We already have an
> >>>>>> open PR that reverts the code introducing the regression.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
> >>>>>>> We have another blocking issue for the RC :
> >>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is
> similar
> >>>>>> to
> >>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
> >>>>> however
> >>>>>>> can lead to the new topic having partitions that a producer cannot
> >>>>> write
> >>>>>> to.
> >>>>>>>
> >>>>>>> --Proven
> >>>>>>>
> >>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> >>>>>> pprovenzano@confluent.io>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
> >>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is
> building
> >>>>> now.
> >>>>>>>> --Proven
> >>>>>>>>
> >>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz>
> wrote:
> >>>>>>>>
> >>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found
> is a
> >>>>>>>>> blocker bug because it *
> >>>>>>>>> *> will generate huge amount of logspam. I guess we didn't find
> it in
> >>>>>>>>> junit
> >>>>>>>>> tests *
> >>>>>>>>> *> since logspam doesn't fail the automated tests. But certainly
> it's
> >>>>>> not
> >>>>>>>>> suitable *
> >>>>>>>>> *> for production. Did you file a JIRA yet?*
> >>>>>>>>>
> >>>>>>>>> Hi Colin,
> >>>>>>>>>
> >>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> >>>>>>>>>
> >>>>>>>>> Thanks & Regards
> >>>>>>>>> Jakub
> >>>>>>>>>
> >>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cmccabe@apache.org
> >
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Stanislav,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
> >>>>>> messing
> >>>>>>>>>> with my mind a bit. I hope this doesn't make people think that
> we're
> >>>>>>>>>> farther along than we are, heh.
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> >>>>>>>>>>> *> Nice catch! It does seem like we should have gated this
> behind
> >>>>> the
> >>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
> >>>>> with
> >>>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
> >>>>>>>>>>>
> >>>>>>>>>>> I did not observe any obvious impact. I was able to send and
> >>>>> receive
> >>>>>>>>>>> messages as normally. But to be honest, I have no idea what
> else
> >>>>>>>>>>> this might impact, so I did not try anything special.
> >>>>>>>>>>>
> >>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
> >>>>> through
> >>>>>>>>> this
> >>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
> >>>>> least
> >>>>>> a
> >>>>>>>>>>> while). So even if it is just a logged exception without any
> other
> >>>>>>>>>> impact I
> >>>>>>>>>>> wonder if it might scare users from upgrading. But I leave it
> to
> >>>>>>>>> others
> >>>>>>>>>> to
> >>>>>>>>>>> decide if this is a blocker or not.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hi Jakub,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for trying the RC. I think what you found is a blocker
> bug
> >>>>>>>>> because
> >>>>>>>>>> it will generate huge amount of logspam. I guess we didn't find
> it
> >>>>> in
> >>>>>>>>> junit
> >>>>>>>>>> tests since logspam doesn't fail the automated tests. But
> certainly
> >>>>>> it's
> >>>>>>>>>> not suitable for production. Did you file a JIRA yet?
> >>>>>>>>>>
> >>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hey Luke,
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is an interesting problem. Given the fact that the KIP
> for
> >>>>>>>>> having a
> >>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
> >>>>> calling
> >>>>>>>>>> this a
> >>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to
> KRaft
> >>>>> in
> >>>>>>>>>> 3.7.0
> >>>>>>>>>>>> (given the inability to rollback safely), but if that's true
> - the
> >>>>>>>>> same
> >>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
> >>>>> expected
> >>>>>>>>> to
> >>>>>>>>>> use a
> >>>>>>>>>>>> patch release for this.
> >>>>>>>>>>
> >>>>>>>>>> Hi Luke,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for testing rollback. I think this is a case where the
> >>>>>>>>>> documentation is wrong. The intention was to for the steps to
> >>>>>> basically
> >>>>>>>>> be:
> >>>>>>>>>>
> >>>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
> >>>>>>>>>> 2. take down the kraft quorum
> >>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
> >>>>>>>>>> 4. roll all the brokers into zk mode without migration enabled
> (if
> >>>>>>>>> desired)
> >>>>>>>>>>
> >>>>>>>>>> With these steps, there isn't really unavailability since a ZK
> >>>>>>>>> controller
> >>>>>>>>>> can be elected quickly after the kraft quorum is gone.
> >>>>>>>>>>
> >>>>>>>>>>>> Further, since we will have a 3.8 release - it is
> >>>>>>>>>>>> likely we will ultimately recommend users upgrade from that
> >>>>> version
> >>>>>>>>>> given
> >>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
> >>>>>>>>>>>> That being said, I am not 100% on this. Let me know whether
> you
> >>>>>> think
> >>>>>>>>>> this
> >>>>>>>>>>>> should block the release, Luke. I am also tagging Colin and
> David
> >>>>> to
> >>>>>>>>>> weigh
> >>>>>>>>>>>> in with their opinions, as they worked on the migration logic.
> >>>>>>>>>>
> >>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong
> is a
> >>>>>>>>> clear
> >>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a
> PR.
> >>>>>>>>>>
> >>>>>>>>>> best,
> >>>>>>>>>> Colin
> >>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hey Kirk and Chris,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a
> bad log
> >>>>>>>>> due
> >>>>>>>>>> to
> >>>>>>>>>>>> improper closing. And the PR description implies this has been
> >>>>>>>>> present
> >>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for
> this to
> >>>>>>>>> block
> >>>>>>>>>>>> the release.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hey Jakub,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nice catch! It does seem like we should have gated this
> behind the
> >>>>>>>>>> metadata
> >>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
> >>>>> multiple
> >>>>>>>>> log
> >>>>>>>>>>>> dirs? What is the impact of the error messages?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Stanislav
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <jakub@scholz.cz
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I was trying the RC2 and run into the following issue ...
> when I
> >>>>>>>>> run
> >>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> >>>>>>>>> metadata
> >>>>>>>>>>>>> version, I seem to be getting repeated errors like this in
> the
> >>>>>>>>>> controller
> >>>>>>>>>>>>> logs:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> >>>>>>>>>>>> assignReplicasToDirs:
> >>>>>>>>>>>>> event failed with UnsupportedVersionException in 15
> microseconds.
> >>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
> >>>>>>>>>>>>> [quorum-controller-0-event-handler]
> >>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
> >>>>> Unexpected
> >>>>>>>>>> error
> >>>>>>>>>>>>> handling request
> RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> headerVersion=2)
> >>>>> --
> >>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> >>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> >>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
> >>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
> >>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with
> context
> >>>>>>>>>>>>>
> >>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> headerVersion=2),
> >>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> >>>>>>>>>> clientAddress=/
> >>>>>>>>>>>>> 172.16.14.217,
> principal=User:CN=my-cluster-kafka,O=io.strimzi,
> >>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
> >>>>> securityProtocol=SSL,
> >>>>>>>>>>>>>
> >>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> >>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> >>>>>>>>>>>>> ])
> >>>>>>>>>>>>> (kafka.server.ControllerApis)
> [quorum-controller-0-event-handler]
> >>>>>>>>>>>>> java.util.concurrent.CompletionException:
> >>>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> >>>>>>>>> Directory
> >>>>>>>>>>>>> assignment is not supported yet.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> >>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Caused by:
> >>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> >>>>>>>>>>>>> Directory assignment is not supported yet.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is that expected? I guess with the metadata version set to
> >>>>>>>>> 3.6-IV2, it
> >>>>>>>>>>>>> makes sense that the request is not supported. But shouldn't
> then
> >>>>>>>>> the
> >>>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened
> a
> >>>>> JIRA
> >>>>>>>>>> for
> >>>>>>>>>>>> it,
> >>>>>>>>>>>>> but I can open one if you agree this is not expected)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks & Regards
> >>>>>>>>>>>>> Jakub
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <showuon@gmail.com
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Stanislav,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
> >>>>> maybe
> >>>>>>>>>> you
> >>>>>>>>>>>>>> missed it.
> >>>>>>>>>>>>>> cross-posting here:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> There is a bug KAFKA-16101
> >>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101>
> reporting
> >>>>>>>>> that
> >>>>>>>>>>>>> "Kafka
> >>>>>>>>>>>>>> cluster will be unavailable during KRaft migration
> rollback".
> >>>>>>>>>>>>>> The impact for this issue is that if brokers try to
> rollback to
> >>>>>>>>> ZK
> >>>>>>>>>> mode
> >>>>>>>>>>>>>> during KRaft migration process, there will be a period of
> time
> >>>>>>>>> the
> >>>>>>>>>>>>> cluster
> >>>>>>>>>>>>>> is unavailable.
> >>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
> >>>>>>>>> feature, I
> >>>>>>>>>>>>> think
> >>>>>>>>>>>>>> this should be addressed soon.
> >>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> >>>>>>>>>> fearthecellos@gmail.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, Kirk!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Chris/Stanislav,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler'
> log
> >>>>>>>>>> problem
> >>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> >>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use
> the
> >>>>>>>>>>>>> quickstart
> >>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I
> wait
> >>>>>>>>> for
> >>>>>>>>>> the
> >>>>>>>>>>>>>> PR's
> >>>>>>>>>>>>>>>> Jenkins job to finish.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Kirk
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> >>>>>>>>>>>>>>>>> Hi Stanislav,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for running this release!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> To verify, I:
> >>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
> >>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> >>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> >>>>>>>>>>>>>>>>>
> >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>>>>>>>> - Checked signatures and checksums
> >>>>>>>>>>>>>>>>> - Ran the quickstart using both:
> >>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> >>>>>>>>>>>>>>>>>
> >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>> Java
> >>>>>>>>>>>>>>>> 11
> >>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
> >>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
> >>>>>>>>> apache/kafka:3.7.0-rc2
> >>>>>>>>>>>>>>>>> - Ran all unit tests
> >>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I found two minor areas for concern:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
> >>>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level
> log
> >>>>>>>>>>>> message
> >>>>>>>>>>>>>>> being
> >>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
> >>>>>>>>>> console
> >>>>>>>>>>>>>>> consumer
> >>>>>>>>>>>>>>>>> via ctrl-C:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> >>>>>>>>>>>>>> clientId=console-consumer,
> >>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> >>>>>>>>>>>> FetchSessionHandler
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> node
> >>>>>>>>>>>>>>>>> 1. Ignoring fetch response
> >>>>>>>>>>>>>>>>>
> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I see that this error message is already reported in
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> >>>>>>>>> think we
> >>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's
> probably
> >>>>>>>>>>>> benign
> >>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>> really not a good look for us when basic operations log
> >>>>>>>>> error
> >>>>>>>>>>>>>> messages,
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> it may give new users some headaches.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
> >>>>>>>>>>>>>>>>> The following unit tests failed the first time around,
> and
> >>>>>>>>>> all of
> >>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>> passed the second time I ran them:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - (clients)
> >>>>>>>>>>>>>>>>
> >>>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> >>>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
> >>>>>>>>>>>>>>>>> - (clients)
> >>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
> >>>>>>>>>>>>>>>>> - (connect)
> >>>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> >>>>>>>>>>>>>>> (I
> >>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
> >>>>>>>>>>>>>>>>> - (core)
> >>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
> >>>>>>>>>> congratulations
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
> >>>>>>>>>>>>>> highly-experimental
> >>>>>>>>>>>>>>>>> single laptop run!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> >>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
> >>>>>>>>>> 3.7.0.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
> >>>>>>>>> that
> >>>>>>>>>> I
> >>>>>>>>>>>> had
> >>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> >>>>>>>>> announcing
> >>>>>>>>>>>> due
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
> >>>>>>>>> tests
> >>>>>>>>>>>>>>>> successfully.
> >>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
> >>>>>>>>> that I
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> spend
> >>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case
> anyone
> >>>>>>>>>>>> notices
> >>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
> >>>>>>>>> me
> >>>>>>>>>> to
> >>>>>>>>>>>>>> follow
> >>>>>>>>>>>>>>>> up in
> >>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Major changes include:
> >>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
> >>>>>>>>>> consumer
> >>>>>>>>>>>>>>>> rebalance
> >>>>>>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> >>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> >>>>>>>>>> standardized
> >>>>>>>>>>>>>>>> interface
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
> >>>>>>>>> 18,
> >>>>>>>>>> 9am
> >>>>>>>>>>>>> PT
> >>>>>>>>>>>>>>> ***
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
> >>>>>>>>> this
> >>>>>>>>>>>>> being
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
> >>>>>>>>> it a
> >>>>>>>>>> bit
> >>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
> >>>>>>>>>>>> release:
> >>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and
> binary):
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
> >>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Javadoc:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0
> tag:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Documentation:
> >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Protocol:
> >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> >>>>>>>>>>>>>>>>>> Unit/integration tests:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> >>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
> >>>>>>>>>> triaging
> >>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
> >>>>>>>>> or
> >>>>>>>>>>>>> simply
> >>>>>>>>>>>>>>>> flakes.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> System tests:
> >>>>>>>>>>>>>>>>
> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
> >>>>>>>>> getting
> >>>>>>>>>> the
> >>>>>>>>>>>>> job
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> run.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for
> 3.7
> >>>>>>>>>>>> branch:
> >>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
> >>>>>>>>>> from
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> Docker
> >>>>>>>>>>>>>>>>>> Build run:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
> >>>>>>>>>>>>> Pipeline:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> /**************************************
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> Stanislav Kozlovski
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Stanislav
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best,
> >>>> Stanislav
> >>>
> >>
> >
>
>

-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
Thanks for the work everybody. Providing a status update at the end of the
week:

- docs change explaining migration
<https://github.com/apache/kafka/pull/15193> was merged
- the blocker KAFKA-16162 <https://github.com/apache/kafka/pull/15270> was
merged
- the blocker KAFKA-14616 <https://github.com/apache/kafka/pull/15230> was
merged
- a small blocker problem with the shadow jar plugin
<https://github.com/apache/kafka/pull/15308>
- the blockers KAFKALESS-16157 & KAFKALESS-16195 aren't merged
- the good-to-have KAFKA-16082 isn't merged

I think we should prioritize merging KAFKALESS-16195 and *call JBOD EA*. I
question whether we may find more blocker bugs in the next RC.
The release is late by approximately a month so far, so I do want to scope
down aggressively to meet the time-based goal.

Best,
Stanislav

On Mon, Jan 29, 2024 at 5:46 PM Omnia Ibrahim <o....@gmail.com>
wrote:

> Hi Stan and Gaurav,
> Just to clarify some points mentioned here before
>  KAFKA-14616: I raised a year ago so it's not related to JBOD work. It is
> rather a blocker bug for KRAFT in general. The PR from Colin should fix
> this. Am not sure if it is a blocker for 3.7 per-say as it was a major bug
> since 3.3 and got missed from all other releases.
>
> Regarding the JBOD's work:
> KAFKA-16082:  Is not a blocker for 3.7 instead it's nice fix. The pr
> https://github.com/apache/kafka/pull/15136 is quite a small one and was
> approved by Proven and I but it is waiting for a committer's approval.
> KAFKA-16162: This is a blocker for 3.7.  Same it’s a small pr
> https://github.com/apache/kafka/pull/15270 and it is approved Proven and
> I and the PR is waiting for committer's approval.
> KAFKA-16157: This is a blocker for 3.7. There is one small suggestion for
> the pr https://github.com/apache/kafka/pull/15263 but I don't think any
> of the current feedback is blocking the pr from getting approved. Assuming
> we get a committer's approval on it.
> KAFKA-16195:  Same it's a blocker but it has approval from Proven and I
> and we are waiting for committer's approval on the pr
> https://github.com/apache/kafka/pull/15262.
>
> If we can’t get a committer approval for KAFKA-16162, KAFKA-16157 and
> KAFKA-16195  in time for 3.7 then we can mark JBOD as early release
> assuming we merge at least KAFKA-16195.
>
> Regards,
> Omnia
>
> > On 26 Jan 2024, at 15:39, kafka@gnarula.com wrote:
> >
> > Apologies, I duplicated KAFKA-16157 twice in my previous message. I
> intended to mention KAFKA-16195
> > with the PR at https://github.com/apache/kafka/pull/15262 as the second
> JIRA.
> >
> > Thanks,
> > Gaurav
> >
> >> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
> >>
> >> Hi Stan,
> >>
> >> I wanted to share some updates about the bugs you shared earlier.
> >>
> >> - KAFKA-14616: I've reviewed and tested the PR from Colin and have
> observed
> >> the fix works as intended.
> >> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the
> proposed fix. I've
> >> therefore raised https://github.com/apache/kafka/pull/15270 following
> a discussion with Luke in JIRA.
> >> - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm
> awaiting
> >> feedback/reviews at https://github.com/apache/kafka/pull/15136
> >>
> >> In addition to the above, there are 2 JIRAs I'd like to bring
> everyone's attention to:
> >>
> >> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a
> blocker. I've raised
> >> https://github.com/apache/kafka/pull/15263 and am awaiting reviews on
> it.
> >> - KAFKA-16157: I raised this yesterday and have addressed feedback from
> Luke. This should
> >> hopefully get merged soon.
> >>
> >> Regards,
> >> Gaurav
> >>
> >>
> >>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
> >>>
> >>> Hi Stanislav,
> >>>
> >>> Thanks for bringing these JIRAs/PRs up.
> >>>
> >>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week
> and I hope to have some feedback
> >>> by Friday. I gather the latter JIRA is marked as a WIP by Proven and
> he's away. I'll try to build on his work in the meantime.
> >>>
> >>> As for KAFKA-16082, we haven't been able to deduce a data loss
> scenario. There's a PR open
> >>> by me for promoting an abandoned future replica with approvals from
> Omnia and Proven,
> >>> so I'd appreciate a committer reviewing it.
> >>>
> >>> Regards,
> >>> Gaurav
> >>>
> >>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID>
> wrote:
> >>>>
> >>>> Hey all, I figured I'd give an update about what known blockers we
> have
> >>>> right now:
> >>>>
> >>>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
> >>>> https://github.com/apache/kafka/pull/15193; This need not block RC
> >>>> creation, but we need the docs updated so that people can test
> properly
> >>>> - KAFKA-14616: Topic recreation with offline broker causes permanent
> URPs -
> >>>> https://github.com/apache/kafka/pull/15230 ; I am of the
> understanding that
> >>>> this is blocking JBOD for 3.7
> >>>> - KAFKA-16162: New created topics are unavailable after upgrading to
> 3.7 -
> >>>> a strict blocker with an open PR
> https://github.com/apache/kafka/pull/15232
> >>>> - although I understand Proveen is out of office
> >>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition -
> I am
> >>>> hearing mixed opinions on whether this is a blocker (
> >>>> https://github.com/apache/kafka/pull/15136)
> >>>>
> >>>> Given that there are 3 JBOD blocker bugs, and I am not confident they
> will
> >>>> all be merged this week - I am on the edge of voting to revert JBOD
> from
> >>>> this release, or mark it early access.
> >>>>
> >>>> By all accounts, it seems that if we keep with JBOD the release will
> have
> >>>> to spill into February, which is a month extra from the time-based
> release
> >>>> plan we had of start of January.
> >>>>
> >>>> Can I ask others for an opinion?
> >>>>
> >>>> Best,
> >>>> Stan
> >>>>
> >>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> I think I've found another blocker issue: KAFKA-16162
> >>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
> >>>>> The impact is after upgrading to 3.7.0, any new created
> topics/partitions
> >>>>> will be unavailable.
> >>>>> I've put my findings in the JIRA.
> >>>>>
> >>>>> Thanks.
> >>>>> Luke
> >>>>>
> >>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >>>>>
> >>>>>> Stan, thanks for driving this all forward! Excellent job.
> >>>>>>
> >>>>>> About
> >>>>>>
> >>>>>>> StreamsStandbyTask -
> https://issues.apache.org/jira/browse/KAFKA-16141
> >>>>>>> StreamsUpgradeTest -
> https://issues.apache.org/jira/browse/KAFKA-16139
> >>>>>>
> >>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be
> fixed
> >>>>>> now in trunk and 3.7 (and actually also in 3.6...)
> >>>>>>
> >>>>>> For `StreamsStandbyTask` the failing test exposes a regression bug,
> so
> >>>>>> it's a blocker. I updated the ticket accordingly. We already have an
> >>>>>> open PR that reverts the code introducing the regression.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
> >>>>>>> We have another blocking issue for the RC :
> >>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is
> similar
> >>>>>> to
> >>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
> >>>>> however
> >>>>>>> can lead to the new topic having partitions that a producer cannot
> >>>>> write
> >>>>>> to.
> >>>>>>>
> >>>>>>> --Proven
> >>>>>>>
> >>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> >>>>>> pprovenzano@confluent.io>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
> >>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is
> building
> >>>>> now.
> >>>>>>>> --Proven
> >>>>>>>>
> >>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz>
> wrote:
> >>>>>>>>
> >>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found
> is a
> >>>>>>>>> blocker bug because it *
> >>>>>>>>> *> will generate huge amount of logspam. I guess we didn't find
> it in
> >>>>>>>>> junit
> >>>>>>>>> tests *
> >>>>>>>>> *> since logspam doesn't fail the automated tests. But certainly
> it's
> >>>>>> not
> >>>>>>>>> suitable *
> >>>>>>>>> *> for production. Did you file a JIRA yet?*
> >>>>>>>>>
> >>>>>>>>> Hi Colin,
> >>>>>>>>>
> >>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> >>>>>>>>>
> >>>>>>>>> Thanks & Regards
> >>>>>>>>> Jakub
> >>>>>>>>>
> >>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cmccabe@apache.org
> >
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Stanislav,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
> >>>>>> messing
> >>>>>>>>>> with my mind a bit. I hope this doesn't make people think that
> we're
> >>>>>>>>>> farther along than we are, heh.
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> >>>>>>>>>>> *> Nice catch! It does seem like we should have gated this
> behind
> >>>>> the
> >>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
> >>>>> with
> >>>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
> >>>>>>>>>>>
> >>>>>>>>>>> I did not observe any obvious impact. I was able to send and
> >>>>> receive
> >>>>>>>>>>> messages as normally. But to be honest, I have no idea what
> else
> >>>>>>>>>>> this might impact, so I did not try anything special.
> >>>>>>>>>>>
> >>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
> >>>>> through
> >>>>>>>>> this
> >>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
> >>>>> least
> >>>>>> a
> >>>>>>>>>>> while). So even if it is just a logged exception without any
> other
> >>>>>>>>>> impact I
> >>>>>>>>>>> wonder if it might scare users from upgrading. But I leave it
> to
> >>>>>>>>> others
> >>>>>>>>>> to
> >>>>>>>>>>> decide if this is a blocker or not.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hi Jakub,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for trying the RC. I think what you found is a blocker
> bug
> >>>>>>>>> because
> >>>>>>>>>> it will generate huge amount of logspam. I guess we didn't find
> it
> >>>>> in
> >>>>>>>>> junit
> >>>>>>>>>> tests since logspam doesn't fail the automated tests. But
> certainly
> >>>>>> it's
> >>>>>>>>>> not suitable for production. Did you file a JIRA yet?
> >>>>>>>>>>
> >>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hey Luke,
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is an interesting problem. Given the fact that the KIP
> for
> >>>>>>>>> having a
> >>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
> >>>>> calling
> >>>>>>>>>> this a
> >>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to
> KRaft
> >>>>> in
> >>>>>>>>>> 3.7.0
> >>>>>>>>>>>> (given the inability to rollback safely), but if that's true
> - the
> >>>>>>>>> same
> >>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
> >>>>> expected
> >>>>>>>>> to
> >>>>>>>>>> use a
> >>>>>>>>>>>> patch release for this.
> >>>>>>>>>>
> >>>>>>>>>> Hi Luke,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for testing rollback. I think this is a case where the
> >>>>>>>>>> documentation is wrong. The intention was to for the steps to
> >>>>>> basically
> >>>>>>>>> be:
> >>>>>>>>>>
> >>>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
> >>>>>>>>>> 2. take down the kraft quorum
> >>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
> >>>>>>>>>> 4. roll all the brokers into zk mode without migration enabled
> (if
> >>>>>>>>> desired)
> >>>>>>>>>>
> >>>>>>>>>> With these steps, there isn't really unavailability since a ZK
> >>>>>>>>> controller
> >>>>>>>>>> can be elected quickly after the kraft quorum is gone.
> >>>>>>>>>>
> >>>>>>>>>>>> Further, since we will have a 3.8 release - it is
> >>>>>>>>>>>> likely we will ultimately recommend users upgrade from that
> >>>>> version
> >>>>>>>>>> given
> >>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
> >>>>>>>>>>>> That being said, I am not 100% on this. Let me know whether
> you
> >>>>>> think
> >>>>>>>>>> this
> >>>>>>>>>>>> should block the release, Luke. I am also tagging Colin and
> David
> >>>>> to
> >>>>>>>>>> weigh
> >>>>>>>>>>>> in with their opinions, as they worked on the migration logic.
> >>>>>>>>>>
> >>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong
> is a
> >>>>>>>>> clear
> >>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a
> PR.
> >>>>>>>>>>
> >>>>>>>>>> best,
> >>>>>>>>>> Colin
> >>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hey Kirk and Chris,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a
> bad log
> >>>>>>>>> due
> >>>>>>>>>> to
> >>>>>>>>>>>> improper closing. And the PR description implies this has been
> >>>>>>>>> present
> >>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for
> this to
> >>>>>>>>> block
> >>>>>>>>>>>> the release.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hey Jakub,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nice catch! It does seem like we should have gated this
> behind the
> >>>>>>>>>> metadata
> >>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
> >>>>> multiple
> >>>>>>>>> log
> >>>>>>>>>>>> dirs? What is the impact of the error messages?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Stanislav
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <jakub@scholz.cz
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I was trying the RC2 and run into the following issue ...
> when I
> >>>>>>>>> run
> >>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> >>>>>>>>> metadata
> >>>>>>>>>>>>> version, I seem to be getting repeated errors like this in
> the
> >>>>>>>>>> controller
> >>>>>>>>>>>>> logs:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> >>>>>>>>>>>> assignReplicasToDirs:
> >>>>>>>>>>>>> event failed with UnsupportedVersionException in 15
> microseconds.
> >>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
> >>>>>>>>>>>>> [quorum-controller-0-event-handler]
> >>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
> >>>>> Unexpected
> >>>>>>>>>> error
> >>>>>>>>>>>>> handling request
> RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> headerVersion=2)
> >>>>> --
> >>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> >>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> >>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
> >>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
> >>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> >>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with
> context
> >>>>>>>>>>>>>
> >>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> headerVersion=2),
> >>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> >>>>>>>>>> clientAddress=/
> >>>>>>>>>>>>> 172.16.14.217,
> principal=User:CN=my-cluster-kafka,O=io.strimzi,
> >>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
> >>>>> securityProtocol=SSL,
> >>>>>>>>>>>>>
> >>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> >>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> >>>>>>>>>>>>> ])
> >>>>>>>>>>>>> (kafka.server.ControllerApis)
> [quorum-controller-0-event-handler]
> >>>>>>>>>>>>> java.util.concurrent.CompletionException:
> >>>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> >>>>>>>>> Directory
> >>>>>>>>>>>>> assignment is not supported yet.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> >>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Caused by:
> >>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> >>>>>>>>>>>>> Directory assignment is not supported yet.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is that expected? I guess with the metadata version set to
> >>>>>>>>> 3.6-IV2, it
> >>>>>>>>>>>>> makes sense that the request is not supported. But shouldn't
> then
> >>>>>>>>> the
> >>>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened
> a
> >>>>> JIRA
> >>>>>>>>>> for
> >>>>>>>>>>>> it,
> >>>>>>>>>>>>> but I can open one if you agree this is not expected)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks & Regards
> >>>>>>>>>>>>> Jakub
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <showuon@gmail.com
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Stanislav,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
> >>>>> maybe
> >>>>>>>>>> you
> >>>>>>>>>>>>>> missed it.
> >>>>>>>>>>>>>> cross-posting here:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> There is a bug KAFKA-16101
> >>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101>
> reporting
> >>>>>>>>> that
> >>>>>>>>>>>>> "Kafka
> >>>>>>>>>>>>>> cluster will be unavailable during KRaft migration
> rollback".
> >>>>>>>>>>>>>> The impact for this issue is that if brokers try to
> rollback to
> >>>>>>>>> ZK
> >>>>>>>>>> mode
> >>>>>>>>>>>>>> during KRaft migration process, there will be a period of
> time
> >>>>>>>>> the
> >>>>>>>>>>>>> cluster
> >>>>>>>>>>>>>> is unavailable.
> >>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
> >>>>>>>>> feature, I
> >>>>>>>>>>>>> think
> >>>>>>>>>>>>>> this should be addressed soon.
> >>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> >>>>>>>>>> fearthecellos@gmail.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, Kirk!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Chris/Stanislav,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler'
> log
> >>>>>>>>>> problem
> >>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> >>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use
> the
> >>>>>>>>>>>>> quickstart
> >>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I
> wait
> >>>>>>>>> for
> >>>>>>>>>> the
> >>>>>>>>>>>>>> PR's
> >>>>>>>>>>>>>>>> Jenkins job to finish.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Kirk
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> >>>>>>>>>>>>>>>>> Hi Stanislav,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for running this release!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> To verify, I:
> >>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
> >>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> >>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> >>>>>>>>>>>>>>>>>
> >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>>>>>>>> - Checked signatures and checksums
> >>>>>>>>>>>>>>>>> - Ran the quickstart using both:
> >>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> >>>>>>>>>>>>>>>>>
> >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>> Java
> >>>>>>>>>>>>>>>> 11
> >>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
> >>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
> >>>>>>>>> apache/kafka:3.7.0-rc2
> >>>>>>>>>>>>>>>>> - Ran all unit tests
> >>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I found two minor areas for concern:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
> >>>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level
> log
> >>>>>>>>>>>> message
> >>>>>>>>>>>>>>> being
> >>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
> >>>>>>>>>> console
> >>>>>>>>>>>>>>> consumer
> >>>>>>>>>>>>>>>>> via ctrl-C:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> >>>>>>>>>>>>>> clientId=console-consumer,
> >>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> >>>>>>>>>>>> FetchSessionHandler
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> node
> >>>>>>>>>>>>>>>>> 1. Ignoring fetch response
> >>>>>>>>>>>>>>>>>
> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I see that this error message is already reported in
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> >>>>>>>>> think we
> >>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's
> probably
> >>>>>>>>>>>> benign
> >>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>> really not a good look for us when basic operations log
> >>>>>>>>> error
> >>>>>>>>>>>>>> messages,
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> it may give new users some headaches.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
> >>>>>>>>>>>>>>>>> The following unit tests failed the first time around,
> and
> >>>>>>>>>> all of
> >>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>> passed the second time I ran them:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - (clients)
> >>>>>>>>>>>>>>>>
> >>>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> >>>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
> >>>>>>>>>>>>>>>>> - (clients)
> >>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
> >>>>>>>>>>>>>>>>> - (connect)
> >>>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> >>>>>>>>>>>>>>> (I
> >>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
> >>>>>>>>>>>>>>>>> - (core)
> >>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
> >>>>>>>>>> congratulations
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
> >>>>>>>>>>>>>> highly-experimental
> >>>>>>>>>>>>>>>>> single laptop run!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> >>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
> >>>>>>>>>> 3.7.0.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
> >>>>>>>>> that
> >>>>>>>>>> I
> >>>>>>>>>>>> had
> >>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> >>>>>>>>> announcing
> >>>>>>>>>>>> due
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
> >>>>>>>>> tests
> >>>>>>>>>>>>>>>> successfully.
> >>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
> >>>>>>>>> that I
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> spend
> >>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case
> anyone
> >>>>>>>>>>>> notices
> >>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
> >>>>>>>>> me
> >>>>>>>>>> to
> >>>>>>>>>>>>>> follow
> >>>>>>>>>>>>>>>> up in
> >>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Major changes include:
> >>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
> >>>>>>>>>> consumer
> >>>>>>>>>>>>>>>> rebalance
> >>>>>>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> >>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> >>>>>>>>>> standardized
> >>>>>>>>>>>>>>>> interface
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
> >>>>>>>>> 18,
> >>>>>>>>>> 9am
> >>>>>>>>>>>>> PT
> >>>>>>>>>>>>>>> ***
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
> >>>>>>>>> this
> >>>>>>>>>>>>> being
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
> >>>>>>>>> it a
> >>>>>>>>>> bit
> >>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
> >>>>>>>>>>>> release:
> >>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and
> binary):
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
> >>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Javadoc:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0
> tag:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Documentation:
> >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Protocol:
> >>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> >>>>>>>>>>>>>>>>>> Unit/integration tests:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> >>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
> >>>>>>>>>> triaging
> >>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
> >>>>>>>>> or
> >>>>>>>>>>>>> simply
> >>>>>>>>>>>>>>>> flakes.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> System tests:
> >>>>>>>>>>>>>>>>
> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
> >>>>>>>>> getting
> >>>>>>>>>> the
> >>>>>>>>>>>>> job
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> run.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for
> 3.7
> >>>>>>>>>>>> branch:
> >>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
> >>>>>>>>>> from
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> Docker
> >>>>>>>>>>>>>>>>>> Build run:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
> >>>>>>>>>>>>> Pipeline:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> /**************************************
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> Stanislav Kozlovski
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Stanislav
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best,
> >>>> Stanislav
> >>>
> >>
> >
>
>

-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Omnia Ibrahim <o....@gmail.com>.
Hi Stan and Gaurav, 
Just to clarify some points mentioned here before 
 KAFKA-14616: I raised a year ago so it's not related to JBOD work. It is rather a blocker bug for KRAFT in general. The PR from Colin should fix this. Am not sure if it is a blocker for 3.7 per-say as it was a major bug since 3.3 and got missed from all other releases.
 
Regarding the JBOD's work: 
KAFKA-16082:  Is not a blocker for 3.7 instead it's nice fix. The pr https://github.com/apache/kafka/pull/15136 is quite a small one and was approved by Proven and I but it is waiting for a committer's approval.
KAFKA-16162: This is a blocker for 3.7.  Same it’s a small pr https://github.com/apache/kafka/pull/15270 and it is approved Proven and I and the PR is waiting for committer's approval. 
KAFKA-16157: This is a blocker for 3.7. There is one small suggestion for the pr https://github.com/apache/kafka/pull/15263 but I don't think any of the current feedback is blocking the pr from getting approved. Assuming we get a committer's approval on it. 
KAFKA-16195:  Same it's a blocker but it has approval from Proven and I and we are waiting for committer's approval on the pr https://github.com/apache/kafka/pull/15262. 

If we can’t get a committer approval for KAFKA-16162, KAFKA-16157 and KAFKA-16195  in time for 3.7 then we can mark JBOD as early release assuming we merge at least KAFKA-16195.

Regards, 
Omnia

> On 26 Jan 2024, at 15:39, kafka@gnarula.com wrote:
> 
> Apologies, I duplicated KAFKA-16157 twice in my previous message. I intended to mention KAFKA-16195
> with the PR at https://github.com/apache/kafka/pull/15262 as the second JIRA.
> 
> Thanks,
> Gaurav
> 
>> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
>> 
>> Hi Stan,
>> 
>> I wanted to share some updates about the bugs you shared earlier.
>> 
>> - KAFKA-14616: I've reviewed and tested the PR from Colin and have observed
>> the fix works as intended.
>> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the proposed fix. I've
>> therefore raised https://github.com/apache/kafka/pull/15270 following a discussion with Luke in JIRA.
>> - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm awaiting
>> feedback/reviews at https://github.com/apache/kafka/pull/15136
>> 
>> In addition to the above, there are 2 JIRAs I'd like to bring everyone's attention to:
>> 
>> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a blocker. I've raised
>> https://github.com/apache/kafka/pull/15263 and am awaiting reviews on it.
>> - KAFKA-16157: I raised this yesterday and have addressed feedback from Luke. This should
>> hopefully get merged soon.
>> 
>> Regards,
>> Gaurav
>> 
>> 
>>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
>>> 
>>> Hi Stanislav,
>>> 
>>> Thanks for bringing these JIRAs/PRs up.
>>> 
>>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
>>> by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.
>>> 
>>> As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
>>> by me for promoting an abandoned future replica with approvals from Omnia and Proven,
>>> so I'd appreciate a committer reviewing it.
>>> 
>>> Regards,
>>> Gaurav
>>> 
>>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
>>>> 
>>>> Hey all, I figured I'd give an update about what known blockers we have
>>>> right now:
>>>> 
>>>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
>>>> https://github.com/apache/kafka/pull/15193; This need not block RC
>>>> creation, but we need the docs updated so that people can test properly
>>>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
>>>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
>>>> this is blocking JBOD for 3.7
>>>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
>>>> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
>>>> - although I understand Proveen is out of office
>>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
>>>> hearing mixed opinions on whether this is a blocker (
>>>> https://github.com/apache/kafka/pull/15136)
>>>> 
>>>> Given that there are 3 JBOD blocker bugs, and I am not confident they will
>>>> all be merged this week - I am on the edge of voting to revert JBOD from
>>>> this release, or mark it early access.
>>>> 
>>>> By all accounts, it seems that if we keep with JBOD the release will have
>>>> to spill into February, which is a month extra from the time-based release
>>>> plan we had of start of January.
>>>> 
>>>> Can I ask others for an opinion?
>>>> 
>>>> Best,
>>>> Stan
>>>> 
>>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I think I've found another blocker issue: KAFKA-16162
>>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>>>>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>>>>> will be unavailable.
>>>>> I've put my findings in the JIRA.
>>>>> 
>>>>> Thanks.
>>>>> Luke
>>>>> 
>>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>>>>> 
>>>>>> Stan, thanks for driving this all forward! Excellent job.
>>>>>> 
>>>>>> About
>>>>>> 
>>>>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>>>>> 
>>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>>>>> now in trunk and 3.7 (and actually also in 3.6...)
>>>>>> 
>>>>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>>>>> it's a blocker. I updated the ticket accordingly. We already have an
>>>>>> open PR that reverts the code introducing the regression.
>>>>>> 
>>>>>> 
>>>>>> -Matthias
>>>>>> 
>>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>>>>> We have another blocking issue for the RC :
>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>>>>> to
>>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>>>>> however
>>>>>>> can lead to the new topic having partitions that a producer cannot
>>>>> write
>>>>>> to.
>>>>>>> 
>>>>>>> --Proven
>>>>>>> 
>>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>>>>> pprovenzano@confluent.io>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>>>>> now.
>>>>>>>> --Proven
>>>>>>>> 
>>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>>>>> 
>>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>>>>> blocker bug because it *
>>>>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>>>>> junit
>>>>>>>>> tests *
>>>>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>>>>> not
>>>>>>>>> suitable *
>>>>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>>>>> 
>>>>>>>>> Hi Colin,
>>>>>>>>> 
>>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>>>>> 
>>>>>>>>> Thanks & Regards
>>>>>>>>> Jakub
>>>>>>>>> 
>>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Stanislav,
>>>>>>>>>> 
>>>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>>>>> messing
>>>>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>>>>> farther along than we are, heh.
>>>>>>>>>> 
>>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>>>>> the
>>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>>>>> with
>>>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>>>>> 
>>>>>>>>>>> I did not observe any obvious impact. I was able to send and
>>>>> receive
>>>>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>>>>> 
>>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>>>>> through
>>>>>>>>> this
>>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>>>>> least
>>>>>> a
>>>>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>>>>> impact I
>>>>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>>>>> others
>>>>>>>>>> to
>>>>>>>>>>> decide if this is a blocker or not.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Hi Jakub,
>>>>>>>>>> 
>>>>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>>>>> because
>>>>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>>>>> in
>>>>>>>>> junit
>>>>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>>>>> it's
>>>>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>>>>> 
>>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hey Luke,
>>>>>>>>>>>> 
>>>>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>>>>> having a
>>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>>>>> calling
>>>>>>>>>> this a
>>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>>>>> 
>>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>>>>> in
>>>>>>>>>> 3.7.0
>>>>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>>>>> same
>>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>>>>> expected
>>>>>>>>> to
>>>>>>>>>> use a
>>>>>>>>>>>> patch release for this.
>>>>>>>>>> 
>>>>>>>>>> Hi Luke,
>>>>>>>>>> 
>>>>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>>>>> documentation is wrong. The intention was to for the steps to
>>>>>> basically
>>>>>>>>> be:
>>>>>>>>>> 
>>>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>>>>> 2. take down the kraft quorum
>>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>>>>> desired)
>>>>>>>>>> 
>>>>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>>>>> controller
>>>>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>>>>> 
>>>>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>>>>> likely we will ultimately recommend users upgrade from that
>>>>> version
>>>>>>>>>> given
>>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>>>>> think
>>>>>>>>>> this
>>>>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>>>>> to
>>>>>>>>>> weigh
>>>>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>>>>> 
>>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>>>>> clear
>>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>>>>> 
>>>>>>>>>> best,
>>>>>>>>>> Colin
>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>>>>> 
>>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>>>>> due
>>>>>>>>>> to
>>>>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>>>>> present
>>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>>>>> block
>>>>>>>>>>>> the release.
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey Jakub,
>>>>>>>>>>>> 
>>>>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>>>>> metadata
>>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>>>>> multiple
>>>>>>>>> log
>>>>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>>>>> 
>>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Stanislav
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>>>>> run
>>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>>>>> metadata
>>>>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>>>>> controller
>>>>>>>>>>>>> logs:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>>>>> Unexpected
>>>>>>>>>> error
>>>>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>>>>> --
>>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>>>>> 
>>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>>>>> clientAddress=/
>>>>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>>>>> securityProtocol=SSL,
>>>>>>>>>>>>> 
>>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>>>>> ])
>>>>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>> Directory
>>>>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Caused by:
>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>>>>> 3.6-IV2, it
>>>>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>>>>> the
>>>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>>>>> JIRA
>>>>>>>>>> for
>>>>>>>>>>>> it,
>>>>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks & Regards
>>>>>>>>>>>>> Jakub
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>>>>> maybe
>>>>>>>>>> you
>>>>>>>>>>>>>> missed it.
>>>>>>>>>>>>>> cross-posting here:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>>>>> that
>>>>>>>>>>>>> "Kafka
>>>>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>>>>> ZK
>>>>>>>>>> mode
>>>>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>>>>> the
>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>> is unavailable.
>>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>>>>> feature, I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>>>>> fearthecellos@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>>>>> problem
>>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>>>>> quickstart
>>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>>> PR's
>>>>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>> with
>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>> 11
>>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>>>>> message
>>>>>>>>>>>>>>> being
>>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>>>>> console
>>>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>>>>> FetchSessionHandler
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>>>>> think we
>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>>>>> benign
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>>>>> error
>>>>>>>>>>>>>> messages,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>>>>> all of
>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>>>>> 
>>>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>>> - (clients)
>>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>>>>> - (core)
>>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>>>>> congratulations
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>>>>> 3.7.0.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>>>>> that
>>>>>>>>>> I
>>>>>>>>>>>> had
>>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>>>>> announcing
>>>>>>>>>>>> due
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>>>>> tests
>>>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>>>>> that I
>>>>>>>>>>>> have
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>>>>> notices
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>>>>> me
>>>>>>>>>> to
>>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>>>>> consumer
>>>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>>>>> standardized
>>>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>>>>> 18,
>>>>>>>>>> 9am
>>>>>>>>>>>>> PT
>>>>>>>>>>>>>>> ***
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>>>>> this
>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>>>>> it a
>>>>>>>>>> bit
>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>>>>> release:
>>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>>>>> triaging
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>>>>> or
>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>>>>> getting
>>>>>>>>>> the
>>>>>>>>>>>>> job
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>>>>> branch:
>>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>>>>> from
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Stanislav
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Best,
>>>> Stanislav
>>> 
>> 
> 


Re: [VOTE] 3.7.0 RC2

Posted by Omnia Ibrahim <o....@gmail.com>.
Hi Stan and Gaurav, 
Just to clarify some points mentioned here before 
 KAFKA-14616: I raised a year ago so it's not related to JBOD work. It is rather a blocker bug for KRAFT in general. The PR from Colin should fix this. Am not sure if it is a blocker for 3.7 per-say as it was a major bug since 3.3 and got missed from all other releases.
 
Regarding the JBOD's work: 
KAFKA-16082:  Is not a blocker for 3.7 instead it's nice fix. The pr https://github.com/apache/kafka/pull/15136 is quite a small one and was approved by Proven and I but it is waiting for a committer's approval.
KAFKA-16162: This is a blocker for 3.7.  Same it’s a small pr https://github.com/apache/kafka/pull/15270 and it is approved Proven and I and the PR is waiting for committer's approval. 
KAFKA-16157: This is a blocker for 3.7. There is one small suggestion for the pr https://github.com/apache/kafka/pull/15263 but I don't think any of the current feedback is blocking the pr from getting approved. Assuming we get a committer's approval on it. 
KAFKA-16195:  Same it's a blocker but it has approval from Proven and I and we are waiting for committer's approval on the pr https://github.com/apache/kafka/pull/15262. 

If we can’t get a committer approval for KAFKA-16162, KAFKA-16157 and KAFKA-16195  in time for 3.7 then we can mark JBOD as early release assuming we merge at least KAFKA-16195.

Regards, 
Omnia

> On 26 Jan 2024, at 15:39, kafka@gnarula.com wrote:
> 
> Apologies, I duplicated KAFKA-16157 twice in my previous message. I intended to mention KAFKA-16195
> with the PR at https://github.com/apache/kafka/pull/15262 as the second JIRA.
> 
> Thanks,
> Gaurav
> 
>> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
>> 
>> Hi Stan,
>> 
>> I wanted to share some updates about the bugs you shared earlier.
>> 
>> - KAFKA-14616: I've reviewed and tested the PR from Colin and have observed
>> the fix works as intended.
>> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the proposed fix. I've
>> therefore raised https://github.com/apache/kafka/pull/15270 following a discussion with Luke in JIRA.
>> - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm awaiting
>> feedback/reviews at https://github.com/apache/kafka/pull/15136
>> 
>> In addition to the above, there are 2 JIRAs I'd like to bring everyone's attention to:
>> 
>> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a blocker. I've raised
>> https://github.com/apache/kafka/pull/15263 and am awaiting reviews on it.
>> - KAFKA-16157: I raised this yesterday and have addressed feedback from Luke. This should
>> hopefully get merged soon.
>> 
>> Regards,
>> Gaurav
>> 
>> 
>>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
>>> 
>>> Hi Stanislav,
>>> 
>>> Thanks for bringing these JIRAs/PRs up.
>>> 
>>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
>>> by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.
>>> 
>>> As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
>>> by me for promoting an abandoned future replica with approvals from Omnia and Proven,
>>> so I'd appreciate a committer reviewing it.
>>> 
>>> Regards,
>>> Gaurav
>>> 
>>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
>>>> 
>>>> Hey all, I figured I'd give an update about what known blockers we have
>>>> right now:
>>>> 
>>>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
>>>> https://github.com/apache/kafka/pull/15193; This need not block RC
>>>> creation, but we need the docs updated so that people can test properly
>>>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
>>>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
>>>> this is blocking JBOD for 3.7
>>>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
>>>> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
>>>> - although I understand Proveen is out of office
>>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
>>>> hearing mixed opinions on whether this is a blocker (
>>>> https://github.com/apache/kafka/pull/15136)
>>>> 
>>>> Given that there are 3 JBOD blocker bugs, and I am not confident they will
>>>> all be merged this week - I am on the edge of voting to revert JBOD from
>>>> this release, or mark it early access.
>>>> 
>>>> By all accounts, it seems that if we keep with JBOD the release will have
>>>> to spill into February, which is a month extra from the time-based release
>>>> plan we had of start of January.
>>>> 
>>>> Can I ask others for an opinion?
>>>> 
>>>> Best,
>>>> Stan
>>>> 
>>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I think I've found another blocker issue: KAFKA-16162
>>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>>>>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>>>>> will be unavailable.
>>>>> I've put my findings in the JIRA.
>>>>> 
>>>>> Thanks.
>>>>> Luke
>>>>> 
>>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>>>>> 
>>>>>> Stan, thanks for driving this all forward! Excellent job.
>>>>>> 
>>>>>> About
>>>>>> 
>>>>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>>>>> 
>>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>>>>> now in trunk and 3.7 (and actually also in 3.6...)
>>>>>> 
>>>>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>>>>> it's a blocker. I updated the ticket accordingly. We already have an
>>>>>> open PR that reverts the code introducing the regression.
>>>>>> 
>>>>>> 
>>>>>> -Matthias
>>>>>> 
>>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>>>>> We have another blocking issue for the RC :
>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>>>>> to
>>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>>>>> however
>>>>>>> can lead to the new topic having partitions that a producer cannot
>>>>> write
>>>>>> to.
>>>>>>> 
>>>>>>> --Proven
>>>>>>> 
>>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>>>>> pprovenzano@confluent.io>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>>>>> now.
>>>>>>>> --Proven
>>>>>>>> 
>>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>>>>> 
>>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>>>>> blocker bug because it *
>>>>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>>>>> junit
>>>>>>>>> tests *
>>>>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>>>>> not
>>>>>>>>> suitable *
>>>>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>>>>> 
>>>>>>>>> Hi Colin,
>>>>>>>>> 
>>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>>>>> 
>>>>>>>>> Thanks & Regards
>>>>>>>>> Jakub
>>>>>>>>> 
>>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Stanislav,
>>>>>>>>>> 
>>>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>>>>> messing
>>>>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>>>>> farther along than we are, heh.
>>>>>>>>>> 
>>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>>>>> the
>>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>>>>> with
>>>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>>>>> 
>>>>>>>>>>> I did not observe any obvious impact. I was able to send and
>>>>> receive
>>>>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>>>>> 
>>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>>>>> through
>>>>>>>>> this
>>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>>>>> least
>>>>>> a
>>>>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>>>>> impact I
>>>>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>>>>> others
>>>>>>>>>> to
>>>>>>>>>>> decide if this is a blocker or not.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Hi Jakub,
>>>>>>>>>> 
>>>>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>>>>> because
>>>>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>>>>> in
>>>>>>>>> junit
>>>>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>>>>> it's
>>>>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>>>>> 
>>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hey Luke,
>>>>>>>>>>>> 
>>>>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>>>>> having a
>>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>>>>> calling
>>>>>>>>>> this a
>>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>>>>> 
>>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>>>>> in
>>>>>>>>>> 3.7.0
>>>>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>>>>> same
>>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>>>>> expected
>>>>>>>>> to
>>>>>>>>>> use a
>>>>>>>>>>>> patch release for this.
>>>>>>>>>> 
>>>>>>>>>> Hi Luke,
>>>>>>>>>> 
>>>>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>>>>> documentation is wrong. The intention was to for the steps to
>>>>>> basically
>>>>>>>>> be:
>>>>>>>>>> 
>>>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>>>>> 2. take down the kraft quorum
>>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>>>>> desired)
>>>>>>>>>> 
>>>>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>>>>> controller
>>>>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>>>>> 
>>>>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>>>>> likely we will ultimately recommend users upgrade from that
>>>>> version
>>>>>>>>>> given
>>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>>>>> think
>>>>>>>>>> this
>>>>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>>>>> to
>>>>>>>>>> weigh
>>>>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>>>>> 
>>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>>>>> clear
>>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>>>>> 
>>>>>>>>>> best,
>>>>>>>>>> Colin
>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>>>>> 
>>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>>>>> due
>>>>>>>>>> to
>>>>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>>>>> present
>>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>>>>> block
>>>>>>>>>>>> the release.
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey Jakub,
>>>>>>>>>>>> 
>>>>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>>>>> metadata
>>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>>>>> multiple
>>>>>>>>> log
>>>>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>>>>> 
>>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Stanislav
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>>>>> run
>>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>>>>> metadata
>>>>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>>>>> controller
>>>>>>>>>>>>> logs:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>>>>> Unexpected
>>>>>>>>>> error
>>>>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>>>>> --
>>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>>>>> 
>>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>>>>> clientAddress=/
>>>>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>>>>> securityProtocol=SSL,
>>>>>>>>>>>>> 
>>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>>>>> ])
>>>>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>> Directory
>>>>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Caused by:
>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>>>>> 3.6-IV2, it
>>>>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>>>>> the
>>>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>>>>> JIRA
>>>>>>>>>> for
>>>>>>>>>>>> it,
>>>>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks & Regards
>>>>>>>>>>>>> Jakub
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>>>>> maybe
>>>>>>>>>> you
>>>>>>>>>>>>>> missed it.
>>>>>>>>>>>>>> cross-posting here:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>>>>> that
>>>>>>>>>>>>> "Kafka
>>>>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>>>>> ZK
>>>>>>>>>> mode
>>>>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>>>>> the
>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>> is unavailable.
>>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>>>>> feature, I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>>>>> fearthecellos@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>>>>> problem
>>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>>>>> quickstart
>>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>>> PR's
>>>>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>> with
>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>> 11
>>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>>>>> message
>>>>>>>>>>>>>>> being
>>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>>>>> console
>>>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>>>>> FetchSessionHandler
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>>>>> think we
>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>>>>> benign
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>>>>> error
>>>>>>>>>>>>>> messages,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>>>>> all of
>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>>>>> 
>>>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>>> - (clients)
>>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>>>>> - (core)
>>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>>>>> congratulations
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>>>>> 3.7.0.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>>>>> that
>>>>>>>>>> I
>>>>>>>>>>>> had
>>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>>>>> announcing
>>>>>>>>>>>> due
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>>>>> tests
>>>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>>>>> that I
>>>>>>>>>>>> have
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>>>>> notices
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>>>>> me
>>>>>>>>>> to
>>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>>>>> consumer
>>>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>>>>> standardized
>>>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>>>>> 18,
>>>>>>>>>> 9am
>>>>>>>>>>>>> PT
>>>>>>>>>>>>>>> ***
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>>>>> this
>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>>>>> it a
>>>>>>>>>> bit
>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>>>>> release:
>>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>>>>> triaging
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>>>>> or
>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>>>>> getting
>>>>>>>>>> the
>>>>>>>>>>>>> job
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>>>>> branch:
>>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>>>>> from
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Stanislav
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Best,
>>>> Stanislav
>>> 
>> 
> 


Re: [VOTE] 3.7.0 RC2

Posted by ka...@gnarula.com.
Apologies, I duplicated KAFKA-16157 twice in my previous message. I intended to mention KAFKA-16195
with the PR at https://github.com/apache/kafka/pull/15262 as the second JIRA.

Thanks,
Gaurav

> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
> 
> Hi Stan,
> 
> I wanted to share some updates about the bugs you shared earlier.
> 
> - KAFKA-14616: I've reviewed and tested the PR from Colin and have observed
> the fix works as intended.
> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the proposed fix. I've
> therefore raised https://github.com/apache/kafka/pull/15270 following a discussion with Luke in JIRA.
> - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm awaiting
> feedback/reviews at https://github.com/apache/kafka/pull/15136
> 
> In addition to the above, there are 2 JIRAs I'd like to bring everyone's attention to:
> 
> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a blocker. I've raised
> https://github.com/apache/kafka/pull/15263 and am awaiting reviews on it.
> - KAFKA-16157: I raised this yesterday and have addressed feedback from Luke. This should
> hopefully get merged soon.
> 
> Regards,
> Gaurav
> 
> 
>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
>> 
>> Hi Stanislav,
>> 
>> Thanks for bringing these JIRAs/PRs up.
>> 
>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
>> by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.
>> 
>> As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
>> by me for promoting an abandoned future replica with approvals from Omnia and Proven,
>> so I'd appreciate a committer reviewing it.
>> 
>> Regards,
>> Gaurav
>> 
>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
>>> 
>>> Hey all, I figured I'd give an update about what known blockers we have
>>> right now:
>>> 
>>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
>>> https://github.com/apache/kafka/pull/15193; This need not block RC
>>> creation, but we need the docs updated so that people can test properly
>>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
>>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
>>> this is blocking JBOD for 3.7
>>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
>>> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
>>> - although I understand Proveen is out of office
>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
>>> hearing mixed opinions on whether this is a blocker (
>>> https://github.com/apache/kafka/pull/15136)
>>> 
>>> Given that there are 3 JBOD blocker bugs, and I am not confident they will
>>> all be merged this week - I am on the edge of voting to revert JBOD from
>>> this release, or mark it early access.
>>> 
>>> By all accounts, it seems that if we keep with JBOD the release will have
>>> to spill into February, which is a month extra from the time-based release
>>> plan we had of start of January.
>>> 
>>> Can I ask others for an opinion?
>>> 
>>> Best,
>>> Stan
>>> 
>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I think I've found another blocker issue: KAFKA-16162
>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>>>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>>>> will be unavailable.
>>>> I've put my findings in the JIRA.
>>>> 
>>>> Thanks.
>>>> Luke
>>>> 
>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>>>> 
>>>>> Stan, thanks for driving this all forward! Excellent job.
>>>>> 
>>>>> About
>>>>> 
>>>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>>>> 
>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>>>> now in trunk and 3.7 (and actually also in 3.6...)
>>>>> 
>>>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>>>> it's a blocker. I updated the ticket accordingly. We already have an
>>>>> open PR that reverts the code introducing the regression.
>>>>> 
>>>>> 
>>>>> -Matthias
>>>>> 
>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>>>> We have another blocking issue for the RC :
>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>>>> to
>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>>>> however
>>>>>> can lead to the new topic having partitions that a producer cannot
>>>> write
>>>>> to.
>>>>>> 
>>>>>> --Proven
>>>>>> 
>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>>>> pprovenzano@confluent.io>
>>>>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>>>> now.
>>>>>>> --Proven
>>>>>>> 
>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>>>> 
>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>>>> blocker bug because it *
>>>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>>>> junit
>>>>>>>> tests *
>>>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>>>> not
>>>>>>>> suitable *
>>>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>>>> 
>>>>>>>> Hi Colin,
>>>>>>>> 
>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>>>> 
>>>>>>>> Thanks & Regards
>>>>>>>> Jakub
>>>>>>>> 
>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Stanislav,
>>>>>>>>> 
>>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>>>> messing
>>>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>>>> farther along than we are, heh.
>>>>>>>>> 
>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>>>> the
>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>>>> with
>>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>>>> 
>>>>>>>>>> I did not observe any obvious impact. I was able to send and
>>>> receive
>>>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>>>> 
>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>>>> through
>>>>>>>> this
>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>>>> least
>>>>> a
>>>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>>>> impact I
>>>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>>>> others
>>>>>>>>> to
>>>>>>>>>> decide if this is a blocker or not.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hi Jakub,
>>>>>>>>> 
>>>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>>>> because
>>>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>>>> in
>>>>>>>> junit
>>>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>>>> it's
>>>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>>>> 
>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey Luke,
>>>>>>>>>>> 
>>>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>>>> having a
>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>>>> calling
>>>>>>>>> this a
>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>>>> 
>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>>>> in
>>>>>>>>> 3.7.0
>>>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>>>> same
>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>>>> expected
>>>>>>>> to
>>>>>>>>> use a
>>>>>>>>>>> patch release for this.
>>>>>>>>> 
>>>>>>>>> Hi Luke,
>>>>>>>>> 
>>>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>>>> documentation is wrong. The intention was to for the steps to
>>>>> basically
>>>>>>>> be:
>>>>>>>>> 
>>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>>>> 2. take down the kraft quorum
>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>>>> desired)
>>>>>>>>> 
>>>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>>>> controller
>>>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>>>> 
>>>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>>>> likely we will ultimately recommend users upgrade from that
>>>> version
>>>>>>>>> given
>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>>>> think
>>>>>>>>> this
>>>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>>>> to
>>>>>>>>> weigh
>>>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>>>> 
>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>>>> clear
>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>>>> 
>>>>>>>>> best,
>>>>>>>>> Colin
>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>>>> 
>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>>>> due
>>>>>>>>> to
>>>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>>>> present
>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>>>> block
>>>>>>>>>>> the release.
>>>>>>>>>>> 
>>>>>>>>>>> Hey Jakub,
>>>>>>>>>>> 
>>>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>>>> metadata
>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>>>> multiple
>>>>>>>> log
>>>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>>>> 
>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Stanislav
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>>>> run
>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>>>> metadata
>>>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>>>> controller
>>>>>>>>>>>> logs:
>>>>>>>>>>>> 
>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>>>> Unexpected
>>>>>>>>> error
>>>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>>>> --
>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>>>> 
>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>>>> clientAddress=/
>>>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>>>> securityProtocol=SSL,
>>>>>>>>>>>> 
>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>>>> ])
>>>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>> Directory
>>>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>>>> 
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>>>> 
>>>>>>>>>>>> Caused by:
>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>>>> 
>>>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>>>> 3.6-IV2, it
>>>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>>>> the
>>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>>>> JIRA
>>>>>>>>> for
>>>>>>>>>>> it,
>>>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks & Regards
>>>>>>>>>>>> Jakub
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>>>> maybe
>>>>>>>>> you
>>>>>>>>>>>>> missed it.
>>>>>>>>>>>>> cross-posting here:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>>>> that
>>>>>>>>>>>> "Kafka
>>>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>>>> ZK
>>>>>>>>> mode
>>>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>>>> the
>>>>>>>>>>>> cluster
>>>>>>>>>>>>> is unavailable.
>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>>>> feature, I
>>>>>>>>>>>> think
>>>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>> Luke
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>>>> fearthecellos@gmail.com
>>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>>>> problem
>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>>>> quickstart
>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>>> PR's
>>>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>>>> 
>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>>>> 
>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>> with
>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>> 11
>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>>>> message
>>>>>>>>>>>>>> being
>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>>>> console
>>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>>>> FetchSessionHandler
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>>>> think we
>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>>>> benign
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>>>> error
>>>>>>>>>>>>> messages,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>>>> all of
>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>>>> 
>>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>> - (clients)
>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>>>> - (core)
>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>>>> congratulations
>>>>>>>>>>> to
>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>>>> 3.7.0.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>>>> that
>>>>>>>>> I
>>>>>>>>>>> had
>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>>>> announcing
>>>>>>>>>>> due
>>>>>>>>>>>> to
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>>>> tests
>>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>>>> that I
>>>>>>>>>>> have
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>>>> notices
>>>>>>>>>>>>> any
>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>>>> me
>>>>>>>>> to
>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>>>> consumer
>>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>>>> standardized
>>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>>>> 18,
>>>>>>>>> 9am
>>>>>>>>>>>> PT
>>>>>>>>>>>>>> ***
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>>>> this
>>>>>>>>>>>> being
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>>>> it a
>>>>>>>>> bit
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>>>> release:
>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>> 
>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>>>> triaging
>>>>>>>>>>>>> some
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>>>> or
>>>>>>>>>>>> simply
>>>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>>>> getting
>>>>>>>>> the
>>>>>>>>>>>> job
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>>>> branch:
>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>>>> from
>>>>>>>>>>> the
>>>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Best,
>>>>>>>>>>> Stanislav
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> -- 
>>> Best,
>>> Stanislav
>> 
> 


Re: [VOTE] 3.7.0 RC2

Posted by ka...@gnarula.com.
Apologies, I duplicated KAFKA-16157 twice in my previous message. I intended to mention KAFKA-16195
with the PR at https://github.com/apache/kafka/pull/15262 as the second JIRA.

Thanks,
Gaurav

> On 26 Jan 2024, at 15:34, kafka@gnarula.com wrote:
> 
> Hi Stan,
> 
> I wanted to share some updates about the bugs you shared earlier.
> 
> - KAFKA-14616: I've reviewed and tested the PR from Colin and have observed
> the fix works as intended.
> - KAFKA-16162: I reviewed Proven's PR and found some gaps in the proposed fix. I've
> therefore raised https://github.com/apache/kafka/pull/15270 following a discussion with Luke in JIRA.
> - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm awaiting
> feedback/reviews at https://github.com/apache/kafka/pull/15136
> 
> In addition to the above, there are 2 JIRAs I'd like to bring everyone's attention to:
> 
> - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a blocker. I've raised
> https://github.com/apache/kafka/pull/15263 and am awaiting reviews on it.
> - KAFKA-16157: I raised this yesterday and have addressed feedback from Luke. This should
> hopefully get merged soon.
> 
> Regards,
> Gaurav
> 
> 
>> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
>> 
>> Hi Stanislav,
>> 
>> Thanks for bringing these JIRAs/PRs up.
>> 
>> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
>> by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.
>> 
>> As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
>> by me for promoting an abandoned future replica with approvals from Omnia and Proven,
>> so I'd appreciate a committer reviewing it.
>> 
>> Regards,
>> Gaurav
>> 
>> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
>>> 
>>> Hey all, I figured I'd give an update about what known blockers we have
>>> right now:
>>> 
>>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
>>> https://github.com/apache/kafka/pull/15193; This need not block RC
>>> creation, but we need the docs updated so that people can test properly
>>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
>>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
>>> this is blocking JBOD for 3.7
>>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
>>> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
>>> - although I understand Proveen is out of office
>>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
>>> hearing mixed opinions on whether this is a blocker (
>>> https://github.com/apache/kafka/pull/15136)
>>> 
>>> Given that there are 3 JBOD blocker bugs, and I am not confident they will
>>> all be merged this week - I am on the edge of voting to revert JBOD from
>>> this release, or mark it early access.
>>> 
>>> By all accounts, it seems that if we keep with JBOD the release will have
>>> to spill into February, which is a month extra from the time-based release
>>> plan we had of start of January.
>>> 
>>> Can I ask others for an opinion?
>>> 
>>> Best,
>>> Stan
>>> 
>>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I think I've found another blocker issue: KAFKA-16162
>>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>>>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>>>> will be unavailable.
>>>> I've put my findings in the JIRA.
>>>> 
>>>> Thanks.
>>>> Luke
>>>> 
>>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>>>> 
>>>>> Stan, thanks for driving this all forward! Excellent job.
>>>>> 
>>>>> About
>>>>> 
>>>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>>>> 
>>>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>>>> now in trunk and 3.7 (and actually also in 3.6...)
>>>>> 
>>>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>>>> it's a blocker. I updated the ticket accordingly. We already have an
>>>>> open PR that reverts the code introducing the regression.
>>>>> 
>>>>> 
>>>>> -Matthias
>>>>> 
>>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>>>> We have another blocking issue for the RC :
>>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>>>> to
>>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>>>> however
>>>>>> can lead to the new topic having partitions that a producer cannot
>>>> write
>>>>> to.
>>>>>> 
>>>>>> --Proven
>>>>>> 
>>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>>>> pprovenzano@confluent.io>
>>>>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>>>> now.
>>>>>>> --Proven
>>>>>>> 
>>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>>>> 
>>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>>>> blocker bug because it *
>>>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>>>> junit
>>>>>>>> tests *
>>>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>>>> not
>>>>>>>> suitable *
>>>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>>>> 
>>>>>>>> Hi Colin,
>>>>>>>> 
>>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>>>> 
>>>>>>>> Thanks & Regards
>>>>>>>> Jakub
>>>>>>>> 
>>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Stanislav,
>>>>>>>>> 
>>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>>>> messing
>>>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>>>> farther along than we are, heh.
>>>>>>>>> 
>>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>>>> the
>>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>>>> with
>>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>>>> 
>>>>>>>>>> I did not observe any obvious impact. I was able to send and
>>>> receive
>>>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>>>> 
>>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>>>> through
>>>>>>>> this
>>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>>>> least
>>>>> a
>>>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>>>> impact I
>>>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>>>> others
>>>>>>>>> to
>>>>>>>>>> decide if this is a blocker or not.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hi Jakub,
>>>>>>>>> 
>>>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>>>> because
>>>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>>>> in
>>>>>>>> junit
>>>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>>>> it's
>>>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>>>> 
>>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey Luke,
>>>>>>>>>>> 
>>>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>>>> having a
>>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>>>> calling
>>>>>>>>> this a
>>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>>>> 
>>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>>>> in
>>>>>>>>> 3.7.0
>>>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>>>> same
>>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>>>> expected
>>>>>>>> to
>>>>>>>>> use a
>>>>>>>>>>> patch release for this.
>>>>>>>>> 
>>>>>>>>> Hi Luke,
>>>>>>>>> 
>>>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>>>> documentation is wrong. The intention was to for the steps to
>>>>> basically
>>>>>>>> be:
>>>>>>>>> 
>>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>>>> 2. take down the kraft quorum
>>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>>>> desired)
>>>>>>>>> 
>>>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>>>> controller
>>>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>>>> 
>>>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>>>> likely we will ultimately recommend users upgrade from that
>>>> version
>>>>>>>>> given
>>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>>>> think
>>>>>>>>> this
>>>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>>>> to
>>>>>>>>> weigh
>>>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>>>> 
>>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>>>> clear
>>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>>>> 
>>>>>>>>> best,
>>>>>>>>> Colin
>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>>>> 
>>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>>>> due
>>>>>>>>> to
>>>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>>>> present
>>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>>>> block
>>>>>>>>>>> the release.
>>>>>>>>>>> 
>>>>>>>>>>> Hey Jakub,
>>>>>>>>>>> 
>>>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>>>> metadata
>>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>>>> multiple
>>>>>>>> log
>>>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>>>> 
>>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Stanislav
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>>>> run
>>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>>>> metadata
>>>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>>>> controller
>>>>>>>>>>>> logs:
>>>>>>>>>>>> 
>>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>>>> Unexpected
>>>>>>>>> error
>>>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>>>> --
>>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>>>> 
>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>>>> clientAddress=/
>>>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>>>> securityProtocol=SSL,
>>>>>>>>>>>> 
>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>>>> ])
>>>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>> Directory
>>>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>>>> 
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>>>> 
>>>>>>>>>>>> Caused by:
>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>>>> 
>>>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>>>> 3.6-IV2, it
>>>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>>>> the
>>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>>>> JIRA
>>>>>>>>> for
>>>>>>>>>>> it,
>>>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks & Regards
>>>>>>>>>>>> Jakub
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>>>> maybe
>>>>>>>>> you
>>>>>>>>>>>>> missed it.
>>>>>>>>>>>>> cross-posting here:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>>>> that
>>>>>>>>>>>> "Kafka
>>>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>>>> ZK
>>>>>>>>> mode
>>>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>>>> the
>>>>>>>>>>>> cluster
>>>>>>>>>>>>> is unavailable.
>>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>>>> feature, I
>>>>>>>>>>>> think
>>>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>> Luke
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>>>> fearthecellos@gmail.com
>>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>>>> problem
>>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>>>> quickstart
>>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>>> PR's
>>>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>>>> 
>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>>>> 
>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>> with
>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>> 11
>>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>>>> message
>>>>>>>>>>>>>> being
>>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>>>> console
>>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>>>> FetchSessionHandler
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>>>> think we
>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>>>> benign
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>>>> error
>>>>>>>>>>>>> messages,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>>>> all of
>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>>>> 
>>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>> - (clients)
>>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>>>> - (core)
>>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>>>> congratulations
>>>>>>>>>>> to
>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>>>> 3.7.0.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>>>> that
>>>>>>>>> I
>>>>>>>>>>> had
>>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>>>> announcing
>>>>>>>>>>> due
>>>>>>>>>>>> to
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>>>> tests
>>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>>>> that I
>>>>>>>>>>> have
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>>>> notices
>>>>>>>>>>>>> any
>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>>>> me
>>>>>>>>> to
>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>>>> consumer
>>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>>>> standardized
>>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>>>> 18,
>>>>>>>>> 9am
>>>>>>>>>>>> PT
>>>>>>>>>>>>>> ***
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>>>> this
>>>>>>>>>>>> being
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>>>> it a
>>>>>>>>> bit
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>>>> release:
>>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>> 
>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>>>> 
>>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>>>> triaging
>>>>>>>>>>>>> some
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>>>> or
>>>>>>>>>>>> simply
>>>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>>>> getting
>>>>>>>>> the
>>>>>>>>>>>> job
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>>>> branch:
>>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>>>> from
>>>>>>>>>>> the
>>>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Best,
>>>>>>>>>>> Stanislav
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> -- 
>>> Best,
>>> Stanislav
>> 
> 


Re: [VOTE] 3.7.0 RC2

Posted by ka...@gnarula.com.
Hi Stan,

I wanted to share some updates about the bugs you shared earlier.

- KAFKA-14616: I've reviewed and tested the PR from Colin and have observed
the fix works as intended.
- KAFKA-16162: I reviewed Proven's PR and found some gaps in the proposed fix. I've
therefore raised https://github.com/apache/kafka/pull/15270 following a discussion with Luke in JIRA.
- KAFKA-16082: I don't think this is marked as a blocker anymore. I'm awaiting
feedback/reviews at https://github.com/apache/kafka/pull/15136

In addition to the above, there are 2 JIRAs I'd like to bring everyone's attention to:

- KAFKA-16157: This is similar to KAFKA-14616 and is marked as a blocker. I've raised
https://github.com/apache/kafka/pull/15263 and am awaiting reviews on it.
- KAFKA-16157: I raised this yesterday and have addressed feedback from Luke. This should
hopefully get merged soon.

Regards,
Gaurav


> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
> 
> Hi Stanislav,
> 
> Thanks for bringing these JIRAs/PRs up.
> 
> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
> by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.
> 
> As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
> by me for promoting an abandoned future replica with approvals from Omnia and Proven,
> so I'd appreciate a committer reviewing it.
> 
> Regards,
> Gaurav
> 
> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
>> 
>> Hey all, I figured I'd give an update about what known blockers we have
>> right now:
>> 
>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
>> https://github.com/apache/kafka/pull/15193; This need not block RC
>> creation, but we need the docs updated so that people can test properly
>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
>> this is blocking JBOD for 3.7
>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
>> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
>> - although I understand Proveen is out of office
>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
>> hearing mixed opinions on whether this is a blocker (
>> https://github.com/apache/kafka/pull/15136)
>> 
>> Given that there are 3 JBOD blocker bugs, and I am not confident they will
>> all be merged this week - I am on the edge of voting to revert JBOD from
>> this release, or mark it early access.
>> 
>> By all accounts, it seems that if we keep with JBOD the release will have
>> to spill into February, which is a month extra from the time-based release
>> plan we had of start of January.
>> 
>> Can I ask others for an opinion?
>> 
>> Best,
>> Stan
>> 
>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>> 
>>> Hi all,
>>> 
>>> I think I've found another blocker issue: KAFKA-16162
>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>>> will be unavailable.
>>> I've put my findings in the JIRA.
>>> 
>>> Thanks.
>>> Luke
>>> 
>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>>> 
>>>> Stan, thanks for driving this all forward! Excellent job.
>>>> 
>>>> About
>>>> 
>>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>>> 
>>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>>> now in trunk and 3.7 (and actually also in 3.6...)
>>>> 
>>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>>> it's a blocker. I updated the ticket accordingly. We already have an
>>>> open PR that reverts the code introducing the regression.
>>>> 
>>>> 
>>>> -Matthias
>>>> 
>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>>> We have another blocking issue for the RC :
>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>>> to
>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>>> however
>>>>> can lead to the new topic having partitions that a producer cannot
>>> write
>>>> to.
>>>>> 
>>>>> --Proven
>>>>> 
>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>>> pprovenzano@confluent.io>
>>>>> wrote:
>>>>> 
>>>>>> 
>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>>> now.
>>>>>> --Proven
>>>>>> 
>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>>> 
>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>>> blocker bug because it *
>>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>>> junit
>>>>>>> tests *
>>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>>> not
>>>>>>> suitable *
>>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>>> 
>>>>>>> Hi Colin,
>>>>>>> 
>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>>> 
>>>>>>> Thanks & Regards
>>>>>>> Jakub
>>>>>>> 
>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>>> Hi Stanislav,
>>>>>>>> 
>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>>> messing
>>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>>> farther along than we are, heh.
>>>>>>>> 
>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>>> the
>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>>> with
>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>>> 
>>>>>>>>> I did not observe any obvious impact. I was able to send and
>>> receive
>>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>>> 
>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>>> through
>>>>>>> this
>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>>> least
>>>> a
>>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>>> impact I
>>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>>> others
>>>>>>>> to
>>>>>>>>> decide if this is a blocker or not.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Hi Jakub,
>>>>>>>> 
>>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>>> because
>>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>>> in
>>>>>>> junit
>>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>>> it's
>>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>>> 
>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey Luke,
>>>>>>>>>> 
>>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>>> having a
>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>>> calling
>>>>>>>> this a
>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>>> 
>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>>> in
>>>>>>>> 3.7.0
>>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>>> same
>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>>> expected
>>>>>>> to
>>>>>>>> use a
>>>>>>>>>> patch release for this.
>>>>>>>> 
>>>>>>>> Hi Luke,
>>>>>>>> 
>>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>>> documentation is wrong. The intention was to for the steps to
>>>> basically
>>>>>>> be:
>>>>>>>> 
>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>>> 2. take down the kraft quorum
>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>>> desired)
>>>>>>>> 
>>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>>> controller
>>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>>> 
>>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>>> likely we will ultimately recommend users upgrade from that
>>> version
>>>>>>>> given
>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>>> think
>>>>>>>> this
>>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>>> to
>>>>>>>> weigh
>>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>>> 
>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>>> clear
>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>>> 
>>>>>>>> best,
>>>>>>>> Colin
>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>>> 
>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>>> due
>>>>>>>> to
>>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>>> present
>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>>> block
>>>>>>>>>> the release.
>>>>>>>>>> 
>>>>>>>>>> Hey Jakub,
>>>>>>>>>> 
>>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>>> metadata
>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>>> multiple
>>>>>>> log
>>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>>> 
>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Stanislav
>>>>>>>>>> 
>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>>> run
>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>>> metadata
>>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>>> controller
>>>>>>>>>>> logs:
>>>>>>>>>>> 
>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>>> Unexpected
>>>>>>>> error
>>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>>> --
>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>>> 
>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>>> clientAddress=/
>>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>>> securityProtocol=SSL,
>>>>>>>>>>> 
>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>>> ])
>>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>> Directory
>>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>>> 
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>>> 
>>>>>>>>>>> Caused by:
>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>>> 
>>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>>> 3.6-IV2, it
>>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>>> the
>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>>> JIRA
>>>>>>>> for
>>>>>>>>>> it,
>>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>>> 
>>>>>>>>>>> Thanks & Regards
>>>>>>>>>>> Jakub
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>> 
>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>>> maybe
>>>>>>>> you
>>>>>>>>>>>> missed it.
>>>>>>>>>>>> cross-posting here:
>>>>>>>>>>>> 
>>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>>> that
>>>>>>>>>>> "Kafka
>>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>>> ZK
>>>>>>>> mode
>>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>>> the
>>>>>>>>>>> cluster
>>>>>>>>>>>> is unavailable.
>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>>> feature, I
>>>>>>>>>>> think
>>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>> Luke
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>>> fearthecellos@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>>> problem
>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>>> quickstart
>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>>> for
>>>>>>>> the
>>>>>>>>>>>> PR's
>>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>>> 
>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>>> 
>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>> with
>>>>>>>>>>>> Java
>>>>>>>>>>>>>> 11
>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>>> message
>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>>> console
>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>>> FetchSessionHandler
>>>>>>>>>>>> for
>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>>> think we
>>>>>>>>>>> should
>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>>> benign
>>>>>>>>>>>> but
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>>> error
>>>>>>>>>>>> messages,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>>> all of
>>>>>>>>>>>> them
>>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>>> 
>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>> - (clients)
>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>>> - (core)
>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>>> congratulations
>>>>>>>>>> to
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>>> 3.7.0.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>>> that
>>>>>>>> I
>>>>>>>>>> had
>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>>> announcing
>>>>>>>>>> due
>>>>>>>>>>> to
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>>> tests
>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>>> that I
>>>>>>>>>> have
>>>>>>>>>>>> to
>>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>>> notices
>>>>>>>>>>>> any
>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>>> me
>>>>>>>> to
>>>>>>>>>>>> follow
>>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>>> consumer
>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>>> standardized
>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>>> 18,
>>>>>>>> 9am
>>>>>>>>>>> PT
>>>>>>>>>>>>> ***
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>>> this
>>>>>>>>>>> being
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>>> it a
>>>>>>>> bit
>>>>>>>>>>>> more
>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>>> release:
>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>>> 
>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>> 
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>>> 
>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>>> triaging
>>>>>>>>>>>> some
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>>> or
>>>>>>>>>>> simply
>>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>>> getting
>>>>>>>> the
>>>>>>>>>>> job
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>>> branch:
>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>>> from
>>>>>>>>>> the
>>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Best,
>>>>>>>>>> Stanislav
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> -- 
>> Best,
>> Stanislav
> 


Re: [VOTE] 3.7.0 RC2

Posted by ka...@gnarula.com.
Hi Stan,

I wanted to share some updates about the bugs you shared earlier.

- KAFKA-14616: I've reviewed and tested the PR from Colin and have observed
the fix works as intended.
- KAFKA-16162: I reviewed Proven's PR and found some gaps in the proposed fix. I've
therefore raised https://github.com/apache/kafka/pull/15270 following a discussion with Luke in JIRA.
- KAFKA-16082: I don't think this is marked as a blocker anymore. I'm awaiting
feedback/reviews at https://github.com/apache/kafka/pull/15136

In addition to the above, there are 2 JIRAs I'd like to bring everyone's attention to:

- KAFKA-16157: This is similar to KAFKA-14616 and is marked as a blocker. I've raised
https://github.com/apache/kafka/pull/15263 and am awaiting reviews on it.
- KAFKA-16157: I raised this yesterday and have addressed feedback from Luke. This should
hopefully get merged soon.

Regards,
Gaurav


> On 24 Jan 2024, at 11:51, kafka@gnarula.com wrote:
> 
> Hi Stanislav,
> 
> Thanks for bringing these JIRAs/PRs up.
> 
> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
> by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.
> 
> As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
> by me for promoting an abandoned future replica with approvals from Omnia and Proven,
> so I'd appreciate a committer reviewing it.
> 
> Regards,
> Gaurav
> 
> On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
>> 
>> Hey all, I figured I'd give an update about what known blockers we have
>> right now:
>> 
>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
>> https://github.com/apache/kafka/pull/15193; This need not block RC
>> creation, but we need the docs updated so that people can test properly
>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
>> this is blocking JBOD for 3.7
>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
>> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
>> - although I understand Proveen is out of office
>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
>> hearing mixed opinions on whether this is a blocker (
>> https://github.com/apache/kafka/pull/15136)
>> 
>> Given that there are 3 JBOD blocker bugs, and I am not confident they will
>> all be merged this week - I am on the edge of voting to revert JBOD from
>> this release, or mark it early access.
>> 
>> By all accounts, it seems that if we keep with JBOD the release will have
>> to spill into February, which is a month extra from the time-based release
>> plan we had of start of January.
>> 
>> Can I ask others for an opinion?
>> 
>> Best,
>> Stan
>> 
>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>> 
>>> Hi all,
>>> 
>>> I think I've found another blocker issue: KAFKA-16162
>>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>>> will be unavailable.
>>> I've put my findings in the JIRA.
>>> 
>>> Thanks.
>>> Luke
>>> 
>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>>> 
>>>> Stan, thanks for driving this all forward! Excellent job.
>>>> 
>>>> About
>>>> 
>>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>>> 
>>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>>> now in trunk and 3.7 (and actually also in 3.6...)
>>>> 
>>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>>> it's a blocker. I updated the ticket accordingly. We already have an
>>>> open PR that reverts the code introducing the regression.
>>>> 
>>>> 
>>>> -Matthias
>>>> 
>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>>> We have another blocking issue for the RC :
>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>>> to
>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>>> however
>>>>> can lead to the new topic having partitions that a producer cannot
>>> write
>>>> to.
>>>>> 
>>>>> --Proven
>>>>> 
>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>>> pprovenzano@confluent.io>
>>>>> wrote:
>>>>> 
>>>>>> 
>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>>> now.
>>>>>> --Proven
>>>>>> 
>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>>> 
>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>>> blocker bug because it *
>>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>>> junit
>>>>>>> tests *
>>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>>> not
>>>>>>> suitable *
>>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>>> 
>>>>>>> Hi Colin,
>>>>>>> 
>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>>> 
>>>>>>> Thanks & Regards
>>>>>>> Jakub
>>>>>>> 
>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>>> Hi Stanislav,
>>>>>>>> 
>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>>> messing
>>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>>> farther along than we are, heh.
>>>>>>>> 
>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>>> the
>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>>> with
>>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>>> 
>>>>>>>>> I did not observe any obvious impact. I was able to send and
>>> receive
>>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>>> 
>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>>> through
>>>>>>> this
>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>>> least
>>>> a
>>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>>> impact I
>>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>>> others
>>>>>>>> to
>>>>>>>>> decide if this is a blocker or not.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Hi Jakub,
>>>>>>>> 
>>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>>> because
>>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>>> in
>>>>>>> junit
>>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>>> it's
>>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>>> 
>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey Luke,
>>>>>>>>>> 
>>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>>> having a
>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>>> calling
>>>>>>>> this a
>>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>>> 
>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>>> in
>>>>>>>> 3.7.0
>>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>>> same
>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>>> expected
>>>>>>> to
>>>>>>>> use a
>>>>>>>>>> patch release for this.
>>>>>>>> 
>>>>>>>> Hi Luke,
>>>>>>>> 
>>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>>> documentation is wrong. The intention was to for the steps to
>>>> basically
>>>>>>> be:
>>>>>>>> 
>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>>> 2. take down the kraft quorum
>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>>> desired)
>>>>>>>> 
>>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>>> controller
>>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>>> 
>>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>>> likely we will ultimately recommend users upgrade from that
>>> version
>>>>>>>> given
>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>>> think
>>>>>>>> this
>>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>>> to
>>>>>>>> weigh
>>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>>> 
>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>>> clear
>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>>> 
>>>>>>>> best,
>>>>>>>> Colin
>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>>> 
>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>>> due
>>>>>>>> to
>>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>>> present
>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>>> block
>>>>>>>>>> the release.
>>>>>>>>>> 
>>>>>>>>>> Hey Jakub,
>>>>>>>>>> 
>>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>>> metadata
>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>>> multiple
>>>>>>> log
>>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>>> 
>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Stanislav
>>>>>>>>>> 
>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>>> run
>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>>> metadata
>>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>>> controller
>>>>>>>>>>> logs:
>>>>>>>>>>> 
>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>>> Unexpected
>>>>>>>> error
>>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>>> --
>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>>> 
>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>>> clientAddress=/
>>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>>> securityProtocol=SSL,
>>>>>>>>>>> 
>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>>> ])
>>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>> Directory
>>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>>> 
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>>> 
>>>>>>>>>>> Caused by:
>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>>> 
>>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>>> 3.6-IV2, it
>>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>>> the
>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>>> JIRA
>>>>>>>> for
>>>>>>>>>> it,
>>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>>> 
>>>>>>>>>>> Thanks & Regards
>>>>>>>>>>> Jakub
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>> 
>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>>> maybe
>>>>>>>> you
>>>>>>>>>>>> missed it.
>>>>>>>>>>>> cross-posting here:
>>>>>>>>>>>> 
>>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>>> that
>>>>>>>>>>> "Kafka
>>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>>> ZK
>>>>>>>> mode
>>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>>> the
>>>>>>>>>>> cluster
>>>>>>>>>>>> is unavailable.
>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>>> feature, I
>>>>>>>>>>> think
>>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>> Luke
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>>> fearthecellos@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>>> problem
>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>>> quickstart
>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>>> for
>>>>>>>> the
>>>>>>>>>>>> PR's
>>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>>> 
>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>>> 
>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>> with
>>>>>>>>>>>> Java
>>>>>>>>>>>>>> 11
>>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>>> message
>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>>> console
>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>>> FetchSessionHandler
>>>>>>>>>>>> for
>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>>> think we
>>>>>>>>>>> should
>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>>> benign
>>>>>>>>>>>> but
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>>> error
>>>>>>>>>>>> messages,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>>> all of
>>>>>>>>>>>> them
>>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>>> 
>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>> - (clients)
>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>>> (I
>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>>> - (core)
>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>>> congratulations
>>>>>>>>>> to
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>>> 3.7.0.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>>> that
>>>>>>>> I
>>>>>>>>>> had
>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>>> announcing
>>>>>>>>>> due
>>>>>>>>>>> to
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>>> tests
>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>>> that I
>>>>>>>>>> have
>>>>>>>>>>>> to
>>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>>> notices
>>>>>>>>>>>> any
>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>>> me
>>>>>>>> to
>>>>>>>>>>>> follow
>>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>>> consumer
>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>>> standardized
>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>>> 18,
>>>>>>>> 9am
>>>>>>>>>>> PT
>>>>>>>>>>>>> ***
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>>> this
>>>>>>>>>>> being
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>>> it a
>>>>>>>> bit
>>>>>>>>>>>> more
>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>>> release:
>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>>> 
>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>> 
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>>> 
>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>>> triaging
>>>>>>>>>>>> some
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>>> or
>>>>>>>>>>> simply
>>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>>> getting
>>>>>>>> the
>>>>>>>>>>> job
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>>> branch:
>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>>> from
>>>>>>>>>> the
>>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Best,
>>>>>>>>>> Stanislav
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> -- 
>> Best,
>> Stanislav
> 


Re: [VOTE] 3.7.0 RC2

Posted by ka...@gnarula.com.
Hi Stanislav,

Thanks for bringing these JIRAs/PRs up.

I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.

As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
by me for promoting an abandoned future replica with approvals from Omnia and Proven,
so I'd appreciate a committer reviewing it.

Regards,
Gaurav

On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
> 
> Hey all, I figured I'd give an update about what known blockers we have
> right now:
> 
> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
> https://github.com/apache/kafka/pull/15193; This need not block RC
> creation, but we need the docs updated so that people can test properly
> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
> this is blocking JBOD for 3.7
> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
> - although I understand Proveen is out of office
> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
> hearing mixed opinions on whether this is a blocker (
> https://github.com/apache/kafka/pull/15136)
> 
> Given that there are 3 JBOD blocker bugs, and I am not confident they will
> all be merged this week - I am on the edge of voting to revert JBOD from
> this release, or mark it early access.
> 
> By all accounts, it seems that if we keep with JBOD the release will have
> to spill into February, which is a month extra from the time-based release
> plan we had of start of January.
> 
> Can I ask others for an opinion?
> 
> Best,
> Stan
> 
> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
> 
>> Hi all,
>> 
>> I think I've found another blocker issue: KAFKA-16162
>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>> will be unavailable.
>> I've put my findings in the JIRA.
>> 
>> Thanks.
>> Luke
>> 
>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>> 
>>> Stan, thanks for driving this all forward! Excellent job.
>>> 
>>> About
>>> 
>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>> 
>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>> now in trunk and 3.7 (and actually also in 3.6...)
>>> 
>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>> it's a blocker. I updated the ticket accordingly. We already have an
>>> open PR that reverts the code introducing the regression.
>>> 
>>> 
>>> -Matthias
>>> 
>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>> We have another blocking issue for the RC :
>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>> to
>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>> however
>>>> can lead to the new topic having partitions that a producer cannot
>> write
>>> to.
>>>> 
>>>> --Proven
>>>> 
>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>> pprovenzano@confluent.io>
>>>> wrote:
>>>> 
>>>>> 
>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>> now.
>>>>> --Proven
>>>>> 
>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>> 
>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>> blocker bug because it *
>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>> junit
>>>>>> tests *
>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>> not
>>>>>> suitable *
>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>> 
>>>>>> Hi Colin,
>>>>>> 
>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>> 
>>>>>> Thanks & Regards
>>>>>> Jakub
>>>>>> 
>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>> wrote:
>>>>>> 
>>>>>>> Hi Stanislav,
>>>>>>> 
>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>> messing
>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>> farther along than we are, heh.
>>>>>>> 
>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>> the
>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>> with
>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>> 
>>>>>>>> I did not observe any obvious impact. I was able to send and
>> receive
>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>> 
>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>> through
>>>>>> this
>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>> least
>>> a
>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>> impact I
>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>> others
>>>>>>> to
>>>>>>>> decide if this is a blocker or not.
>>>>>>>> 
>>>>>>> 
>>>>>>> Hi Jakub,
>>>>>>> 
>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>> because
>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>> in
>>>>>> junit
>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>> it's
>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>> 
>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>> 
>>>>>>>>> Hey Luke,
>>>>>>>>> 
>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>> having a
>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>> calling
>>>>>>> this a
>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>> 
>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>> in
>>>>>>> 3.7.0
>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>> same
>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>> expected
>>>>>> to
>>>>>>> use a
>>>>>>>>> patch release for this.
>>>>>>> 
>>>>>>> Hi Luke,
>>>>>>> 
>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>> documentation is wrong. The intention was to for the steps to
>>> basically
>>>>>> be:
>>>>>>> 
>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>> 2. take down the kraft quorum
>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>> desired)
>>>>>>> 
>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>> controller
>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>> 
>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>> likely we will ultimately recommend users upgrade from that
>> version
>>>>>>> given
>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>> think
>>>>>>> this
>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>> to
>>>>>>> weigh
>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>> 
>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>> clear
>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>> 
>>>>>>> best,
>>>>>>> Colin
>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>> 
>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>> due
>>>>>>> to
>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>> present
>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>> block
>>>>>>>>> the release.
>>>>>>>>> 
>>>>>>>>> Hey Jakub,
>>>>>>>>> 
>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>> metadata
>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>> multiple
>>>>>> log
>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>> 
>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Stanislav
>>>>>>>>> 
>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>> run
>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>> metadata
>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>> controller
>>>>>>>>>> logs:
>>>>>>>>>> 
>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>> Unexpected
>>>>>>> error
>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>> --
>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>> 
>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>> clientAddress=/
>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>> securityProtocol=SSL,
>>>>>>>>>> 
>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>> ])
>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>> Directory
>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>> 
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>  at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>> 
>>>>>>>>>> Caused by:
>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>> 
>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>> 3.6-IV2, it
>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>> the
>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>> JIRA
>>>>>>> for
>>>>>>>>> it,
>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>> 
>>>>>>>>>> Thanks & Regards
>>>>>>>>>> Jakub
>>>>>>>>>> 
>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>> 
>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>> maybe
>>>>>>> you
>>>>>>>>>>> missed it.
>>>>>>>>>>> cross-posting here:
>>>>>>>>>>> 
>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>> that
>>>>>>>>>> "Kafka
>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>> ZK
>>>>>>> mode
>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>> the
>>>>>>>>>> cluster
>>>>>>>>>>> is unavailable.
>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>> feature, I
>>>>>>>>>> think
>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks.
>>>>>>>>>>> Luke
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>> fearthecellos@gmail.com
>>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>> 
>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>> problem
>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>> quickstart
>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>> for
>>>>>>> the
>>>>>>>>>>> PR's
>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>> 
>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>> 
>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>> with
>>>>>>>>>>> Java
>>>>>>>>>>>>> 11
>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>> message
>>>>>>>>>>>> being
>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>> console
>>>>>>>>>>>> consumer
>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>> FetchSessionHandler
>>>>>>>>>>> for
>>>>>>>>>>>>> node
>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>> think we
>>>>>>>>>> should
>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>> benign
>>>>>>>>>>> but
>>>>>>>>>>>>> it's
>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>> error
>>>>>>>>>>> messages,
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>> all of
>>>>>>>>>>> them
>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>> 
>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>> - (clients)
>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>> (I
>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>> - (core)
>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>> congratulations
>>>>>>>>> to
>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>> 3.7.0.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>> that
>>>>>>> I
>>>>>>>>> had
>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>> announcing
>>>>>>>>> due
>>>>>>>>>> to
>>>>>>>>>>>> new
>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>> tests
>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>> that I
>>>>>>>>> have
>>>>>>>>>>> to
>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>> notices
>>>>>>>>>>> any
>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>> me
>>>>>>> to
>>>>>>>>>>> follow
>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>> consumer
>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>> standardized
>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>> 18,
>>>>>>> 9am
>>>>>>>>>> PT
>>>>>>>>>>>> ***
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>> this
>>>>>>>>>> being
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>> it a
>>>>>>> bit
>>>>>>>>>>> more
>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>> release:
>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>> 
>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> 
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>> 
>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>> triaging
>>>>>>>>>>> some
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>> or
>>>>>>>>>> simply
>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>> getting
>>>>>>> the
>>>>>>>>>> job
>>>>>>>>>>>> to
>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>> branch:
>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>> from
>>>>>>>>> the
>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Best,
>>>>>>>>> Stanislav
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> 
> -- 
> Best,
> Stanislav


Re: [VOTE] 3.7.0 RC2

Posted by ka...@gnarula.com.
Hi Stanislav,

Thanks for bringing these JIRAs/PRs up.

I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I hope to have some feedback
by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's away. I'll try to build on his work in the meantime.

As for KAFKA-16082, we haven't been able to deduce a data loss scenario. There's a PR open
by me for promoting an abandoned future replica with approvals from Omnia and Proven,
so I'd appreciate a committer reviewing it.

Regards,
Gaurav

On 23 Jan 2024, at 20:17, Stanislav Kozlovski <st...@confluent.io.INVALID> wrote:
> 
> Hey all, I figured I'd give an update about what known blockers we have
> right now:
> 
> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
> https://github.com/apache/kafka/pull/15193; This need not block RC
> creation, but we need the docs updated so that people can test properly
> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
> this is blocking JBOD for 3.7
> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
> - although I understand Proveen is out of office
> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
> hearing mixed opinions on whether this is a blocker (
> https://github.com/apache/kafka/pull/15136)
> 
> Given that there are 3 JBOD blocker bugs, and I am not confident they will
> all be merged this week - I am on the edge of voting to revert JBOD from
> this release, or mark it early access.
> 
> By all accounts, it seems that if we keep with JBOD the release will have
> to spill into February, which is a month extra from the time-based release
> plan we had of start of January.
> 
> Can I ask others for an opinion?
> 
> Best,
> Stan
> 
> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
> 
>> Hi all,
>> 
>> I think I've found another blocker issue: KAFKA-16162
>> <https://issues.apache.org/jira/browse/KAFKA-16162> .
>> The impact is after upgrading to 3.7.0, any new created topics/partitions
>> will be unavailable.
>> I've put my findings in the JIRA.
>> 
>> Thanks.
>> Luke
>> 
>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>> 
>>> Stan, thanks for driving this all forward! Excellent job.
>>> 
>>> About
>>> 
>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>>> 
>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>>> now in trunk and 3.7 (and actually also in 3.6...)
>>> 
>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so
>>> it's a blocker. I updated the ticket accordingly. We already have an
>>> open PR that reverts the code introducing the regression.
>>> 
>>> 
>>> -Matthias
>>> 
>>> On 1/17/24 9:44 AM, Proven Provenzano wrote:
>>>> We have another blocking issue for the RC :
>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
>>> to
>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>> however
>>>> can lead to the new topic having partitions that a producer cannot
>> write
>>> to.
>>>> 
>>>> --Proven
>>>> 
>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>>> pprovenzano@confluent.io>
>>>> wrote:
>>>> 
>>>>> 
>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for
>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>> now.
>>>>> --Proven
>>>>> 
>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>>>> 
>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>>>>> blocker bug because it *
>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>>>>> junit
>>>>>> tests *
>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's
>>> not
>>>>>> suitable *
>>>>>> *> for production. Did you file a JIRA yet?*
>>>>>> 
>>>>>> Hi Colin,
>>>>>> 
>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>>>> 
>>>>>> Thanks & Regards
>>>>>> Jakub
>>>>>> 
>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>>> wrote:
>>>>>> 
>>>>>>> Hi Stanislav,
>>>>>>> 
>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is
>>> messing
>>>>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>>>>> farther along than we are, heh.
>>>>>>> 
>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>>>>> *> Nice catch! It does seem like we should have gated this behind
>> the
>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured
>> with
>>>>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>>>> 
>>>>>>>> I did not observe any obvious impact. I was able to send and
>> receive
>>>>>>>> messages as normally. But to be honest, I have no idea what else
>>>>>>>> this might impact, so I did not try anything special.
>>>>>>>> 
>>>>>>>> I think everyone upgrading an existing KRaft cluster will go
>> through
>>>>>> this
>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at
>> least
>>> a
>>>>>>>> while). So even if it is just a logged exception without any other
>>>>>>> impact I
>>>>>>>> wonder if it might scare users from upgrading. But I leave it to
>>>>>> others
>>>>>>> to
>>>>>>>> decide if this is a blocker or not.
>>>>>>>> 
>>>>>>> 
>>>>>>> Hi Jakub,
>>>>>>> 
>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>>>>> because
>>>>>>> it will generate huge amount of logspam. I guess we didn't find it
>> in
>>>>>> junit
>>>>>>> tests since logspam doesn't fail the automated tests. But certainly
>>> it's
>>>>>>> not suitable for production. Did you file a JIRA yet?
>>>>>>> 
>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>> 
>>>>>>>>> Hey Luke,
>>>>>>>>> 
>>>>>>>>> This is an interesting problem. Given the fact that the KIP for
>>>>>> having a
>>>>>>>>> 3.8 release passed, I think it weights the scale towards not
>> calling
>>>>>>> this a
>>>>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>>>> 
>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
>> in
>>>>>>> 3.7.0
>>>>>>>>> (given the inability to rollback safely), but if that's true - the
>>>>>> same
>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be
>> expected
>>>>>> to
>>>>>>> use a
>>>>>>>>> patch release for this.
>>>>>>> 
>>>>>>> Hi Luke,
>>>>>>> 
>>>>>>> Thanks for testing rollback. I think this is a case where the
>>>>>>> documentation is wrong. The intention was to for the steps to
>>> basically
>>>>>> be:
>>>>>>> 
>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>>>>> 2. take down the kraft quorum
>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>>>>> desired)
>>>>>>> 
>>>>>>> With these steps, there isn't really unavailability since a ZK
>>>>>> controller
>>>>>>> can be elected quickly after the kraft quorum is gone.
>>>>>>> 
>>>>>>>>> Further, since we will have a 3.8 release - it is
>>>>>>>>> likely we will ultimately recommend users upgrade from that
>> version
>>>>>>> given
>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>>>>> That being said, I am not 100% on this. Let me know whether you
>>> think
>>>>>>> this
>>>>>>>>> should block the release, Luke. I am also tagging Colin and David
>> to
>>>>>>> weigh
>>>>>>>>> in with their opinions, as they worked on the migration logic.
>>>>>>> 
>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>>>>> clear
>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>>>> 
>>>>>>> best,
>>>>>>> Colin
>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hey Kirk and Chris,
>>>>>>>>> 
>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>>>>> due
>>>>>>> to
>>>>>>>>> improper closing. And the PR description implies this has been
>>>>>> present
>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>>>>> block
>>>>>>>>> the release.
>>>>>>>>> 
>>>>>>>>> Hey Jakub,
>>>>>>>>> 
>>>>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>>>>> metadata
>>>>>>>>> version as KIP-858 implies. Is the cluster configured with
>> multiple
>>>>>> log
>>>>>>>>> dirs? What is the impact of the error messages?
>>>>>>>>> 
>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Stanislav
>>>>>>>>> 
>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>>>>> run
>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>>>>> metadata
>>>>>>>>>> version, I seem to be getting repeated errors like this in the
>>>>>>> controller
>>>>>>>>>> logs:
>>>>>>>>>> 
>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>>>>> assignReplicasToDirs:
>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>>>>> [quorum-controller-0-event-handler]
>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>> Unexpected
>>>>>>> error
>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
>> --
>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>>>>> 
>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>>>>> clientAddress=/
>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>> securityProtocol=SSL,
>>>>>>>>>> 
>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>>>>> ])
>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>>>>> java.util.concurrent.CompletionException:
>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>> Directory
>>>>>>>>>> assignment is not supported yet.
>>>>>>>>>> 
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>>>>  at
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>>>>  at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>>>> 
>>>>>>>>>> Caused by:
>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>>>>> Directory assignment is not supported yet.
>>>>>>>>>> 
>>>>>>>>>> Is that expected? I guess with the metadata version set to
>>>>>> 3.6-IV2, it
>>>>>>>>>> makes sense that the request is not supported. But shouldn't then
>>>>>> the
>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a
>> JIRA
>>>>>>> for
>>>>>>>>> it,
>>>>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>>>> 
>>>>>>>>>> Thanks & Regards
>>>>>>>>>> Jakub
>>>>>>>>>> 
>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>> 
>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>> maybe
>>>>>>> you
>>>>>>>>>>> missed it.
>>>>>>>>>>> cross-posting here:
>>>>>>>>>>> 
>>>>>>>>>>> There is a bug KAFKA-16101
>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>>>>> that
>>>>>>>>>> "Kafka
>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>>>>> ZK
>>>>>>> mode
>>>>>>>>>>> during KRaft migration process, there will be a period of time
>>>>>> the
>>>>>>>>>> cluster
>>>>>>>>>>> is unavailable.
>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>>>>> feature, I
>>>>>>>>>> think
>>>>>>>>>>> this should be addressed soon.
>>>>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks.
>>>>>>>>>>> Luke
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>>>>> fearthecellos@gmail.com
>>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Thanks, Kirk!
>>>>>>>>>>>> 
>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>>>>> problem
>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>>>>> quickstart
>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>>>>> for
>>>>>>> the
>>>>>>>>>>> PR's
>>>>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Kirk
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> To verify, I:
>>>>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>>>> 
>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>>>> 
>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>> with
>>>>>>>>>>> Java
>>>>>>>>>>>>> 11
>>>>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>>>>> - - Our shiny new broker Docker image,
>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>>>>> message
>>>>>>>>>>>> being
>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>>>>> console
>>>>>>>>>>>> consumer
>>>>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>>>>> clientId=console-consumer,
>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>>>>> FetchSessionHandler
>>>>>>>>>>> for
>>>>>>>>>>>>> node
>>>>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>>>>> think we
>>>>>>>>>> should
>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>>>>> benign
>>>>>>>>>>> but
>>>>>>>>>>>>> it's
>>>>>>>>>>>>>> really not a good look for us when basic operations log
>>>>>> error
>>>>>>>>>>> messages,
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>>>>> The following unit tests failed the first time around, and
>>>>>>> all of
>>>>>>>>>>> them
>>>>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> - (clients)
>>>>>>>>>>>>> 
>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>> - (clients)
>>>>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>>>>> - (connect)
>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>>>>> (I
>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>>>>> - (core)
>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks again for your work on this release, and
>>>>>>> congratulations
>>>>>>>>> to
>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>>>>> highly-experimental
>>>>>>>>>>>>>> single laptop run!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>>>>> 3.7.0.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>>>>> that
>>>>>>> I
>>>>>>>>> had
>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>>>>> announcing
>>>>>>>>> due
>>>>>>>>>> to
>>>>>>>>>>>> new
>>>>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>>>>> tests
>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>>>>> that I
>>>>>>>>> have
>>>>>>>>>>> to
>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>>>>> notices
>>>>>>>>>>> any
>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>>>>> me
>>>>>>> to
>>>>>>>>>>> follow
>>>>>>>>>>>>> up in
>>>>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>>>>> consumer
>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>>>>> standardized
>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>>>>> 18,
>>>>>>> 9am
>>>>>>>>>> PT
>>>>>>>>>>>> ***
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>>>>> this
>>>>>>>>>> being
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>>>>> it a
>>>>>>> bit
>>>>>>>>>>> more
>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>>>>> release:
>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>>>> 
>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> 
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>>>> 
>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>>>>> triaging
>>>>>>>>>>> some
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>>>>> or
>>>>>>>>>> simply
>>>>>>>>>>>>> flakes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> System tests:
>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> No successful system test runs yet. I am working on
>>>>>> getting
>>>>>>> the
>>>>>>>>>> job
>>>>>>>>>>>> to
>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>>>>> branch:
>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>>>>> from
>>>>>>>>> the
>>>>>>>>>>>>> Docker
>>>>>>>>>>>>>>> Build run:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>>>>> Pipeline:
>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Best,
>>>>>>>>> Stanislav
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> 
> -- 
> Best,
> Stanislav


Re: [VOTE] 3.7.0 RC2

Posted by Justine Olshan <jo...@confluent.io.INVALID>.
Oops sorry I got confused between
https://issues.apache.org/jira/browse/KAFKA-16120 (migration issue) and
https://issues.apache.org/jira/browse/KAFKA-14616 (not migration issue)

However, both do not seem related to JBOD based on the jira and PRs

Justine

On Tue, Jan 23, 2024 at 1:51 PM Justine Olshan <jo...@confluent.io> wrote:

> Hey Stan,
>
> Just wanted to clarify -- KAFKA-14616 is not particularly related to JBOD
> but to ZK -> KRaft migration.
>
> There were some other related migration bugs like
> https://issues.apache.org/jira/browse/KAFKA-16180.
>
> This may or may not influence decisions, but wanted to paint the full
> picture of the blocker bugs and their causes.
>
> Justine
>
>
> On Tue, Jan 23, 2024 at 12:17 PM Stanislav Kozlovski
> <st...@confluent.io.invalid> wrote:
>
>> Hey all, I figured I'd give an update about what known blockers we have
>> right now:
>>
>> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
>> https://github.com/apache/kafka/pull/15193; This need not block RC
>> creation, but we need the docs updated so that people can test properly
>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs
>> -
>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding
>> that
>> this is blocking JBOD for 3.7
>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
>> a strict blocker with an open PR
>> https://github.com/apache/kafka/pull/15232
>> - although I understand Proveen is out of office
>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
>> hearing mixed opinions on whether this is a blocker (
>> https://github.com/apache/kafka/pull/15136)
>>
>> Given that there are 3 JBOD blocker bugs, and I am not confident they will
>> all be merged this week - I am on the edge of voting to revert JBOD from
>> this release, or mark it early access.
>>
>> By all accounts, it seems that if we keep with JBOD the release will have
>> to spill into February, which is a month extra from the time-based release
>> plan we had of start of January.
>>
>> Can I ask others for an opinion?
>>
>> Best,
>> Stan
>>
>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I think I've found another blocker issue: KAFKA-16162
>> > <https://issues.apache.org/jira/browse/KAFKA-16162> .
>> > The impact is after upgrading to 3.7.0, any new created
>> topics/partitions
>> > will be unavailable.
>> > I've put my findings in the JIRA.
>> >
>> > Thanks.
>> > Luke
>> >
>> > On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org>
>> wrote:
>> >
>> > > Stan, thanks for driving this all forward! Excellent job.
>> > >
>> > > About
>> > >
>> > > > StreamsStandbyTask -
>> https://issues.apache.org/jira/browse/KAFKA-16141
>> > > > StreamsUpgradeTest -
>> https://issues.apache.org/jira/browse/KAFKA-16139
>> > >
>> > > For `StreamsUpgradeTest` it was a test setup issue and should be fixed
>> > > now in trunk and 3.7 (and actually also in 3.6...)
>> > >
>> > > For `StreamsStandbyTask` the failing test exposes a regression bug, so
>> > > it's a blocker. I updated the ticket accordingly. We already have an
>> > > open PR that reverts the code introducing the regression.
>> > >
>> > >
>> > > -Matthias
>> > >
>> > > On 1/17/24 9:44 AM, Proven Provenzano wrote:
>> > > > We have another blocking issue for the RC :
>> > > > https://issues.apache.org/jira/browse/KAFKA-16157. This bug is
>> similar
>> > > to
>> > > > https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
>> > however
>> > > > can lead to the new topic having partitions that a producer cannot
>> > write
>> > > to.
>> > > >
>> > > > --Proven
>> > > >
>> > > > On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
>> > > pprovenzano@confluent.io>
>> > > > wrote:
>> > > >
>> > > >>
>> > > >> I have a PR https://github.com/apache/kafka/pull/15197 for
>> > > >> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
>> > now.
>> > > >> --Proven
>> > > >>
>> > > >> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz>
>> wrote:
>> > > >>
>> > > >>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found
>> is a
>> > > >>> blocker bug because it *
>> > > >>> *> will generate huge amount of logspam. I guess we didn't find
>> it in
>> > > >>> junit
>> > > >>> tests *
>> > > >>> *> since logspam doesn't fail the automated tests. But certainly
>> it's
>> > > not
>> > > >>> suitable *
>> > > >>> *> for production. Did you file a JIRA yet?*
>> > > >>>
>> > > >>> Hi Colin,
>> > > >>>
>> > > >>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>> > > >>>
>> > > >>> Thanks & Regards
>> > > >>> Jakub
>> > > >>>
>> > > >>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
>> > > wrote:
>> > > >>>
>> > > >>>> Hi Stanislav,
>> > > >>>>
>> > > >>>> Thanks for making the first RC. The fact that it's titled RC2 is
>> > > messing
>> > > >>>> with my mind a bit. I hope this doesn't make people think that
>> we're
>> > > >>>> farther along than we are, heh.
>> > > >>>>
>> > > >>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>> > > >>>>> *> Nice catch! It does seem like we should have gated this
>> behind
>> > the
>> > > >>>>> metadata> version as KIP-858 implies. Is the cluster configured
>> > with
>> > > >>>>> multiple log> dirs? What is the impact of the error messages?*
>> > > >>>>>
>> > > >>>>> I did not observe any obvious impact. I was able to send and
>> > receive
>> > > >>>>> messages as normally. But to be honest, I have no idea what else
>> > > >>>>> this might impact, so I did not try anything special.
>> > > >>>>>
>> > > >>>>> I think everyone upgrading an existing KRaft cluster will go
>> > through
>> > > >>> this
>> > > >>>>> stage (running Kafka 3.7 with an older metadata version for at
>> > least
>> > > a
>> > > >>>>> while). So even if it is just a logged exception without any
>> other
>> > > >>>> impact I
>> > > >>>>> wonder if it might scare users from upgrading. But I leave it to
>> > > >>> others
>> > > >>>> to
>> > > >>>>> decide if this is a blocker or not.
>> > > >>>>>
>> > > >>>>
>> > > >>>> Hi Jakub,
>> > > >>>>
>> > > >>>> Thanks for trying the RC. I think what you found is a blocker bug
>> > > >>> because
>> > > >>>> it will generate huge amount of logspam. I guess we didn't find
>> it
>> > in
>> > > >>> junit
>> > > >>>> tests since logspam doesn't fail the automated tests. But
>> certainly
>> > > it's
>> > > >>>> not suitable for production. Did you file a JIRA yet?
>> > > >>>>
>> > > >>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>> > > >>>>> <st...@confluent.io.invalid> wrote:
>> > > >>>>>
>> > > >>>>>> Hey Luke,
>> > > >>>>>>
>> > > >>>>>> This is an interesting problem. Given the fact that the KIP for
>> > > >>> having a
>> > > >>>>>> 3.8 release passed, I think it weights the scale towards not
>> > calling
>> > > >>>> this a
>> > > >>>>>> blocker and expecting it to be solved in 3.7.1.
>> > > >>>>>>
>> > > >>>>>> It is unfortunate that it would not seem safe to migrate to
>> KRaft
>> > in
>> > > >>>> 3.7.0
>> > > >>>>>> (given the inability to rollback safely), but if that's true -
>> the
>> > > >>> same
>> > > >>>>>> case would apply for 3.6.0. So in any case users w\ould be
>> > expected
>> > > >>> to
>> > > >>>> use a
>> > > >>>>>> patch release for this.
>> > > >>>>
>> > > >>>> Hi Luke,
>> > > >>>>
>> > > >>>> Thanks for testing rollback. I think this is a case where the
>> > > >>>> documentation is wrong. The intention was to for the steps to
>> > > basically
>> > > >>> be:
>> > > >>>>
>> > > >>>> 1. roll all the brokers into zk mode, but with migration enabled
>> > > >>>> 2. take down the kraft quorum
>> > > >>>> 3. rmr /controller, allowing a hybrid broker to take over.
>> > > >>>> 4. roll all the brokers into zk mode without migration enabled
>> (if
>> > > >>> desired)
>> > > >>>>
>> > > >>>> With these steps, there isn't really unavailability since a ZK
>> > > >>> controller
>> > > >>>> can be elected quickly after the kraft quorum is gone.
>> > > >>>>
>> > > >>>>>> Further, since we will have a 3.8 release - it is
>> > > >>>>>> likely we will ultimately recommend users upgrade from that
>> > version
>> > > >>>> given
>> > > >>>>>> its aim is to have strategic KRaft feature parity with ZK.
>> > > >>>>>> That being said, I am not 100% on this. Let me know whether you
>> > > think
>> > > >>>> this
>> > > >>>>>> should block the release, Luke. I am also tagging Colin and
>> David
>> > to
>> > > >>>> weigh
>> > > >>>>>> in with their opinions, as they worked on the migration logic.
>> > > >>>>
>> > > >>>> The rollback docs are new in 3.7 so the fact that they're wrong
>> is a
>> > > >>> clear
>> > > >>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>> > > >>>>
>> > > >>>> best,
>> > > >>>> Colin
>> > > >>>>
>> > > >>>>>>
>> > > >>>>>> Hey Kirk and Chris,
>> > > >>>>>>
>> > > >>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad
>> log
>> > > >>> due
>> > > >>>> to
>> > > >>>>>> improper closing. And the PR description implies this has been
>> > > >>> present
>> > > >>>>>> since 3.5. While annoying, I don't see a strong reason for
>> this to
>> > > >>> block
>> > > >>>>>> the release.
>> > > >>>>>>
>> > > >>>>>> Hey Jakub,
>> > > >>>>>>
>> > > >>>>>> Nice catch! It does seem like we should have gated this behind
>> the
>> > > >>>> metadata
>> > > >>>>>> version as KIP-858 implies. Is the cluster configured with
>> > multiple
>> > > >>> log
>> > > >>>>>> dirs? What is the impact of the error messages?
>> > > >>>>>>
>> > > >>>>>> Tagging Igor (the author of the KIP) to weigh in.
>> > > >>>>>>
>> > > >>>>>> Best,
>> > > >>>>>> Stanislav
>> > > >>>>>>
>> > > >>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>> > > >>> wrote:
>> > > >>>>>>
>> > > >>>>>>> Hi,
>> > > >>>>>>>
>> > > >>>>>>> I was trying the RC2 and run into the following issue ...
>> when I
>> > > >>> run
>> > > >>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>> > > >>> metadata
>> > > >>>>>>> version, I seem to be getting repeated errors like this in the
>> > > >>>> controller
>> > > >>>>>>> logs:
>> > > >>>>>>>
>> > > >>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>> > > >>>>>> assignReplicasToDirs:
>> > > >>>>>>> event failed with UnsupportedVersionException in 15
>> microseconds.
>> > > >>>>>>> (org.apache.kafka.controller.QuorumController)
>> > > >>>>>>> [quorum-controller-0-event-handler]
>> > > >>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
>> > Unexpected
>> > > >>>> error
>> > > >>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>> > > >>>>>>> apiVersion=0, clientId=1000, correlationId=14,
>> headerVersion=2)
>> > --
>> > > >>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>> > > >>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>> > > >>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>> > > >>>>>>> partitions=[PartitionData(partitionIndex=2),
>> > > >>>>>>> PartitionData(partitionIndex=1)]),
>> > > >>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>> > > >>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>> > > >>>>>>>
>> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>> > > >>>>>>> apiVersion=0, clientId=1000, correlationId=14,
>> headerVersion=2),
>> > > >>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>> > > >>>> clientAddress=/
>> > > >>>>>>> 172.16.14.217,
>> principal=User:CN=my-cluster-kafka,O=io.strimzi,
>> > > >>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
>> > securityProtocol=SSL,
>> > > >>>>>>>
>> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
>> > > >>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>> > > >>>>>>> ])
>> > > >>>>>>> (kafka.server.ControllerApis)
>> [quorum-controller-0-event-handler]
>> > > >>>>>>> java.util.concurrent.CompletionException:
>> > > >>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>> > > >>> Directory
>> > > >>>>>>> assignment is not supported yet.
>> > > >>>>>>>
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>> > > >>>>>>>   at
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>> > > >>>>>>>   at java.base/java.lang.Thread.run(Thread.java:840)
>> > > >>>>>>>
>> > > >>>>>>> Caused by:
>> > > >>> org.apache.kafka.common.errors.UnsupportedVersionException:
>> > > >>>>>>> Directory assignment is not supported yet.
>> > > >>>>>>>
>> > > >>>>>>> Is that expected? I guess with the metadata version set to
>> > > >>> 3.6-IV2, it
>> > > >>>>>>> makes sense that the request is not supported. But shouldn't
>> then
>> > > >>> the
>> > > >>>>>>> request not be sent at all by the brokers? (I did not opened a
>> > JIRA
>> > > >>>> for
>> > > >>>>>> it,
>> > > >>>>>>> but I can open one if you agree this is not expected)
>> > > >>>>>>>
>> > > >>>>>>> Thanks & Regards
>> > > >>>>>>> Jakub
>> > > >>>>>>>
>> > > >>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>> > > >>> wrote:
>> > > >>>>>>>
>> > > >>>>>>>> Hi Stanislav,
>> > > >>>>>>>>
>> > > >>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
>> > maybe
>> > > >>>> you
>> > > >>>>>>>> missed it.
>> > > >>>>>>>> cross-posting here:
>> > > >>>>>>>>
>> > > >>>>>>>> There is a bug KAFKA-16101
>> > > >>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101>
>> reporting
>> > > >>> that
>> > > >>>>>>> "Kafka
>> > > >>>>>>>> cluster will be unavailable during KRaft migration rollback".
>> > > >>>>>>>> The impact for this issue is that if brokers try to rollback
>> to
>> > > >>> ZK
>> > > >>>> mode
>> > > >>>>>>>> during KRaft migration process, there will be a period of
>> time
>> > > >>> the
>> > > >>>>>>> cluster
>> > > >>>>>>>> is unavailable.
>> > > >>>>>>>> Since ZK migrating to KRaft feature is a production ready
>> > > >>> feature, I
>> > > >>>>>>> think
>> > > >>>>>>>> this should be addressed soon.
>> > > >>>>>>>> Do you think this is a blocker for v3.7.0?
>> > > >>>>>>>>
>> > > >>>>>>>> Thanks.
>> > > >>>>>>>> Luke
>> > > >>>>>>>>
>> > > >>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>> > > >>>> fearthecellos@gmail.com
>> > > >>>>>>>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> Thanks, Kirk!
>> > > >>>>>>>>>
>> > > >>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>> > > >>>>>>>>>
>> > > >>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>> > > >>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Hi Chris/Stanislav,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>> > > >>>> problem
>> > > >>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>> > > >>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use
>> the
>> > > >>>>>>> quickstart
>> > > >>>>>>>>>> approach as a second means to reproduce/verify while I wait
>> > > >>> for
>> > > >>>> the
>> > > >>>>>>>> PR's
>> > > >>>>>>>>>> Jenkins job to finish.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Thanks,
>> > > >>>>>>>>>> Kirk
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>> > > >>>>>>>>>>> Hi Stanislav,
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Thanks for running this release!
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> To verify, I:
>> > > >>>>>>>>>>> - Built from source using Java 11 with both:
>> > > >>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>> > > >>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>> > > >>>>>>>>>>>
>> > > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > > >>>>>>>>>>> - Checked signatures and checksums
>> > > >>>>>>>>>>> - Ran the quickstart using both:
>> > > >>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>> > > >>>>>>>>>>>
>> > > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > > >>>>>> with
>> > > >>>>>>>> Java
>> > > >>>>>>>>>> 11
>> > > >>>>>>>>>>> and Scala 13 in KRaft mode
>> > > >>>>>>>>>>> - - Our shiny new broker Docker image,
>> > > >>> apache/kafka:3.7.0-rc2
>> > > >>>>>>>>>>> - Ran all unit tests
>> > > >>>>>>>>>>> - Ran all integration tests for Connect and MM2
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I found two minor areas for concern:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> 1. (Possibly a blocker)
>> > > >>>>>>>>>>> When running the quickstart, I noticed this ERROR-level
>> log
>> > > >>>>>> message
>> > > >>>>>>>>> being
>> > > >>>>>>>>>>> emitted frequently (not not every time) when I killed my
>> > > >>>> console
>> > > >>>>>>>>> consumer
>> > > >>>>>>>>>>> via ctrl-C:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>> > > >>>>>>>> clientId=console-consumer,
>> > > >>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>> > > >>>>>> FetchSessionHandler
>> > > >>>>>>>> for
>> > > >>>>>>>>>> node
>> > > >>>>>>>>>>> 1. Ignoring fetch response
>> > > >>>>>>>>>>>
>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I see that this error message is already reported in
>> > > >>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>> > > >>> think we
>> > > >>>>>>> should
>> > > >>>>>>>>>>> prioritize fixing it for this release. I know it's
>> probably
>> > > >>>>>> benign
>> > > >>>>>>>> but
>> > > >>>>>>>>>> it's
>> > > >>>>>>>>>>> really not a good look for us when basic operations log
>> > > >>> error
>> > > >>>>>>>> messages,
>> > > >>>>>>>>>> and
>> > > >>>>>>>>>>> it may give new users some headaches.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> 2. (Probably not a blocker)
>> > > >>>>>>>>>>> The following unit tests failed the first time around, and
>> > > >>>> all of
>> > > >>>>>>>> them
>> > > >>>>>>>>>>> passed the second time I ran them:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> - (clients)
>> > > >>>>>>>>>>
>> > > >>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>> > > >>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>> > > >>>>>>>>>>> - (clients)
>> > > >>> Tls13SelectorTest.testConnectionsByClientMetric()
>> > > >>>>>>>>>>> - (connect)
>> > > >>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>> > > >>>>>>>>> (I
>> > > >>>>>>>>>>> thought I fixed this one! 🤬🤬)
>> > > >>>>>>>>>>> - (core)
>> > > >>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Thanks again for your work on this release, and
>> > > >>>> congratulations
>> > > >>>>>> to
>> > > >>>>>>>>> Kafka
>> > > >>>>>>>>>>> Streams for having zero flaky unit tests during my
>> > > >>>>>>>> highly-experimental
>> > > >>>>>>>>>>> single laptop run!
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Cheers,
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Chris
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>> > > >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>> > > >>>> 3.7.0.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>> > > >>> that
>> > > >>>> I
>> > > >>>>>> had
>> > > >>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>> > > >>> announcing
>> > > >>>>>> due
>> > > >>>>>>> to
>> > > >>>>>>>>> new
>> > > >>>>>>>>>>>> blockers arriving before I could even announce them.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Further - I haven't yet been able to set up the system
>> > > >>> tests
>> > > >>>>>>>>>> successfully.
>> > > >>>>>>>>>>>> And the integration/unit tests do have a few failures
>> > > >>> that I
>> > > >>>>>> have
>> > > >>>>>>>> to
>> > > >>>>>>>>>> spend
>> > > >>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>> > > >>>>>> notices
>> > > >>>>>>>> any
>> > > >>>>>>>>>> tests
>> > > >>>>>>>>>>>> failing that they're subject matters experts in. Expect
>> > > >>> me
>> > > >>>> to
>> > > >>>>>>>> follow
>> > > >>>>>>>>>> up in
>> > > >>>>>>>>>>>> a day or two with more detailed analysis.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Major changes include:
>> > > >>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>> > > >>>> consumer
>> > > >>>>>>>>>> rebalance
>> > > >>>>>>>>>>>> protocol
>> > > >>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>> > > >>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>> > > >>>> standardized
>> > > >>>>>>>>>> interface
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Check more information in the WIP blog post:
>> > > >>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Release notes for the 3.7.0 release:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>> > > >>> 18,
>> > > >>>> 9am
>> > > >>>>>>> PT
>> > > >>>>>>>>> ***
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>> > > >>> this
>> > > >>>>>>> being
>> > > >>>>>>>>> the
>> > > >>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>> > > >>> it a
>> > > >>>> bit
>> > > >>>>>>>> more
>> > > >>>>>>>>>> time.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>> > > >>>>>> release:
>> > > >>>>>>>>>>>> https://kafka.apache.org/KEYS
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>> > > >>>>>>>>>>>>
>> > > >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Docker release artifact to be voted upon:
>> > > >>>>>>>>>>>> apache/kafka:3.7.0-rc2
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Maven artifacts to be voted upon:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>
>> > > >>>
>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Javadoc:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>
>> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>> > > >>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Documentation:
>> > > >>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Protocol:
>> > > >>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>> > > >>>>>>>>>>>> Unit/integration tests:
>> > > >>>>>>>>>>>>
>> > > >>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>> > > >>>>>>>>>>>> There are failing tests here. I have to follow up with
>> > > >>>> triaging
>> > > >>>>>>>> some
>> > > >>>>>>>>> of
>> > > >>>>>>>>>>>> the failures and figuring out if they're actual problems
>> > > >>> or
>> > > >>>>>>> simply
>> > > >>>>>>>>>> flakes.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> System tests:
>> > > >>>>>>>>>>
>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> No successful system test runs yet. I am working on
>> > > >>> getting
>> > > >>>> the
>> > > >>>>>>> job
>> > > >>>>>>>>> to
>> > > >>>>>>>>>> run.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>> > > >>>>>> branch:
>> > > >>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>> > > >>>> from
>> > > >>>>>> the
>> > > >>>>>>>>>> Docker
>> > > >>>>>>>>>>>> Build run:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>
>> > >
>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> And the final docker image build job - Docker Build Test
>> > > >>>>>>> Pipeline:
>> > > >>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> /**************************************
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Thanks,
>> > > >>>>>>>>>>>> Stanislav Kozlovski
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> --
>> > > >>>>>> Best,
>> > > >>>>>> Stanislav
>> > > >>>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > > >
>> > >
>> >
>>
>>
>> --
>> Best,
>> Stanislav
>>
>

Re: [VOTE] 3.7.0 RC2

Posted by Justine Olshan <jo...@confluent.io.INVALID>.
Hey Stan,

Just wanted to clarify -- KAFKA-14616 is not particularly related to JBOD
but to ZK -> KRaft migration.

There were some other related migration bugs like
https://issues.apache.org/jira/browse/KAFKA-16180.

This may or may not influence decisions, but wanted to paint the full
picture of the blocker bugs and their causes.

Justine


On Tue, Jan 23, 2024 at 12:17 PM Stanislav Kozlovski
<st...@confluent.io.invalid> wrote:

> Hey all, I figured I'd give an update about what known blockers we have
> right now:
>
> - KAFKA-16101: KRaft migration rollback documentation is incorrect -
> https://github.com/apache/kafka/pull/15193; This need not block RC
> creation, but we need the docs updated so that people can test properly
> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
> https://github.com/apache/kafka/pull/15230 ; I am of the understanding
> that
> this is blocking JBOD for 3.7
> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
> a strict blocker with an open PR
> https://github.com/apache/kafka/pull/15232
> - although I understand Proveen is out of office
> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
> hearing mixed opinions on whether this is a blocker (
> https://github.com/apache/kafka/pull/15136)
>
> Given that there are 3 JBOD blocker bugs, and I am not confident they will
> all be merged this week - I am on the edge of voting to revert JBOD from
> this release, or mark it early access.
>
> By all accounts, it seems that if we keep with JBOD the release will have
> to spill into February, which is a month extra from the time-based release
> plan we had of start of January.
>
> Can I ask others for an opinion?
>
> Best,
> Stan
>
> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi all,
> >
> > I think I've found another blocker issue: KAFKA-16162
> > <https://issues.apache.org/jira/browse/KAFKA-16162> .
> > The impact is after upgrading to 3.7.0, any new created topics/partitions
> > will be unavailable.
> > I've put my findings in the JIRA.
> >
> > Thanks.
> > Luke
> >
> > On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > Stan, thanks for driving this all forward! Excellent job.
> > >
> > > About
> > >
> > > > StreamsStandbyTask -
> https://issues.apache.org/jira/browse/KAFKA-16141
> > > > StreamsUpgradeTest -
> https://issues.apache.org/jira/browse/KAFKA-16139
> > >
> > > For `StreamsUpgradeTest` it was a test setup issue and should be fixed
> > > now in trunk and 3.7 (and actually also in 3.6...)
> > >
> > > For `StreamsStandbyTask` the failing test exposes a regression bug, so
> > > it's a blocker. I updated the ticket accordingly. We already have an
> > > open PR that reverts the code introducing the regression.
> > >
> > >
> > > -Matthias
> > >
> > > On 1/17/24 9:44 AM, Proven Provenzano wrote:
> > > > We have another blocking issue for the RC :
> > > > https://issues.apache.org/jira/browse/KAFKA-16157. This bug is
> similar
> > > to
> > > > https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
> > however
> > > > can lead to the new topic having partitions that a producer cannot
> > write
> > > to.
> > > >
> > > > --Proven
> > > >
> > > > On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> > > pprovenzano@confluent.io>
> > > > wrote:
> > > >
> > > >>
> > > >> I have a PR https://github.com/apache/kafka/pull/15197 for
> > > >> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
> > now.
> > > >> --Proven
> > > >>
> > > >> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz>
> wrote:
> > > >>
> > > >>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found
> is a
> > > >>> blocker bug because it *
> > > >>> *> will generate huge amount of logspam. I guess we didn't find it
> in
> > > >>> junit
> > > >>> tests *
> > > >>> *> since logspam doesn't fail the automated tests. But certainly
> it's
> > > not
> > > >>> suitable *
> > > >>> *> for production. Did you file a JIRA yet?*
> > > >>>
> > > >>> Hi Colin,
> > > >>>
> > > >>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> > > >>>
> > > >>> Thanks & Regards
> > > >>> Jakub
> > > >>>
> > > >>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > >>>
> > > >>>> Hi Stanislav,
> > > >>>>
> > > >>>> Thanks for making the first RC. The fact that it's titled RC2 is
> > > messing
> > > >>>> with my mind a bit. I hope this doesn't make people think that
> we're
> > > >>>> farther along than we are, heh.
> > > >>>>
> > > >>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > > >>>>> *> Nice catch! It does seem like we should have gated this behind
> > the
> > > >>>>> metadata> version as KIP-858 implies. Is the cluster configured
> > with
> > > >>>>> multiple log> dirs? What is the impact of the error messages?*
> > > >>>>>
> > > >>>>> I did not observe any obvious impact. I was able to send and
> > receive
> > > >>>>> messages as normally. But to be honest, I have no idea what else
> > > >>>>> this might impact, so I did not try anything special.
> > > >>>>>
> > > >>>>> I think everyone upgrading an existing KRaft cluster will go
> > through
> > > >>> this
> > > >>>>> stage (running Kafka 3.7 with an older metadata version for at
> > least
> > > a
> > > >>>>> while). So even if it is just a logged exception without any
> other
> > > >>>> impact I
> > > >>>>> wonder if it might scare users from upgrading. But I leave it to
> > > >>> others
> > > >>>> to
> > > >>>>> decide if this is a blocker or not.
> > > >>>>>
> > > >>>>
> > > >>>> Hi Jakub,
> > > >>>>
> > > >>>> Thanks for trying the RC. I think what you found is a blocker bug
> > > >>> because
> > > >>>> it will generate huge amount of logspam. I guess we didn't find it
> > in
> > > >>> junit
> > > >>>> tests since logspam doesn't fail the automated tests. But
> certainly
> > > it's
> > > >>>> not suitable for production. Did you file a JIRA yet?
> > > >>>>
> > > >>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > > >>>>> <st...@confluent.io.invalid> wrote:
> > > >>>>>
> > > >>>>>> Hey Luke,
> > > >>>>>>
> > > >>>>>> This is an interesting problem. Given the fact that the KIP for
> > > >>> having a
> > > >>>>>> 3.8 release passed, I think it weights the scale towards not
> > calling
> > > >>>> this a
> > > >>>>>> blocker and expecting it to be solved in 3.7.1.
> > > >>>>>>
> > > >>>>>> It is unfortunate that it would not seem safe to migrate to
> KRaft
> > in
> > > >>>> 3.7.0
> > > >>>>>> (given the inability to rollback safely), but if that's true -
> the
> > > >>> same
> > > >>>>>> case would apply for 3.6.0. So in any case users w\ould be
> > expected
> > > >>> to
> > > >>>> use a
> > > >>>>>> patch release for this.
> > > >>>>
> > > >>>> Hi Luke,
> > > >>>>
> > > >>>> Thanks for testing rollback. I think this is a case where the
> > > >>>> documentation is wrong. The intention was to for the steps to
> > > basically
> > > >>> be:
> > > >>>>
> > > >>>> 1. roll all the brokers into zk mode, but with migration enabled
> > > >>>> 2. take down the kraft quorum
> > > >>>> 3. rmr /controller, allowing a hybrid broker to take over.
> > > >>>> 4. roll all the brokers into zk mode without migration enabled (if
> > > >>> desired)
> > > >>>>
> > > >>>> With these steps, there isn't really unavailability since a ZK
> > > >>> controller
> > > >>>> can be elected quickly after the kraft quorum is gone.
> > > >>>>
> > > >>>>>> Further, since we will have a 3.8 release - it is
> > > >>>>>> likely we will ultimately recommend users upgrade from that
> > version
> > > >>>> given
> > > >>>>>> its aim is to have strategic KRaft feature parity with ZK.
> > > >>>>>> That being said, I am not 100% on this. Let me know whether you
> > > think
> > > >>>> this
> > > >>>>>> should block the release, Luke. I am also tagging Colin and
> David
> > to
> > > >>>> weigh
> > > >>>>>> in with their opinions, as they worked on the migration logic.
> > > >>>>
> > > >>>> The rollback docs are new in 3.7 so the fact that they're wrong
> is a
> > > >>> clear
> > > >>>> blocker, I think. But easy to fix, I believe. I will create a PR.
> > > >>>>
> > > >>>> best,
> > > >>>> Colin
> > > >>>>
> > > >>>>>>
> > > >>>>>> Hey Kirk and Chris,
> > > >>>>>>
> > > >>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad
> log
> > > >>> due
> > > >>>> to
> > > >>>>>> improper closing. And the PR description implies this has been
> > > >>> present
> > > >>>>>> since 3.5. While annoying, I don't see a strong reason for this
> to
> > > >>> block
> > > >>>>>> the release.
> > > >>>>>>
> > > >>>>>> Hey Jakub,
> > > >>>>>>
> > > >>>>>> Nice catch! It does seem like we should have gated this behind
> the
> > > >>>> metadata
> > > >>>>>> version as KIP-858 implies. Is the cluster configured with
> > multiple
> > > >>> log
> > > >>>>>> dirs? What is the impact of the error messages?
> > > >>>>>>
> > > >>>>>> Tagging Igor (the author of the KIP) to weigh in.
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Stanislav
> > > >>>>>>
> > > >>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
> > > >>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi,
> > > >>>>>>>
> > > >>>>>>> I was trying the RC2 and run into the following issue ... when
> I
> > > >>> run
> > > >>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> > > >>> metadata
> > > >>>>>>> version, I seem to be getting repeated errors like this in the
> > > >>>> controller
> > > >>>>>>> logs:
> > > >>>>>>>
> > > >>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > > >>>>>> assignReplicasToDirs:
> > > >>>>>>> event failed with UnsupportedVersionException in 15
> microseconds.
> > > >>>>>>> (org.apache.kafka.controller.QuorumController)
> > > >>>>>>> [quorum-controller-0-event-handler]
> > > >>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
> > Unexpected
> > > >>>> error
> > > >>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > > >>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
> > --
> > > >>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > > >>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > > >>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > > >>>>>>> partitions=[PartitionData(partitionIndex=2),
> > > >>>>>>> PartitionData(partitionIndex=1)]),
> > > >>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > > >>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
> > > >>>>>>>
> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > > >>>>>>> apiVersion=0, clientId=1000, correlationId=14,
> headerVersion=2),
> > > >>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > > >>>> clientAddress=/
> > > >>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > > >>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
> > securityProtocol=SSL,
> > > >>>>>>>
> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > > >>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > > >>>>>>> ])
> > > >>>>>>> (kafka.server.ControllerApis)
> [quorum-controller-0-event-handler]
> > > >>>>>>> java.util.concurrent.CompletionException:
> > > >>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > > >>> Directory
> > > >>>>>>> assignment is not supported yet.
> > > >>>>>>>
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > > >>>>>>>   at
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > > >>>>>>>   at java.base/java.lang.Thread.run(Thread.java:840)
> > > >>>>>>>
> > > >>>>>>> Caused by:
> > > >>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > > >>>>>>> Directory assignment is not supported yet.
> > > >>>>>>>
> > > >>>>>>> Is that expected? I guess with the metadata version set to
> > > >>> 3.6-IV2, it
> > > >>>>>>> makes sense that the request is not supported. But shouldn't
> then
> > > >>> the
> > > >>>>>>> request not be sent at all by the brokers? (I did not opened a
> > JIRA
> > > >>>> for
> > > >>>>>> it,
> > > >>>>>>> but I can open one if you agree this is not expected)
> > > >>>>>>>
> > > >>>>>>> Thanks & Regards
> > > >>>>>>> Jakub
> > > >>>>>>>
> > > >>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi Stanislav,
> > > >>>>>>>>
> > > >>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
> > maybe
> > > >>>> you
> > > >>>>>>>> missed it.
> > > >>>>>>>> cross-posting here:
> > > >>>>>>>>
> > > >>>>>>>> There is a bug KAFKA-16101
> > > >>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> > > >>> that
> > > >>>>>>> "Kafka
> > > >>>>>>>> cluster will be unavailable during KRaft migration rollback".
> > > >>>>>>>> The impact for this issue is that if brokers try to rollback
> to
> > > >>> ZK
> > > >>>> mode
> > > >>>>>>>> during KRaft migration process, there will be a period of time
> > > >>> the
> > > >>>>>>> cluster
> > > >>>>>>>> is unavailable.
> > > >>>>>>>> Since ZK migrating to KRaft feature is a production ready
> > > >>> feature, I
> > > >>>>>>> think
> > > >>>>>>>> this should be addressed soon.
> > > >>>>>>>> Do you think this is a blocker for v3.7.0?
> > > >>>>>>>>
> > > >>>>>>>> Thanks.
> > > >>>>>>>> Luke
> > > >>>>>>>>
> > > >>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > > >>>> fearthecellos@gmail.com
> > > >>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks, Kirk!
> > > >>>>>>>>>
> > > >>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> > > >>>>>>>>>
> > > >>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> > > >>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Chris/Stanislav,
> > > >>>>>>>>>>
> > > >>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
> > > >>>> problem
> > > >>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> > > >>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
> > > >>>>>>> quickstart
> > > >>>>>>>>>> approach as a second means to reproduce/verify while I wait
> > > >>> for
> > > >>>> the
> > > >>>>>>>> PR's
> > > >>>>>>>>>> Jenkins job to finish.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> Kirk
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > > >>>>>>>>>>> Hi Stanislav,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for running this release!
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> To verify, I:
> > > >>>>>>>>>>> - Built from source using Java 11 with both:
> > > >>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> > > >>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> > > >>>>>>>>>>>
> > > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > >>>>>>>>>>> - Checked signatures and checksums
> > > >>>>>>>>>>> - Ran the quickstart using both:
> > > >>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> > > >>>>>>>>>>>
> > > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > >>>>>> with
> > > >>>>>>>> Java
> > > >>>>>>>>>> 11
> > > >>>>>>>>>>> and Scala 13 in KRaft mode
> > > >>>>>>>>>>> - - Our shiny new broker Docker image,
> > > >>> apache/kafka:3.7.0-rc2
> > > >>>>>>>>>>> - Ran all unit tests
> > > >>>>>>>>>>> - Ran all integration tests for Connect and MM2
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I found two minor areas for concern:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 1. (Possibly a blocker)
> > > >>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
> > > >>>>>> message
> > > >>>>>>>>> being
> > > >>>>>>>>>>> emitted frequently (not not every time) when I killed my
> > > >>>> console
> > > >>>>>>>>> consumer
> > > >>>>>>>>>>> via ctrl-C:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> > > >>>>>>>> clientId=console-consumer,
> > > >>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> > > >>>>>> FetchSessionHandler
> > > >>>>>>>> for
> > > >>>>>>>>>> node
> > > >>>>>>>>>>> 1. Ignoring fetch response
> > > >>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I see that this error message is already reported in
> > > >>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> > > >>> think we
> > > >>>>>>> should
> > > >>>>>>>>>>> prioritize fixing it for this release. I know it's probably
> > > >>>>>> benign
> > > >>>>>>>> but
> > > >>>>>>>>>> it's
> > > >>>>>>>>>>> really not a good look for us when basic operations log
> > > >>> error
> > > >>>>>>>> messages,
> > > >>>>>>>>>> and
> > > >>>>>>>>>>> it may give new users some headaches.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 2. (Probably not a blocker)
> > > >>>>>>>>>>> The following unit tests failed the first time around, and
> > > >>>> all of
> > > >>>>>>>> them
> > > >>>>>>>>>>> passed the second time I ran them:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - (clients)
> > > >>>>>>>>>>
> > > >>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > > >>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
> > > >>>>>>>>>>> - (clients)
> > > >>> Tls13SelectorTest.testConnectionsByClientMetric()
> > > >>>>>>>>>>> - (connect)
> > > >>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > > >>>>>>>>> (I
> > > >>>>>>>>>>> thought I fixed this one! 🤬🤬)
> > > >>>>>>>>>>> - (core)
> > > >>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks again for your work on this release, and
> > > >>>> congratulations
> > > >>>>>> to
> > > >>>>>>>>> Kafka
> > > >>>>>>>>>>> Streams for having zero flaky unit tests during my
> > > >>>>>>>> highly-experimental
> > > >>>>>>>>>>> single laptop run!
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Chris
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > > >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> This is the first candidate for release of Apache Kafka
> > > >>>> 3.7.0.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
> > > >>> that
> > > >>>> I
> > > >>>>>> had
> > > >>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> > > >>> announcing
> > > >>>>>> due
> > > >>>>>>> to
> > > >>>>>>>>> new
> > > >>>>>>>>>>>> blockers arriving before I could even announce them.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Further - I haven't yet been able to set up the system
> > > >>> tests
> > > >>>>>>>>>> successfully.
> > > >>>>>>>>>>>> And the integration/unit tests do have a few failures
> > > >>> that I
> > > >>>>>> have
> > > >>>>>>>> to
> > > >>>>>>>>>> spend
> > > >>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
> > > >>>>>> notices
> > > >>>>>>>> any
> > > >>>>>>>>>> tests
> > > >>>>>>>>>>>> failing that they're subject matters experts in. Expect
> > > >>> me
> > > >>>> to
> > > >>>>>>>> follow
> > > >>>>>>>>>> up in
> > > >>>>>>>>>>>> a day or two with more detailed analysis.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Major changes include:
> > > >>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
> > > >>>> consumer
> > > >>>>>>>>>> rebalance
> > > >>>>>>>>>>>> protocol
> > > >>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> > > >>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> > > >>>> standardized
> > > >>>>>>>>>> interface
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Check more information in the WIP blog post:
> > > >>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Release notes for the 3.7.0 release:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> *** Please download, test and vote by Thursday, January
> > > >>> 18,
> > > >>>> 9am
> > > >>>>>>> PT
> > > >>>>>>>>> ***
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
> > > >>> this
> > > >>>>>>> being
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
> > > >>> it a
> > > >>>> bit
> > > >>>>>>>> more
> > > >>>>>>>>>> time.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
> > > >>>>>> release:
> > > >>>>>>>>>>>> https://kafka.apache.org/KEYS
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
> > > >>>>>>>>>>>>
> > > >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Docker release artifact to be voted upon:
> > > >>>>>>>>>>>> apache/kafka:3.7.0-rc2
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Maven artifacts to be voted upon:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Javadoc:
> > > >>>>>>>>>>>>
> > > >>>>>>>>
> > > >>>>
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > > >>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Documentation:
> > > >>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Protocol:
> > > >>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> > > >>>>>>>>>>>> Unit/integration tests:
> > > >>>>>>>>>>>>
> > > >>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > > >>>>>>>>>>>> There are failing tests here. I have to follow up with
> > > >>>> triaging
> > > >>>>>>>> some
> > > >>>>>>>>> of
> > > >>>>>>>>>>>> the failures and figuring out if they're actual problems
> > > >>> or
> > > >>>>>>> simply
> > > >>>>>>>>>> flakes.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> System tests:
> > > >>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> No successful system test runs yet. I am working on
> > > >>> getting
> > > >>>> the
> > > >>>>>>> job
> > > >>>>>>>>> to
> > > >>>>>>>>>> run.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
> > > >>>>>> branch:
> > > >>>>>>>>>>>> Attached are the scan_report and report_jvm output files
> > > >>>> from
> > > >>>>>> the
> > > >>>>>>>>>> Docker
> > > >>>>>>>>>>>> Build run:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> And the final docker image build job - Docker Build Test
> > > >>>>>>> Pipeline:
> > > >>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> /**************************************
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>> Stanislav Kozlovski
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Best,
> > > >>>>>> Stanislav
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> >
>
>
> --
> Best,
> Stanislav
>

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
Hey all, I figured I'd give an update about what known blockers we have
right now:

- KAFKA-16101: KRaft migration rollback documentation is incorrect -
https://github.com/apache/kafka/pull/15193; This need not block RC
creation, but we need the docs updated so that people can test properly
- KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
this is blocking JBOD for 3.7
- KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
- although I understand Proveen is out of office
- KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
hearing mixed opinions on whether this is a blocker (
https://github.com/apache/kafka/pull/15136)

Given that there are 3 JBOD blocker bugs, and I am not confident they will
all be merged this week - I am on the edge of voting to revert JBOD from
this release, or mark it early access.

By all accounts, it seems that if we keep with JBOD the release will have
to spill into February, which is a month extra from the time-based release
plan we had of start of January.

Can I ask others for an opinion?

Best,
Stan

On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:

> Hi all,
>
> I think I've found another blocker issue: KAFKA-16162
> <https://issues.apache.org/jira/browse/KAFKA-16162> .
> The impact is after upgrading to 3.7.0, any new created topics/partitions
> will be unavailable.
> I've put my findings in the JIRA.
>
> Thanks.
> Luke
>
> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>
> > Stan, thanks for driving this all forward! Excellent job.
> >
> > About
> >
> > > StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
> > > StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
> >
> > For `StreamsUpgradeTest` it was a test setup issue and should be fixed
> > now in trunk and 3.7 (and actually also in 3.6...)
> >
> > For `StreamsStandbyTask` the failing test exposes a regression bug, so
> > it's a blocker. I updated the ticket accordingly. We already have an
> > open PR that reverts the code introducing the regression.
> >
> >
> > -Matthias
> >
> > On 1/17/24 9:44 AM, Proven Provenzano wrote:
> > > We have another blocking issue for the RC :
> > > https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
> > to
> > > https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
> however
> > > can lead to the new topic having partitions that a producer cannot
> write
> > to.
> > >
> > > --Proven
> > >
> > > On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> > pprovenzano@confluent.io>
> > > wrote:
> > >
> > >>
> > >> I have a PR https://github.com/apache/kafka/pull/15197 for
> > >> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
> now.
> > >> --Proven
> > >>
> > >> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
> > >>
> > >>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
> > >>> blocker bug because it *
> > >>> *> will generate huge amount of logspam. I guess we didn't find it in
> > >>> junit
> > >>> tests *
> > >>> *> since logspam doesn't fail the automated tests. But certainly it's
> > not
> > >>> suitable *
> > >>> *> for production. Did you file a JIRA yet?*
> > >>>
> > >>> Hi Colin,
> > >>>
> > >>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> > >>>
> > >>> Thanks & Regards
> > >>> Jakub
> > >>>
> > >>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > >>>
> > >>>> Hi Stanislav,
> > >>>>
> > >>>> Thanks for making the first RC. The fact that it's titled RC2 is
> > messing
> > >>>> with my mind a bit. I hope this doesn't make people think that we're
> > >>>> farther along than we are, heh.
> > >>>>
> > >>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > >>>>> *> Nice catch! It does seem like we should have gated this behind
> the
> > >>>>> metadata> version as KIP-858 implies. Is the cluster configured
> with
> > >>>>> multiple log> dirs? What is the impact of the error messages?*
> > >>>>>
> > >>>>> I did not observe any obvious impact. I was able to send and
> receive
> > >>>>> messages as normally. But to be honest, I have no idea what else
> > >>>>> this might impact, so I did not try anything special.
> > >>>>>
> > >>>>> I think everyone upgrading an existing KRaft cluster will go
> through
> > >>> this
> > >>>>> stage (running Kafka 3.7 with an older metadata version for at
> least
> > a
> > >>>>> while). So even if it is just a logged exception without any other
> > >>>> impact I
> > >>>>> wonder if it might scare users from upgrading. But I leave it to
> > >>> others
> > >>>> to
> > >>>>> decide if this is a blocker or not.
> > >>>>>
> > >>>>
> > >>>> Hi Jakub,
> > >>>>
> > >>>> Thanks for trying the RC. I think what you found is a blocker bug
> > >>> because
> > >>>> it will generate huge amount of logspam. I guess we didn't find it
> in
> > >>> junit
> > >>>> tests since logspam doesn't fail the automated tests. But certainly
> > it's
> > >>>> not suitable for production. Did you file a JIRA yet?
> > >>>>
> > >>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > >>>>> <st...@confluent.io.invalid> wrote:
> > >>>>>
> > >>>>>> Hey Luke,
> > >>>>>>
> > >>>>>> This is an interesting problem. Given the fact that the KIP for
> > >>> having a
> > >>>>>> 3.8 release passed, I think it weights the scale towards not
> calling
> > >>>> this a
> > >>>>>> blocker and expecting it to be solved in 3.7.1.
> > >>>>>>
> > >>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
> in
> > >>>> 3.7.0
> > >>>>>> (given the inability to rollback safely), but if that's true - the
> > >>> same
> > >>>>>> case would apply for 3.6.0. So in any case users w\ould be
> expected
> > >>> to
> > >>>> use a
> > >>>>>> patch release for this.
> > >>>>
> > >>>> Hi Luke,
> > >>>>
> > >>>> Thanks for testing rollback. I think this is a case where the
> > >>>> documentation is wrong. The intention was to for the steps to
> > basically
> > >>> be:
> > >>>>
> > >>>> 1. roll all the brokers into zk mode, but with migration enabled
> > >>>> 2. take down the kraft quorum
> > >>>> 3. rmr /controller, allowing a hybrid broker to take over.
> > >>>> 4. roll all the brokers into zk mode without migration enabled (if
> > >>> desired)
> > >>>>
> > >>>> With these steps, there isn't really unavailability since a ZK
> > >>> controller
> > >>>> can be elected quickly after the kraft quorum is gone.
> > >>>>
> > >>>>>> Further, since we will have a 3.8 release - it is
> > >>>>>> likely we will ultimately recommend users upgrade from that
> version
> > >>>> given
> > >>>>>> its aim is to have strategic KRaft feature parity with ZK.
> > >>>>>> That being said, I am not 100% on this. Let me know whether you
> > think
> > >>>> this
> > >>>>>> should block the release, Luke. I am also tagging Colin and David
> to
> > >>>> weigh
> > >>>>>> in with their opinions, as they worked on the migration logic.
> > >>>>
> > >>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
> > >>> clear
> > >>>> blocker, I think. But easy to fix, I believe. I will create a PR.
> > >>>>
> > >>>> best,
> > >>>> Colin
> > >>>>
> > >>>>>>
> > >>>>>> Hey Kirk and Chris,
> > >>>>>>
> > >>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
> > >>> due
> > >>>> to
> > >>>>>> improper closing. And the PR description implies this has been
> > >>> present
> > >>>>>> since 3.5. While annoying, I don't see a strong reason for this to
> > >>> block
> > >>>>>> the release.
> > >>>>>>
> > >>>>>> Hey Jakub,
> > >>>>>>
> > >>>>>> Nice catch! It does seem like we should have gated this behind the
> > >>>> metadata
> > >>>>>> version as KIP-858 implies. Is the cluster configured with
> multiple
> > >>> log
> > >>>>>> dirs? What is the impact of the error messages?
> > >>>>>>
> > >>>>>> Tagging Igor (the author of the KIP) to weigh in.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Stanislav
> > >>>>>>
> > >>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> I was trying the RC2 and run into the following issue ... when I
> > >>> run
> > >>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> > >>> metadata
> > >>>>>>> version, I seem to be getting repeated errors like this in the
> > >>>> controller
> > >>>>>>> logs:
> > >>>>>>>
> > >>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > >>>>>> assignReplicasToDirs:
> > >>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
> > >>>>>>> (org.apache.kafka.controller.QuorumController)
> > >>>>>>> [quorum-controller-0-event-handler]
> > >>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
> Unexpected
> > >>>> error
> > >>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
> --
> > >>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > >>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > >>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > >>>>>>> partitions=[PartitionData(partitionIndex=2),
> > >>>>>>> PartitionData(partitionIndex=1)]),
> > >>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > >>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
> > >>>>>>>
> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> > >>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > >>>> clientAddress=/
> > >>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > >>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
> securityProtocol=SSL,
> > >>>>>>>
> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > >>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > >>>>>>> ])
> > >>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> > >>>>>>> java.util.concurrent.CompletionException:
> > >>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >>> Directory
> > >>>>>>> assignment is not supported yet.
> > >>>>>>>
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > >>>>>>>   at java.base/java.lang.Thread.run(Thread.java:840)
> > >>>>>>>
> > >>>>>>> Caused by:
> > >>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >>>>>>> Directory assignment is not supported yet.
> > >>>>>>>
> > >>>>>>> Is that expected? I guess with the metadata version set to
> > >>> 3.6-IV2, it
> > >>>>>>> makes sense that the request is not supported. But shouldn't then
> > >>> the
> > >>>>>>> request not be sent at all by the brokers? (I did not opened a
> JIRA
> > >>>> for
> > >>>>>> it,
> > >>>>>>> but I can open one if you agree this is not expected)
> > >>>>>>>
> > >>>>>>> Thanks & Regards
> > >>>>>>> Jakub
> > >>>>>>>
> > >>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> > >>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Stanislav,
> > >>>>>>>>
> > >>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
> maybe
> > >>>> you
> > >>>>>>>> missed it.
> > >>>>>>>> cross-posting here:
> > >>>>>>>>
> > >>>>>>>> There is a bug KAFKA-16101
> > >>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> > >>> that
> > >>>>>>> "Kafka
> > >>>>>>>> cluster will be unavailable during KRaft migration rollback".
> > >>>>>>>> The impact for this issue is that if brokers try to rollback to
> > >>> ZK
> > >>>> mode
> > >>>>>>>> during KRaft migration process, there will be a period of time
> > >>> the
> > >>>>>>> cluster
> > >>>>>>>> is unavailable.
> > >>>>>>>> Since ZK migrating to KRaft feature is a production ready
> > >>> feature, I
> > >>>>>>> think
> > >>>>>>>> this should be addressed soon.
> > >>>>>>>> Do you think this is a blocker for v3.7.0?
> > >>>>>>>>
> > >>>>>>>> Thanks.
> > >>>>>>>> Luke
> > >>>>>>>>
> > >>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > >>>> fearthecellos@gmail.com
> > >>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks, Kirk!
> > >>>>>>>>>
> > >>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> > >>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Chris/Stanislav,
> > >>>>>>>>>>
> > >>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
> > >>>> problem
> > >>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> > >>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
> > >>>>>>> quickstart
> > >>>>>>>>>> approach as a second means to reproduce/verify while I wait
> > >>> for
> > >>>> the
> > >>>>>>>> PR's
> > >>>>>>>>>> Jenkins job to finish.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> Kirk
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > >>>>>>>>>>> Hi Stanislav,
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for running this release!
> > >>>>>>>>>>>
> > >>>>>>>>>>> To verify, I:
> > >>>>>>>>>>> - Built from source using Java 11 with both:
> > >>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> > >>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> > >>>>>>>>>>>
> > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>>>>>>> - Checked signatures and checksums
> > >>>>>>>>>>> - Ran the quickstart using both:
> > >>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> > >>>>>>>>>>>
> > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>> with
> > >>>>>>>> Java
> > >>>>>>>>>> 11
> > >>>>>>>>>>> and Scala 13 in KRaft mode
> > >>>>>>>>>>> - - Our shiny new broker Docker image,
> > >>> apache/kafka:3.7.0-rc2
> > >>>>>>>>>>> - Ran all unit tests
> > >>>>>>>>>>> - Ran all integration tests for Connect and MM2
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> I found two minor areas for concern:
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. (Possibly a blocker)
> > >>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
> > >>>>>> message
> > >>>>>>>>> being
> > >>>>>>>>>>> emitted frequently (not not every time) when I killed my
> > >>>> console
> > >>>>>>>>> consumer
> > >>>>>>>>>>> via ctrl-C:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> > >>>>>>>> clientId=console-consumer,
> > >>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> > >>>>>> FetchSessionHandler
> > >>>>>>>> for
> > >>>>>>>>>> node
> > >>>>>>>>>>> 1. Ignoring fetch response
> > >>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >>>>>>>>>>>
> > >>>>>>>>>>> I see that this error message is already reported in
> > >>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> > >>> think we
> > >>>>>>> should
> > >>>>>>>>>>> prioritize fixing it for this release. I know it's probably
> > >>>>>> benign
> > >>>>>>>> but
> > >>>>>>>>>> it's
> > >>>>>>>>>>> really not a good look for us when basic operations log
> > >>> error
> > >>>>>>>> messages,
> > >>>>>>>>>> and
> > >>>>>>>>>>> it may give new users some headaches.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. (Probably not a blocker)
> > >>>>>>>>>>> The following unit tests failed the first time around, and
> > >>>> all of
> > >>>>>>>> them
> > >>>>>>>>>>> passed the second time I ran them:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - (clients)
> > >>>>>>>>>>
> > >>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > >>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
> > >>>>>>>>>>> - (clients)
> > >>> Tls13SelectorTest.testConnectionsByClientMetric()
> > >>>>>>>>>>> - (connect)
> > >>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > >>>>>>>>> (I
> > >>>>>>>>>>> thought I fixed this one! 🤬🤬)
> > >>>>>>>>>>> - (core)
> > >>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks again for your work on this release, and
> > >>>> congratulations
> > >>>>>> to
> > >>>>>>>>> Kafka
> > >>>>>>>>>>> Streams for having zero flaky unit tests during my
> > >>>>>>>> highly-experimental
> > >>>>>>>>>>> single laptop run!
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Cheers,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Chris
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This is the first candidate for release of Apache Kafka
> > >>>> 3.7.0.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
> > >>> that
> > >>>> I
> > >>>>>> had
> > >>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> > >>> announcing
> > >>>>>> due
> > >>>>>>> to
> > >>>>>>>>> new
> > >>>>>>>>>>>> blockers arriving before I could even announce them.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Further - I haven't yet been able to set up the system
> > >>> tests
> > >>>>>>>>>> successfully.
> > >>>>>>>>>>>> And the integration/unit tests do have a few failures
> > >>> that I
> > >>>>>> have
> > >>>>>>>> to
> > >>>>>>>>>> spend
> > >>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
> > >>>>>> notices
> > >>>>>>>> any
> > >>>>>>>>>> tests
> > >>>>>>>>>>>> failing that they're subject matters experts in. Expect
> > >>> me
> > >>>> to
> > >>>>>>>> follow
> > >>>>>>>>>> up in
> > >>>>>>>>>>>> a day or two with more detailed analysis.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Major changes include:
> > >>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
> > >>>> consumer
> > >>>>>>>>>> rebalance
> > >>>>>>>>>>>> protocol
> > >>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> > >>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> > >>>> standardized
> > >>>>>>>>>> interface
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Check more information in the WIP blog post:
> > >>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Release notes for the 3.7.0 release:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> *** Please download, test and vote by Thursday, January
> > >>> 18,
> > >>>> 9am
> > >>>>>>> PT
> > >>>>>>>>> ***
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
> > >>> this
> > >>>>>>> being
> > >>>>>>>>> the
> > >>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
> > >>> it a
> > >>>> bit
> > >>>>>>>> more
> > >>>>>>>>>> time.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
> > >>>>>> release:
> > >>>>>>>>>>>> https://kafka.apache.org/KEYS
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
> > >>>>>>>>>>>>
> > >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Docker release artifact to be voted upon:
> > >>>>>>>>>>>> apache/kafka:3.7.0-rc2
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Maven artifacts to be voted upon:
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Javadoc:
> > >>>>>>>>>>>>
> > >>>>>>>>
> > >>>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > >>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Documentation:
> > >>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Protocol:
> > >>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> > >>>>>>>>>>>> Unit/integration tests:
> > >>>>>>>>>>>>
> > >>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > >>>>>>>>>>>> There are failing tests here. I have to follow up with
> > >>>> triaging
> > >>>>>>>> some
> > >>>>>>>>> of
> > >>>>>>>>>>>> the failures and figuring out if they're actual problems
> > >>> or
> > >>>>>>> simply
> > >>>>>>>>>> flakes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> System tests:
> > >>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> No successful system test runs yet. I am working on
> > >>> getting
> > >>>> the
> > >>>>>>> job
> > >>>>>>>>> to
> > >>>>>>>>>> run.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
> > >>>>>> branch:
> > >>>>>>>>>>>> Attached are the scan_report and report_jvm output files
> > >>>> from
> > >>>>>> the
> > >>>>>>>>>> Docker
> > >>>>>>>>>>>> Build run:
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>
> > https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> And the final docker image build job - Docker Build Test
> > >>>>>>> Pipeline:
> > >>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> /**************************************
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Stanislav Kozlovski
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best,
> > >>>>>> Stanislav
> > >>>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>


-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
Hey all, I figured I'd give an update about what known blockers we have
right now:

- KAFKA-16101: KRaft migration rollback documentation is incorrect -
https://github.com/apache/kafka/pull/15193; This need not block RC
creation, but we need the docs updated so that people can test properly
- KAFKA-14616: Topic recreation with offline broker causes permanent URPs -
https://github.com/apache/kafka/pull/15230 ; I am of the understanding that
this is blocking JBOD for 3.7
- KAFKA-16162: New created topics are unavailable after upgrading to 3.7 -
a strict blocker with an open PR https://github.com/apache/kafka/pull/15232
- although I understand Proveen is out of office
- KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am
hearing mixed opinions on whether this is a blocker (
https://github.com/apache/kafka/pull/15136)

Given that there are 3 JBOD blocker bugs, and I am not confident they will
all be merged this week - I am on the edge of voting to revert JBOD from
this release, or mark it early access.

By all accounts, it seems that if we keep with JBOD the release will have
to spill into February, which is a month extra from the time-based release
plan we had of start of January.

Can I ask others for an opinion?

Best,
Stan

On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <sh...@gmail.com> wrote:

> Hi all,
>
> I think I've found another blocker issue: KAFKA-16162
> <https://issues.apache.org/jira/browse/KAFKA-16162> .
> The impact is after upgrading to 3.7.0, any new created topics/partitions
> will be unavailable.
> I've put my findings in the JIRA.
>
> Thanks.
> Luke
>
> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:
>
> > Stan, thanks for driving this all forward! Excellent job.
> >
> > About
> >
> > > StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
> > > StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
> >
> > For `StreamsUpgradeTest` it was a test setup issue and should be fixed
> > now in trunk and 3.7 (and actually also in 3.6...)
> >
> > For `StreamsStandbyTask` the failing test exposes a regression bug, so
> > it's a blocker. I updated the ticket accordingly. We already have an
> > open PR that reverts the code introducing the regression.
> >
> >
> > -Matthias
> >
> > On 1/17/24 9:44 AM, Proven Provenzano wrote:
> > > We have another blocking issue for the RC :
> > > https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
> > to
> > > https://issues.apache.org/jira/browse/KAFKA-14616. The new issue
> however
> > > can lead to the new topic having partitions that a producer cannot
> write
> > to.
> > >
> > > --Proven
> > >
> > > On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> > pprovenzano@confluent.io>
> > > wrote:
> > >
> > >>
> > >> I have a PR https://github.com/apache/kafka/pull/15197 for
> > >> https://issues.apache.org/jira/browse/KAFKA-16131 that is building
> now.
> > >> --Proven
> > >>
> > >> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
> > >>
> > >>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
> > >>> blocker bug because it *
> > >>> *> will generate huge amount of logspam. I guess we didn't find it in
> > >>> junit
> > >>> tests *
> > >>> *> since logspam doesn't fail the automated tests. But certainly it's
> > not
> > >>> suitable *
> > >>> *> for production. Did you file a JIRA yet?*
> > >>>
> > >>> Hi Colin,
> > >>>
> > >>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> > >>>
> > >>> Thanks & Regards
> > >>> Jakub
> > >>>
> > >>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > >>>
> > >>>> Hi Stanislav,
> > >>>>
> > >>>> Thanks for making the first RC. The fact that it's titled RC2 is
> > messing
> > >>>> with my mind a bit. I hope this doesn't make people think that we're
> > >>>> farther along than we are, heh.
> > >>>>
> > >>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > >>>>> *> Nice catch! It does seem like we should have gated this behind
> the
> > >>>>> metadata> version as KIP-858 implies. Is the cluster configured
> with
> > >>>>> multiple log> dirs? What is the impact of the error messages?*
> > >>>>>
> > >>>>> I did not observe any obvious impact. I was able to send and
> receive
> > >>>>> messages as normally. But to be honest, I have no idea what else
> > >>>>> this might impact, so I did not try anything special.
> > >>>>>
> > >>>>> I think everyone upgrading an existing KRaft cluster will go
> through
> > >>> this
> > >>>>> stage (running Kafka 3.7 with an older metadata version for at
> least
> > a
> > >>>>> while). So even if it is just a logged exception without any other
> > >>>> impact I
> > >>>>> wonder if it might scare users from upgrading. But I leave it to
> > >>> others
> > >>>> to
> > >>>>> decide if this is a blocker or not.
> > >>>>>
> > >>>>
> > >>>> Hi Jakub,
> > >>>>
> > >>>> Thanks for trying the RC. I think what you found is a blocker bug
> > >>> because
> > >>>> it will generate huge amount of logspam. I guess we didn't find it
> in
> > >>> junit
> > >>>> tests since logspam doesn't fail the automated tests. But certainly
> > it's
> > >>>> not suitable for production. Did you file a JIRA yet?
> > >>>>
> > >>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > >>>>> <st...@confluent.io.invalid> wrote:
> > >>>>>
> > >>>>>> Hey Luke,
> > >>>>>>
> > >>>>>> This is an interesting problem. Given the fact that the KIP for
> > >>> having a
> > >>>>>> 3.8 release passed, I think it weights the scale towards not
> calling
> > >>>> this a
> > >>>>>> blocker and expecting it to be solved in 3.7.1.
> > >>>>>>
> > >>>>>> It is unfortunate that it would not seem safe to migrate to KRaft
> in
> > >>>> 3.7.0
> > >>>>>> (given the inability to rollback safely), but if that's true - the
> > >>> same
> > >>>>>> case would apply for 3.6.0. So in any case users w\ould be
> expected
> > >>> to
> > >>>> use a
> > >>>>>> patch release for this.
> > >>>>
> > >>>> Hi Luke,
> > >>>>
> > >>>> Thanks for testing rollback. I think this is a case where the
> > >>>> documentation is wrong. The intention was to for the steps to
> > basically
> > >>> be:
> > >>>>
> > >>>> 1. roll all the brokers into zk mode, but with migration enabled
> > >>>> 2. take down the kraft quorum
> > >>>> 3. rmr /controller, allowing a hybrid broker to take over.
> > >>>> 4. roll all the brokers into zk mode without migration enabled (if
> > >>> desired)
> > >>>>
> > >>>> With these steps, there isn't really unavailability since a ZK
> > >>> controller
> > >>>> can be elected quickly after the kraft quorum is gone.
> > >>>>
> > >>>>>> Further, since we will have a 3.8 release - it is
> > >>>>>> likely we will ultimately recommend users upgrade from that
> version
> > >>>> given
> > >>>>>> its aim is to have strategic KRaft feature parity with ZK.
> > >>>>>> That being said, I am not 100% on this. Let me know whether you
> > think
> > >>>> this
> > >>>>>> should block the release, Luke. I am also tagging Colin and David
> to
> > >>>> weigh
> > >>>>>> in with their opinions, as they worked on the migration logic.
> > >>>>
> > >>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
> > >>> clear
> > >>>> blocker, I think. But easy to fix, I believe. I will create a PR.
> > >>>>
> > >>>> best,
> > >>>> Colin
> > >>>>
> > >>>>>>
> > >>>>>> Hey Kirk and Chris,
> > >>>>>>
> > >>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
> > >>> due
> > >>>> to
> > >>>>>> improper closing. And the PR description implies this has been
> > >>> present
> > >>>>>> since 3.5. While annoying, I don't see a strong reason for this to
> > >>> block
> > >>>>>> the release.
> > >>>>>>
> > >>>>>> Hey Jakub,
> > >>>>>>
> > >>>>>> Nice catch! It does seem like we should have gated this behind the
> > >>>> metadata
> > >>>>>> version as KIP-858 implies. Is the cluster configured with
> multiple
> > >>> log
> > >>>>>> dirs? What is the impact of the error messages?
> > >>>>>>
> > >>>>>> Tagging Igor (the author of the KIP) to weigh in.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Stanislav
> > >>>>>>
> > >>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> I was trying the RC2 and run into the following issue ... when I
> > >>> run
> > >>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> > >>> metadata
> > >>>>>>> version, I seem to be getting repeated errors like this in the
> > >>>> controller
> > >>>>>>> logs:
> > >>>>>>>
> > >>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > >>>>>> assignReplicasToDirs:
> > >>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
> > >>>>>>> (org.apache.kafka.controller.QuorumController)
> > >>>>>>> [quorum-controller-0-event-handler]
> > >>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0]
> Unexpected
> > >>>> error
> > >>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2)
> --
> > >>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > >>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > >>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > >>>>>>> partitions=[PartitionData(partitionIndex=2),
> > >>>>>>> PartitionData(partitionIndex=1)]),
> > >>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > >>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
> > >>>>>>>
> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> > >>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > >>>> clientAddress=/
> > >>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > >>>>>>> listenerName=ListenerName(CONTROLPLANE-9090),
> securityProtocol=SSL,
> > >>>>>>>
> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > >>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > >>>>>>> ])
> > >>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> > >>>>>>> java.util.concurrent.CompletionException:
> > >>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >>> Directory
> > >>>>>>> assignment is not supported yet.
> > >>>>>>>
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > >>>>>>>   at
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > >>>>>>>   at java.base/java.lang.Thread.run(Thread.java:840)
> > >>>>>>>
> > >>>>>>> Caused by:
> > >>> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >>>>>>> Directory assignment is not supported yet.
> > >>>>>>>
> > >>>>>>> Is that expected? I guess with the metadata version set to
> > >>> 3.6-IV2, it
> > >>>>>>> makes sense that the request is not supported. But shouldn't then
> > >>> the
> > >>>>>>> request not be sent at all by the brokers? (I did not opened a
> JIRA
> > >>>> for
> > >>>>>> it,
> > >>>>>>> but I can open one if you agree this is not expected)
> > >>>>>>>
> > >>>>>>> Thanks & Regards
> > >>>>>>> Jakub
> > >>>>>>>
> > >>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> > >>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Stanislav,
> > >>>>>>>>
> > >>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but
> maybe
> > >>>> you
> > >>>>>>>> missed it.
> > >>>>>>>> cross-posting here:
> > >>>>>>>>
> > >>>>>>>> There is a bug KAFKA-16101
> > >>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> > >>> that
> > >>>>>>> "Kafka
> > >>>>>>>> cluster will be unavailable during KRaft migration rollback".
> > >>>>>>>> The impact for this issue is that if brokers try to rollback to
> > >>> ZK
> > >>>> mode
> > >>>>>>>> during KRaft migration process, there will be a period of time
> > >>> the
> > >>>>>>> cluster
> > >>>>>>>> is unavailable.
> > >>>>>>>> Since ZK migrating to KRaft feature is a production ready
> > >>> feature, I
> > >>>>>>> think
> > >>>>>>>> this should be addressed soon.
> > >>>>>>>> Do you think this is a blocker for v3.7.0?
> > >>>>>>>>
> > >>>>>>>> Thanks.
> > >>>>>>>> Luke
> > >>>>>>>>
> > >>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > >>>> fearthecellos@gmail.com
> > >>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks, Kirk!
> > >>>>>>>>>
> > >>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> > >>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Chris/Stanislav,
> > >>>>>>>>>>
> > >>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
> > >>>> problem
> > >>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> > >>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
> > >>>>>>> quickstart
> > >>>>>>>>>> approach as a second means to reproduce/verify while I wait
> > >>> for
> > >>>> the
> > >>>>>>>> PR's
> > >>>>>>>>>> Jenkins job to finish.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> Kirk
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > >>>>>>>>>>> Hi Stanislav,
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for running this release!
> > >>>>>>>>>>>
> > >>>>>>>>>>> To verify, I:
> > >>>>>>>>>>> - Built from source using Java 11 with both:
> > >>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> > >>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> > >>>>>>>>>>>
> > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>>>>>>> - Checked signatures and checksums
> > >>>>>>>>>>> - Ran the quickstart using both:
> > >>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> > >>>>>>>>>>>
> > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>> with
> > >>>>>>>> Java
> > >>>>>>>>>> 11
> > >>>>>>>>>>> and Scala 13 in KRaft mode
> > >>>>>>>>>>> - - Our shiny new broker Docker image,
> > >>> apache/kafka:3.7.0-rc2
> > >>>>>>>>>>> - Ran all unit tests
> > >>>>>>>>>>> - Ran all integration tests for Connect and MM2
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> I found two minor areas for concern:
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. (Possibly a blocker)
> > >>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
> > >>>>>> message
> > >>>>>>>>> being
> > >>>>>>>>>>> emitted frequently (not not every time) when I killed my
> > >>>> console
> > >>>>>>>>> consumer
> > >>>>>>>>>>> via ctrl-C:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> > >>>>>>>> clientId=console-consumer,
> > >>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> > >>>>>> FetchSessionHandler
> > >>>>>>>> for
> > >>>>>>>>>> node
> > >>>>>>>>>>> 1. Ignoring fetch response
> > >>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >>>>>>>>>>>
> > >>>>>>>>>>> I see that this error message is already reported in
> > >>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> > >>> think we
> > >>>>>>> should
> > >>>>>>>>>>> prioritize fixing it for this release. I know it's probably
> > >>>>>> benign
> > >>>>>>>> but
> > >>>>>>>>>> it's
> > >>>>>>>>>>> really not a good look for us when basic operations log
> > >>> error
> > >>>>>>>> messages,
> > >>>>>>>>>> and
> > >>>>>>>>>>> it may give new users some headaches.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. (Probably not a blocker)
> > >>>>>>>>>>> The following unit tests failed the first time around, and
> > >>>> all of
> > >>>>>>>> them
> > >>>>>>>>>>> passed the second time I ran them:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - (clients)
> > >>>>>>>>>>
> > >>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > >>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
> > >>>>>>>>>>> - (clients)
> > >>> Tls13SelectorTest.testConnectionsByClientMetric()
> > >>>>>>>>>>> - (connect)
> > >>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > >>>>>>>>> (I
> > >>>>>>>>>>> thought I fixed this one! 🤬🤬)
> > >>>>>>>>>>> - (core)
> > >>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks again for your work on this release, and
> > >>>> congratulations
> > >>>>>> to
> > >>>>>>>>> Kafka
> > >>>>>>>>>>> Streams for having zero flaky unit tests during my
> > >>>>>>>> highly-experimental
> > >>>>>>>>>>> single laptop run!
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Cheers,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Chris
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This is the first candidate for release of Apache Kafka
> > >>>> 3.7.0.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
> > >>> that
> > >>>> I
> > >>>>>> had
> > >>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> > >>> announcing
> > >>>>>> due
> > >>>>>>> to
> > >>>>>>>>> new
> > >>>>>>>>>>>> blockers arriving before I could even announce them.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Further - I haven't yet been able to set up the system
> > >>> tests
> > >>>>>>>>>> successfully.
> > >>>>>>>>>>>> And the integration/unit tests do have a few failures
> > >>> that I
> > >>>>>> have
> > >>>>>>>> to
> > >>>>>>>>>> spend
> > >>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
> > >>>>>> notices
> > >>>>>>>> any
> > >>>>>>>>>> tests
> > >>>>>>>>>>>> failing that they're subject matters experts in. Expect
> > >>> me
> > >>>> to
> > >>>>>>>> follow
> > >>>>>>>>>> up in
> > >>>>>>>>>>>> a day or two with more detailed analysis.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Major changes include:
> > >>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
> > >>>> consumer
> > >>>>>>>>>> rebalance
> > >>>>>>>>>>>> protocol
> > >>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> > >>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> > >>>> standardized
> > >>>>>>>>>> interface
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Check more information in the WIP blog post:
> > >>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Release notes for the 3.7.0 release:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> *** Please download, test and vote by Thursday, January
> > >>> 18,
> > >>>> 9am
> > >>>>>>> PT
> > >>>>>>>>> ***
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
> > >>> this
> > >>>>>>> being
> > >>>>>>>>> the
> > >>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
> > >>> it a
> > >>>> bit
> > >>>>>>>> more
> > >>>>>>>>>> time.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
> > >>>>>> release:
> > >>>>>>>>>>>> https://kafka.apache.org/KEYS
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
> > >>>>>>>>>>>>
> > >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Docker release artifact to be voted upon:
> > >>>>>>>>>>>> apache/kafka:3.7.0-rc2
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Maven artifacts to be voted upon:
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Javadoc:
> > >>>>>>>>>>>>
> > >>>>>>>>
> > >>>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > >>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Documentation:
> > >>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Protocol:
> > >>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> > >>>>>>>>>>>> Unit/integration tests:
> > >>>>>>>>>>>>
> > >>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > >>>>>>>>>>>> There are failing tests here. I have to follow up with
> > >>>> triaging
> > >>>>>>>> some
> > >>>>>>>>> of
> > >>>>>>>>>>>> the failures and figuring out if they're actual problems
> > >>> or
> > >>>>>>> simply
> > >>>>>>>>>> flakes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> System tests:
> > >>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> No successful system test runs yet. I am working on
> > >>> getting
> > >>>> the
> > >>>>>>> job
> > >>>>>>>>> to
> > >>>>>>>>>> run.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
> > >>>>>> branch:
> > >>>>>>>>>>>> Attached are the scan_report and report_jvm output files
> > >>>> from
> > >>>>>> the
> > >>>>>>>>>> Docker
> > >>>>>>>>>>>> Build run:
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>
> > https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> And the final docker image build job - Docker Build Test
> > >>>>>>> Pipeline:
> > >>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> /**************************************
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Stanislav Kozlovski
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best,
> > >>>>>> Stanislav
> > >>>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>


-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Luke Chen <sh...@gmail.com>.
Hi all,

I think I've found another blocker issue: KAFKA-16162
<https://issues.apache.org/jira/browse/KAFKA-16162> .
The impact is after upgrading to 3.7.0, any new created topics/partitions
will be unavailable.
I've put my findings in the JIRA.

Thanks.
Luke

On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote:

> Stan, thanks for driving this all forward! Excellent job.
>
> About
>
> > StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
> > StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139
>
> For `StreamsUpgradeTest` it was a test setup issue and should be fixed
> now in trunk and 3.7 (and actually also in 3.6...)
>
> For `StreamsStandbyTask` the failing test exposes a regression bug, so
> it's a blocker. I updated the ticket accordingly. We already have an
> open PR that reverts the code introducing the regression.
>
>
> -Matthias
>
> On 1/17/24 9:44 AM, Proven Provenzano wrote:
> > We have another blocking issue for the RC :
> > https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar
> to
> > https://issues.apache.org/jira/browse/KAFKA-14616. The new issue however
> > can lead to the new topic having partitions that a producer cannot write
> to.
> >
> > --Proven
> >
> > On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <
> pprovenzano@confluent.io>
> > wrote:
> >
> >>
> >> I have a PR https://github.com/apache/kafka/pull/15197 for
> >> https://issues.apache.org/jira/browse/KAFKA-16131 that is building now.
> >> --Proven
> >>
> >> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
> >>
> >>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
> >>> blocker bug because it *
> >>> *> will generate huge amount of logspam. I guess we didn't find it in
> >>> junit
> >>> tests *
> >>> *> since logspam doesn't fail the automated tests. But certainly it's
> not
> >>> suitable *
> >>> *> for production. Did you file a JIRA yet?*
> >>>
> >>> Hi Colin,
> >>>
> >>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
> >>>
> >>> Thanks & Regards
> >>> Jakub
> >>>
> >>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org>
> wrote:
> >>>
> >>>> Hi Stanislav,
> >>>>
> >>>> Thanks for making the first RC. The fact that it's titled RC2 is
> messing
> >>>> with my mind a bit. I hope this doesn't make people think that we're
> >>>> farther along than we are, heh.
> >>>>
> >>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> >>>>> *> Nice catch! It does seem like we should have gated this behind the
> >>>>> metadata> version as KIP-858 implies. Is the cluster configured with
> >>>>> multiple log> dirs? What is the impact of the error messages?*
> >>>>>
> >>>>> I did not observe any obvious impact. I was able to send and receive
> >>>>> messages as normally. But to be honest, I have no idea what else
> >>>>> this might impact, so I did not try anything special.
> >>>>>
> >>>>> I think everyone upgrading an existing KRaft cluster will go through
> >>> this
> >>>>> stage (running Kafka 3.7 with an older metadata version for at least
> a
> >>>>> while). So even if it is just a logged exception without any other
> >>>> impact I
> >>>>> wonder if it might scare users from upgrading. But I leave it to
> >>> others
> >>>> to
> >>>>> decide if this is a blocker or not.
> >>>>>
> >>>>
> >>>> Hi Jakub,
> >>>>
> >>>> Thanks for trying the RC. I think what you found is a blocker bug
> >>> because
> >>>> it will generate huge amount of logspam. I guess we didn't find it in
> >>> junit
> >>>> tests since logspam doesn't fail the automated tests. But certainly
> it's
> >>>> not suitable for production. Did you file a JIRA yet?
> >>>>
> >>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> >>>>> <st...@confluent.io.invalid> wrote:
> >>>>>
> >>>>>> Hey Luke,
> >>>>>>
> >>>>>> This is an interesting problem. Given the fact that the KIP for
> >>> having a
> >>>>>> 3.8 release passed, I think it weights the scale towards not calling
> >>>> this a
> >>>>>> blocker and expecting it to be solved in 3.7.1.
> >>>>>>
> >>>>>> It is unfortunate that it would not seem safe to migrate to KRaft in
> >>>> 3.7.0
> >>>>>> (given the inability to rollback safely), but if that's true - the
> >>> same
> >>>>>> case would apply for 3.6.0. So in any case users w\ould be expected
> >>> to
> >>>> use a
> >>>>>> patch release for this.
> >>>>
> >>>> Hi Luke,
> >>>>
> >>>> Thanks for testing rollback. I think this is a case where the
> >>>> documentation is wrong. The intention was to for the steps to
> basically
> >>> be:
> >>>>
> >>>> 1. roll all the brokers into zk mode, but with migration enabled
> >>>> 2. take down the kraft quorum
> >>>> 3. rmr /controller, allowing a hybrid broker to take over.
> >>>> 4. roll all the brokers into zk mode without migration enabled (if
> >>> desired)
> >>>>
> >>>> With these steps, there isn't really unavailability since a ZK
> >>> controller
> >>>> can be elected quickly after the kraft quorum is gone.
> >>>>
> >>>>>> Further, since we will have a 3.8 release - it is
> >>>>>> likely we will ultimately recommend users upgrade from that version
> >>>> given
> >>>>>> its aim is to have strategic KRaft feature parity with ZK.
> >>>>>> That being said, I am not 100% on this. Let me know whether you
> think
> >>>> this
> >>>>>> should block the release, Luke. I am also tagging Colin and David to
> >>>> weigh
> >>>>>> in with their opinions, as they worked on the migration logic.
> >>>>
> >>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
> >>> clear
> >>>> blocker, I think. But easy to fix, I believe. I will create a PR.
> >>>>
> >>>> best,
> >>>> Colin
> >>>>
> >>>>>>
> >>>>>> Hey Kirk and Chris,
> >>>>>>
> >>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
> >>> due
> >>>> to
> >>>>>> improper closing. And the PR description implies this has been
> >>> present
> >>>>>> since 3.5. While annoying, I don't see a strong reason for this to
> >>> block
> >>>>>> the release.
> >>>>>>
> >>>>>> Hey Jakub,
> >>>>>>
> >>>>>> Nice catch! It does seem like we should have gated this behind the
> >>>> metadata
> >>>>>> version as KIP-858 implies. Is the cluster configured with multiple
> >>> log
> >>>>>> dirs? What is the impact of the error messages?
> >>>>>>
> >>>>>> Tagging Igor (the author of the KIP) to weigh in.
> >>>>>>
> >>>>>> Best,
> >>>>>> Stanislav
> >>>>>>
> >>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
> >>> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I was trying the RC2 and run into the following issue ... when I
> >>> run
> >>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> >>> metadata
> >>>>>>> version, I seem to be getting repeated errors like this in the
> >>>> controller
> >>>>>>> logs:
> >>>>>>>
> >>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> >>>>>> assignReplicasToDirs:
> >>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
> >>>>>>> (org.apache.kafka.controller.QuorumController)
> >>>>>>> [quorum-controller-0-event-handler]
> >>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
> >>>> error
> >>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> >>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> >>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> >>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> >>>>>>> partitions=[PartitionData(partitionIndex=2),
> >>>>>>> PartitionData(partitionIndex=1)]),
> >>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> >>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
> >>>>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> >>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> >>>> clientAddress=/
> >>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> >>>>>>> listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> >>>>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> >>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> >>>>>>> ])
> >>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> >>>>>>> java.util.concurrent.CompletionException:
> >>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
> >>> Directory
> >>>>>>> assignment is not supported yet.
> >>>>>>>
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> >>>>>>>   at
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> >>>>>>>   at java.base/java.lang.Thread.run(Thread.java:840)
> >>>>>>>
> >>>>>>> Caused by:
> >>> org.apache.kafka.common.errors.UnsupportedVersionException:
> >>>>>>> Directory assignment is not supported yet.
> >>>>>>>
> >>>>>>> Is that expected? I guess with the metadata version set to
> >>> 3.6-IV2, it
> >>>>>>> makes sense that the request is not supported. But shouldn't then
> >>> the
> >>>>>>> request not be sent at all by the brokers? (I did not opened a JIRA
> >>>> for
> >>>>>> it,
> >>>>>>> but I can open one if you agree this is not expected)
> >>>>>>>
> >>>>>>> Thanks & Regards
> >>>>>>> Jakub
> >>>>>>>
> >>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> >>> wrote:
> >>>>>>>
> >>>>>>>> Hi Stanislav,
> >>>>>>>>
> >>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
> >>>> you
> >>>>>>>> missed it.
> >>>>>>>> cross-posting here:
> >>>>>>>>
> >>>>>>>> There is a bug KAFKA-16101
> >>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> >>> that
> >>>>>>> "Kafka
> >>>>>>>> cluster will be unavailable during KRaft migration rollback".
> >>>>>>>> The impact for this issue is that if brokers try to rollback to
> >>> ZK
> >>>> mode
> >>>>>>>> during KRaft migration process, there will be a period of time
> >>> the
> >>>>>>> cluster
> >>>>>>>> is unavailable.
> >>>>>>>> Since ZK migrating to KRaft feature is a production ready
> >>> feature, I
> >>>>>>> think
> >>>>>>>> this should be addressed soon.
> >>>>>>>> Do you think this is a blocker for v3.7.0?
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>> Luke
> >>>>>>>>
> >>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> >>>> fearthecellos@gmail.com
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks, Kirk!
> >>>>>>>>>
> >>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
> >>>>>>>>>
> >>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Chris/Stanislav,
> >>>>>>>>>>
> >>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
> >>>> problem
> >>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
> >>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
> >>>>>>> quickstart
> >>>>>>>>>> approach as a second means to reproduce/verify while I wait
> >>> for
> >>>> the
> >>>>>>>> PR's
> >>>>>>>>>> Jenkins job to finish.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Kirk
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> >>>>>>>>>>> Hi Stanislav,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for running this release!
> >>>>>>>>>>>
> >>>>>>>>>>> To verify, I:
> >>>>>>>>>>> - Built from source using Java 11 with both:
> >>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
> >>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
> >>>>>>>>>>>
> >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>> - Checked signatures and checksums
> >>>>>>>>>>> - Ran the quickstart using both:
> >>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
> >>>>>>>>>>>
> >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>> with
> >>>>>>>> Java
> >>>>>>>>>> 11
> >>>>>>>>>>> and Scala 13 in KRaft mode
> >>>>>>>>>>> - - Our shiny new broker Docker image,
> >>> apache/kafka:3.7.0-rc2
> >>>>>>>>>>> - Ran all unit tests
> >>>>>>>>>>> - Ran all integration tests for Connect and MM2
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I found two minor areas for concern:
> >>>>>>>>>>>
> >>>>>>>>>>> 1. (Possibly a blocker)
> >>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
> >>>>>> message
> >>>>>>>>> being
> >>>>>>>>>>> emitted frequently (not not every time) when I killed my
> >>>> console
> >>>>>>>>> consumer
> >>>>>>>>>>> via ctrl-C:
> >>>>>>>>>>>
> >>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
> >>>>>>>> clientId=console-consumer,
> >>>>>>>>>>> groupId=console-consumer-74388] Unable to find
> >>>>>> FetchSessionHandler
> >>>>>>>> for
> >>>>>>>>>> node
> >>>>>>>>>>> 1. Ignoring fetch response
> >>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> >>>>>>>>>>>
> >>>>>>>>>>> I see that this error message is already reported in
> >>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
> >>> think we
> >>>>>>> should
> >>>>>>>>>>> prioritize fixing it for this release. I know it's probably
> >>>>>> benign
> >>>>>>>> but
> >>>>>>>>>> it's
> >>>>>>>>>>> really not a good look for us when basic operations log
> >>> error
> >>>>>>>> messages,
> >>>>>>>>>> and
> >>>>>>>>>>> it may give new users some headaches.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 2. (Probably not a blocker)
> >>>>>>>>>>> The following unit tests failed the first time around, and
> >>>> all of
> >>>>>>>> them
> >>>>>>>>>>> passed the second time I ran them:
> >>>>>>>>>>>
> >>>>>>>>>>> - (clients)
> >>>>>>>>>>
> >>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> >>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
> >>>>>>>>>>> - (clients)
> >>> Tls13SelectorTest.testConnectionsByClientMetric()
> >>>>>>>>>>> - (connect)
> >>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> >>>>>>>>> (I
> >>>>>>>>>>> thought I fixed this one! 🤬🤬)
> >>>>>>>>>>> - (core)
> >>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks again for your work on this release, and
> >>>> congratulations
> >>>>>> to
> >>>>>>>>> Kafka
> >>>>>>>>>>> Streams for having zero flaky unit tests during my
> >>>>>>>> highly-experimental
> >>>>>>>>>>> single laptop run!
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>>
> >>>>>>>>>>> Chris
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> >>>>>>>>>>> <st...@confluent.io.invalid> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is the first candidate for release of Apache Kafka
> >>>> 3.7.0.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
> >>> that
> >>>> I
> >>>>>> had
> >>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
> >>> announcing
> >>>>>> due
> >>>>>>> to
> >>>>>>>>> new
> >>>>>>>>>>>> blockers arriving before I could even announce them.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Further - I haven't yet been able to set up the system
> >>> tests
> >>>>>>>>>> successfully.
> >>>>>>>>>>>> And the integration/unit tests do have a few failures
> >>> that I
> >>>>>> have
> >>>>>>>> to
> >>>>>>>>>> spend
> >>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
> >>>>>> notices
> >>>>>>>> any
> >>>>>>>>>> tests
> >>>>>>>>>>>> failing that they're subject matters experts in. Expect
> >>> me
> >>>> to
> >>>>>>>> follow
> >>>>>>>>>> up in
> >>>>>>>>>>>> a day or two with more detailed analysis.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Major changes include:
> >>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
> >>>> consumer
> >>>>>>>>>> rebalance
> >>>>>>>>>>>> protocol
> >>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
> >>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
> >>>> standardized
> >>>>>>>>>> interface
> >>>>>>>>>>>>
> >>>>>>>>>>>> Check more information in the WIP blog post:
> >>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
> >>>>>>>>>>>>
> >>>>>>>>>>>> Release notes for the 3.7.0 release:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> >>>>>>>>>>>>
> >>>>>>>>>>>> *** Please download, test and vote by Thursday, January
> >>> 18,
> >>>> 9am
> >>>>>>> PT
> >>>>>>>>> ***
> >>>>>>>>>>>>
> >>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
> >>> this
> >>>>>>> being
> >>>>>>>>> the
> >>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
> >>> it a
> >>>> bit
> >>>>>>>> more
> >>>>>>>>>> time.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
> >>>>>> release:
> >>>>>>>>>>>> https://kafka.apache.org/KEYS
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
> >>>>>>>>>>>>
> >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Docker release artifact to be voted upon:
> >>>>>>>>>>>> apache/kafka:3.7.0-rc2
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Maven artifacts to be voted upon:
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Javadoc:
> >>>>>>>>>>>>
> >>>>>>>>
> >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> >>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Documentation:
> >>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Protocol:
> >>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
> >>>>>>>>>>>> Unit/integration tests:
> >>>>>>>>>>>>
> >>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> >>>>>>>>>>>> There are failing tests here. I have to follow up with
> >>>> triaging
> >>>>>>>> some
> >>>>>>>>> of
> >>>>>>>>>>>> the failures and figuring out if they're actual problems
> >>> or
> >>>>>>> simply
> >>>>>>>>>> flakes.
> >>>>>>>>>>>>
> >>>>>>>>>>>> System tests:
> >>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> >>>>>>>>>>>>
> >>>>>>>>>>>> No successful system test runs yet. I am working on
> >>> getting
> >>>> the
> >>>>>>> job
> >>>>>>>>> to
> >>>>>>>>>> run.
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
> >>>>>> branch:
> >>>>>>>>>>>> Attached are the scan_report and report_jvm output files
> >>>> from
> >>>>>> the
> >>>>>>>>>> Docker
> >>>>>>>>>>>> Build run:
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> >>>>>>>>>>>>
> >>>>>>>>>>>> And the final docker image build job - Docker Build Test
> >>>>>>> Pipeline:
> >>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
> >>>>>>>>>>>>
> >>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> >>>>>>>>>>>>
> >>>>>>>>>>>> /**************************************
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Stanislav Kozlovski
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best,
> >>>>>> Stanislav
> >>>>>>
> >>>>
> >>>
> >>
> >
>

Re: [VOTE] 3.7.0 RC2

Posted by "Matthias J. Sax" <mj...@apache.org>.
Stan, thanks for driving this all forward! Excellent job.

About

> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139

For `StreamsUpgradeTest` it was a test setup issue and should be fixed 
now in trunk and 3.7 (and actually also in 3.6...)

For `StreamsStandbyTask` the failing test exposes a regression bug, so 
it's a blocker. I updated the ticket accordingly. We already have an 
open PR that reverts the code introducing the regression.


-Matthias

On 1/17/24 9:44 AM, Proven Provenzano wrote:
> We have another blocking issue for the RC :
> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar to
> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue however
> can lead to the new topic having partitions that a producer cannot write to.
> 
> --Proven
> 
> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <pp...@confluent.io>
> wrote:
> 
>>
>> I have a PR https://github.com/apache/kafka/pull/15197 for
>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building now.
>> --Proven
>>
>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>>
>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>>> blocker bug because it *
>>> *> will generate huge amount of logspam. I guess we didn't find it in
>>> junit
>>> tests *
>>> *> since logspam doesn't fail the automated tests. But certainly it's not
>>> suitable *
>>> *> for production. Did you file a JIRA yet?*
>>>
>>> Hi Colin,
>>>
>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>>
>>> Thanks & Regards
>>> Jakub
>>>
>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org> wrote:
>>>
>>>> Hi Stanislav,
>>>>
>>>> Thanks for making the first RC. The fact that it's titled RC2 is messing
>>>> with my mind a bit. I hope this doesn't make people think that we're
>>>> farther along than we are, heh.
>>>>
>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>>>>> *> Nice catch! It does seem like we should have gated this behind the
>>>>> metadata> version as KIP-858 implies. Is the cluster configured with
>>>>> multiple log> dirs? What is the impact of the error messages?*
>>>>>
>>>>> I did not observe any obvious impact. I was able to send and receive
>>>>> messages as normally. But to be honest, I have no idea what else
>>>>> this might impact, so I did not try anything special.
>>>>>
>>>>> I think everyone upgrading an existing KRaft cluster will go through
>>> this
>>>>> stage (running Kafka 3.7 with an older metadata version for at least a
>>>>> while). So even if it is just a logged exception without any other
>>>> impact I
>>>>> wonder if it might scare users from upgrading. But I leave it to
>>> others
>>>> to
>>>>> decide if this is a blocker or not.
>>>>>
>>>>
>>>> Hi Jakub,
>>>>
>>>> Thanks for trying the RC. I think what you found is a blocker bug
>>> because
>>>> it will generate huge amount of logspam. I guess we didn't find it in
>>> junit
>>>> tests since logspam doesn't fail the automated tests. But certainly it's
>>>> not suitable for production. Did you file a JIRA yet?
>>>>
>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>>>>> <st...@confluent.io.invalid> wrote:
>>>>>
>>>>>> Hey Luke,
>>>>>>
>>>>>> This is an interesting problem. Given the fact that the KIP for
>>> having a
>>>>>> 3.8 release passed, I think it weights the scale towards not calling
>>>> this a
>>>>>> blocker and expecting it to be solved in 3.7.1.
>>>>>>
>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft in
>>>> 3.7.0
>>>>>> (given the inability to rollback safely), but if that's true - the
>>> same
>>>>>> case would apply for 3.6.0. So in any case users w\ould be expected
>>> to
>>>> use a
>>>>>> patch release for this.
>>>>
>>>> Hi Luke,
>>>>
>>>> Thanks for testing rollback. I think this is a case where the
>>>> documentation is wrong. The intention was to for the steps to basically
>>> be:
>>>>
>>>> 1. roll all the brokers into zk mode, but with migration enabled
>>>> 2. take down the kraft quorum
>>>> 3. rmr /controller, allowing a hybrid broker to take over.
>>>> 4. roll all the brokers into zk mode without migration enabled (if
>>> desired)
>>>>
>>>> With these steps, there isn't really unavailability since a ZK
>>> controller
>>>> can be elected quickly after the kraft quorum is gone.
>>>>
>>>>>> Further, since we will have a 3.8 release - it is
>>>>>> likely we will ultimately recommend users upgrade from that version
>>>> given
>>>>>> its aim is to have strategic KRaft feature parity with ZK.
>>>>>> That being said, I am not 100% on this. Let me know whether you think
>>>> this
>>>>>> should block the release, Luke. I am also tagging Colin and David to
>>>> weigh
>>>>>> in with their opinions, as they worked on the migration logic.
>>>>
>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a
>>> clear
>>>> blocker, I think. But easy to fix, I believe. I will create a PR.
>>>>
>>>> best,
>>>> Colin
>>>>
>>>>>>
>>>>>> Hey Kirk and Chris,
>>>>>>
>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>>> due
>>>> to
>>>>>> improper closing. And the PR description implies this has been
>>> present
>>>>>> since 3.5. While annoying, I don't see a strong reason for this to
>>> block
>>>>>> the release.
>>>>>>
>>>>>> Hey Jakub,
>>>>>>
>>>>>> Nice catch! It does seem like we should have gated this behind the
>>>> metadata
>>>>>> version as KIP-858 implies. Is the cluster configured with multiple
>>> log
>>>>>> dirs? What is the impact of the error messages?
>>>>>>
>>>>>> Tagging Igor (the author of the KIP) to weigh in.
>>>>>>
>>>>>> Best,
>>>>>> Stanislav
>>>>>>
>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I was trying the RC2 and run into the following issue ... when I
>>> run
>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>>> metadata
>>>>>>> version, I seem to be getting repeated errors like this in the
>>>> controller
>>>>>>> logs:
>>>>>>>
>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>>>>> assignReplicasToDirs:
>>>>>>> event failed with UnsupportedVersionException in 15 microseconds.
>>>>>>> (org.apache.kafka.controller.QuorumController)
>>>>>>> [quorum-controller-0-event-handler]
>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
>>>> error
>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>>>>>> partitions=[PartitionData(partitionIndex=2),
>>>>>>> PartitionData(partitionIndex=1)]),
>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context
>>>>>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>>>> clientAddress=/
>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
>>>>>>> clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false,
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>>>>>> ])
>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>>>>>> java.util.concurrent.CompletionException:
>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>> Directory
>>>>>>> assignment is not supported yet.
>>>>>>>
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>>>>>>   at
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>>>>>>   at java.base/java.lang.Thread.run(Thread.java:840)
>>>>>>>
>>>>>>> Caused by:
>>> org.apache.kafka.common.errors.UnsupportedVersionException:
>>>>>>> Directory assignment is not supported yet.
>>>>>>>
>>>>>>> Is that expected? I guess with the metadata version set to
>>> 3.6-IV2, it
>>>>>>> makes sense that the request is not supported. But shouldn't then
>>> the
>>>>>>> request not be sent at all by the brokers? (I did not opened a JIRA
>>>> for
>>>>>> it,
>>>>>>> but I can open one if you agree this is not expected)
>>>>>>>
>>>>>>> Thanks & Regards
>>>>>>> Jakub
>>>>>>>
>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>>> wrote:
>>>>>>>
>>>>>>>> Hi Stanislav,
>>>>>>>>
>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
>>>> you
>>>>>>>> missed it.
>>>>>>>> cross-posting here:
>>>>>>>>
>>>>>>>> There is a bug KAFKA-16101
>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>>> that
>>>>>>> "Kafka
>>>>>>>> cluster will be unavailable during KRaft migration rollback".
>>>>>>>> The impact for this issue is that if brokers try to rollback to
>>> ZK
>>>> mode
>>>>>>>> during KRaft migration process, there will be a period of time
>>> the
>>>>>>> cluster
>>>>>>>> is unavailable.
>>>>>>>> Since ZK migrating to KRaft feature is a production ready
>>> feature, I
>>>>>>> think
>>>>>>>> this should be addressed soon.
>>>>>>>> Do you think this is a blocker for v3.7.0?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>> Luke
>>>>>>>>
>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>>>> fearthecellos@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks, Kirk!
>>>>>>>>>
>>>>>>>>> @Stanislav--do you believe that this warrants a new RC?
>>>>>>>>>
>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Chris/Stanislav,
>>>>>>>>>>
>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log
>>>> problem
>>>>>>>>>> (KAFKA-16029) and have put out a draft PR (
>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the
>>>>>>> quickstart
>>>>>>>>>> approach as a second means to reproduce/verify while I wait
>>> for
>>>> the
>>>>>>>> PR's
>>>>>>>>>> Jenkins job to finish.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Kirk
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>>>>>>>>>> Hi Stanislav,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks for running this release!
>>>>>>>>>>>
>>>>>>>>>>> To verify, I:
>>>>>>>>>>> - Built from source using Java 11 with both:
>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub
>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from
>>>>>>>>>>>
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>> - Checked signatures and checksums
>>>>>>>>>>> - Ran the quickstart using both:
>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from
>>>>>>>>>>>
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>> with
>>>>>>>> Java
>>>>>>>>>> 11
>>>>>>>>>>> and Scala 13 in KRaft mode
>>>>>>>>>>> - - Our shiny new broker Docker image,
>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>> - Ran all unit tests
>>>>>>>>>>> - Ran all integration tests for Connect and MM2
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I found two minor areas for concern:
>>>>>>>>>>>
>>>>>>>>>>> 1. (Possibly a blocker)
>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log
>>>>>> message
>>>>>>>>> being
>>>>>>>>>>> emitted frequently (not not every time) when I killed my
>>>> console
>>>>>>>>> consumer
>>>>>>>>>>> via ctrl-C:
>>>>>>>>>>>
>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer
>>>>>>>> clientId=console-consumer,
>>>>>>>>>>> groupId=console-consumer-74388] Unable to find
>>>>>> FetchSessionHandler
>>>>>>>> for
>>>>>>>>>> node
>>>>>>>>>>> 1. Ignoring fetch response
>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>>>>>>>>>>
>>>>>>>>>>> I see that this error message is already reported in
>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I
>>> think we
>>>>>>> should
>>>>>>>>>>> prioritize fixing it for this release. I know it's probably
>>>>>> benign
>>>>>>>> but
>>>>>>>>>> it's
>>>>>>>>>>> really not a good look for us when basic operations log
>>> error
>>>>>>>> messages,
>>>>>>>>>> and
>>>>>>>>>>> it may give new users some headaches.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2. (Probably not a blocker)
>>>>>>>>>>> The following unit tests failed the first time around, and
>>>> all of
>>>>>>>> them
>>>>>>>>>>> passed the second time I ran them:
>>>>>>>>>>>
>>>>>>>>>>> - (clients)
>>>>>>>>>>
>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>> - (clients)
>>> Tls13SelectorTest.testConnectionsByClientMetric()
>>>>>>>>>>> - (connect)
>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>>>>>>>> (I
>>>>>>>>>>> thought I fixed this one! 🤬🤬)
>>>>>>>>>>> - (core)
>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks again for your work on this release, and
>>>> congratulations
>>>>>> to
>>>>>>>>> Kafka
>>>>>>>>>>> Streams for having zero flaky unit tests during my
>>>>>>>> highly-experimental
>>>>>>>>>>> single laptop run!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Chris
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>>>>>>>>>> <st...@confluent.io.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello Kafka users, developers, and client-developers,
>>>>>>>>>>>>
>>>>>>>>>>>> This is the first candidate for release of Apache Kafka
>>>> 3.7.0.
>>>>>>>>>>>>
>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs
>>> that
>>>> I
>>>>>> had
>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to
>>> announcing
>>>>>> due
>>>>>>> to
>>>>>>>>> new
>>>>>>>>>>>> blockers arriving before I could even announce them.
>>>>>>>>>>>>
>>>>>>>>>>>> Further - I haven't yet been able to set up the system
>>> tests
>>>>>>>>>> successfully.
>>>>>>>>>>>> And the integration/unit tests do have a few failures
>>> that I
>>>>>> have
>>>>>>>> to
>>>>>>>>>> spend
>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone
>>>>>> notices
>>>>>>>> any
>>>>>>>>>> tests
>>>>>>>>>>>> failing that they're subject matters experts in. Expect
>>> me
>>>> to
>>>>>>>> follow
>>>>>>>>>> up in
>>>>>>>>>>>> a day or two with more detailed analysis.
>>>>>>>>>>>>
>>>>>>>>>>>> Major changes include:
>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the
>>>> consumer
>>>>>>>>>> rebalance
>>>>>>>>>>>> protocol
>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft
>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a
>>>> standardized
>>>>>>>>>> interface
>>>>>>>>>>>>
>>>>>>>>>>>> Check more information in the WIP blog post:
>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578
>>>>>>>>>>>>
>>>>>>>>>>>> Release notes for the 3.7.0 release:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>>>>>>>>>>>
>>>>>>>>>>>> *** Please download, test and vote by Thursday, January
>>> 18,
>>>> 9am
>>>>>>> PT
>>>>>>>>> ***
>>>>>>>>>>>>
>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to
>>> this
>>>>>>> being
>>>>>>>>> the
>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving
>>> it a
>>>> bit
>>>>>>>> more
>>>>>>>>>> time.
>>>>>>>>>>>>
>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the
>>>>>> release:
>>>>>>>>>>>> https://kafka.apache.org/KEYS
>>>>>>>>>>>>
>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary):
>>>>>>>>>>>>
>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>>>>>>>>>>>
>>>>>>>>>>>> * Docker release artifact to be voted upon:
>>>>>>>>>>>> apache/kafka:3.7.0-rc2
>>>>>>>>>>>>
>>>>>>>>>>>> * Maven artifacts to be voted upon:
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>>>>>>>>>>
>>>>>>>>>>>> * Javadoc:
>>>>>>>>>>>>
>>>>>>>>
>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>>>>>>>>>>>
>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>>>>>>>>>>>
>>>>>>>>>>>> * Documentation:
>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html
>>>>>>>>>>>>
>>>>>>>>>>>> * Protocol:
>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html
>>>>>>>>>>>>
>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch:
>>>>>>>>>>>> Unit/integration tests:
>>>>>>>>>>>>
>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>>>>>>>>>>> There are failing tests here. I have to follow up with
>>>> triaging
>>>>>>>> some
>>>>>>>>> of
>>>>>>>>>>>> the failures and figuring out if they're actual problems
>>> or
>>>>>>> simply
>>>>>>>>>> flakes.
>>>>>>>>>>>>
>>>>>>>>>>>> System tests:
>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>>>>>>>>>>>
>>>>>>>>>>>> No successful system test runs yet. I am working on
>>> getting
>>>> the
>>>>>>> job
>>>>>>>>> to
>>>>>>>>>> run.
>>>>>>>>>>>>
>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7
>>>>>> branch:
>>>>>>>>>>>> Attached are the scan_report and report_jvm output files
>>>> from
>>>>>> the
>>>>>>>>>> Docker
>>>>>>>>>>>> Build run:
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>>>>>>>>>>>
>>>>>>>>>>>> And the final docker image build job - Docker Build Test
>>>>>>> Pipeline:
>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277
>>>>>>>>>>>>
>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 -
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>>>>>>>>>>>
>>>>>>>>>>>> /**************************************
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Stanislav Kozlovski
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best,
>>>>>> Stanislav
>>>>>>
>>>>
>>>
>>
> 

Re: [VOTE] 3.7.0 RC2

Posted by Proven Provenzano <pp...@confluent.io.INVALID>.
We have another blocking issue for the RC :
https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar to
https://issues.apache.org/jira/browse/KAFKA-14616. The new issue however
can lead to the new topic having partitions that a producer cannot write to.

--Proven

On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano <pp...@confluent.io>
wrote:

>
> I have a PR https://github.com/apache/kafka/pull/15197 for
> https://issues.apache.org/jira/browse/KAFKA-16131 that is building now.
> --Proven
>
> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:
>
>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>> blocker bug because it *
>> *> will generate huge amount of logspam. I guess we didn't find it in
>> junit
>> tests *
>> *> since logspam doesn't fail the automated tests. But certainly it's not
>> suitable *
>> *> for production. Did you file a JIRA yet?*
>>
>> Hi Colin,
>>
>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>
>> Thanks & Regards
>> Jakub
>>
>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org> wrote:
>>
>> > Hi Stanislav,
>> >
>> > Thanks for making the first RC. The fact that it's titled RC2 is messing
>> > with my mind a bit. I hope this doesn't make people think that we're
>> > farther along than we are, heh.
>> >
>> > On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>> > > *> Nice catch! It does seem like we should have gated this behind the
>> > > metadata> version as KIP-858 implies. Is the cluster configured with
>> > > multiple log> dirs? What is the impact of the error messages?*
>> > >
>> > > I did not observe any obvious impact. I was able to send and receive
>> > > messages as normally. But to be honest, I have no idea what else
>> > > this might impact, so I did not try anything special.
>> > >
>> > > I think everyone upgrading an existing KRaft cluster will go through
>> this
>> > > stage (running Kafka 3.7 with an older metadata version for at least a
>> > > while). So even if it is just a logged exception without any other
>> > impact I
>> > > wonder if it might scare users from upgrading. But I leave it to
>> others
>> > to
>> > > decide if this is a blocker or not.
>> > >
>> >
>> > Hi Jakub,
>> >
>> > Thanks for trying the RC. I think what you found is a blocker bug
>> because
>> > it will generate huge amount of logspam. I guess we didn't find it in
>> junit
>> > tests since logspam doesn't fail the automated tests. But certainly it's
>> > not suitable for production. Did you file a JIRA yet?
>> >
>> > > On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>> > > <st...@confluent.io.invalid> wrote:
>> > >
>> > >> Hey Luke,
>> > >>
>> > >> This is an interesting problem. Given the fact that the KIP for
>> having a
>> > >> 3.8 release passed, I think it weights the scale towards not calling
>> > this a
>> > >> blocker and expecting it to be solved in 3.7.1.
>> > >>
>> > >> It is unfortunate that it would not seem safe to migrate to KRaft in
>> > 3.7.0
>> > >> (given the inability to rollback safely), but if that's true - the
>> same
>> > >> case would apply for 3.6.0. So in any case users w\ould be expected
>> to
>> > use a
>> > >> patch release for this.
>> >
>> > Hi Luke,
>> >
>> > Thanks for testing rollback. I think this is a case where the
>> > documentation is wrong. The intention was to for the steps to basically
>> be:
>> >
>> > 1. roll all the brokers into zk mode, but with migration enabled
>> > 2. take down the kraft quorum
>> > 3. rmr /controller, allowing a hybrid broker to take over.
>> > 4. roll all the brokers into zk mode without migration enabled (if
>> desired)
>> >
>> > With these steps, there isn't really unavailability since a ZK
>> controller
>> > can be elected quickly after the kraft quorum is gone.
>> >
>> > >> Further, since we will have a 3.8 release - it is
>> > >> likely we will ultimately recommend users upgrade from that version
>> > given
>> > >> its aim is to have strategic KRaft feature parity with ZK.
>> > >> That being said, I am not 100% on this. Let me know whether you think
>> > this
>> > >> should block the release, Luke. I am also tagging Colin and David to
>> > weigh
>> > >> in with their opinions, as they worked on the migration logic.
>> >
>> > The rollback docs are new in 3.7 so the fact that they're wrong is a
>> clear
>> > blocker, I think. But easy to fix, I believe. I will create a PR.
>> >
>> > best,
>> > Colin
>> >
>> > >>
>> > >> Hey Kirk and Chris,
>> > >>
>> > >> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>> due
>> > to
>> > >> improper closing. And the PR description implies this has been
>> present
>> > >> since 3.5. While annoying, I don't see a strong reason for this to
>> block
>> > >> the release.
>> > >>
>> > >> Hey Jakub,
>> > >>
>> > >> Nice catch! It does seem like we should have gated this behind the
>> > metadata
>> > >> version as KIP-858 implies. Is the cluster configured with multiple
>> log
>> > >> dirs? What is the impact of the error messages?
>> > >>
>> > >> Tagging Igor (the author of the KIP) to weigh in.
>> > >>
>> > >> Best,
>> > >> Stanislav
>> > >>
>> > >> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
>> wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > I was trying the RC2 and run into the following issue ... when I
>> run
>> > >> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
>> metadata
>> > >> > version, I seem to be getting repeated errors like this in the
>> > controller
>> > >> > logs:
>> > >> >
>> > >> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>> > >> assignReplicasToDirs:
>> > >> > event failed with UnsupportedVersionException in 15 microseconds.
>> > >> > (org.apache.kafka.controller.QuorumController)
>> > >> > [quorum-controller-0-event-handler]
>> > >> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
>> > error
>> > >> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
>> > >> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>> > >> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>> > >> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>> > >> > partitions=[PartitionData(partitionIndex=2),
>> > >> > PartitionData(partitionIndex=1)]),
>> > >> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>> > >> > partitions=[PartitionData(partitionIndex=0)])])]) with context
>> > >> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>> > >> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
>> > clientAddress=/
>> > >> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>> > >> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
>> > >> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
>> > >> > softwareVersion=3.7.0), fromPrivilegedListener=false,
>> > >> >
>> > >> >
>> > >>
>> >
>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>> > >> > ])
>> > >> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>> > >> > java.util.concurrent.CompletionException:
>> > >> > org.apache.kafka.common.errors.UnsupportedVersionException:
>> Directory
>> > >> > assignment is not supported yet.
>> > >> >
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>> > >> >  at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>> > >> >  at java.base/java.lang.Thread.run(Thread.java:840)
>> > >> >
>> > >> > Caused by:
>> org.apache.kafka.common.errors.UnsupportedVersionException:
>> > >> > Directory assignment is not supported yet.
>> > >> >
>> > >> > Is that expected? I guess with the metadata version set to
>> 3.6-IV2, it
>> > >> > makes sense that the request is not supported. But shouldn't then
>> the
>> > >> > request not be sent at all by the brokers? (I did not opened a JIRA
>> > for
>> > >> it,
>> > >> > but I can open one if you agree this is not expected)
>> > >> >
>> > >> > Thanks & Regards
>> > >> > Jakub
>> > >> >
>> > >> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
>> wrote:
>> > >> >
>> > >> > > Hi Stanislav,
>> > >> > >
>> > >> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
>> > you
>> > >> > > missed it.
>> > >> > > cross-posting here:
>> > >> > >
>> > >> > > There is a bug KAFKA-16101
>> > >> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
>> that
>> > >> > "Kafka
>> > >> > > cluster will be unavailable during KRaft migration rollback".
>> > >> > > The impact for this issue is that if brokers try to rollback to
>> ZK
>> > mode
>> > >> > > during KRaft migration process, there will be a period of time
>> the
>> > >> > cluster
>> > >> > > is unavailable.
>> > >> > > Since ZK migrating to KRaft feature is a production ready
>> feature, I
>> > >> > think
>> > >> > > this should be addressed soon.
>> > >> > > Do you think this is a blocker for v3.7.0?
>> > >> > >
>> > >> > > Thanks.
>> > >> > > Luke
>> > >> > >
>> > >> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
>> > fearthecellos@gmail.com
>> > >> >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Thanks, Kirk!
>> > >> > > >
>> > >> > > > @Stanislav--do you believe that this warrants a new RC?
>> > >> > > >
>> > >> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
>> wrote:
>> > >> > > >
>> > >> > > > > Hi Chris/Stanislav,
>> > >> > > > >
>> > >> > > > > I'm working on the 'Unable to find FetchSessionHandler' log
>> > problem
>> > >> > > > > (KAFKA-16029) and have put out a draft PR (
>> > >> > > > > https://github.com/apache/kafka/pull/15186). I will use the
>> > >> > quickstart
>> > >> > > > > approach as a second means to reproduce/verify while I wait
>> for
>> > the
>> > >> > > PR's
>> > >> > > > > Jenkins job to finish.
>> > >> > > > >
>> > >> > > > > Thanks,
>> > >> > > > > Kirk
>> > >> > > > >
>> > >> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>> > >> > > > > > Hi Stanislav,
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > Thanks for running this release!
>> > >> > > > > >
>> > >> > > > > > To verify, I:
>> > >> > > > > > - Built from source using Java 11 with both:
>> > >> > > > > > - - the 3.7.0-rc2 tag on GitHub
>> > >> > > > > > - - the kafka-3.7.0-src.tgz artifact from
>> > >> > > > > >
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > >> > > > > > - Checked signatures and checksums
>> > >> > > > > > - Ran the quickstart using both:
>> > >> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
>> > >> > > > > >
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > >> with
>> > >> > > Java
>> > >> > > > > 11
>> > >> > > > > > and Scala 13 in KRaft mode
>> > >> > > > > > - - Our shiny new broker Docker image,
>> apache/kafka:3.7.0-rc2
>> > >> > > > > > - Ran all unit tests
>> > >> > > > > > - Ran all integration tests for Connect and MM2
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > I found two minor areas for concern:
>> > >> > > > > >
>> > >> > > > > > 1. (Possibly a blocker)
>> > >> > > > > > When running the quickstart, I noticed this ERROR-level log
>> > >> message
>> > >> > > > being
>> > >> > > > > > emitted frequently (not not every time) when I killed my
>> > console
>> > >> > > > consumer
>> > >> > > > > > via ctrl-C:
>> > >> > > > > >
>> > >> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
>> > >> > > clientId=console-consumer,
>> > >> > > > > > groupId=console-consumer-74388] Unable to find
>> > >> FetchSessionHandler
>> > >> > > for
>> > >> > > > > node
>> > >> > > > > > 1. Ignoring fetch response
>> > >> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>> > >> > > > > >
>> > >> > > > > > I see that this error message is already reported in
>> > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I
>> think we
>> > >> > should
>> > >> > > > > > prioritize fixing it for this release. I know it's probably
>> > >> benign
>> > >> > > but
>> > >> > > > > it's
>> > >> > > > > > really not a good look for us when basic operations log
>> error
>> > >> > > messages,
>> > >> > > > > and
>> > >> > > > > > it may give new users some headaches.
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > 2. (Probably not a blocker)
>> > >> > > > > > The following unit tests failed the first time around, and
>> > all of
>> > >> > > them
>> > >> > > > > > passed the second time I ran them:
>> > >> > > > > >
>> > >> > > > > > - (clients)
>> > >> > > > >
>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>> > >> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
>> > >> > > > > > - (clients)
>> Tls13SelectorTest.testConnectionsByClientMetric()
>> > >> > > > > > - (connect)
>> > >> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>> > >> > > > (I
>> > >> > > > > > thought I fixed this one! 🤬🤬)
>> > >> > > > > > - (core)
>> > ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > Thanks again for your work on this release, and
>> > congratulations
>> > >> to
>> > >> > > > Kafka
>> > >> > > > > > Streams for having zero flaky unit tests during my
>> > >> > > highly-experimental
>> > >> > > > > > single laptop run!
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > Cheers,
>> > >> > > > > >
>> > >> > > > > > Chris
>> > >> > > > > >
>> > >> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>> > >> > > > > > <st...@confluent.io.invalid> wrote:
>> > >> > > > > >
>> > >> > > > > > > Hello Kafka users, developers, and client-developers,
>> > >> > > > > > >
>> > >> > > > > > > This is the first candidate for release of Apache Kafka
>> > 3.7.0.
>> > >> > > > > > >
>> > >> > > > > > > Note it's named "RC2" because I had a few "failed" RCs
>> that
>> > I
>> > >> had
>> > >> > > > > > > cut/uploaded but ultimately had to scrap prior to
>> announcing
>> > >> due
>> > >> > to
>> > >> > > > new
>> > >> > > > > > > blockers arriving before I could even announce them.
>> > >> > > > > > >
>> > >> > > > > > > Further - I haven't yet been able to set up the system
>> tests
>> > >> > > > > successfully.
>> > >> > > > > > > And the integration/unit tests do have a few failures
>> that I
>> > >> have
>> > >> > > to
>> > >> > > > > spend
>> > >> > > > > > > time triaging. I would appreciate any help in case anyone
>> > >> notices
>> > >> > > any
>> > >> > > > > tests
>> > >> > > > > > > failing that they're subject matters experts in. Expect
>> me
>> > to
>> > >> > > follow
>> > >> > > > > up in
>> > >> > > > > > > a day or two with more detailed analysis.
>> > >> > > > > > >
>> > >> > > > > > > Major changes include:
>> > >> > > > > > > - Early Access to KIP-848 - the next generation of the
>> > consumer
>> > >> > > > > rebalance
>> > >> > > > > > > protocol
>> > >> > > > > > > - KIP-858: Adding JBOD support to KRaft
>> > >> > > > > > > - KIP-714: Observability into Client metrics via a
>> > standardized
>> > >> > > > > interface
>> > >> > > > > > >
>> > >> > > > > > > Check more information in the WIP blog post:
>> > >> > > > > > > https://github.com/apache/kafka-site/pull/578
>> > >> > > > > > >
>> > >> > > > > > > Release notes for the 3.7.0 release:
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>> > >> > > > > > >
>> > >> > > > > > > *** Please download, test and vote by Thursday, January
>> 18,
>> > 9am
>> > >> > PT
>> > >> > > > ***
>> > >> > > > > > >
>> > >> > > > > > > Usually these deadlines tend to be 2-3 days, but due to
>> this
>> > >> > being
>> > >> > > > the
>> > >> > > > > > > first RC and the tests not having ran yet, I am giving
>> it a
>> > bit
>> > >> > > more
>> > >> > > > > time.
>> > >> > > > > > >
>> > >> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
>> > >> release:
>> > >> > > > > > > https://kafka.apache.org/KEYS
>> > >> > > > > > >
>> > >> > > > > > > * Release artifacts to be voted upon (source and binary):
>> > >> > > > > > >
>> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > >> > > > > > >
>> > >> > > > > > > * Docker release artifact to be voted upon:
>> > >> > > > > > > apache/kafka:3.7.0-rc2
>> > >> > > > > > >
>> > >> > > > > > > * Maven artifacts to be voted upon:
>> > >> > > > > > >
>> > >> > > >
>> > >>
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > >> > > > > > >
>> > >> > > > > > > * Javadoc:
>> > >> > > > > > >
>> > >> > >
>> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>> > >> > > > > > >
>> > >> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>> > >> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>> > >> > > > > > >
>> > >> > > > > > > * Documentation:
>> > >> > > > > > > https://kafka.apache.org/37/documentation.html
>> > >> > > > > > >
>> > >> > > > > > > * Protocol:
>> > >> > > > > > > https://kafka.apache.org/37/protocol.html
>> > >> > > > > > >
>> > >> > > > > > > * Successful Jenkins builds for the 3.7 branch:
>> > >> > > > > > > Unit/integration tests:
>> > >> > > > > > >
>> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>> > >> > > > > > > There are failing tests here. I have to follow up with
>> > triaging
>> > >> > > some
>> > >> > > > of
>> > >> > > > > > > the failures and figuring out if they're actual problems
>> or
>> > >> > simply
>> > >> > > > > flakes.
>> > >> > > > > > >
>> > >> > > > > > > System tests:
>> > >> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>> > >> > > > > > >
>> > >> > > > > > > No successful system test runs yet. I am working on
>> getting
>> > the
>> > >> > job
>> > >> > > > to
>> > >> > > > > run.
>> > >> > > > > > >
>> > >> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
>> > >> branch:
>> > >> > > > > > > Attached are the scan_report and report_jvm output files
>> > from
>> > >> the
>> > >> > > > > Docker
>> > >> > > > > > > Build run:
>> > >> > > > > > >
>> > >> > > > >
>> > >> > >
>> > >>
>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>> > >> > > > > > >
>> > >> > > > > > > And the final docker image build job - Docker Build Test
>> > >> > Pipeline:
>> > >> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
>> > >> > > > > > >
>> > >> > > > > > > The image is apache/kafka:3.7.0-rc2 -
>> > >> > > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>> > >> > > > > > >
>> > >> > > > > > > /**************************************
>> > >> > > > > > >
>> > >> > > > > > > Thanks,
>> > >> > > > > > > Stanislav Kozlovski
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> Best,
>> > >> Stanislav
>> > >>
>> >
>>
>

Re: [VOTE] 3.7.0 RC2

Posted by Proven Provenzano <pp...@confluent.io.INVALID>.
I have a PR https://github.com/apache/kafka/pull/15197 for
https://issues.apache.org/jira/browse/KAFKA-16131 that is building now.
--Proven

On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote:

> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
> blocker bug because it *
> *> will generate huge amount of logspam. I guess we didn't find it in junit
> tests *
> *> since logspam doesn't fail the automated tests. But certainly it's not
> suitable *
> *> for production. Did you file a JIRA yet?*
>
> Hi Colin,
>
> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>
> Thanks & Regards
> Jakub
>
> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for making the first RC. The fact that it's titled RC2 is messing
> > with my mind a bit. I hope this doesn't make people think that we're
> > farther along than we are, heh.
> >
> > On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > > *> Nice catch! It does seem like we should have gated this behind the
> > > metadata> version as KIP-858 implies. Is the cluster configured with
> > > multiple log> dirs? What is the impact of the error messages?*
> > >
> > > I did not observe any obvious impact. I was able to send and receive
> > > messages as normally. But to be honest, I have no idea what else
> > > this might impact, so I did not try anything special.
> > >
> > > I think everyone upgrading an existing KRaft cluster will go through
> this
> > > stage (running Kafka 3.7 with an older metadata version for at least a
> > > while). So even if it is just a logged exception without any other
> > impact I
> > > wonder if it might scare users from upgrading. But I leave it to others
> > to
> > > decide if this is a blocker or not.
> > >
> >
> > Hi Jakub,
> >
> > Thanks for trying the RC. I think what you found is a blocker bug because
> > it will generate huge amount of logspam. I guess we didn't find it in
> junit
> > tests since logspam doesn't fail the automated tests. But certainly it's
> > not suitable for production. Did you file a JIRA yet?
> >
> > > On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > > <st...@confluent.io.invalid> wrote:
> > >
> > >> Hey Luke,
> > >>
> > >> This is an interesting problem. Given the fact that the KIP for
> having a
> > >> 3.8 release passed, I think it weights the scale towards not calling
> > this a
> > >> blocker and expecting it to be solved in 3.7.1.
> > >>
> > >> It is unfortunate that it would not seem safe to migrate to KRaft in
> > 3.7.0
> > >> (given the inability to rollback safely), but if that's true - the
> same
> > >> case would apply for 3.6.0. So in any case users w\ould be expected to
> > use a
> > >> patch release for this.
> >
> > Hi Luke,
> >
> > Thanks for testing rollback. I think this is a case where the
> > documentation is wrong. The intention was to for the steps to basically
> be:
> >
> > 1. roll all the brokers into zk mode, but with migration enabled
> > 2. take down the kraft quorum
> > 3. rmr /controller, allowing a hybrid broker to take over.
> > 4. roll all the brokers into zk mode without migration enabled (if
> desired)
> >
> > With these steps, there isn't really unavailability since a ZK controller
> > can be elected quickly after the kraft quorum is gone.
> >
> > >> Further, since we will have a 3.8 release - it is
> > >> likely we will ultimately recommend users upgrade from that version
> > given
> > >> its aim is to have strategic KRaft feature parity with ZK.
> > >> That being said, I am not 100% on this. Let me know whether you think
> > this
> > >> should block the release, Luke. I am also tagging Colin and David to
> > weigh
> > >> in with their opinions, as they worked on the migration logic.
> >
> > The rollback docs are new in 3.7 so the fact that they're wrong is a
> clear
> > blocker, I think. But easy to fix, I believe. I will create a PR.
> >
> > best,
> > Colin
> >
> > >>
> > >> Hey Kirk and Chris,
> > >>
> > >> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due
> > to
> > >> improper closing. And the PR description implies this has been present
> > >> since 3.5. While annoying, I don't see a strong reason for this to
> block
> > >> the release.
> > >>
> > >> Hey Jakub,
> > >>
> > >> Nice catch! It does seem like we should have gated this behind the
> > metadata
> > >> version as KIP-858 implies. Is the cluster configured with multiple
> log
> > >> dirs? What is the impact of the error messages?
> > >>
> > >> Tagging Igor (the author of the KIP) to weigh in.
> > >>
> > >> Best,
> > >> Stanislav
> > >>
> > >> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I was trying the RC2 and run into the following issue ... when I run
> > >> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> metadata
> > >> > version, I seem to be getting repeated errors like this in the
> > controller
> > >> > logs:
> > >> >
> > >> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > >> assignReplicasToDirs:
> > >> > event failed with UnsupportedVersionException in 15 microseconds.
> > >> > (org.apache.kafka.controller.QuorumController)
> > >> > [quorum-controller-0-event-handler]
> > >> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
> > error
> > >> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> > >> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > >> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > >> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > >> > partitions=[PartitionData(partitionIndex=2),
> > >> > PartitionData(partitionIndex=1)]),
> > >> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > >> > partitions=[PartitionData(partitionIndex=0)])])]) with context
> > >> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> > >> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > clientAddress=/
> > >> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > >> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> > >> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > >> > softwareVersion=3.7.0), fromPrivilegedListener=false,
> > >> >
> > >> >
> > >>
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > >> > ])
> > >> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> > >> > java.util.concurrent.CompletionException:
> > >> > org.apache.kafka.common.errors.UnsupportedVersionException:
> Directory
> > >> > assignment is not supported yet.
> > >> >
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > >> >  at
> > >> >
> > >> >
> > >>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > >> >  at java.base/java.lang.Thread.run(Thread.java:840)
> > >> >
> > >> > Caused by:
> org.apache.kafka.common.errors.UnsupportedVersionException:
> > >> > Directory assignment is not supported yet.
> > >> >
> > >> > Is that expected? I guess with the metadata version set to 3.6-IV2,
> it
> > >> > makes sense that the request is not supported. But shouldn't then
> the
> > >> > request not be sent at all by the brokers? (I did not opened a JIRA
> > for
> > >> it,
> > >> > but I can open one if you agree this is not expected)
> > >> >
> > >> > Thanks & Regards
> > >> > Jakub
> > >> >
> > >> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> wrote:
> > >> >
> > >> > > Hi Stanislav,
> > >> > >
> > >> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
> > you
> > >> > > missed it.
> > >> > > cross-posting here:
> > >> > >
> > >> > > There is a bug KAFKA-16101
> > >> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> that
> > >> > "Kafka
> > >> > > cluster will be unavailable during KRaft migration rollback".
> > >> > > The impact for this issue is that if brokers try to rollback to ZK
> > mode
> > >> > > during KRaft migration process, there will be a period of time the
> > >> > cluster
> > >> > > is unavailable.
> > >> > > Since ZK migrating to KRaft feature is a production ready
> feature, I
> > >> > think
> > >> > > this should be addressed soon.
> > >> > > Do you think this is a blocker for v3.7.0?
> > >> > >
> > >> > > Thanks.
> > >> > > Luke
> > >> > >
> > >> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > fearthecellos@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks, Kirk!
> > >> > > >
> > >> > > > @Stanislav--do you believe that this warrants a new RC?
> > >> > > >
> > >> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> wrote:
> > >> > > >
> > >> > > > > Hi Chris/Stanislav,
> > >> > > > >
> > >> > > > > I'm working on the 'Unable to find FetchSessionHandler' log
> > problem
> > >> > > > > (KAFKA-16029) and have put out a draft PR (
> > >> > > > > https://github.com/apache/kafka/pull/15186). I will use the
> > >> > quickstart
> > >> > > > > approach as a second means to reproduce/verify while I wait
> for
> > the
> > >> > > PR's
> > >> > > > > Jenkins job to finish.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Kirk
> > >> > > > >
> > >> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > >> > > > > > Hi Stanislav,
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks for running this release!
> > >> > > > > >
> > >> > > > > > To verify, I:
> > >> > > > > > - Built from source using Java 11 with both:
> > >> > > > > > - - the 3.7.0-rc2 tag on GitHub
> > >> > > > > > - - the kafka-3.7.0-src.tgz artifact from
> > >> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> > > > > > - Checked signatures and checksums
> > >> > > > > > - Ran the quickstart using both:
> > >> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> > >> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> with
> > >> > > Java
> > >> > > > > 11
> > >> > > > > > and Scala 13 in KRaft mode
> > >> > > > > > - - Our shiny new broker Docker image,
> apache/kafka:3.7.0-rc2
> > >> > > > > > - Ran all unit tests
> > >> > > > > > - Ran all integration tests for Connect and MM2
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > I found two minor areas for concern:
> > >> > > > > >
> > >> > > > > > 1. (Possibly a blocker)
> > >> > > > > > When running the quickstart, I noticed this ERROR-level log
> > >> message
> > >> > > > being
> > >> > > > > > emitted frequently (not not every time) when I killed my
> > console
> > >> > > > consumer
> > >> > > > > > via ctrl-C:
> > >> > > > > >
> > >> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> > >> > > clientId=console-consumer,
> > >> > > > > > groupId=console-consumer-74388] Unable to find
> > >> FetchSessionHandler
> > >> > > for
> > >> > > > > node
> > >> > > > > > 1. Ignoring fetch response
> > >> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >> > > > > >
> > >> > > > > > I see that this error message is already reported in
> > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think
> we
> > >> > should
> > >> > > > > > prioritize fixing it for this release. I know it's probably
> > >> benign
> > >> > > but
> > >> > > > > it's
> > >> > > > > > really not a good look for us when basic operations log
> error
> > >> > > messages,
> > >> > > > > and
> > >> > > > > > it may give new users some headaches.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > 2. (Probably not a blocker)
> > >> > > > > > The following unit tests failed the first time around, and
> > all of
> > >> > > them
> > >> > > > > > passed the second time I ran them:
> > >> > > > > >
> > >> > > > > > - (clients)
> > >> > > > >
> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > >> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > >> > > > > > - (clients)
> Tls13SelectorTest.testConnectionsByClientMetric()
> > >> > > > > > - (connect)
> > >> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > >> > > > (I
> > >> > > > > > thought I fixed this one! 🤬🤬)
> > >> > > > > > - (core)
> > ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks again for your work on this release, and
> > congratulations
> > >> to
> > >> > > > Kafka
> > >> > > > > > Streams for having zero flaky unit tests during my
> > >> > > highly-experimental
> > >> > > > > > single laptop run!
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Cheers,
> > >> > > > > >
> > >> > > > > > Chris
> > >> > > > > >
> > >> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > >> > > > > > <st...@confluent.io.invalid> wrote:
> > >> > > > > >
> > >> > > > > > > Hello Kafka users, developers, and client-developers,
> > >> > > > > > >
> > >> > > > > > > This is the first candidate for release of Apache Kafka
> > 3.7.0.
> > >> > > > > > >
> > >> > > > > > > Note it's named "RC2" because I had a few "failed" RCs
> that
> > I
> > >> had
> > >> > > > > > > cut/uploaded but ultimately had to scrap prior to
> announcing
> > >> due
> > >> > to
> > >> > > > new
> > >> > > > > > > blockers arriving before I could even announce them.
> > >> > > > > > >
> > >> > > > > > > Further - I haven't yet been able to set up the system
> tests
> > >> > > > > successfully.
> > >> > > > > > > And the integration/unit tests do have a few failures
> that I
> > >> have
> > >> > > to
> > >> > > > > spend
> > >> > > > > > > time triaging. I would appreciate any help in case anyone
> > >> notices
> > >> > > any
> > >> > > > > tests
> > >> > > > > > > failing that they're subject matters experts in. Expect me
> > to
> > >> > > follow
> > >> > > > > up in
> > >> > > > > > > a day or two with more detailed analysis.
> > >> > > > > > >
> > >> > > > > > > Major changes include:
> > >> > > > > > > - Early Access to KIP-848 - the next generation of the
> > consumer
> > >> > > > > rebalance
> > >> > > > > > > protocol
> > >> > > > > > > - KIP-858: Adding JBOD support to KRaft
> > >> > > > > > > - KIP-714: Observability into Client metrics via a
> > standardized
> > >> > > > > interface
> > >> > > > > > >
> > >> > > > > > > Check more information in the WIP blog post:
> > >> > > > > > > https://github.com/apache/kafka-site/pull/578
> > >> > > > > > >
> > >> > > > > > > Release notes for the 3.7.0 release:
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >> > > > > > >
> > >> > > > > > > *** Please download, test and vote by Thursday, January
> 18,
> > 9am
> > >> > PT
> > >> > > > ***
> > >> > > > > > >
> > >> > > > > > > Usually these deadlines tend to be 2-3 days, but due to
> this
> > >> > being
> > >> > > > the
> > >> > > > > > > first RC and the tests not having ran yet, I am giving it
> a
> > bit
> > >> > > more
> > >> > > > > time.
> > >> > > > > > >
> > >> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > >> release:
> > >> > > > > > > https://kafka.apache.org/KEYS
> > >> > > > > > >
> > >> > > > > > > * Release artifacts to be voted upon (source and binary):
> > >> > > > > > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >> > > > > > >
> > >> > > > > > > * Docker release artifact to be voted upon:
> > >> > > > > > > apache/kafka:3.7.0-rc2
> > >> > > > > > >
> > >> > > > > > > * Maven artifacts to be voted upon:
> > >> > > > > > >
> > >> > > >
> > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > > > > > >
> > >> > > > > > > * Javadoc:
> > >> > > > > > >
> > >> > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >> > > > > > >
> > >> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > >> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >> > > > > > >
> > >> > > > > > > * Documentation:
> > >> > > > > > > https://kafka.apache.org/37/documentation.html
> > >> > > > > > >
> > >> > > > > > > * Protocol:
> > >> > > > > > > https://kafka.apache.org/37/protocol.html
> > >> > > > > > >
> > >> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> > >> > > > > > > Unit/integration tests:
> > >> > > > > > >
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > >> > > > > > > There are failing tests here. I have to follow up with
> > triaging
> > >> > > some
> > >> > > > of
> > >> > > > > > > the failures and figuring out if they're actual problems
> or
> > >> > simply
> > >> > > > > flakes.
> > >> > > > > > >
> > >> > > > > > > System tests:
> > >> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >> > > > > > >
> > >> > > > > > > No successful system test runs yet. I am working on
> getting
> > the
> > >> > job
> > >> > > > to
> > >> > > > > run.
> > >> > > > > > >
> > >> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> > >> branch:
> > >> > > > > > > Attached are the scan_report and report_jvm output files
> > from
> > >> the
> > >> > > > > Docker
> > >> > > > > > > Build run:
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >> > > > > > >
> > >> > > > > > > And the final docker image build job - Docker Build Test
> > >> > Pipeline:
> > >> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> > >> > > > > > >
> > >> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >> > > > > > >
> > >> > > > > > > /**************************************
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Stanislav Kozlovski
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Best,
> > >> Stanislav
> > >>
> >
>

Re: [VOTE] 3.7.0 RC2

Posted by Jakub Scholz <ja...@scholz.cz>.
*> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
blocker bug because it *
*> will generate huge amount of logspam. I guess we didn't find it in junit
tests *
*> since logspam doesn't fail the automated tests. But certainly it's not
suitable *
*> for production. Did you file a JIRA yet?*

Hi Colin,

I opened https://issues.apache.org/jira/browse/KAFKA-16131.

Thanks & Regards
Jakub

On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cm...@apache.org> wrote:

> Hi Stanislav,
>
> Thanks for making the first RC. The fact that it's titled RC2 is messing
> with my mind a bit. I hope this doesn't make people think that we're
> farther along than we are, heh.
>
> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > *> Nice catch! It does seem like we should have gated this behind the
> > metadata> version as KIP-858 implies. Is the cluster configured with
> > multiple log> dirs? What is the impact of the error messages?*
> >
> > I did not observe any obvious impact. I was able to send and receive
> > messages as normally. But to be honest, I have no idea what else
> > this might impact, so I did not try anything special.
> >
> > I think everyone upgrading an existing KRaft cluster will go through this
> > stage (running Kafka 3.7 with an older metadata version for at least a
> > while). So even if it is just a logged exception without any other
> impact I
> > wonder if it might scare users from upgrading. But I leave it to others
> to
> > decide if this is a blocker or not.
> >
>
> Hi Jakub,
>
> Thanks for trying the RC. I think what you found is a blocker bug because
> it will generate huge amount of logspam. I guess we didn't find it in junit
> tests since logspam doesn't fail the automated tests. But certainly it's
> not suitable for production. Did you file a JIRA yet?
>
> > On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > <st...@confluent.io.invalid> wrote:
> >
> >> Hey Luke,
> >>
> >> This is an interesting problem. Given the fact that the KIP for having a
> >> 3.8 release passed, I think it weights the scale towards not calling
> this a
> >> blocker and expecting it to be solved in 3.7.1.
> >>
> >> It is unfortunate that it would not seem safe to migrate to KRaft in
> 3.7.0
> >> (given the inability to rollback safely), but if that's true - the same
> >> case would apply for 3.6.0. So in any case users w\ould be expected to
> use a
> >> patch release for this.
>
> Hi Luke,
>
> Thanks for testing rollback. I think this is a case where the
> documentation is wrong. The intention was to for the steps to basically be:
>
> 1. roll all the brokers into zk mode, but with migration enabled
> 2. take down the kraft quorum
> 3. rmr /controller, allowing a hybrid broker to take over.
> 4. roll all the brokers into zk mode without migration enabled (if desired)
>
> With these steps, there isn't really unavailability since a ZK controller
> can be elected quickly after the kraft quorum is gone.
>
> >> Further, since we will have a 3.8 release - it is
> >> likely we will ultimately recommend users upgrade from that version
> given
> >> its aim is to have strategic KRaft feature parity with ZK.
> >> That being said, I am not 100% on this. Let me know whether you think
> this
> >> should block the release, Luke. I am also tagging Colin and David to
> weigh
> >> in with their opinions, as they worked on the migration logic.
>
> The rollback docs are new in 3.7 so the fact that they're wrong is a clear
> blocker, I think. But easy to fix, I believe. I will create a PR.
>
> best,
> Colin
>
> >>
> >> Hey Kirk and Chris,
> >>
> >> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due
> to
> >> improper closing. And the PR description implies this has been present
> >> since 3.5. While annoying, I don't see a strong reason for this to block
> >> the release.
> >>
> >> Hey Jakub,
> >>
> >> Nice catch! It does seem like we should have gated this behind the
> metadata
> >> version as KIP-858 implies. Is the cluster configured with multiple log
> >> dirs? What is the impact of the error messages?
> >>
> >> Tagging Igor (the author of the KIP) to weigh in.
> >>
> >> Best,
> >> Stanislav
> >>
> >> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
> >>
> >> > Hi,
> >> >
> >> > I was trying the RC2 and run into the following issue ... when I run
> >> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
> >> > version, I seem to be getting repeated errors like this in the
> controller
> >> > logs:
> >> >
> >> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> >> assignReplicasToDirs:
> >> > event failed with UnsupportedVersionException in 15 microseconds.
> >> > (org.apache.kafka.controller.QuorumController)
> >> > [quorum-controller-0-event-handler]
> >> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
> error
> >> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> >> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> >> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> >> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> >> > partitions=[PartitionData(partitionIndex=2),
> >> > PartitionData(partitionIndex=1)]),
> >> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> >> > partitions=[PartitionData(partitionIndex=0)])])]) with context
> >> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> >> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> clientAddress=/
> >> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> >> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> >> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> >> > softwareVersion=3.7.0), fromPrivilegedListener=false,
> >> >
> >> >
> >>
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> >> > ])
> >> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> >> > java.util.concurrent.CompletionException:
> >> > org.apache.kafka.common.errors.UnsupportedVersionException: Directory
> >> > assignment is not supported yet.
> >> >
> >> >  at
> >> >
> >> >
> >>
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> >> >  at
> >> >
> >> >
> >>
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> >> >  at
> >> >
> >> >
> >>
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> >> >  at
> >> >
> >> >
> >>
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> >> >  at
> >> >
> >> >
> >>
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> >> >  at
> >> >
> >> >
> >>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> >> >  at
> >> >
> >> >
> >>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> >> >  at
> >> >
> >> >
> >>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> >> >  at
> >> >
> >> >
> >>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> >> >  at
> >> >
> >> >
> >>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> >> >  at
> >> >
> >> >
> >>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> >> >  at java.base/java.lang.Thread.run(Thread.java:840)
> >> >
> >> > Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> >> > Directory assignment is not supported yet.
> >> >
> >> > Is that expected? I guess with the metadata version set to 3.6-IV2, it
> >> > makes sense that the request is not supported. But shouldn't then the
> >> > request not be sent at all by the brokers? (I did not opened a JIRA
> for
> >> it,
> >> > but I can open one if you agree this is not expected)
> >> >
> >> > Thanks & Regards
> >> > Jakub
> >> >
> >> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com> wrote:
> >> >
> >> > > Hi Stanislav,
> >> > >
> >> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
> you
> >> > > missed it.
> >> > > cross-posting here:
> >> > >
> >> > > There is a bug KAFKA-16101
> >> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that
> >> > "Kafka
> >> > > cluster will be unavailable during KRaft migration rollback".
> >> > > The impact for this issue is that if brokers try to rollback to ZK
> mode
> >> > > during KRaft migration process, there will be a period of time the
> >> > cluster
> >> > > is unavailable.
> >> > > Since ZK migrating to KRaft feature is a production ready feature, I
> >> > think
> >> > > this should be addressed soon.
> >> > > Do you think this is a blocker for v3.7.0?
> >> > >
> >> > > Thanks.
> >> > > Luke
> >> > >
> >> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> fearthecellos@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > Thanks, Kirk!
> >> > > >
> >> > > > @Stanislav--do you believe that this warrants a new RC?
> >> > > >
> >> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
> >> > > >
> >> > > > > Hi Chris/Stanislav,
> >> > > > >
> >> > > > > I'm working on the 'Unable to find FetchSessionHandler' log
> problem
> >> > > > > (KAFKA-16029) and have put out a draft PR (
> >> > > > > https://github.com/apache/kafka/pull/15186). I will use the
> >> > quickstart
> >> > > > > approach as a second means to reproduce/verify while I wait for
> the
> >> > > PR's
> >> > > > > Jenkins job to finish.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Kirk
> >> > > > >
> >> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> >> > > > > > Hi Stanislav,
> >> > > > > >
> >> > > > > >
> >> > > > > > Thanks for running this release!
> >> > > > > >
> >> > > > > > To verify, I:
> >> > > > > > - Built from source using Java 11 with both:
> >> > > > > > - - the 3.7.0-rc2 tag on GitHub
> >> > > > > > - - the kafka-3.7.0-src.tgz artifact from
> >> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >> > > > > > - Checked signatures and checksums
> >> > > > > > - Ran the quickstart using both:
> >> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> >> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >> with
> >> > > Java
> >> > > > > 11
> >> > > > > > and Scala 13 in KRaft mode
> >> > > > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> >> > > > > > - Ran all unit tests
> >> > > > > > - Ran all integration tests for Connect and MM2
> >> > > > > >
> >> > > > > >
> >> > > > > > I found two minor areas for concern:
> >> > > > > >
> >> > > > > > 1. (Possibly a blocker)
> >> > > > > > When running the quickstart, I noticed this ERROR-level log
> >> message
> >> > > > being
> >> > > > > > emitted frequently (not not every time) when I killed my
> console
> >> > > > consumer
> >> > > > > > via ctrl-C:
> >> > > > > >
> >> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> >> > > clientId=console-consumer,
> >> > > > > > groupId=console-consumer-74388] Unable to find
> >> FetchSessionHandler
> >> > > for
> >> > > > > node
> >> > > > > > 1. Ignoring fetch response
> >> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> >> > > > > >
> >> > > > > > I see that this error message is already reported in
> >> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think we
> >> > should
> >> > > > > > prioritize fixing it for this release. I know it's probably
> >> benign
> >> > > but
> >> > > > > it's
> >> > > > > > really not a good look for us when basic operations log error
> >> > > messages,
> >> > > > > and
> >> > > > > > it may give new users some headaches.
> >> > > > > >
> >> > > > > >
> >> > > > > > 2. (Probably not a blocker)
> >> > > > > > The following unit tests failed the first time around, and
> all of
> >> > > them
> >> > > > > > passed the second time I ran them:
> >> > > > > >
> >> > > > > > - (clients)
> >> > > > > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> >> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> >> > > > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> >> > > > > > - (connect)
> >> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> >> > > > (I
> >> > > > > > thought I fixed this one! 🤬🤬)
> >> > > > > > - (core)
> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> >> > > > > >
> >> > > > > >
> >> > > > > > Thanks again for your work on this release, and
> congratulations
> >> to
> >> > > > Kafka
> >> > > > > > Streams for having zero flaky unit tests during my
> >> > > highly-experimental
> >> > > > > > single laptop run!
> >> > > > > >
> >> > > > > >
> >> > > > > > Cheers,
> >> > > > > >
> >> > > > > > Chris
> >> > > > > >
> >> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> >> > > > > > <st...@confluent.io.invalid> wrote:
> >> > > > > >
> >> > > > > > > Hello Kafka users, developers, and client-developers,
> >> > > > > > >
> >> > > > > > > This is the first candidate for release of Apache Kafka
> 3.7.0.
> >> > > > > > >
> >> > > > > > > Note it's named "RC2" because I had a few "failed" RCs that
> I
> >> had
> >> > > > > > > cut/uploaded but ultimately had to scrap prior to announcing
> >> due
> >> > to
> >> > > > new
> >> > > > > > > blockers arriving before I could even announce them.
> >> > > > > > >
> >> > > > > > > Further - I haven't yet been able to set up the system tests
> >> > > > > successfully.
> >> > > > > > > And the integration/unit tests do have a few failures that I
> >> have
> >> > > to
> >> > > > > spend
> >> > > > > > > time triaging. I would appreciate any help in case anyone
> >> notices
> >> > > any
> >> > > > > tests
> >> > > > > > > failing that they're subject matters experts in. Expect me
> to
> >> > > follow
> >> > > > > up in
> >> > > > > > > a day or two with more detailed analysis.
> >> > > > > > >
> >> > > > > > > Major changes include:
> >> > > > > > > - Early Access to KIP-848 - the next generation of the
> consumer
> >> > > > > rebalance
> >> > > > > > > protocol
> >> > > > > > > - KIP-858: Adding JBOD support to KRaft
> >> > > > > > > - KIP-714: Observability into Client metrics via a
> standardized
> >> > > > > interface
> >> > > > > > >
> >> > > > > > > Check more information in the WIP blog post:
> >> > > > > > > https://github.com/apache/kafka-site/pull/578
> >> > > > > > >
> >> > > > > > > Release notes for the 3.7.0 release:
> >> > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> >> > > > > > >
> >> > > > > > > *** Please download, test and vote by Thursday, January 18,
> 9am
> >> > PT
> >> > > > ***
> >> > > > > > >
> >> > > > > > > Usually these deadlines tend to be 2-3 days, but due to this
> >> > being
> >> > > > the
> >> > > > > > > first RC and the tests not having ran yet, I am giving it a
> bit
> >> > > more
> >> > > > > time.
> >> > > > > > >
> >> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> >> release:
> >> > > > > > > https://kafka.apache.org/KEYS
> >> > > > > > >
> >> > > > > > > * Release artifacts to be voted upon (source and binary):
> >> > > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >> > > > > > >
> >> > > > > > > * Docker release artifact to be voted upon:
> >> > > > > > > apache/kafka:3.7.0-rc2
> >> > > > > > >
> >> > > > > > > * Maven artifacts to be voted upon:
> >> > > > > > >
> >> > > >
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >> > > > > > >
> >> > > > > > > * Javadoc:
> >> > > > > > >
> >> > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> >> > > > > > >
> >> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> >> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> >> > > > > > >
> >> > > > > > > * Documentation:
> >> > > > > > > https://kafka.apache.org/37/documentation.html
> >> > > > > > >
> >> > > > > > > * Protocol:
> >> > > > > > > https://kafka.apache.org/37/protocol.html
> >> > > > > > >
> >> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> >> > > > > > > Unit/integration tests:
> >> > > > > > >
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> >> > > > > > > There are failing tests here. I have to follow up with
> triaging
> >> > > some
> >> > > > of
> >> > > > > > > the failures and figuring out if they're actual problems or
> >> > simply
> >> > > > > flakes.
> >> > > > > > >
> >> > > > > > > System tests:
> >> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> >> > > > > > >
> >> > > > > > > No successful system test runs yet. I am working on getting
> the
> >> > job
> >> > > > to
> >> > > > > run.
> >> > > > > > >
> >> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> >> branch:
> >> > > > > > > Attached are the scan_report and report_jvm output files
> from
> >> the
> >> > > > > Docker
> >> > > > > > > Build run:
> >> > > > > > >
> >> > > > >
> >> > >
> >> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> >> > > > > > >
> >> > > > > > > And the final docker image build job - Docker Build Test
> >> > Pipeline:
> >> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> >> > > > > > >
> >> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> >> > > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> >> > > > > > >
> >> > > > > > > /**************************************
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Stanislav Kozlovski
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Best,
> >> Stanislav
> >>
>

Re: [VOTE] 3.7.0 RC2

Posted by Luke Chen <sh...@gmail.com>.
Hi Paolo, Colin,

Let's discuss detail about this issue in the PR:
https://github.com/apache/kafka/pull/15193 .

On Mon, Jan 15, 2024 at 4:21 PM Paolo Patierno <pa...@gmail.com>
wrote:

> Hi Colin,
> I was the one raising the issue about rollback and I also already tried
> what you mentioned but with no success.
> During the first rolling, I left
> the zookeeper.metadata.migration.enable=true but
> removed controller.quorum.voters and controller.listener.names.
> This is what I get from the brokers on restarting:
>
> 2024-01-15 09:19:14,172] ERROR Exiting Kafka due to fatal exception
> (kafka.Kafka$)
> org.apache.kafka.common.config.ConfigException: If using
> zookeeper.metadata.migration.enable, controller.quorum.voters must contain
> a parseable set of voters.
>         at
>
> kafka.server.KafkaConfig.validateNonEmptyQuorumVotersForMigration$1(KafkaConfig.scala:2286)
>         at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2371)
>         at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2233)
>         at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1604)
>         at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1527)
>         at kafka.Kafka$.buildServer(Kafka.scala:72)
>         at kafka.Kafka$.main(Kafka.scala:91)
>         at kafka.Kafka.main(Kafka.scala)
>
> Did you try it?
> Am I missing anything in your procedure?
>
> Thanks,
> Paolo
>
> On Mon, 15 Jan 2024 at 09:13, Colin McCabe <co...@cmccabe.xyz> wrote:
>
> > Docs fix discussed in the thread is here:
> > https://github.com/apache/kafka/pull/15193
> >
> > best,
> > Colin
> >
> >
> > On Sun, Jan 14, 2024, at 23:56, Colin McCabe wrote:
> > > Hi Stanislav,
> > >
> > > Thanks for making the first RC. The fact that it's titled RC2 is
> > > messing with my mind a bit. I hope this doesn't make people think that
> > > we're farther along than we are, heh.
> > >
> > > On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> > >> *> Nice catch! It does seem like we should have gated this behind the
> > >> metadata> version as KIP-858 implies. Is the cluster configured with
> > >> multiple log> dirs? What is the impact of the error messages?*
> > >>
> > >> I did not observe any obvious impact. I was able to send and receive
> > >> messages as normally. But to be honest, I have no idea what else
> > >> this might impact, so I did not try anything special.
> > >>
> > >> I think everyone upgrading an existing KRaft cluster will go through
> > this
> > >> stage (running Kafka 3.7 with an older metadata version for at least a
> > >> while). So even if it is just a logged exception without any other
> > impact I
> > >> wonder if it might scare users from upgrading. But I leave it to
> others
> > to
> > >> decide if this is a blocker or not.
> > >>
> > >
> > > Hi Jakub,
> > >
> > > Thanks for trying the RC. I think what you found is a blocker bug
> > > because it will generate huge amount of logspam. I guess we didn't find
> > > it in junit tests since logspam doesn't fail the automated tests. But
> > > certainly it's not suitable for production. Did you file a JIRA yet?
> > >
> > >> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> > >> <st...@confluent.io.invalid> wrote:
> > >>
> > >>> Hey Luke,
> > >>>
> > >>> This is an interesting problem. Given the fact that the KIP for
> having
> > a
> > >>> 3.8 release passed, I think it weights the scale towards not calling
> > this a
> > >>> blocker and expecting it to be solved in 3.7.1.
> > >>>
> > >>> It is unfortunate that it would not seem safe to migrate to KRaft in
> > 3.7.0
> > >>> (given the inability to rollback safely), but if that's true - the
> same
> > >>> case would apply for 3.6.0. So in any case users w\ould be expected
> to
> > use a
> > >>> patch release for this.
> > >
> > > Hi Luke,
> > >
> > > Thanks for testing rollback. I think this is a case where the
> > > documentation is wrong. The intention was to for the steps to basically
> > > be:
> > >
> > > 1. roll all the brokers into zk mode, but with migration enabled
> > > 2. take down the kraft quorum
> > > 3. rmr /controller, allowing a hybrid broker to take over.
> > > 4. roll all the brokers into zk mode without migration enabled (if
> > desired)
> > >
> > > With these steps, there isn't really unavailability since a ZK
> > > controller can be elected quickly after the kraft quorum is gone.
> > >
> > >>> Further, since we will have a 3.8 release - it is
> > >>> likely we will ultimately recommend users upgrade from that version
> > given
> > >>> its aim is to have strategic KRaft feature parity with ZK.
> > >>> That being said, I am not 100% on this. Let me know whether you think
> > this
> > >>> should block the release, Luke. I am also tagging Colin and David to
> > weigh
> > >>> in with their opinions, as they worked on the migration logic.
> > >
> > > The rollback docs are new in 3.7 so the fact that they're wrong is a
> > > clear blocker, I think. But easy to fix, I believe. I will create a PR.
> > >
> > > best,
> > > Colin
> > >
> > >>>
> > >>> Hey Kirk and Chris,
> > >>>
> > >>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
> due
> > to
> > >>> improper closing. And the PR description implies this has been
> present
> > >>> since 3.5. While annoying, I don't see a strong reason for this to
> > block
> > >>> the release.
> > >>>
> > >>> Hey Jakub,
> > >>>
> > >>> Nice catch! It does seem like we should have gated this behind the
> > metadata
> > >>> version as KIP-858 implies. Is the cluster configured with multiple
> log
> > >>> dirs? What is the impact of the error messages?
> > >>>
> > >>> Tagging Igor (the author of the KIP) to weigh in.
> > >>>
> > >>> Best,
> > >>> Stanislav
> > >>>
> > >>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
> wrote:
> > >>>
> > >>> > Hi,
> > >>> >
> > >>> > I was trying the RC2 and run into the following issue ... when I
> run
> > >>> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
> metadata
> > >>> > version, I seem to be getting repeated errors like this in the
> > controller
> > >>> > logs:
> > >>> >
> > >>> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> > >>> assignReplicasToDirs:
> > >>> > event failed with UnsupportedVersionException in 15 microseconds.
> > >>> > (org.apache.kafka.controller.QuorumController)
> > >>> > [quorum-controller-0-event-handler]
> > >>> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
> > error
> > >>> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> > >>> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > >>> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > >>> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > >>> > partitions=[PartitionData(partitionIndex=2),
> > >>> > PartitionData(partitionIndex=1)]),
> > >>> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > >>> > partitions=[PartitionData(partitionIndex=0)])])]) with context
> > >>> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > >>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> > >>> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> > clientAddress=/
> > >>> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > >>> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> > >>> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > >>> > softwareVersion=3.7.0), fromPrivilegedListener=false,
> > >>> >
> > >>> >
> > >>>
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > >>> > ])
> > >>> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> > >>> > java.util.concurrent.CompletionException:
> > >>> > org.apache.kafka.common.errors.UnsupportedVersionException:
> Directory
> > >>> > assignment is not supported yet.
> > >>> >
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> > >>> >  at
> > >>> >
> > >>> >
> > >>>
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> > >>> >  at java.base/java.lang.Thread.run(Thread.java:840)
> > >>> >
> > >>> > Caused by:
> > org.apache.kafka.common.errors.UnsupportedVersionException:
> > >>> > Directory assignment is not supported yet.
> > >>> >
> > >>> > Is that expected? I guess with the metadata version set to 3.6-IV2,
> > it
> > >>> > makes sense that the request is not supported. But shouldn't then
> the
> > >>> > request not be sent at all by the brokers? (I did not opened a JIRA
> > for
> > >>> it,
> > >>> > but I can open one if you agree this is not expected)
> > >>> >
> > >>> > Thanks & Regards
> > >>> > Jakub
> > >>> >
> > >>> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com>
> wrote:
> > >>> >
> > >>> > > Hi Stanislav,
> > >>> > >
> > >>> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
> > you
> > >>> > > missed it.
> > >>> > > cross-posting here:
> > >>> > >
> > >>> > > There is a bug KAFKA-16101
> > >>> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting
> that
> > >>> > "Kafka
> > >>> > > cluster will be unavailable during KRaft migration rollback".
> > >>> > > The impact for this issue is that if brokers try to rollback to
> ZK
> > mode
> > >>> > > during KRaft migration process, there will be a period of time
> the
> > >>> > cluster
> > >>> > > is unavailable.
> > >>> > > Since ZK migrating to KRaft feature is a production ready
> feature,
> > I
> > >>> > think
> > >>> > > this should be addressed soon.
> > >>> > > Do you think this is a blocker for v3.7.0?
> > >>> > >
> > >>> > > Thanks.
> > >>> > > Luke
> > >>> > >
> > >>> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> > fearthecellos@gmail.com
> > >>> >
> > >>> > > wrote:
> > >>> > >
> > >>> > > > Thanks, Kirk!
> > >>> > > >
> > >>> > > > @Stanislav--do you believe that this warrants a new RC?
> > >>> > > >
> > >>> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro>
> wrote:
> > >>> > > >
> > >>> > > > > Hi Chris/Stanislav,
> > >>> > > > >
> > >>> > > > > I'm working on the 'Unable to find FetchSessionHandler' log
> > problem
> > >>> > > > > (KAFKA-16029) and have put out a draft PR (
> > >>> > > > > https://github.com/apache/kafka/pull/15186). I will use the
> > >>> > quickstart
> > >>> > > > > approach as a second means to reproduce/verify while I wait
> > for the
> > >>> > > PR's
> > >>> > > > > Jenkins job to finish.
> > >>> > > > >
> > >>> > > > > Thanks,
> > >>> > > > > Kirk
> > >>> > > > >
> > >>> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > >>> > > > > > Hi Stanislav,
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > Thanks for running this release!
> > >>> > > > > >
> > >>> > > > > > To verify, I:
> > >>> > > > > > - Built from source using Java 11 with both:
> > >>> > > > > > - - the 3.7.0-rc2 tag on GitHub
> > >>> > > > > > - - the kafka-3.7.0-src.tgz artifact from
> > >>> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>> > > > > > - Checked signatures and checksums
> > >>> > > > > > - Ran the quickstart using both:
> > >>> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> > >>> > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>> with
> > >>> > > Java
> > >>> > > > > 11
> > >>> > > > > > and Scala 13 in KRaft mode
> > >>> > > > > > - - Our shiny new broker Docker image,
> apache/kafka:3.7.0-rc2
> > >>> > > > > > - Ran all unit tests
> > >>> > > > > > - Ran all integration tests for Connect and MM2
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > I found two minor areas for concern:
> > >>> > > > > >
> > >>> > > > > > 1. (Possibly a blocker)
> > >>> > > > > > When running the quickstart, I noticed this ERROR-level log
> > >>> message
> > >>> > > > being
> > >>> > > > > > emitted frequently (not not every time) when I killed my
> > console
> > >>> > > > consumer
> > >>> > > > > > via ctrl-C:
> > >>> > > > > >
> > >>> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> > >>> > > clientId=console-consumer,
> > >>> > > > > > groupId=console-consumer-74388] Unable to find
> > >>> FetchSessionHandler
> > >>> > > for
> > >>> > > > > node
> > >>> > > > > > 1. Ignoring fetch response
> > >>> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >>> > > > > >
> > >>> > > > > > I see that this error message is already reported in
> > >>> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think
> > we
> > >>> > should
> > >>> > > > > > prioritize fixing it for this release. I know it's probably
> > >>> benign
> > >>> > > but
> > >>> > > > > it's
> > >>> > > > > > really not a good look for us when basic operations log
> error
> > >>> > > messages,
> > >>> > > > > and
> > >>> > > > > > it may give new users some headaches.
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > 2. (Probably not a blocker)
> > >>> > > > > > The following unit tests failed the first time around, and
> > all of
> > >>> > > them
> > >>> > > > > > passed the second time I ran them:
> > >>> > > > > >
> > >>> > > > > > - (clients)
> > >>> > > > >
> > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > >>> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > >>> > > > > > - (clients)
> Tls13SelectorTest.testConnectionsByClientMetric()
> > >>> > > > > > - (connect)
> > >>> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > >>> > > > (I
> > >>> > > > > > thought I fixed this one! 🤬🤬)
> > >>> > > > > > - (core)
> > ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > Thanks again for your work on this release, and
> > congratulations
> > >>> to
> > >>> > > > Kafka
> > >>> > > > > > Streams for having zero flaky unit tests during my
> > >>> > > highly-experimental
> > >>> > > > > > single laptop run!
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > Cheers,
> > >>> > > > > >
> > >>> > > > > > Chris
> > >>> > > > > >
> > >>> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > >>> > > > > > <st...@confluent.io.invalid> wrote:
> > >>> > > > > >
> > >>> > > > > > > Hello Kafka users, developers, and client-developers,
> > >>> > > > > > >
> > >>> > > > > > > This is the first candidate for release of Apache Kafka
> > 3.7.0.
> > >>> > > > > > >
> > >>> > > > > > > Note it's named "RC2" because I had a few "failed" RCs
> > that I
> > >>> had
> > >>> > > > > > > cut/uploaded but ultimately had to scrap prior to
> > announcing
> > >>> due
> > >>> > to
> > >>> > > > new
> > >>> > > > > > > blockers arriving before I could even announce them.
> > >>> > > > > > >
> > >>> > > > > > > Further - I haven't yet been able to set up the system
> > tests
> > >>> > > > > successfully.
> > >>> > > > > > > And the integration/unit tests do have a few failures
> that
> > I
> > >>> have
> > >>> > > to
> > >>> > > > > spend
> > >>> > > > > > > time triaging. I would appreciate any help in case anyone
> > >>> notices
> > >>> > > any
> > >>> > > > > tests
> > >>> > > > > > > failing that they're subject matters experts in. Expect
> me
> > to
> > >>> > > follow
> > >>> > > > > up in
> > >>> > > > > > > a day or two with more detailed analysis.
> > >>> > > > > > >
> > >>> > > > > > > Major changes include:
> > >>> > > > > > > - Early Access to KIP-848 - the next generation of the
> > consumer
> > >>> > > > > rebalance
> > >>> > > > > > > protocol
> > >>> > > > > > > - KIP-858: Adding JBOD support to KRaft
> > >>> > > > > > > - KIP-714: Observability into Client metrics via a
> > standardized
> > >>> > > > > interface
> > >>> > > > > > >
> > >>> > > > > > > Check more information in the WIP blog post:
> > >>> > > > > > > https://github.com/apache/kafka-site/pull/578
> > >>> > > > > > >
> > >>> > > > > > > Release notes for the 3.7.0 release:
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >>> > > > > > >
> > >>> > > > > > > *** Please download, test and vote by Thursday, January
> > 18, 9am
> > >>> > PT
> > >>> > > > ***
> > >>> > > > > > >
> > >>> > > > > > > Usually these deadlines tend to be 2-3 days, but due to
> > this
> > >>> > being
> > >>> > > > the
> > >>> > > > > > > first RC and the tests not having ran yet, I am giving it
> > a bit
> > >>> > > more
> > >>> > > > > time.
> > >>> > > > > > >
> > >>> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > >>> release:
> > >>> > > > > > > https://kafka.apache.org/KEYS
> > >>> > > > > > >
> > >>> > > > > > > * Release artifacts to be voted upon (source and binary):
> > >>> > > > > > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >>> > > > > > >
> > >>> > > > > > > * Docker release artifact to be voted upon:
> > >>> > > > > > > apache/kafka:3.7.0-rc2
> > >>> > > > > > >
> > >>> > > > > > > * Maven artifacts to be voted upon:
> > >>> > > > > > >
> > >>> > > >
> > >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>> > > > > > >
> > >>> > > > > > > * Javadoc:
> > >>> > > > > > >
> > >>> > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >>> > > > > > >
> > >>> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > >>> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >>> > > > > > >
> > >>> > > > > > > * Documentation:
> > >>> > > > > > > https://kafka.apache.org/37/documentation.html
> > >>> > > > > > >
> > >>> > > > > > > * Protocol:
> > >>> > > > > > > https://kafka.apache.org/37/protocol.html
> > >>> > > > > > >
> > >>> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> > >>> > > > > > > Unit/integration tests:
> > >>> > > > > > >
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > >>> > > > > > > There are failing tests here. I have to follow up with
> > triaging
> > >>> > > some
> > >>> > > > of
> > >>> > > > > > > the failures and figuring out if they're actual problems
> or
> > >>> > simply
> > >>> > > > > flakes.
> > >>> > > > > > >
> > >>> > > > > > > System tests:
> > >>> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >>> > > > > > >
> > >>> > > > > > > No successful system test runs yet. I am working on
> > getting the
> > >>> > job
> > >>> > > > to
> > >>> > > > > run.
> > >>> > > > > > >
> > >>> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> > >>> branch:
> > >>> > > > > > > Attached are the scan_report and report_jvm output files
> > from
> > >>> the
> > >>> > > > > Docker
> > >>> > > > > > > Build run:
> > >>> > > > > > >
> > >>> > > > >
> > >>> > >
> > >>>
> > https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >>> > > > > > >
> > >>> > > > > > > And the final docker image build job - Docker Build Test
> > >>> > Pipeline:
> > >>> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> > >>> > > > > > >
> > >>> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> > >>> > > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >>> > > > > > >
> > >>> > > > > > > /**************************************
> > >>> > > > > > >
> > >>> > > > > > > Thanks,
> > >>> > > > > > > Stanislav Kozlovski
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>>
> > >>> --
> > >>> Best,
> > >>> Stanislav
> > >>>
> >
>
>
> --
> Paolo Patierno
>
> *Senior Principal Software Engineer @ Red Hat**Microsoft MVP on **Azure*
>
> Twitter : @ppatierno <http://twitter.com/ppatierno>
> Linkedin : paolopatierno <http://it.linkedin.com/in/paolopatierno>
> GitHub : ppatierno <https://github.com/ppatierno>
>

Re: [VOTE] 3.7.0 RC2

Posted by Paolo Patierno <pa...@gmail.com>.
Hi Colin,
I was the one raising the issue about rollback and I also already tried
what you mentioned but with no success.
During the first rolling, I left
the zookeeper.metadata.migration.enable=true but
removed controller.quorum.voters and controller.listener.names.
This is what I get from the brokers on restarting:

2024-01-15 09:19:14,172] ERROR Exiting Kafka due to fatal exception
(kafka.Kafka$)
org.apache.kafka.common.config.ConfigException: If using
zookeeper.metadata.migration.enable, controller.quorum.voters must contain
a parseable set of voters.
        at
kafka.server.KafkaConfig.validateNonEmptyQuorumVotersForMigration$1(KafkaConfig.scala:2286)
        at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2371)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2233)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1604)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1527)
        at kafka.Kafka$.buildServer(Kafka.scala:72)
        at kafka.Kafka$.main(Kafka.scala:91)
        at kafka.Kafka.main(Kafka.scala)

Did you try it?
Am I missing anything in your procedure?

Thanks,
Paolo

On Mon, 15 Jan 2024 at 09:13, Colin McCabe <co...@cmccabe.xyz> wrote:

> Docs fix discussed in the thread is here:
> https://github.com/apache/kafka/pull/15193
>
> best,
> Colin
>
>
> On Sun, Jan 14, 2024, at 23:56, Colin McCabe wrote:
> > Hi Stanislav,
> >
> > Thanks for making the first RC. The fact that it's titled RC2 is
> > messing with my mind a bit. I hope this doesn't make people think that
> > we're farther along than we are, heh.
> >
> > On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> >> *> Nice catch! It does seem like we should have gated this behind the
> >> metadata> version as KIP-858 implies. Is the cluster configured with
> >> multiple log> dirs? What is the impact of the error messages?*
> >>
> >> I did not observe any obvious impact. I was able to send and receive
> >> messages as normally. But to be honest, I have no idea what else
> >> this might impact, so I did not try anything special.
> >>
> >> I think everyone upgrading an existing KRaft cluster will go through
> this
> >> stage (running Kafka 3.7 with an older metadata version for at least a
> >> while). So even if it is just a logged exception without any other
> impact I
> >> wonder if it might scare users from upgrading. But I leave it to others
> to
> >> decide if this is a blocker or not.
> >>
> >
> > Hi Jakub,
> >
> > Thanks for trying the RC. I think what you found is a blocker bug
> > because it will generate huge amount of logspam. I guess we didn't find
> > it in junit tests since logspam doesn't fail the automated tests. But
> > certainly it's not suitable for production. Did you file a JIRA yet?
> >
> >> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> >> <st...@confluent.io.invalid> wrote:
> >>
> >>> Hey Luke,
> >>>
> >>> This is an interesting problem. Given the fact that the KIP for having
> a
> >>> 3.8 release passed, I think it weights the scale towards not calling
> this a
> >>> blocker and expecting it to be solved in 3.7.1.
> >>>
> >>> It is unfortunate that it would not seem safe to migrate to KRaft in
> 3.7.0
> >>> (given the inability to rollback safely), but if that's true - the same
> >>> case would apply for 3.6.0. So in any case users w\ould be expected to
> use a
> >>> patch release for this.
> >
> > Hi Luke,
> >
> > Thanks for testing rollback. I think this is a case where the
> > documentation is wrong. The intention was to for the steps to basically
> > be:
> >
> > 1. roll all the brokers into zk mode, but with migration enabled
> > 2. take down the kraft quorum
> > 3. rmr /controller, allowing a hybrid broker to take over.
> > 4. roll all the brokers into zk mode without migration enabled (if
> desired)
> >
> > With these steps, there isn't really unavailability since a ZK
> > controller can be elected quickly after the kraft quorum is gone.
> >
> >>> Further, since we will have a 3.8 release - it is
> >>> likely we will ultimately recommend users upgrade from that version
> given
> >>> its aim is to have strategic KRaft feature parity with ZK.
> >>> That being said, I am not 100% on this. Let me know whether you think
> this
> >>> should block the release, Luke. I am also tagging Colin and David to
> weigh
> >>> in with their opinions, as they worked on the migration logic.
> >
> > The rollback docs are new in 3.7 so the fact that they're wrong is a
> > clear blocker, I think. But easy to fix, I believe. I will create a PR.
> >
> > best,
> > Colin
> >
> >>>
> >>> Hey Kirk and Chris,
> >>>
> >>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due
> to
> >>> improper closing. And the PR description implies this has been present
> >>> since 3.5. While annoying, I don't see a strong reason for this to
> block
> >>> the release.
> >>>
> >>> Hey Jakub,
> >>>
> >>> Nice catch! It does seem like we should have gated this behind the
> metadata
> >>> version as KIP-858 implies. Is the cluster configured with multiple log
> >>> dirs? What is the impact of the error messages?
> >>>
> >>> Tagging Igor (the author of the KIP) to weigh in.
> >>>
> >>> Best,
> >>> Stanislav
> >>>
> >>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > I was trying the RC2 and run into the following issue ... when I run
> >>> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
> >>> > version, I seem to be getting repeated errors like this in the
> controller
> >>> > logs:
> >>> >
> >>> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> >>> assignReplicasToDirs:
> >>> > event failed with UnsupportedVersionException in 15 microseconds.
> >>> > (org.apache.kafka.controller.QuorumController)
> >>> > [quorum-controller-0-event-handler]
> >>> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
> error
> >>> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> >>> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> >>> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> >>> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> >>> > partitions=[PartitionData(partitionIndex=2),
> >>> > PartitionData(partitionIndex=1)]),
> >>> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> >>> > partitions=[PartitionData(partitionIndex=0)])])]) with context
> >>> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> >>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> >>> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
> clientAddress=/
> >>> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> >>> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> >>> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> >>> > softwareVersion=3.7.0), fromPrivilegedListener=false,
> >>> >
> >>> >
> >>>
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> >>> > ])
> >>> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> >>> > java.util.concurrent.CompletionException:
> >>> > org.apache.kafka.common.errors.UnsupportedVersionException: Directory
> >>> > assignment is not supported yet.
> >>> >
> >>> >  at
> >>> >
> >>> >
> >>>
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> >>> >  at
> >>> >
> >>> >
> >>>
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> >>> >  at
> >>> >
> >>> >
> >>>
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> >>> >  at
> >>> >
> >>> >
> >>>
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> >>> >  at
> >>> >
> >>> >
> >>>
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> >>> >  at
> >>> >
> >>> >
> >>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> >>> >  at
> >>> >
> >>> >
> >>>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> >>> >  at
> >>> >
> >>> >
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> >>> >  at
> >>> >
> >>> >
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> >>> >  at
> >>> >
> >>> >
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> >>> >  at
> >>> >
> >>> >
> >>>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> >>> >  at java.base/java.lang.Thread.run(Thread.java:840)
> >>> >
> >>> > Caused by:
> org.apache.kafka.common.errors.UnsupportedVersionException:
> >>> > Directory assignment is not supported yet.
> >>> >
> >>> > Is that expected? I guess with the metadata version set to 3.6-IV2,
> it
> >>> > makes sense that the request is not supported. But shouldn't then the
> >>> > request not be sent at all by the brokers? (I did not opened a JIRA
> for
> >>> it,
> >>> > but I can open one if you agree this is not expected)
> >>> >
> >>> > Thanks & Regards
> >>> > Jakub
> >>> >
> >>> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com> wrote:
> >>> >
> >>> > > Hi Stanislav,
> >>> > >
> >>> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
> you
> >>> > > missed it.
> >>> > > cross-posting here:
> >>> > >
> >>> > > There is a bug KAFKA-16101
> >>> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that
> >>> > "Kafka
> >>> > > cluster will be unavailable during KRaft migration rollback".
> >>> > > The impact for this issue is that if brokers try to rollback to ZK
> mode
> >>> > > during KRaft migration process, there will be a period of time the
> >>> > cluster
> >>> > > is unavailable.
> >>> > > Since ZK migrating to KRaft feature is a production ready feature,
> I
> >>> > think
> >>> > > this should be addressed soon.
> >>> > > Do you think this is a blocker for v3.7.0?
> >>> > >
> >>> > > Thanks.
> >>> > > Luke
> >>> > >
> >>> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
> fearthecellos@gmail.com
> >>> >
> >>> > > wrote:
> >>> > >
> >>> > > > Thanks, Kirk!
> >>> > > >
> >>> > > > @Stanislav--do you believe that this warrants a new RC?
> >>> > > >
> >>> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
> >>> > > >
> >>> > > > > Hi Chris/Stanislav,
> >>> > > > >
> >>> > > > > I'm working on the 'Unable to find FetchSessionHandler' log
> problem
> >>> > > > > (KAFKA-16029) and have put out a draft PR (
> >>> > > > > https://github.com/apache/kafka/pull/15186). I will use the
> >>> > quickstart
> >>> > > > > approach as a second means to reproduce/verify while I wait
> for the
> >>> > > PR's
> >>> > > > > Jenkins job to finish.
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > > Kirk
> >>> > > > >
> >>> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> >>> > > > > > Hi Stanislav,
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Thanks for running this release!
> >>> > > > > >
> >>> > > > > > To verify, I:
> >>> > > > > > - Built from source using Java 11 with both:
> >>> > > > > > - - the 3.7.0-rc2 tag on GitHub
> >>> > > > > > - - the kafka-3.7.0-src.tgz artifact from
> >>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>> > > > > > - Checked signatures and checksums
> >>> > > > > > - Ran the quickstart using both:
> >>> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> >>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>> with
> >>> > > Java
> >>> > > > > 11
> >>> > > > > > and Scala 13 in KRaft mode
> >>> > > > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> >>> > > > > > - Ran all unit tests
> >>> > > > > > - Ran all integration tests for Connect and MM2
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > I found two minor areas for concern:
> >>> > > > > >
> >>> > > > > > 1. (Possibly a blocker)
> >>> > > > > > When running the quickstart, I noticed this ERROR-level log
> >>> message
> >>> > > > being
> >>> > > > > > emitted frequently (not not every time) when I killed my
> console
> >>> > > > consumer
> >>> > > > > > via ctrl-C:
> >>> > > > > >
> >>> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> >>> > > clientId=console-consumer,
> >>> > > > > > groupId=console-consumer-74388] Unable to find
> >>> FetchSessionHandler
> >>> > > for
> >>> > > > > node
> >>> > > > > > 1. Ignoring fetch response
> >>> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> >>> > > > > >
> >>> > > > > > I see that this error message is already reported in
> >>> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think
> we
> >>> > should
> >>> > > > > > prioritize fixing it for this release. I know it's probably
> >>> benign
> >>> > > but
> >>> > > > > it's
> >>> > > > > > really not a good look for us when basic operations log error
> >>> > > messages,
> >>> > > > > and
> >>> > > > > > it may give new users some headaches.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > 2. (Probably not a blocker)
> >>> > > > > > The following unit tests failed the first time around, and
> all of
> >>> > > them
> >>> > > > > > passed the second time I ran them:
> >>> > > > > >
> >>> > > > > > - (clients)
> >>> > > > >
> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> >>> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> >>> > > > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> >>> > > > > > - (connect)
> >>> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> >>> > > > (I
> >>> > > > > > thought I fixed this one! 🤬🤬)
> >>> > > > > > - (core)
> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Thanks again for your work on this release, and
> congratulations
> >>> to
> >>> > > > Kafka
> >>> > > > > > Streams for having zero flaky unit tests during my
> >>> > > highly-experimental
> >>> > > > > > single laptop run!
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Cheers,
> >>> > > > > >
> >>> > > > > > Chris
> >>> > > > > >
> >>> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> >>> > > > > > <st...@confluent.io.invalid> wrote:
> >>> > > > > >
> >>> > > > > > > Hello Kafka users, developers, and client-developers,
> >>> > > > > > >
> >>> > > > > > > This is the first candidate for release of Apache Kafka
> 3.7.0.
> >>> > > > > > >
> >>> > > > > > > Note it's named "RC2" because I had a few "failed" RCs
> that I
> >>> had
> >>> > > > > > > cut/uploaded but ultimately had to scrap prior to
> announcing
> >>> due
> >>> > to
> >>> > > > new
> >>> > > > > > > blockers arriving before I could even announce them.
> >>> > > > > > >
> >>> > > > > > > Further - I haven't yet been able to set up the system
> tests
> >>> > > > > successfully.
> >>> > > > > > > And the integration/unit tests do have a few failures that
> I
> >>> have
> >>> > > to
> >>> > > > > spend
> >>> > > > > > > time triaging. I would appreciate any help in case anyone
> >>> notices
> >>> > > any
> >>> > > > > tests
> >>> > > > > > > failing that they're subject matters experts in. Expect me
> to
> >>> > > follow
> >>> > > > > up in
> >>> > > > > > > a day or two with more detailed analysis.
> >>> > > > > > >
> >>> > > > > > > Major changes include:
> >>> > > > > > > - Early Access to KIP-848 - the next generation of the
> consumer
> >>> > > > > rebalance
> >>> > > > > > > protocol
> >>> > > > > > > - KIP-858: Adding JBOD support to KRaft
> >>> > > > > > > - KIP-714: Observability into Client metrics via a
> standardized
> >>> > > > > interface
> >>> > > > > > >
> >>> > > > > > > Check more information in the WIP blog post:
> >>> > > > > > > https://github.com/apache/kafka-site/pull/578
> >>> > > > > > >
> >>> > > > > > > Release notes for the 3.7.0 release:
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> >>> > > > > > >
> >>> > > > > > > *** Please download, test and vote by Thursday, January
> 18, 9am
> >>> > PT
> >>> > > > ***
> >>> > > > > > >
> >>> > > > > > > Usually these deadlines tend to be 2-3 days, but due to
> this
> >>> > being
> >>> > > > the
> >>> > > > > > > first RC and the tests not having ran yet, I am giving it
> a bit
> >>> > > more
> >>> > > > > time.
> >>> > > > > > >
> >>> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> >>> release:
> >>> > > > > > > https://kafka.apache.org/KEYS
> >>> > > > > > >
> >>> > > > > > > * Release artifacts to be voted upon (source and binary):
> >>> > > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>> > > > > > >
> >>> > > > > > > * Docker release artifact to be voted upon:
> >>> > > > > > > apache/kafka:3.7.0-rc2
> >>> > > > > > >
> >>> > > > > > > * Maven artifacts to be voted upon:
> >>> > > > > > >
> >>> > > >
> >>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>> > > > > > >
> >>> > > > > > > * Javadoc:
> >>> > > > > > >
> >>> > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> >>> > > > > > >
> >>> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> >>> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> >>> > > > > > >
> >>> > > > > > > * Documentation:
> >>> > > > > > > https://kafka.apache.org/37/documentation.html
> >>> > > > > > >
> >>> > > > > > > * Protocol:
> >>> > > > > > > https://kafka.apache.org/37/protocol.html
> >>> > > > > > >
> >>> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> >>> > > > > > > Unit/integration tests:
> >>> > > > > > >
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> >>> > > > > > > There are failing tests here. I have to follow up with
> triaging
> >>> > > some
> >>> > > > of
> >>> > > > > > > the failures and figuring out if they're actual problems or
> >>> > simply
> >>> > > > > flakes.
> >>> > > > > > >
> >>> > > > > > > System tests:
> >>> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> >>> > > > > > >
> >>> > > > > > > No successful system test runs yet. I am working on
> getting the
> >>> > job
> >>> > > > to
> >>> > > > > run.
> >>> > > > > > >
> >>> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> >>> branch:
> >>> > > > > > > Attached are the scan_report and report_jvm output files
> from
> >>> the
> >>> > > > > Docker
> >>> > > > > > > Build run:
> >>> > > > > > >
> >>> > > > >
> >>> > >
> >>>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> >>> > > > > > >
> >>> > > > > > > And the final docker image build job - Docker Build Test
> >>> > Pipeline:
> >>> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> >>> > > > > > >
> >>> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> >>> > > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> >>> > > > > > >
> >>> > > > > > > /**************************************
> >>> > > > > > >
> >>> > > > > > > Thanks,
> >>> > > > > > > Stanislav Kozlovski
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>>
> >>> --
> >>> Best,
> >>> Stanislav
> >>>
>


-- 
Paolo Patierno

*Senior Principal Software Engineer @ Red Hat**Microsoft MVP on **Azure*

Twitter : @ppatierno <http://twitter.com/ppatierno>
Linkedin : paolopatierno <http://it.linkedin.com/in/paolopatierno>
GitHub : ppatierno <https://github.com/ppatierno>

Re: [VOTE] 3.7.0 RC2

Posted by Colin McCabe <co...@cmccabe.xyz>.
Docs fix discussed in the thread is here: https://github.com/apache/kafka/pull/15193

best,
Colin


On Sun, Jan 14, 2024, at 23:56, Colin McCabe wrote:
> Hi Stanislav,
>
> Thanks for making the first RC. The fact that it's titled RC2 is 
> messing with my mind a bit. I hope this doesn't make people think that 
> we're farther along than we are, heh.
>
> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>> *> Nice catch! It does seem like we should have gated this behind the
>> metadata> version as KIP-858 implies. Is the cluster configured with
>> multiple log> dirs? What is the impact of the error messages?*
>>
>> I did not observe any obvious impact. I was able to send and receive
>> messages as normally. But to be honest, I have no idea what else
>> this might impact, so I did not try anything special.
>>
>> I think everyone upgrading an existing KRaft cluster will go through this
>> stage (running Kafka 3.7 with an older metadata version for at least a
>> while). So even if it is just a logged exception without any other impact I
>> wonder if it might scare users from upgrading. But I leave it to others to
>> decide if this is a blocker or not.
>>
>
> Hi Jakub,
>
> Thanks for trying the RC. I think what you found is a blocker bug 
> because it will generate huge amount of logspam. I guess we didn't find 
> it in junit tests since logspam doesn't fail the automated tests. But 
> certainly it's not suitable for production. Did you file a JIRA yet?
>
>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>> <st...@confluent.io.invalid> wrote:
>>
>>> Hey Luke,
>>>
>>> This is an interesting problem. Given the fact that the KIP for having a
>>> 3.8 release passed, I think it weights the scale towards not calling this a
>>> blocker and expecting it to be solved in 3.7.1.
>>>
>>> It is unfortunate that it would not seem safe to migrate to KRaft in 3.7.0
>>> (given the inability to rollback safely), but if that's true - the same
>>> case would apply for 3.6.0. So in any case users w\ould be expected to use a
>>> patch release for this.
>
> Hi Luke,
>
> Thanks for testing rollback. I think this is a case where the 
> documentation is wrong. The intention was to for the steps to basically 
> be:
>
> 1. roll all the brokers into zk mode, but with migration enabled
> 2. take down the kraft quorum
> 3. rmr /controller, allowing a hybrid broker to take over.
> 4. roll all the brokers into zk mode without migration enabled (if desired)
>
> With these steps, there isn't really unavailability since a ZK 
> controller can be elected quickly after the kraft quorum is gone.
>
>>> Further, since we will have a 3.8 release - it is
>>> likely we will ultimately recommend users upgrade from that version given
>>> its aim is to have strategic KRaft feature parity with ZK.
>>> That being said, I am not 100% on this. Let me know whether you think this
>>> should block the release, Luke. I am also tagging Colin and David to weigh
>>> in with their opinions, as they worked on the migration logic.
>
> The rollback docs are new in 3.7 so the fact that they're wrong is a 
> clear blocker, I think. But easy to fix, I believe. I will create a PR.
>
> best,
> Colin
>
>>>
>>> Hey Kirk and Chris,
>>>
>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due to
>>> improper closing. And the PR description implies this has been present
>>> since 3.5. While annoying, I don't see a strong reason for this to block
>>> the release.
>>>
>>> Hey Jakub,
>>>
>>> Nice catch! It does seem like we should have gated this behind the metadata
>>> version as KIP-858 implies. Is the cluster configured with multiple log
>>> dirs? What is the impact of the error messages?
>>>
>>> Tagging Igor (the author of the KIP) to weigh in.
>>>
>>> Best,
>>> Stanislav
>>>
>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
>>>
>>> > Hi,
>>> >
>>> > I was trying the RC2 and run into the following issue ... when I run
>>> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
>>> > version, I seem to be getting repeated errors like this in the controller
>>> > logs:
>>> >
>>> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>>> assignReplicasToDirs:
>>> > event failed with UnsupportedVersionException in 15 microseconds.
>>> > (org.apache.kafka.controller.QuorumController)
>>> > [quorum-controller-0-event-handler]
>>> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
>>> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
>>> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>>> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>>> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>>> > partitions=[PartitionData(partitionIndex=2),
>>> > PartitionData(partitionIndex=1)]),
>>> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>>> > partitions=[PartitionData(partitionIndex=0)])])]) with context
>>> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>>> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7', clientAddress=/
>>> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>>> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
>>> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
>>> > softwareVersion=3.7.0), fromPrivilegedListener=false,
>>> >
>>> >
>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>>> > ])
>>> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>>> > java.util.concurrent.CompletionException:
>>> > org.apache.kafka.common.errors.UnsupportedVersionException: Directory
>>> > assignment is not supported yet.
>>> >
>>> >  at
>>> >
>>> >
>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>>> >  at
>>> >
>>> >
>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>>> >  at
>>> >
>>> >
>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>>> >  at
>>> >
>>> >
>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>>> >  at
>>> >
>>> >
>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>>> >  at
>>> >
>>> >
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>>> >  at
>>> >
>>> >
>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>>> >  at
>>> >
>>> >
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>>> >  at
>>> >
>>> >
>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>>> >  at
>>> >
>>> >
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>>> >  at
>>> >
>>> >
>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>>> >  at java.base/java.lang.Thread.run(Thread.java:840)
>>> >
>>> > Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
>>> > Directory assignment is not supported yet.
>>> >
>>> > Is that expected? I guess with the metadata version set to 3.6-IV2, it
>>> > makes sense that the request is not supported. But shouldn't then the
>>> > request not be sent at all by the brokers? (I did not opened a JIRA for
>>> it,
>>> > but I can open one if you agree this is not expected)
>>> >
>>> > Thanks & Regards
>>> > Jakub
>>> >
>>> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com> wrote:
>>> >
>>> > > Hi Stanislav,
>>> > >
>>> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe you
>>> > > missed it.
>>> > > cross-posting here:
>>> > >
>>> > > There is a bug KAFKA-16101
>>> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that
>>> > "Kafka
>>> > > cluster will be unavailable during KRaft migration rollback".
>>> > > The impact for this issue is that if brokers try to rollback to ZK mode
>>> > > during KRaft migration process, there will be a period of time the
>>> > cluster
>>> > > is unavailable.
>>> > > Since ZK migrating to KRaft feature is a production ready feature, I
>>> > think
>>> > > this should be addressed soon.
>>> > > Do you think this is a blocker for v3.7.0?
>>> > >
>>> > > Thanks.
>>> > > Luke
>>> > >
>>> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <fearthecellos@gmail.com
>>> >
>>> > > wrote:
>>> > >
>>> > > > Thanks, Kirk!
>>> > > >
>>> > > > @Stanislav--do you believe that this warrants a new RC?
>>> > > >
>>> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
>>> > > >
>>> > > > > Hi Chris/Stanislav,
>>> > > > >
>>> > > > > I'm working on the 'Unable to find FetchSessionHandler' log problem
>>> > > > > (KAFKA-16029) and have put out a draft PR (
>>> > > > > https://github.com/apache/kafka/pull/15186). I will use the
>>> > quickstart
>>> > > > > approach as a second means to reproduce/verify while I wait for the
>>> > > PR's
>>> > > > > Jenkins job to finish.
>>> > > > >
>>> > > > > Thanks,
>>> > > > > Kirk
>>> > > > >
>>> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>>> > > > > > Hi Stanislav,
>>> > > > > >
>>> > > > > >
>>> > > > > > Thanks for running this release!
>>> > > > > >
>>> > > > > > To verify, I:
>>> > > > > > - Built from source using Java 11 with both:
>>> > > > > > - - the 3.7.0-rc2 tag on GitHub
>>> > > > > > - - the kafka-3.7.0-src.tgz artifact from
>>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>> > > > > > - Checked signatures and checksums
>>> > > > > > - Ran the quickstart using both:
>>> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
>>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>> with
>>> > > Java
>>> > > > > 11
>>> > > > > > and Scala 13 in KRaft mode
>>> > > > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
>>> > > > > > - Ran all unit tests
>>> > > > > > - Ran all integration tests for Connect and MM2
>>> > > > > >
>>> > > > > >
>>> > > > > > I found two minor areas for concern:
>>> > > > > >
>>> > > > > > 1. (Possibly a blocker)
>>> > > > > > When running the quickstart, I noticed this ERROR-level log
>>> message
>>> > > > being
>>> > > > > > emitted frequently (not not every time) when I killed my console
>>> > > > consumer
>>> > > > > > via ctrl-C:
>>> > > > > >
>>> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
>>> > > clientId=console-consumer,
>>> > > > > > groupId=console-consumer-74388] Unable to find
>>> FetchSessionHandler
>>> > > for
>>> > > > > node
>>> > > > > > 1. Ignoring fetch response
>>> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>>> > > > > >
>>> > > > > > I see that this error message is already reported in
>>> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think we
>>> > should
>>> > > > > > prioritize fixing it for this release. I know it's probably
>>> benign
>>> > > but
>>> > > > > it's
>>> > > > > > really not a good look for us when basic operations log error
>>> > > messages,
>>> > > > > and
>>> > > > > > it may give new users some headaches.
>>> > > > > >
>>> > > > > >
>>> > > > > > 2. (Probably not a blocker)
>>> > > > > > The following unit tests failed the first time around, and all of
>>> > > them
>>> > > > > > passed the second time I ran them:
>>> > > > > >
>>> > > > > > - (clients)
>>> > > > > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>>> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
>>> > > > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
>>> > > > > > - (connect)
>>> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>>> > > > (I
>>> > > > > > thought I fixed this one! 🤬🤬)
>>> > > > > > - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>>> > > > > >
>>> > > > > >
>>> > > > > > Thanks again for your work on this release, and congratulations
>>> to
>>> > > > Kafka
>>> > > > > > Streams for having zero flaky unit tests during my
>>> > > highly-experimental
>>> > > > > > single laptop run!
>>> > > > > >
>>> > > > > >
>>> > > > > > Cheers,
>>> > > > > >
>>> > > > > > Chris
>>> > > > > >
>>> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>>> > > > > > <st...@confluent.io.invalid> wrote:
>>> > > > > >
>>> > > > > > > Hello Kafka users, developers, and client-developers,
>>> > > > > > >
>>> > > > > > > This is the first candidate for release of Apache Kafka 3.7.0.
>>> > > > > > >
>>> > > > > > > Note it's named "RC2" because I had a few "failed" RCs that I
>>> had
>>> > > > > > > cut/uploaded but ultimately had to scrap prior to announcing
>>> due
>>> > to
>>> > > > new
>>> > > > > > > blockers arriving before I could even announce them.
>>> > > > > > >
>>> > > > > > > Further - I haven't yet been able to set up the system tests
>>> > > > > successfully.
>>> > > > > > > And the integration/unit tests do have a few failures that I
>>> have
>>> > > to
>>> > > > > spend
>>> > > > > > > time triaging. I would appreciate any help in case anyone
>>> notices
>>> > > any
>>> > > > > tests
>>> > > > > > > failing that they're subject matters experts in. Expect me to
>>> > > follow
>>> > > > > up in
>>> > > > > > > a day or two with more detailed analysis.
>>> > > > > > >
>>> > > > > > > Major changes include:
>>> > > > > > > - Early Access to KIP-848 - the next generation of the consumer
>>> > > > > rebalance
>>> > > > > > > protocol
>>> > > > > > > - KIP-858: Adding JBOD support to KRaft
>>> > > > > > > - KIP-714: Observability into Client metrics via a standardized
>>> > > > > interface
>>> > > > > > >
>>> > > > > > > Check more information in the WIP blog post:
>>> > > > > > > https://github.com/apache/kafka-site/pull/578
>>> > > > > > >
>>> > > > > > > Release notes for the 3.7.0 release:
>>> > > > > > >
>>> > > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>>> > > > > > >
>>> > > > > > > *** Please download, test and vote by Thursday, January 18, 9am
>>> > PT
>>> > > > ***
>>> > > > > > >
>>> > > > > > > Usually these deadlines tend to be 2-3 days, but due to this
>>> > being
>>> > > > the
>>> > > > > > > first RC and the tests not having ran yet, I am giving it a bit
>>> > > more
>>> > > > > time.
>>> > > > > > >
>>> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
>>> release:
>>> > > > > > > https://kafka.apache.org/KEYS
>>> > > > > > >
>>> > > > > > > * Release artifacts to be voted upon (source and binary):
>>> > > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>>> > > > > > >
>>> > > > > > > * Docker release artifact to be voted upon:
>>> > > > > > > apache/kafka:3.7.0-rc2
>>> > > > > > >
>>> > > > > > > * Maven artifacts to be voted upon:
>>> > > > > > >
>>> > > >
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> > > > > > >
>>> > > > > > > * Javadoc:
>>> > > > > > >
>>> > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>>> > > > > > >
>>> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>>> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>>> > > > > > >
>>> > > > > > > * Documentation:
>>> > > > > > > https://kafka.apache.org/37/documentation.html
>>> > > > > > >
>>> > > > > > > * Protocol:
>>> > > > > > > https://kafka.apache.org/37/protocol.html
>>> > > > > > >
>>> > > > > > > * Successful Jenkins builds for the 3.7 branch:
>>> > > > > > > Unit/integration tests:
>>> > > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>>> > > > > > > There are failing tests here. I have to follow up with triaging
>>> > > some
>>> > > > of
>>> > > > > > > the failures and figuring out if they're actual problems or
>>> > simply
>>> > > > > flakes.
>>> > > > > > >
>>> > > > > > > System tests:
>>> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>>> > > > > > >
>>> > > > > > > No successful system test runs yet. I am working on getting the
>>> > job
>>> > > > to
>>> > > > > run.
>>> > > > > > >
>>> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
>>> branch:
>>> > > > > > > Attached are the scan_report and report_jvm output files from
>>> the
>>> > > > > Docker
>>> > > > > > > Build run:
>>> > > > > > >
>>> > > > >
>>> > >
>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>>> > > > > > >
>>> > > > > > > And the final docker image build job - Docker Build Test
>>> > Pipeline:
>>> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
>>> > > > > > >
>>> > > > > > > The image is apache/kafka:3.7.0-rc2 -
>>> > > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>>> > > > > > >
>>> > > > > > > /**************************************
>>> > > > > > >
>>> > > > > > > Thanks,
>>> > > > > > > Stanislav Kozlovski
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>>
>>> --
>>> Best,
>>> Stanislav
>>>

Re: [VOTE] 3.7.0 RC2

Posted by Colin McCabe <cm...@apache.org>.
Hi Stanislav,

Thanks for making the first RC. The fact that it's titled RC2 is messing with my mind a bit. I hope this doesn't make people think that we're farther along than we are, heh.

On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
> *> Nice catch! It does seem like we should have gated this behind the
> metadata> version as KIP-858 implies. Is the cluster configured with
> multiple log> dirs? What is the impact of the error messages?*
>
> I did not observe any obvious impact. I was able to send and receive
> messages as normally. But to be honest, I have no idea what else
> this might impact, so I did not try anything special.
>
> I think everyone upgrading an existing KRaft cluster will go through this
> stage (running Kafka 3.7 with an older metadata version for at least a
> while). So even if it is just a logged exception without any other impact I
> wonder if it might scare users from upgrading. But I leave it to others to
> decide if this is a blocker or not.
>

Hi Jakub,

Thanks for trying the RC. I think what you found is a blocker bug because it will generate huge amount of logspam. I guess we didn't find it in junit tests since logspam doesn't fail the automated tests. But certainly it's not suitable for production. Did you file a JIRA yet?

> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
> <st...@confluent.io.invalid> wrote:
>
>> Hey Luke,
>>
>> This is an interesting problem. Given the fact that the KIP for having a
>> 3.8 release passed, I think it weights the scale towards not calling this a
>> blocker and expecting it to be solved in 3.7.1.
>>
>> It is unfortunate that it would not seem safe to migrate to KRaft in 3.7.0
>> (given the inability to rollback safely), but if that's true - the same
>> case would apply for 3.6.0. So in any case users w\ould be expected to use a
>> patch release for this.

Hi Luke,

Thanks for testing rollback. I think this is a case where the documentation is wrong. The intention was to for the steps to basically be:

1. roll all the brokers into zk mode, but with migration enabled
2. take down the kraft quorum
3. rmr /controller, allowing a hybrid broker to take over.
4. roll all the brokers into zk mode without migration enabled (if desired)

With these steps, there isn't really unavailability since a ZK controller can be elected quickly after the kraft quorum is gone.

>> Further, since we will have a 3.8 release - it is
>> likely we will ultimately recommend users upgrade from that version given
>> its aim is to have strategic KRaft feature parity with ZK.
>> That being said, I am not 100% on this. Let me know whether you think this
>> should block the release, Luke. I am also tagging Colin and David to weigh
>> in with their opinions, as they worked on the migration logic.

The rollback docs are new in 3.7 so the fact that they're wrong is a clear blocker, I think. But easy to fix, I believe. I will create a PR.

best,
Colin

>>
>> Hey Kirk and Chris,
>>
>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due to
>> improper closing. And the PR description implies this has been present
>> since 3.5. While annoying, I don't see a strong reason for this to block
>> the release.
>>
>> Hey Jakub,
>>
>> Nice catch! It does seem like we should have gated this behind the metadata
>> version as KIP-858 implies. Is the cluster configured with multiple log
>> dirs? What is the impact of the error messages?
>>
>> Tagging Igor (the author of the KIP) to weigh in.
>>
>> Best,
>> Stanislav
>>
>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
>>
>> > Hi,
>> >
>> > I was trying the RC2 and run into the following issue ... when I run
>> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
>> > version, I seem to be getting repeated errors like this in the controller
>> > logs:
>> >
>> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
>> assignReplicasToDirs:
>> > event failed with UnsupportedVersionException in 15 microseconds.
>> > (org.apache.kafka.controller.QuorumController)
>> > [quorum-controller-0-event-handler]
>> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
>> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
>> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
>> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
>> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
>> > partitions=[PartitionData(partitionIndex=2),
>> > PartitionData(partitionIndex=1)]),
>> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
>> > partitions=[PartitionData(partitionIndex=0)])])]) with context
>> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
>> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
>> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7', clientAddress=/
>> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
>> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
>> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
>> > softwareVersion=3.7.0), fromPrivilegedListener=false,
>> >
>> >
>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
>> > ])
>> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
>> > java.util.concurrent.CompletionException:
>> > org.apache.kafka.common.errors.UnsupportedVersionException: Directory
>> > assignment is not supported yet.
>> >
>> >  at
>> >
>> >
>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>> >  at
>> >
>> >
>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>> >  at
>> >
>> >
>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>> >  at
>> >
>> >
>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>> >  at
>> >
>> >
>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>> >  at
>> >
>> >
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>> >  at
>> >
>> >
>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>> >  at
>> >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>> >  at
>> >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>> >  at
>> >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>> >  at
>> >
>> >
>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>> >  at java.base/java.lang.Thread.run(Thread.java:840)
>> >
>> > Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
>> > Directory assignment is not supported yet.
>> >
>> > Is that expected? I guess with the metadata version set to 3.6-IV2, it
>> > makes sense that the request is not supported. But shouldn't then the
>> > request not be sent at all by the brokers? (I did not opened a JIRA for
>> it,
>> > but I can open one if you agree this is not expected)
>> >
>> > Thanks & Regards
>> > Jakub
>> >
>> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com> wrote:
>> >
>> > > Hi Stanislav,
>> > >
>> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe you
>> > > missed it.
>> > > cross-posting here:
>> > >
>> > > There is a bug KAFKA-16101
>> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that
>> > "Kafka
>> > > cluster will be unavailable during KRaft migration rollback".
>> > > The impact for this issue is that if brokers try to rollback to ZK mode
>> > > during KRaft migration process, there will be a period of time the
>> > cluster
>> > > is unavailable.
>> > > Since ZK migrating to KRaft feature is a production ready feature, I
>> > think
>> > > this should be addressed soon.
>> > > Do you think this is a blocker for v3.7.0?
>> > >
>> > > Thanks.
>> > > Luke
>> > >
>> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <fearthecellos@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Thanks, Kirk!
>> > > >
>> > > > @Stanislav--do you believe that this warrants a new RC?
>> > > >
>> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
>> > > >
>> > > > > Hi Chris/Stanislav,
>> > > > >
>> > > > > I'm working on the 'Unable to find FetchSessionHandler' log problem
>> > > > > (KAFKA-16029) and have put out a draft PR (
>> > > > > https://github.com/apache/kafka/pull/15186). I will use the
>> > quickstart
>> > > > > approach as a second means to reproduce/verify while I wait for the
>> > > PR's
>> > > > > Jenkins job to finish.
>> > > > >
>> > > > > Thanks,
>> > > > > Kirk
>> > > > >
>> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
>> > > > > > Hi Stanislav,
>> > > > > >
>> > > > > >
>> > > > > > Thanks for running this release!
>> > > > > >
>> > > > > > To verify, I:
>> > > > > > - Built from source using Java 11 with both:
>> > > > > > - - the 3.7.0-rc2 tag on GitHub
>> > > > > > - - the kafka-3.7.0-src.tgz artifact from
>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > > > > > - Checked signatures and checksums
>> > > > > > - Ran the quickstart using both:
>> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> with
>> > > Java
>> > > > > 11
>> > > > > > and Scala 13 in KRaft mode
>> > > > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
>> > > > > > - Ran all unit tests
>> > > > > > - Ran all integration tests for Connect and MM2
>> > > > > >
>> > > > > >
>> > > > > > I found two minor areas for concern:
>> > > > > >
>> > > > > > 1. (Possibly a blocker)
>> > > > > > When running the quickstart, I noticed this ERROR-level log
>> message
>> > > > being
>> > > > > > emitted frequently (not not every time) when I killed my console
>> > > > consumer
>> > > > > > via ctrl-C:
>> > > > > >
>> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
>> > > clientId=console-consumer,
>> > > > > > groupId=console-consumer-74388] Unable to find
>> FetchSessionHandler
>> > > for
>> > > > > node
>> > > > > > 1. Ignoring fetch response
>> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
>> > > > > >
>> > > > > > I see that this error message is already reported in
>> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think we
>> > should
>> > > > > > prioritize fixing it for this release. I know it's probably
>> benign
>> > > but
>> > > > > it's
>> > > > > > really not a good look for us when basic operations log error
>> > > messages,
>> > > > > and
>> > > > > > it may give new users some headaches.
>> > > > > >
>> > > > > >
>> > > > > > 2. (Probably not a blocker)
>> > > > > > The following unit tests failed the first time around, and all of
>> > > them
>> > > > > > passed the second time I ran them:
>> > > > > >
>> > > > > > - (clients)
>> > > > > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
>> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
>> > > > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
>> > > > > > - (connect)
>> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
>> > > > (I
>> > > > > > thought I fixed this one! 🤬🤬)
>> > > > > > - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
>> > > > > >
>> > > > > >
>> > > > > > Thanks again for your work on this release, and congratulations
>> to
>> > > > Kafka
>> > > > > > Streams for having zero flaky unit tests during my
>> > > highly-experimental
>> > > > > > single laptop run!
>> > > > > >
>> > > > > >
>> > > > > > Cheers,
>> > > > > >
>> > > > > > Chris
>> > > > > >
>> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
>> > > > > > <st...@confluent.io.invalid> wrote:
>> > > > > >
>> > > > > > > Hello Kafka users, developers, and client-developers,
>> > > > > > >
>> > > > > > > This is the first candidate for release of Apache Kafka 3.7.0.
>> > > > > > >
>> > > > > > > Note it's named "RC2" because I had a few "failed" RCs that I
>> had
>> > > > > > > cut/uploaded but ultimately had to scrap prior to announcing
>> due
>> > to
>> > > > new
>> > > > > > > blockers arriving before I could even announce them.
>> > > > > > >
>> > > > > > > Further - I haven't yet been able to set up the system tests
>> > > > > successfully.
>> > > > > > > And the integration/unit tests do have a few failures that I
>> have
>> > > to
>> > > > > spend
>> > > > > > > time triaging. I would appreciate any help in case anyone
>> notices
>> > > any
>> > > > > tests
>> > > > > > > failing that they're subject matters experts in. Expect me to
>> > > follow
>> > > > > up in
>> > > > > > > a day or two with more detailed analysis.
>> > > > > > >
>> > > > > > > Major changes include:
>> > > > > > > - Early Access to KIP-848 - the next generation of the consumer
>> > > > > rebalance
>> > > > > > > protocol
>> > > > > > > - KIP-858: Adding JBOD support to KRaft
>> > > > > > > - KIP-714: Observability into Client metrics via a standardized
>> > > > > interface
>> > > > > > >
>> > > > > > > Check more information in the WIP blog post:
>> > > > > > > https://github.com/apache/kafka-site/pull/578
>> > > > > > >
>> > > > > > > Release notes for the 3.7.0 release:
>> > > > > > >
>> > > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>> > > > > > >
>> > > > > > > *** Please download, test and vote by Thursday, January 18, 9am
>> > PT
>> > > > ***
>> > > > > > >
>> > > > > > > Usually these deadlines tend to be 2-3 days, but due to this
>> > being
>> > > > the
>> > > > > > > first RC and the tests not having ran yet, I am giving it a bit
>> > > more
>> > > > > time.
>> > > > > > >
>> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
>> release:
>> > > > > > > https://kafka.apache.org/KEYS
>> > > > > > >
>> > > > > > > * Release artifacts to be voted upon (source and binary):
>> > > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>> > > > > > >
>> > > > > > > * Docker release artifact to be voted upon:
>> > > > > > > apache/kafka:3.7.0-rc2
>> > > > > > >
>> > > > > > > * Maven artifacts to be voted upon:
>> > > > > > >
>> > > >
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > > > > > >
>> > > > > > > * Javadoc:
>> > > > > > >
>> > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>> > > > > > >
>> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>> > > > > > >
>> > > > > > > * Documentation:
>> > > > > > > https://kafka.apache.org/37/documentation.html
>> > > > > > >
>> > > > > > > * Protocol:
>> > > > > > > https://kafka.apache.org/37/protocol.html
>> > > > > > >
>> > > > > > > * Successful Jenkins builds for the 3.7 branch:
>> > > > > > > Unit/integration tests:
>> > > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
>> > > > > > > There are failing tests here. I have to follow up with triaging
>> > > some
>> > > > of
>> > > > > > > the failures and figuring out if they're actual problems or
>> > simply
>> > > > > flakes.
>> > > > > > >
>> > > > > > > System tests:
>> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>> > > > > > >
>> > > > > > > No successful system test runs yet. I am working on getting the
>> > job
>> > > > to
>> > > > > run.
>> > > > > > >
>> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
>> branch:
>> > > > > > > Attached are the scan_report and report_jvm output files from
>> the
>> > > > > Docker
>> > > > > > > Build run:
>> > > > > > >
>> > > > >
>> > >
>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>> > > > > > >
>> > > > > > > And the final docker image build job - Docker Build Test
>> > Pipeline:
>> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
>> > > > > > >
>> > > > > > > The image is apache/kafka:3.7.0-rc2 -
>> > > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>> > > > > > >
>> > > > > > > /**************************************
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Stanislav Kozlovski
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Best,
>> Stanislav
>>

Re: [VOTE] 3.7.0 RC2

Posted by Jakub Scholz <ja...@scholz.cz>.
*> Nice catch! It does seem like we should have gated this behind the
metadata> version as KIP-858 implies. Is the cluster configured with
multiple log> dirs? What is the impact of the error messages?*

I did not observe any obvious impact. I was able to send and receive
messages as normally. But to be honest, I have no idea what else
this might impact, so I did not try anything special.

I think everyone upgrading an existing KRaft cluster will go through this
stage (running Kafka 3.7 with an older metadata version for at least a
while). So even if it is just a logged exception without any other impact I
wonder if it might scare users from upgrading. But I leave it to others to
decide if this is a blocker or not.

Jakub


On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
<st...@confluent.io.invalid> wrote:

> Hey Luke,
>
> This is an interesting problem. Given the fact that the KIP for having a
> 3.8 release passed, I think it weights the scale towards not calling this a
> blocker and expecting it to be solved in 3.7.1.
>
> It is unfortunate that it would not seem safe to migrate to KRaft in 3.7.0
> (given the inability to rollback safely), but if that's true - the same
> case would apply for 3.6.0. So in any case users would be expected to use a
> patch release for this. Further, since we will have a 3.8 release - it is
> likely we will ultimately recommend users upgrade from that version given
> its aim is to have strategic KRaft feature parity with ZK.
> That being said, I am not 100% on this. Let me know whether you think this
> should block the release, Luke. I am also tagging Colin and David to weigh
> in with their opinions, as they worked on the migration logic.
>
> Hey Kirk and Chris,
>
> Unless I'm missing something - KAFKALESS-16029 is simply a bad log due to
> improper closing. And the PR description implies this has been present
> since 3.5. While annoying, I don't see a strong reason for this to block
> the release.
>
> Hey Jakub,
>
> Nice catch! It does seem like we should have gated this behind the metadata
> version as KIP-858 implies. Is the cluster configured with multiple log
> dirs? What is the impact of the error messages?
>
> Tagging Igor (the author of the KIP) to weigh in.
>
> Best,
> Stanislav
>
> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:
>
> > Hi,
> >
> > I was trying the RC2 and run into the following issue ... when I run
> > 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
> > version, I seem to be getting repeated errors like this in the controller
> > logs:
> >
> > 2024-01-13 16:58:01,197 INFO [QuorumController id=0]
> assignReplicasToDirs:
> > event failed with UnsupportedVersionException in 15 microseconds.
> > (org.apache.kafka.controller.QuorumController)
> > [quorum-controller-0-event-handler]
> > 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
> > handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> > AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> > directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> > topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> > partitions=[PartitionData(partitionIndex=2),
> > PartitionData(partitionIndex=1)]),
> > TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> > partitions=[PartitionData(partitionIndex=0)])])]) with context
> > RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> > apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> > connectionId='172.16.14.219:9090-172.16.14.217:53590-7', clientAddress=/
> > 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> > listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> > clientInformation=ClientInformation(softwareName=apache-kafka-java,
> > softwareVersion=3.7.0), fromPrivilegedListener=false,
> >
> >
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> > ])
> > (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> > java.util.concurrent.CompletionException:
> > org.apache.kafka.common.errors.UnsupportedVersionException: Directory
> > assignment is not supported yet.
> >
> >  at
> >
> >
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> >  at
> >
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> >  at
> >
> >
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> >  at
> >
> >
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> >  at
> >
> >
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> >  at
> >
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
> >  at
> >
> >
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
> >  at
> >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
> >  at
> >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
> >  at
> >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> >  at
> >
> >
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> >  at java.base/java.lang.Thread.run(Thread.java:840)
> >
> > Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> > Directory assignment is not supported yet.
> >
> > Is that expected? I guess with the metadata version set to 3.6-IV2, it
> > makes sense that the request is not supported. But shouldn't then the
> > request not be sent at all by the brokers? (I did not opened a JIRA for
> it,
> > but I can open one if you agree this is not expected)
> >
> > Thanks & Regards
> > Jakub
> >
> > On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi Stanislav,
> > >
> > > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe you
> > > missed it.
> > > cross-posting here:
> > >
> > > There is a bug KAFKA-16101
> > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that
> > "Kafka
> > > cluster will be unavailable during KRaft migration rollback".
> > > The impact for this issue is that if brokers try to rollback to ZK mode
> > > during KRaft migration process, there will be a period of time the
> > cluster
> > > is unavailable.
> > > Since ZK migrating to KRaft feature is a production ready feature, I
> > think
> > > this should be addressed soon.
> > > Do you think this is a blocker for v3.7.0?
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <fearthecellos@gmail.com
> >
> > > wrote:
> > >
> > > > Thanks, Kirk!
> > > >
> > > > @Stanislav--do you believe that this warrants a new RC?
> > > >
> > > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
> > > >
> > > > > Hi Chris/Stanislav,
> > > > >
> > > > > I'm working on the 'Unable to find FetchSessionHandler' log problem
> > > > > (KAFKA-16029) and have put out a draft PR (
> > > > > https://github.com/apache/kafka/pull/15186). I will use the
> > quickstart
> > > > > approach as a second means to reproduce/verify while I wait for the
> > > PR's
> > > > > Jenkins job to finish.
> > > > >
> > > > > Thanks,
> > > > > Kirk
> > > > >
> > > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > > > > > Hi Stanislav,
> > > > > >
> > > > > >
> > > > > > Thanks for running this release!
> > > > > >
> > > > > > To verify, I:
> > > > > > - Built from source using Java 11 with both:
> > > > > > - - the 3.7.0-rc2 tag on GitHub
> > > > > > - - the kafka-3.7.0-src.tgz artifact from
> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > > > > - Checked signatures and checksums
> > > > > > - Ran the quickstart using both:
> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> with
> > > Java
> > > > > 11
> > > > > > and Scala 13 in KRaft mode
> > > > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> > > > > > - Ran all unit tests
> > > > > > - Ran all integration tests for Connect and MM2
> > > > > >
> > > > > >
> > > > > > I found two minor areas for concern:
> > > > > >
> > > > > > 1. (Possibly a blocker)
> > > > > > When running the quickstart, I noticed this ERROR-level log
> message
> > > > being
> > > > > > emitted frequently (not not every time) when I killed my console
> > > > consumer
> > > > > > via ctrl-C:
> > > > > >
> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> > > clientId=console-consumer,
> > > > > > groupId=console-consumer-74388] Unable to find
> FetchSessionHandler
> > > for
> > > > > node
> > > > > > 1. Ignoring fetch response
> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > > > > >
> > > > > > I see that this error message is already reported in
> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think we
> > should
> > > > > > prioritize fixing it for this release. I know it's probably
> benign
> > > but
> > > > > it's
> > > > > > really not a good look for us when basic operations log error
> > > messages,
> > > > > and
> > > > > > it may give new users some headaches.
> > > > > >
> > > > > >
> > > > > > 2. (Probably not a blocker)
> > > > > > The following unit tests failed the first time around, and all of
> > > them
> > > > > > passed the second time I ran them:
> > > > > >
> > > > > > - (clients)
> > > > > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > > > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> > > > > > - (connect)
> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > > > (I
> > > > > > thought I fixed this one! 🤬🤬)
> > > > > > - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > > > > >
> > > > > >
> > > > > > Thanks again for your work on this release, and congratulations
> to
> > > > Kafka
> > > > > > Streams for having zero flaky unit tests during my
> > > highly-experimental
> > > > > > single laptop run!
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > > > > > <st...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > > Hello Kafka users, developers, and client-developers,
> > > > > > >
> > > > > > > This is the first candidate for release of Apache Kafka 3.7.0.
> > > > > > >
> > > > > > > Note it's named "RC2" because I had a few "failed" RCs that I
> had
> > > > > > > cut/uploaded but ultimately had to scrap prior to announcing
> due
> > to
> > > > new
> > > > > > > blockers arriving before I could even announce them.
> > > > > > >
> > > > > > > Further - I haven't yet been able to set up the system tests
> > > > > successfully.
> > > > > > > And the integration/unit tests do have a few failures that I
> have
> > > to
> > > > > spend
> > > > > > > time triaging. I would appreciate any help in case anyone
> notices
> > > any
> > > > > tests
> > > > > > > failing that they're subject matters experts in. Expect me to
> > > follow
> > > > > up in
> > > > > > > a day or two with more detailed analysis.
> > > > > > >
> > > > > > > Major changes include:
> > > > > > > - Early Access to KIP-848 - the next generation of the consumer
> > > > > rebalance
> > > > > > > protocol
> > > > > > > - KIP-858: Adding JBOD support to KRaft
> > > > > > > - KIP-714: Observability into Client metrics via a standardized
> > > > > interface
> > > > > > >
> > > > > > > Check more information in the WIP blog post:
> > > > > > > https://github.com/apache/kafka-site/pull/578
> > > > > > >
> > > > > > > Release notes for the 3.7.0 release:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > > > > > >
> > > > > > > *** Please download, test and vote by Thursday, January 18, 9am
> > PT
> > > > ***
> > > > > > >
> > > > > > > Usually these deadlines tend to be 2-3 days, but due to this
> > being
> > > > the
> > > > > > > first RC and the tests not having ran yet, I am giving it a bit
> > > more
> > > > > time.
> > > > > > >
> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > > > > > > https://kafka.apache.org/KEYS
> > > > > > >
> > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > > > > >
> > > > > > > * Docker release artifact to be voted upon:
> > > > > > > apache/kafka:3.7.0-rc2
> > > > > > >
> > > > > > > * Maven artifacts to be voted upon:
> > > > > > >
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > >
> > > > > > > * Javadoc:
> > > > > > >
> > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > > > > > >
> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > > > > > >
> > > > > > > * Documentation:
> > > > > > > https://kafka.apache.org/37/documentation.html
> > > > > > >
> > > > > > > * Protocol:
> > > > > > > https://kafka.apache.org/37/protocol.html
> > > > > > >
> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> > > > > > > Unit/integration tests:
> > > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > > > > > > There are failing tests here. I have to follow up with triaging
> > > some
> > > > of
> > > > > > > the failures and figuring out if they're actual problems or
> > simply
> > > > > flakes.
> > > > > > >
> > > > > > > System tests:
> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > > > > > >
> > > > > > > No successful system test runs yet. I am working on getting the
> > job
> > > > to
> > > > > run.
> > > > > > >
> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> branch:
> > > > > > > Attached are the scan_report and report_jvm output files from
> the
> > > > > Docker
> > > > > > > Build run:
> > > > > > >
> > > > >
> > >
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > > > > > >
> > > > > > > And the final docker image build job - Docker Build Test
> > Pipeline:
> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> > > > > > >
> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > > > > > >
> > > > > > > /**************************************
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Stanislav Kozlovski
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best,
> Stanislav
>

Re: [VOTE] 3.7.0 RC2

Posted by Stanislav Kozlovski <st...@confluent.io.INVALID>.
Hey Luke,

This is an interesting problem. Given the fact that the KIP for having a
3.8 release passed, I think it weights the scale towards not calling this a
blocker and expecting it to be solved in 3.7.1.

It is unfortunate that it would not seem safe to migrate to KRaft in 3.7.0
(given the inability to rollback safely), but if that's true - the same
case would apply for 3.6.0. So in any case users would be expected to use a
patch release for this. Further, since we will have a 3.8 release - it is
likely we will ultimately recommend users upgrade from that version given
its aim is to have strategic KRaft feature parity with ZK.
That being said, I am not 100% on this. Let me know whether you think this
should block the release, Luke. I am also tagging Colin and David to weigh
in with their opinions, as they worked on the migration logic.

Hey Kirk and Chris,

Unless I'm missing something - KAFKALESS-16029 is simply a bad log due to
improper closing. And the PR description implies this has been present
since 3.5. While annoying, I don't see a strong reason for this to block
the release.

Hey Jakub,

Nice catch! It does seem like we should have gated this behind the metadata
version as KIP-858 implies. Is the cluster configured with multiple log
dirs? What is the impact of the error messages?

Tagging Igor (the author of the KIP) to weigh in.

Best,
Stanislav

On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> wrote:

> Hi,
>
> I was trying the RC2 and run into the following issue ... when I run
> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
> version, I seem to be getting repeated errors like this in the controller
> logs:
>
> 2024-01-13 16:58:01,197 INFO [QuorumController id=0] assignReplicasToDirs:
> event failed with UnsupportedVersionException in 15 microseconds.
> (org.apache.kafka.controller.QuorumController)
> [quorum-controller-0-event-handler]
> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
> partitions=[PartitionData(partitionIndex=2),
> PartitionData(partitionIndex=1)]),
> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
> partitions=[PartitionData(partitionIndex=0)])])]) with context
> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
> connectionId='172.16.14.219:9090-172.16.14.217:53590-7', clientAddress=/
> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
> listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
> clientInformation=ClientInformation(softwareName=apache-kafka-java,
> softwareVersion=3.7.0), fromPrivilegedListener=false,
>
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
> ])
> (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> java.util.concurrent.CompletionException:
> org.apache.kafka.common.errors.UnsupportedVersionException: Directory
> assignment is not supported yet.
>
>  at
>
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>  at
>
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>  at
>
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>  at
>
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>  at
>
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>  at
>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>  at
>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>  at
>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>  at
>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>  at
>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>  at
>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>  at java.base/java.lang.Thread.run(Thread.java:840)
>
> Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> Directory assignment is not supported yet.
>
> Is that expected? I guess with the metadata version set to 3.6-IV2, it
> makes sense that the request is not supported. But shouldn't then the
> request not be sent at all by the brokers? (I did not opened a JIRA for it,
> but I can open one if you agree this is not expected)
>
> Thanks & Regards
> Jakub
>
> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Stanislav,
> >
> > I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe you
> > missed it.
> > cross-posting here:
> >
> > There is a bug KAFKA-16101
> > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that
> "Kafka
> > cluster will be unavailable during KRaft migration rollback".
> > The impact for this issue is that if brokers try to rollback to ZK mode
> > during KRaft migration process, there will be a period of time the
> cluster
> > is unavailable.
> > Since ZK migrating to KRaft feature is a production ready feature, I
> think
> > this should be addressed soon.
> > Do you think this is a blocker for v3.7.0?
> >
> > Thanks.
> > Luke
> >
> > On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <fe...@gmail.com>
> > wrote:
> >
> > > Thanks, Kirk!
> > >
> > > @Stanislav--do you believe that this warrants a new RC?
> > >
> > > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
> > >
> > > > Hi Chris/Stanislav,
> > > >
> > > > I'm working on the 'Unable to find FetchSessionHandler' log problem
> > > > (KAFKA-16029) and have put out a draft PR (
> > > > https://github.com/apache/kafka/pull/15186). I will use the
> quickstart
> > > > approach as a second means to reproduce/verify while I wait for the
> > PR's
> > > > Jenkins job to finish.
> > > >
> > > > Thanks,
> > > > Kirk
> > > >
> > > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > > > > Hi Stanislav,
> > > > >
> > > > >
> > > > > Thanks for running this release!
> > > > >
> > > > > To verify, I:
> > > > > - Built from source using Java 11 with both:
> > > > > - - the 3.7.0-rc2 tag on GitHub
> > > > > - - the kafka-3.7.0-src.tgz artifact from
> > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > > > - Checked signatures and checksums
> > > > > - Ran the quickstart using both:
> > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ with
> > Java
> > > > 11
> > > > > and Scala 13 in KRaft mode
> > > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> > > > > - Ran all unit tests
> > > > > - Ran all integration tests for Connect and MM2
> > > > >
> > > > >
> > > > > I found two minor areas for concern:
> > > > >
> > > > > 1. (Possibly a blocker)
> > > > > When running the quickstart, I noticed this ERROR-level log message
> > > being
> > > > > emitted frequently (not not every time) when I killed my console
> > > consumer
> > > > > via ctrl-C:
> > > > >
> > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> > clientId=console-consumer,
> > > > > groupId=console-consumer-74388] Unable to find FetchSessionHandler
> > for
> > > > node
> > > > > 1. Ignoring fetch response
> > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > > > >
> > > > > I see that this error message is already reported in
> > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think we
> should
> > > > > prioritize fixing it for this release. I know it's probably benign
> > but
> > > > it's
> > > > > really not a good look for us when basic operations log error
> > messages,
> > > > and
> > > > > it may give new users some headaches.
> > > > >
> > > > >
> > > > > 2. (Probably not a blocker)
> > > > > The following unit tests failed the first time around, and all of
> > them
> > > > > passed the second time I ran them:
> > > > >
> > > > > - (clients)
> > > > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> > > > > - (connect)
> > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > > (I
> > > > > thought I fixed this one! 🤬🤬)
> > > > > - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > > > >
> > > > >
> > > > > Thanks again for your work on this release, and congratulations to
> > > Kafka
> > > > > Streams for having zero flaky unit tests during my
> > highly-experimental
> > > > > single laptop run!
> > > > >
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > > > > <st...@confluent.io.invalid> wrote:
> > > > >
> > > > > > Hello Kafka users, developers, and client-developers,
> > > > > >
> > > > > > This is the first candidate for release of Apache Kafka 3.7.0.
> > > > > >
> > > > > > Note it's named "RC2" because I had a few "failed" RCs that I had
> > > > > > cut/uploaded but ultimately had to scrap prior to announcing due
> to
> > > new
> > > > > > blockers arriving before I could even announce them.
> > > > > >
> > > > > > Further - I haven't yet been able to set up the system tests
> > > > successfully.
> > > > > > And the integration/unit tests do have a few failures that I have
> > to
> > > > spend
> > > > > > time triaging. I would appreciate any help in case anyone notices
> > any
> > > > tests
> > > > > > failing that they're subject matters experts in. Expect me to
> > follow
> > > > up in
> > > > > > a day or two with more detailed analysis.
> > > > > >
> > > > > > Major changes include:
> > > > > > - Early Access to KIP-848 - the next generation of the consumer
> > > > rebalance
> > > > > > protocol
> > > > > > - KIP-858: Adding JBOD support to KRaft
> > > > > > - KIP-714: Observability into Client metrics via a standardized
> > > > interface
> > > > > >
> > > > > > Check more information in the WIP blog post:
> > > > > > https://github.com/apache/kafka-site/pull/578
> > > > > >
> > > > > > Release notes for the 3.7.0 release:
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by Thursday, January 18, 9am
> PT
> > > ***
> > > > > >
> > > > > > Usually these deadlines tend to be 2-3 days, but due to this
> being
> > > the
> > > > > > first RC and the tests not having ran yet, I am giving it a bit
> > more
> > > > time.
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > > > >
> > > > > > * Docker release artifact to be voted upon:
> > > > > > apache/kafka:3.7.0-rc2
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > > > > >
> > > > > > * Documentation:
> > > > > > https://kafka.apache.org/37/documentation.html
> > > > > >
> > > > > > * Protocol:
> > > > > > https://kafka.apache.org/37/protocol.html
> > > > > >
> > > > > > * Successful Jenkins builds for the 3.7 branch:
> > > > > > Unit/integration tests:
> > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > > > > > There are failing tests here. I have to follow up with triaging
> > some
> > > of
> > > > > > the failures and figuring out if they're actual problems or
> simply
> > > > flakes.
> > > > > >
> > > > > > System tests:
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > > > > >
> > > > > > No successful system test runs yet. I am working on getting the
> job
> > > to
> > > > run.
> > > > > >
> > > > > > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > > > > > Attached are the scan_report and report_jvm output files from the
> > > > Docker
> > > > > > Build run:
> > > > > >
> > > >
> > https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > > > > >
> > > > > > And the final docker image build job - Docker Build Test
> Pipeline:
> > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> > > > > >
> > > > > > The image is apache/kafka:3.7.0-rc2 -
> > > > > >
> > > >
> > >
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > > > > >
> > > > > > /**************************************
> > > > > >
> > > > > > Thanks,
> > > > > > Stanislav Kozlovski
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Best,
Stanislav

Re: [VOTE] 3.7.0 RC2

Posted by Jakub Scholz <ja...@scholz.cz>.
Hi,

I was trying the RC2 and run into the following issue ... when I run
3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
version, I seem to be getting repeated errors like this in the controller
logs:

2024-01-13 16:58:01,197 INFO [QuorumController id=0] assignReplicasToDirs:
event failed with UnsupportedVersionException in 15 microseconds.
(org.apache.kafka.controller.QuorumController)
[quorum-controller-0-event-handler]
2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
partitions=[PartitionData(partitionIndex=2),
PartitionData(partitionIndex=1)]),
TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
partitions=[PartitionData(partitionIndex=0)])])]) with context
RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
connectionId='172.16.14.219:9090-172.16.14.217:53590-7', clientAddress=/
172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
clientInformation=ClientInformation(softwareName=apache-kafka-java,
softwareVersion=3.7.0), fromPrivilegedListener=false,
principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2])
(kafka.server.ControllerApis) [quorum-controller-0-event-handler]
java.util.concurrent.CompletionException:
org.apache.kafka.common.errors.UnsupportedVersionException: Directory
assignment is not supported yet.

 at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
 at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
 at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
 at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
 at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
 at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
 at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
 at
org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
 at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
 at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
 at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
 at java.base/java.lang.Thread.run(Thread.java:840)

Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
Directory assignment is not supported yet.

Is that expected? I guess with the metadata version set to 3.6-IV2, it
makes sense that the request is not supported. But shouldn't then the
request not be sent at all by the brokers? (I did not opened a JIRA for it,
but I can open one if you agree this is not expected)

Thanks & Regards
Jakub

On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <sh...@gmail.com> wrote:

> Hi Stanislav,
>
> I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe you
> missed it.
> cross-posting here:
>
> There is a bug KAFKA-16101
> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that "Kafka
> cluster will be unavailable during KRaft migration rollback".
> The impact for this issue is that if brokers try to rollback to ZK mode
> during KRaft migration process, there will be a period of time the cluster
> is unavailable.
> Since ZK migrating to KRaft feature is a production ready feature, I think
> this should be addressed soon.
> Do you think this is a blocker for v3.7.0?
>
> Thanks.
> Luke
>
> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <fe...@gmail.com>
> wrote:
>
> > Thanks, Kirk!
> >
> > @Stanislav--do you believe that this warrants a new RC?
> >
> > On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
> >
> > > Hi Chris/Stanislav,
> > >
> > > I'm working on the 'Unable to find FetchSessionHandler' log problem
> > > (KAFKA-16029) and have put out a draft PR (
> > > https://github.com/apache/kafka/pull/15186). I will use the quickstart
> > > approach as a second means to reproduce/verify while I wait for the
> PR's
> > > Jenkins job to finish.
> > >
> > > Thanks,
> > > Kirk
> > >
> > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > > > Hi Stanislav,
> > > >
> > > >
> > > > Thanks for running this release!
> > > >
> > > > To verify, I:
> > > > - Built from source using Java 11 with both:
> > > > - - the 3.7.0-rc2 tag on GitHub
> > > > - - the kafka-3.7.0-src.tgz artifact from
> > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > > - Checked signatures and checksums
> > > > - Ran the quickstart using both:
> > > > - - The kafka_2.13-3.7.0.tgz artifact from
> > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ with
> Java
> > > 11
> > > > and Scala 13 in KRaft mode
> > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> > > > - Ran all unit tests
> > > > - Ran all integration tests for Connect and MM2
> > > >
> > > >
> > > > I found two minor areas for concern:
> > > >
> > > > 1. (Possibly a blocker)
> > > > When running the quickstart, I noticed this ERROR-level log message
> > being
> > > > emitted frequently (not not every time) when I killed my console
> > consumer
> > > > via ctrl-C:
> > > >
> > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> clientId=console-consumer,
> > > > groupId=console-consumer-74388] Unable to find FetchSessionHandler
> for
> > > node
> > > > 1. Ignoring fetch response
> > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > > >
> > > > I see that this error message is already reported in
> > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think we should
> > > > prioritize fixing it for this release. I know it's probably benign
> but
> > > it's
> > > > really not a good look for us when basic operations log error
> messages,
> > > and
> > > > it may give new users some headaches.
> > > >
> > > >
> > > > 2. (Probably not a blocker)
> > > > The following unit tests failed the first time around, and all of
> them
> > > > passed the second time I ran them:
> > > >
> > > > - (clients)
> > > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> > > > - (connect)
> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> > (I
> > > > thought I fixed this one! 🤬🤬)
> > > > - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > > >
> > > >
> > > > Thanks again for your work on this release, and congratulations to
> > Kafka
> > > > Streams for having zero flaky unit tests during my
> highly-experimental
> > > > single laptop run!
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > > > <st...@confluent.io.invalid> wrote:
> > > >
> > > > > Hello Kafka users, developers, and client-developers,
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 3.7.0.
> > > > >
> > > > > Note it's named "RC2" because I had a few "failed" RCs that I had
> > > > > cut/uploaded but ultimately had to scrap prior to announcing due to
> > new
> > > > > blockers arriving before I could even announce them.
> > > > >
> > > > > Further - I haven't yet been able to set up the system tests
> > > successfully.
> > > > > And the integration/unit tests do have a few failures that I have
> to
> > > spend
> > > > > time triaging. I would appreciate any help in case anyone notices
> any
> > > tests
> > > > > failing that they're subject matters experts in. Expect me to
> follow
> > > up in
> > > > > a day or two with more detailed analysis.
> > > > >
> > > > > Major changes include:
> > > > > - Early Access to KIP-848 - the next generation of the consumer
> > > rebalance
> > > > > protocol
> > > > > - KIP-858: Adding JBOD support to KRaft
> > > > > - KIP-714: Observability into Client metrics via a standardized
> > > interface
> > > > >
> > > > > Check more information in the WIP blog post:
> > > > > https://github.com/apache/kafka-site/pull/578
> > > > >
> > > > > Release notes for the 3.7.0 release:
> > > > >
> > > > >
> > >
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Thursday, January 18, 9am PT
> > ***
> > > > >
> > > > > Usually these deadlines tend to be 2-3 days, but due to this being
> > the
> > > > > first RC and the tests not having ran yet, I am giving it a bit
> more
> > > time.
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > > >
> > > > > * Docker release artifact to be voted upon:
> > > > > apache/kafka:3.7.0-rc2
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/37/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/37/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.7 branch:
> > > > > Unit/integration tests:
> > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > > > > There are failing tests here. I have to follow up with triaging
> some
> > of
> > > > > the failures and figuring out if they're actual problems or simply
> > > flakes.
> > > > >
> > > > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > > > >
> > > > > No successful system test runs yet. I am working on getting the job
> > to
> > > run.
> > > > >
> > > > > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > > > > Attached are the scan_report and report_jvm output files from the
> > > Docker
> > > > > Build run:
> > > > >
> > >
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > > > >
> > > > > And the final docker image build job - Docker Build Test Pipeline:
> > > > > https://github.com/apache/kafka/actions/runs/7486178277
> > > > >
> > > > > The image is apache/kafka:3.7.0-rc2 -
> > > > >
> > >
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > > > >
> > > > > /**************************************
> > > > >
> > > > > Thanks,
> > > > > Stanislav Kozlovski
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] 3.7.0 RC2

Posted by Luke Chen <sh...@gmail.com>.
Hi Stanislav,

I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe you
missed it.
cross-posting here:

There is a bug KAFKA-16101
<https://issues.apache.org/jira/browse/KAFKA-16101> reporting that "Kafka
cluster will be unavailable during KRaft migration rollback".
The impact for this issue is that if brokers try to rollback to ZK mode
during KRaft migration process, there will be a period of time the cluster
is unavailable.
Since ZK migrating to KRaft feature is a production ready feature, I think
this should be addressed soon.
Do you think this is a blocker for v3.7.0?

Thanks.
Luke

On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <fe...@gmail.com>
wrote:

> Thanks, Kirk!
>
> @Stanislav--do you believe that this warrants a new RC?
>
> On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:
>
> > Hi Chris/Stanislav,
> >
> > I'm working on the 'Unable to find FetchSessionHandler' log problem
> > (KAFKA-16029) and have put out a draft PR (
> > https://github.com/apache/kafka/pull/15186). I will use the quickstart
> > approach as a second means to reproduce/verify while I wait for the PR's
> > Jenkins job to finish.
> >
> > Thanks,
> > Kirk
> >
> > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > > Hi Stanislav,
> > >
> > >
> > > Thanks for running this release!
> > >
> > > To verify, I:
> > > - Built from source using Java 11 with both:
> > > - - the 3.7.0-rc2 tag on GitHub
> > > - - the kafka-3.7.0-src.tgz artifact from
> > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > - Checked signatures and checksums
> > > - Ran the quickstart using both:
> > > - - The kafka_2.13-3.7.0.tgz artifact from
> > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ with Java
> > 11
> > > and Scala 13 in KRaft mode
> > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> > > - Ran all unit tests
> > > - Ran all integration tests for Connect and MM2
> > >
> > >
> > > I found two minor areas for concern:
> > >
> > > 1. (Possibly a blocker)
> > > When running the quickstart, I noticed this ERROR-level log message
> being
> > > emitted frequently (not not every time) when I killed my console
> consumer
> > > via ctrl-C:
> > >
> > > > [2024-01-12 11:00:31,088] ERROR [Consumer clientId=console-consumer,
> > > groupId=console-consumer-74388] Unable to find FetchSessionHandler for
> > node
> > > 1. Ignoring fetch response
> > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> > >
> > > I see that this error message is already reported in
> > > https://issues.apache.org/jira/browse/KAFKA-16029. I think we should
> > > prioritize fixing it for this release. I know it's probably benign but
> > it's
> > > really not a good look for us when basic operations log error messages,
> > and
> > > it may give new users some headaches.
> > >
> > >
> > > 2. (Probably not a blocker)
> > > The following unit tests failed the first time around, and all of them
> > > passed the second time I ran them:
> > >
> > > - (clients)
> > ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > > - (clients) SelectorTest.testConnectionsByClientMetric()
> > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> > > - (connect) TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> (I
> > > thought I fixed this one! 🤬🤬)
> > > - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> > >
> > >
> > > Thanks again for your work on this release, and congratulations to
> Kafka
> > > Streams for having zero flaky unit tests during my highly-experimental
> > > single laptop run!
> > >
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > > <st...@confluent.io.invalid> wrote:
> > >
> > > > Hello Kafka users, developers, and client-developers,
> > > >
> > > > This is the first candidate for release of Apache Kafka 3.7.0.
> > > >
> > > > Note it's named "RC2" because I had a few "failed" RCs that I had
> > > > cut/uploaded but ultimately had to scrap prior to announcing due to
> new
> > > > blockers arriving before I could even announce them.
> > > >
> > > > Further - I haven't yet been able to set up the system tests
> > successfully.
> > > > And the integration/unit tests do have a few failures that I have to
> > spend
> > > > time triaging. I would appreciate any help in case anyone notices any
> > tests
> > > > failing that they're subject matters experts in. Expect me to follow
> > up in
> > > > a day or two with more detailed analysis.
> > > >
> > > > Major changes include:
> > > > - Early Access to KIP-848 - the next generation of the consumer
> > rebalance
> > > > protocol
> > > > - KIP-858: Adding JBOD support to KRaft
> > > > - KIP-714: Observability into Client metrics via a standardized
> > interface
> > > >
> > > > Check more information in the WIP blog post:
> > > > https://github.com/apache/kafka-site/pull/578
> > > >
> > > > Release notes for the 3.7.0 release:
> > > >
> > > >
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Thursday, January 18, 9am PT
> ***
> > > >
> > > > Usually these deadlines tend to be 2-3 days, but due to this being
> the
> > > > first RC and the tests not having ran yet, I am giving it a bit more
> > time.
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > >
> > > > * Docker release artifact to be voted upon:
> > > > apache/kafka:3.7.0-rc2
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/37/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/37/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.7 branch:
> > > > Unit/integration tests:
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > > > There are failing tests here. I have to follow up with triaging some
> of
> > > > the failures and figuring out if they're actual problems or simply
> > flakes.
> > > >
> > > > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > > >
> > > > No successful system test runs yet. I am working on getting the job
> to
> > run.
> > > >
> > > > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > > > Attached are the scan_report and report_jvm output files from the
> > Docker
> > > > Build run:
> > > >
> > https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > > >
> > > > And the final docker image build job - Docker Build Test Pipeline:
> > > > https://github.com/apache/kafka/actions/runs/7486178277
> > > >
> > > > The image is apache/kafka:3.7.0-rc2 -
> > > >
> >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > > >
> > > > /**************************************
> > > >
> > > > Thanks,
> > > > Stanislav Kozlovski
> > > >
> > >
> >
>

Re: [VOTE] 3.7.0 RC2

Posted by Chris Egerton <fe...@gmail.com>.
Thanks, Kirk!

@Stanislav--do you believe that this warrants a new RC?

On Fri, Jan 12, 2024, 19:08 Kirk True <ki...@kirktrue.pro> wrote:

> Hi Chris/Stanislav,
>
> I'm working on the 'Unable to find FetchSessionHandler' log problem
> (KAFKA-16029) and have put out a draft PR (
> https://github.com/apache/kafka/pull/15186). I will use the quickstart
> approach as a second means to reproduce/verify while I wait for the PR's
> Jenkins job to finish.
>
> Thanks,
> Kirk
>
> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > Hi Stanislav,
> >
> >
> > Thanks for running this release!
> >
> > To verify, I:
> > - Built from source using Java 11 with both:
> > - - the 3.7.0-rc2 tag on GitHub
> > - - the kafka-3.7.0-src.tgz artifact from
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > - Checked signatures and checksums
> > - Ran the quickstart using both:
> > - - The kafka_2.13-3.7.0.tgz artifact from
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ with Java
> 11
> > and Scala 13 in KRaft mode
> > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> > - Ran all unit tests
> > - Ran all integration tests for Connect and MM2
> >
> >
> > I found two minor areas for concern:
> >
> > 1. (Possibly a blocker)
> > When running the quickstart, I noticed this ERROR-level log message being
> > emitted frequently (not not every time) when I killed my console consumer
> > via ctrl-C:
> >
> > > [2024-01-12 11:00:31,088] ERROR [Consumer clientId=console-consumer,
> > groupId=console-consumer-74388] Unable to find FetchSessionHandler for
> node
> > 1. Ignoring fetch response
> > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> >
> > I see that this error message is already reported in
> > https://issues.apache.org/jira/browse/KAFKA-16029. I think we should
> > prioritize fixing it for this release. I know it's probably benign but
> it's
> > really not a good look for us when basic operations log error messages,
> and
> > it may give new users some headaches.
> >
> >
> > 2. (Probably not a blocker)
> > The following unit tests failed the first time around, and all of them
> > passed the second time I ran them:
> >
> > - (clients)
> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> > - (clients) SelectorTest.testConnectionsByClientMetric()
> > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> > - (connect) TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound (I
> > thought I fixed this one! 🤬🤬)
> > - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> >
> >
> > Thanks again for your work on this release, and congratulations to Kafka
> > Streams for having zero flaky unit tests during my highly-experimental
> > single laptop run!
> >
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> > <st...@confluent.io.invalid> wrote:
> >
> > > Hello Kafka users, developers, and client-developers,
> > >
> > > This is the first candidate for release of Apache Kafka 3.7.0.
> > >
> > > Note it's named "RC2" because I had a few "failed" RCs that I had
> > > cut/uploaded but ultimately had to scrap prior to announcing due to new
> > > blockers arriving before I could even announce them.
> > >
> > > Further - I haven't yet been able to set up the system tests
> successfully.
> > > And the integration/unit tests do have a few failures that I have to
> spend
> > > time triaging. I would appreciate any help in case anyone notices any
> tests
> > > failing that they're subject matters experts in. Expect me to follow
> up in
> > > a day or two with more detailed analysis.
> > >
> > > Major changes include:
> > > - Early Access to KIP-848 - the next generation of the consumer
> rebalance
> > > protocol
> > > - KIP-858: Adding JBOD support to KRaft
> > > - KIP-714: Observability into Client metrics via a standardized
> interface
> > >
> > > Check more information in the WIP blog post:
> > > https://github.com/apache/kafka-site/pull/578
> > >
> > > Release notes for the 3.7.0 release:
> > >
> > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Thursday, January 18, 9am PT ***
> > >
> > > Usually these deadlines tend to be 2-3 days, but due to this being the
> > > first RC and the tests not having ran yet, I am giving it a bit more
> time.
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > >
> > > * Docker release artifact to be voted upon:
> > > apache/kafka:3.7.0-rc2
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> > >
> > > * Documentation:
> > > https://kafka.apache.org/37/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/37/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.7 branch:
> > > Unit/integration tests:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > > There are failing tests here. I have to follow up with triaging some of
> > > the failures and figuring out if they're actual problems or simply
> flakes.
> > >
> > > System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> > >
> > > No successful system test runs yet. I am working on getting the job to
> run.
> > >
> > > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > > Attached are the scan_report and report_jvm output files from the
> Docker
> > > Build run:
> > >
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> > >
> > > And the final docker image build job - Docker Build Test Pipeline:
> > > https://github.com/apache/kafka/actions/runs/7486178277
> > >
> > > The image is apache/kafka:3.7.0-rc2 -
> > >
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> > >
> > > /**************************************
> > >
> > > Thanks,
> > > Stanislav Kozlovski
> > >
> >
>

Re: [VOTE] 3.7.0 RC2

Posted by Kirk True <ki...@kirktrue.pro>.
Hi Chris/Stanislav,

I'm working on the 'Unable to find FetchSessionHandler' log problem (KAFKA-16029) and have put out a draft PR (https://github.com/apache/kafka/pull/15186). I will use the quickstart approach as a second means to reproduce/verify while I wait for the PR's Jenkins job to finish.   

Thanks,
Kirk

On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> Hi Stanislav,
> 
> 
> Thanks for running this release!
> 
> To verify, I:
> - Built from source using Java 11 with both:
> - - the 3.7.0-rc2 tag on GitHub
> - - the kafka-3.7.0-src.tgz artifact from
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> - Checked signatures and checksums
> - Ran the quickstart using both:
> - - The kafka_2.13-3.7.0.tgz artifact from
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ with Java 11
> and Scala 13 in KRaft mode
> - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> - Ran all unit tests
> - Ran all integration tests for Connect and MM2
> 
> 
> I found two minor areas for concern:
> 
> 1. (Possibly a blocker)
> When running the quickstart, I noticed this ERROR-level log message being
> emitted frequently (not not every time) when I killed my console consumer
> via ctrl-C:
> 
> > [2024-01-12 11:00:31,088] ERROR [Consumer clientId=console-consumer,
> groupId=console-consumer-74388] Unable to find FetchSessionHandler for node
> 1. Ignoring fetch response
> (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> 
> I see that this error message is already reported in
> https://issues.apache.org/jira/browse/KAFKA-16029. I think we should
> prioritize fixing it for this release. I know it's probably benign but it's
> really not a good look for us when basic operations log error messages, and
> it may give new users some headaches.
> 
> 
> 2. (Probably not a blocker)
> The following unit tests failed the first time around, and all of them
> passed the second time I ran them:
> 
> - (clients) ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> - (clients) SelectorTest.testConnectionsByClientMetric()
> - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> - (connect) TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound (I
> thought I fixed this one! 🤬🤬)
> - (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> 
> 
> Thanks again for your work on this release, and congratulations to Kafka
> Streams for having zero flaky unit tests during my highly-experimental
> single laptop run!
> 
> 
> Cheers,
> 
> Chris
> 
> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> <st...@confluent.io.invalid> wrote:
> 
> > Hello Kafka users, developers, and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.7.0.
> >
> > Note it's named "RC2" because I had a few "failed" RCs that I had
> > cut/uploaded but ultimately had to scrap prior to announcing due to new
> > blockers arriving before I could even announce them.
> >
> > Further - I haven't yet been able to set up the system tests successfully.
> > And the integration/unit tests do have a few failures that I have to spend
> > time triaging. I would appreciate any help in case anyone notices any tests
> > failing that they're subject matters experts in. Expect me to follow up in
> > a day or two with more detailed analysis.
> >
> > Major changes include:
> > - Early Access to KIP-848 - the next generation of the consumer rebalance
> > protocol
> > - KIP-858: Adding JBOD support to KRaft
> > - KIP-714: Observability into Client metrics via a standardized interface
> >
> > Check more information in the WIP blog post:
> > https://github.com/apache/kafka-site/pull/578
> >
> > Release notes for the 3.7.0 release:
> >
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Thursday, January 18, 9am PT ***
> >
> > Usually these deadlines tend to be 2-3 days, but due to this being the
> > first RC and the tests not having ran yet, I am giving it a bit more time.
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >
> > * Docker release artifact to be voted upon:
> > apache/kafka:3.7.0-rc2
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> >
> > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> >
> > * Documentation:
> > https://kafka.apache.org/37/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/37/protocol.html
> >
> > * Successful Jenkins builds for the 3.7 branch:
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> > There are failing tests here. I have to follow up with triaging some of
> > the failures and figuring out if they're actual problems or simply flakes.
> >
> > System tests: https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> >
> > No successful system test runs yet. I am working on getting the job to run.
> >
> > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > Attached are the scan_report and report_jvm output files from the Docker
> > Build run:
> > https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> >
> > And the final docker image build job - Docker Build Test Pipeline:
> > https://github.com/apache/kafka/actions/runs/7486178277
> >
> > The image is apache/kafka:3.7.0-rc2 -
> > https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> >
> > /**************************************
> >
> > Thanks,
> > Stanislav Kozlovski
> >
> 

Re: [VOTE] 3.7.0 RC2

Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Hi Stanislav,


Thanks for running this release!

To verify, I:
- Built from source using Java 11 with both:
- - the 3.7.0-rc2 tag on GitHub
- - the kafka-3.7.0-src.tgz artifact from
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
- Checked signatures and checksums
- Ran the quickstart using both:
- - The kafka_2.13-3.7.0.tgz artifact from
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ with Java 11
and Scala 13 in KRaft mode
- - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
- Ran all unit tests
- Ran all integration tests for Connect and MM2


I found two minor areas for concern:

1. (Possibly a blocker)
When running the quickstart, I noticed this ERROR-level log message being
emitted frequently (not not every time) when I killed my console consumer
via ctrl-C:

> [2024-01-12 11:00:31,088] ERROR [Consumer clientId=console-consumer,
groupId=console-consumer-74388] Unable to find FetchSessionHandler for node
1. Ignoring fetch response
(org.apache.kafka.clients.consumer.internals.AbstractFetch)

I see that this error message is already reported in
https://issues.apache.org/jira/browse/KAFKA-16029. I think we should
prioritize fixing it for this release. I know it's probably benign but it's
really not a good look for us when basic operations log error messages, and
it may give new users some headaches.


2. (Probably not a blocker)
The following unit tests failed the first time around, and all of them
passed the second time I ran them:

- (clients) ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
- (clients) SelectorTest.testConnectionsByClientMetric()
- (clients) Tls13SelectorTest.testConnectionsByClientMetric()
- (connect) TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound (I
thought I fixed this one! 🤬🤬)
- (core) ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]


Thanks again for your work on this release, and congratulations to Kafka
Streams for having zero flaky unit tests during my highly-experimental
single laptop run!


Cheers,

Chris

On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
<st...@confluent.io.invalid> wrote:

> Hello Kafka users, developers, and client-developers,
>
> This is the first candidate for release of Apache Kafka 3.7.0.
>
> Note it's named "RC2" because I had a few "failed" RCs that I had
> cut/uploaded but ultimately had to scrap prior to announcing due to new
> blockers arriving before I could even announce them.
>
> Further - I haven't yet been able to set up the system tests successfully.
> And the integration/unit tests do have a few failures that I have to spend
> time triaging. I would appreciate any help in case anyone notices any tests
> failing that they're subject matters experts in. Expect me to follow up in
> a day or two with more detailed analysis.
>
> Major changes include:
> - Early Access to KIP-848 - the next generation of the consumer rebalance
> protocol
> - KIP-858: Adding JBOD support to KRaft
> - KIP-714: Observability into Client metrics via a standardized interface
>
> Check more information in the WIP blog post:
> https://github.com/apache/kafka-site/pull/578
>
> Release notes for the 3.7.0 release:
>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, January 18, 9am PT ***
>
> Usually these deadlines tend to be 2-3 days, but due to this being the
> first RC and the tests not having ran yet, I am giving it a bit more time.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
>
> * Docker release artifact to be voted upon:
> apache/kafka:3.7.0-rc2
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
>
> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> https://github.com/apache/kafka/releases/tag/3.7.0-rc2
>
> * Documentation:
> https://kafka.apache.org/37/documentation.html
>
> * Protocol:
> https://kafka.apache.org/37/protocol.html
>
> * Successful Jenkins builds for the 3.7 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> There are failing tests here. I have to follow up with triaging some of
> the failures and figuring out if they're actual problems or simply flakes.
>
> System tests: https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
>
> No successful system test runs yet. I am working on getting the job to run.
>
> * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> Attached are the scan_report and report_jvm output files from the Docker
> Build run:
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
>
> And the final docker image build job - Docker Build Test Pipeline:
> https://github.com/apache/kafka/actions/runs/7486178277
>
> The image is apache/kafka:3.7.0-rc2 -
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
>
> /**************************************
>
> Thanks,
> Stanislav Kozlovski
>