Re: Increase in flaky test failures

[Kirk True <ki...@kirktrue.pro>]

Hi Divij,

I’m hitting something similar in my pull request test run (https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13990/26/pipeline/10):

> Task :clients:testClasses
Unexpected exception thrown.
org.gradle.internal.remote.internal.MessageIOException: Could not read message from '/127.0.0.1:53070'.
	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:94)
	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException
	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:72)
	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
	... 6 more
> Task :clients:spotbugsTest SKIPPED
> Task :connect:mirror-client:compileTestJava
> Task :connect:mirror-client:testClasses
> Task :storage:api:compileTestJava
> Task :storage:api:testClasses
> Task :log4j-appender:compileTestJava
> Task :log4j-appender:testClasses
org.gradle.internal.remote.internal.ConnectException: Could not connect to server [871f8055-03e7-4c5c-bc92-6b367d765631 port:37149, addresses:[/127.0.0.1]]. Tried addresses: [/127.0.0.1].
	at org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:67)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedClient.getConnection(MessageHubBackedClient.java:36)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:103)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:774)
	at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:120)
	at org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.tryConnect(TcpOutgoingConnector.java:81)
	at org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:54)
	... 5 more
> Task :streams:examples:compileTestJava
> Task :connect:mirror-client:spotbugsTest SKIPPED
> Task :storage:api:spotbugsTest SKIPPED
> Task :streams:examples:testClasses
> Task :clients:checkstyleTest FAILED

This only pops up occasionally, of course :(

If you got the chance to look into this before, did you find anything promising to investigate/research?

Thanks,
Kirk

> On Jun 15, 2023, at 11:32 AM, Divij Vaidya <di...@gmail.com> wrote:
> 
> Hey folks
> 
> We are having increased flaky failures for the past one day.
> 
> Mostly, streams upgrade tests (:streams:upgrade-system-tests-20:unitTest)
> are failing with
> 
> org.gradle.internal.remote.internal.ConnectException: Could not
> connect to server [90d54b62-a664-438e-b241-20847ead1eab port:34765,
> addresses:[/127.0.0.1]]. Tried addresses: [/127.0.0.1].
> [2023-06-15T14:50:31.779Z] 	at
> org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:67)
> 
> 
> And other network related tests are failing with errors such as
> 
> java.net.ConnectException: Connection timed out
> at java.base/sun.nio.ch.Net.connect0(Native Method)
> at java.base/sun.nio.ch.Net.connect(Net.java:579)
> at java.base/sun.nio.ch.Net.connect(Net.java:568)
> at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
> at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
> at java.base/java.net.Socket.connect(Socket.java:633)
> at java.base/java.net.Socket.connect(Socket.java:583)
> at java.base/java.net.Socket.<init>(Socket.java:507)
> at java.base/java.net.Socket.<init>(Socket.java:319)
> at
> org.apache.kafka.common.network.PlaintextSender.lambda$new$0(PlaintextSender.java:30)
> 
> 
> These stack traces probably hints at some infra problem to me? I will take
> a deeper look tomorrow in my day time but meanwhile, if someone gets a
> chance, please try to debug!
> 
> Example CI run -
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13858/2/tests
> 
> 
> --
> Divij Vaidya