You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/04/03 02:55:00 UTC
[jira] [Commented] (KAFKA-9812) Integration tests hang and timeout
the entire PR build on jenkins
[ https://issues.apache.org/jira/browse/KAFKA-9812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074231#comment-17074231 ]
ASF GitHub Bot commented on KAFKA-9812:
---------------------------------------
vvcephei commented on pull request #8411: KAFKA-9812: fix infinite loop in test code
URL: https://github.com/apache/kafka/pull/8411
If the EosIntegrationTest fails an assertion after setting `gcInjected:=true` but before it gets to set `doGC=false`, then it would never set the flag, and the transformer would become an infinite loop.
There are a couple of ways to tackle this, but I opted to just check inside the loop that we're not currently shutting down.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Integration tests hang and timeout the entire PR build on jenkins
> ------------------------------------------------------------------
>
> Key: KAFKA-9812
> URL: https://issues.apache.org/jira/browse/KAFKA-9812
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Konstantine Karantasis
> Assignee: John Roesler
> Priority: Major
> Labels: flaky, flaky-build
>
> Test {{org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta]}} and possibly others was been spotted to hang indefinitely without entering a {{FAILED}} state, which resulted in the whole jenkins to time out hours later without a list of failures.
> A suggested fix could possibly involve using a class wide {{Timeout}} rule with reasonable timeouts for integration tests, as described here: [https://github.com/junit-team/junit4/wiki/Timeout-for-tests]
> Snippet from the build log [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/console] :
> {code:bash}
> org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] STARTED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] PASSED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] STARTED 15:12:14 Build timed out (after 270 minutes). Marking the build as aborted. 15:12:15 Build was aborted 15:12:15 [FINDBUGS] Skipping publisher since build result is ABORTED 15:12:15 Recording test results 15:12:15 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:15 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:15 15:12:15 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] SKIPPED 15:12:16 15:12:16 > Task :streams:integrationTest FAILED 15:12:17 The message received from the daemon indicates that the daemon has disappeared. 15:12:17 Build request sent: Build{id=4f5e3086-ab55-48c5-a3a4-6f213565ac61, currentDir=/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.12} 15:12:17 Attempting to read last messages from the daemon log... 15:12:17 Daemon pid: 27019 15:12:17 log file: /home/jenkins/.gradle/daemon/5.6.2/daemon-27019.out.log 15:12:17 ----- Last 20 lines from daemon log file - daemon-27019.out.log ----- 15:12:17 at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:237) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:214) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:364) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87) 15:12:17 at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) 15:12:17 at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) 15:12:17 at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) 15:12:17 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 15:12:17 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 15:12:17 at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) 15:12:17 at java.lang.Thread.run(Thread.java:748) 15:12:17 Caused by: java.lang.IllegalStateException: Shutdown in progress 15:12:17 at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) 15:12:17 at java.lang.Runtime.removeShutdownHook(Runtime.java:239) 15:12:17 at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:33) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:204) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:360) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) 15:12:17 ... 7 more 15:12:17 ----- End of the daemon log ----- 15:12:17 15:12:17 15:12:17 FAILURE: Build failed with an exception. 15:12:17 15:12:17 * What went wrong: 15:12:17 Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) 15:12:17 15:12:17 * Try: 15:12:17 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 15:12:17 15:12:17 * Get more help at https://help.gradle.org 15:12:17 > Task :tools:integrationTest 15:12:17 Test steps failed 15:12:17 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:17 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Adding one-line test results to commit status... 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting status of 45ec78a919d6220521b55ca63582d365664b85c1 to FAILURE with url https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/ and message: 'FAILURE 15:12:19 13289 tests run, 67 skipped, 0 failed.' 15:12:19 Using context: JDK 8 and Scala 2.12 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Finished: ABORTED
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)