You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Konstantine Karantasis (Jira)" <ji...@apache.org> on 2020/04/03 00:10:00 UTC
[jira] [Updated] (KAFKA-9812) Integration tests hang and timeout
the entire PR build on jenkins
[ https://issues.apache.org/jira/browse/KAFKA-9812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantine Karantasis updated KAFKA-9812:
------------------------------------------
Description:
Test {{org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta]}} and possibly others was been spotted to hang indefinitely without entering a {{FAILED}} state, which resulted in the whole jenkins to time out hours later without a list of failures.
A suggested fix could possibly involve using a class wide {{Timeout}} rule with reasonable timeouts for integration tests, as described here: [https://github.com/junit-team/junit4/wiki/Timeout-for-tests
]
Snippet from the build log [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/console] :
{noformat}
org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] STARTED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] PASSED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] STARTED 15:12:14 Build timed out (after 270 minutes). Marking the build as aborted. 15:12:15 Build was aborted 15:12:15 [FINDBUGS] Skipping publisher since build result is ABORTED 15:12:15 Recording test results 15:12:15 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:15 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:15 15:12:15 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] SKIPPED 15:12:16 15:12:16 > Task :streams:integrationTest FAILED 15:12:17 The message received from the daemon indicates that the daemon has disappeared. 15:12:17 Build request sent: Build{id=4f5e3086-ab55-48c5-a3a4-6f213565ac61, currentDir=/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.12} 15:12:17 Attempting to read last messages from the daemon log... 15:12:17 Daemon pid: 27019 15:12:17 log file: /home/jenkins/.gradle/daemon/5.6.2/daemon-27019.out.log 15:12:17 ----- Last 20 lines from daemon log file - daemon-27019.out.log ----- 15:12:17 at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:237) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:214) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:364) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87) 15:12:17 at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) 15:12:17 at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) 15:12:17 at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) 15:12:17 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 15:12:17 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 15:12:17 at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) 15:12:17 at java.lang.Thread.run(Thread.java:748) 15:12:17 Caused by: java.lang.IllegalStateException: Shutdown in progress 15:12:17 at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) 15:12:17 at java.lang.Runtime.removeShutdownHook(Runtime.java:239) 15:12:17 at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:33) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:204) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:360) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) 15:12:17 ... 7 more 15:12:17 ----- End of the daemon log ----- 15:12:17 15:12:17 15:12:17 FAILURE: Build failed with an exception. 15:12:17 15:12:17 * What went wrong: 15:12:17 Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) 15:12:17 15:12:17 * Try: 15:12:17 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 15:12:17 15:12:17 * Get more help at https://help.gradle.org 15:12:17 > Task :tools:integrationTest 15:12:17 Test steps failed 15:12:17 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:17 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Adding one-line test results to commit status... 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting status of 45ec78a919d6220521b55ca63582d365664b85c1 to FAILURE with url https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/ and message: 'FAILURE 15:12:19 13289 tests run, 67 skipped, 0 failed.' 15:12:19 Using context: JDK 8 and Scala 2.12 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Finished: ABORTED
{noformat}
was:
Test {{org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta]}} and possibly others was been spotted to hang indefinitely without entering a {{FAILED}} state, which resulted in the whole jenkins to time out hours later without a list of failures.
A suggested fix could possibly involve using a class wide {{Timeout}} rule with reasonable timeouts for integration tests, as described here: [https://github.com/junit-team/junit4/wiki/Timeout-for-tests]
> Integration tests hang and timeout the entire PR build on jenkins
> ------------------------------------------------------------------
>
> Key: KAFKA-9812
> URL: https://issues.apache.org/jira/browse/KAFKA-9812
> Project: Kafka
> Issue Type: Bug
> Reporter: Konstantine Karantasis
> Assignee: Boyang Chen
> Priority: Major
>
> Test {{org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta]}} and possibly others was been spotted to hang indefinitely without entering a {{FAILED}} state, which resulted in the whole jenkins to time out hours later without a list of failures.
> A suggested fix could possibly involve using a class wide {{Timeout}} rule with reasonable timeouts for integration tests, as described here: [https://github.com/junit-team/junit4/wiki/Timeout-for-tests
> ]
> Snippet from the build log [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/console] :
> {noformat}
> org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] STARTED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] PASSED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] STARTED 15:12:14 Build timed out (after 270 minutes). Marking the build as aborted. 15:12:15 Build was aborted 15:12:15 [FINDBUGS] Skipping publisher since build result is ABORTED 15:12:15 Recording test results 15:12:15 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:15 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:15 15:12:15 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] SKIPPED 15:12:16 15:12:16 > Task :streams:integrationTest FAILED 15:12:17 The message received from the daemon indicates that the daemon has disappeared. 15:12:17 Build request sent: Build{id=4f5e3086-ab55-48c5-a3a4-6f213565ac61, currentDir=/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.12} 15:12:17 Attempting to read last messages from the daemon log... 15:12:17 Daemon pid: 27019 15:12:17 log file: /home/jenkins/.gradle/daemon/5.6.2/daemon-27019.out.log 15:12:17 ----- Last 20 lines from daemon log file - daemon-27019.out.log ----- 15:12:17 at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:237) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:214) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:364) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87) 15:12:17 at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) 15:12:17 at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) 15:12:17 at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) 15:12:17 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 15:12:17 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 15:12:17 at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) 15:12:17 at java.lang.Thread.run(Thread.java:748) 15:12:17 Caused by: java.lang.IllegalStateException: Shutdown in progress 15:12:17 at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) 15:12:17 at java.lang.Runtime.removeShutdownHook(Runtime.java:239) 15:12:17 at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:33) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:204) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:360) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) 15:12:17 ... 7 more 15:12:17 ----- End of the daemon log ----- 15:12:17 15:12:17 15:12:17 FAILURE: Build failed with an exception. 15:12:17 15:12:17 * What went wrong: 15:12:17 Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) 15:12:17 15:12:17 * Try: 15:12:17 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 15:12:17 15:12:17 * Get more help at https://help.gradle.org 15:12:17 > Task :tools:integrationTest 15:12:17 Test steps failed 15:12:17 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:17 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Adding one-line test results to commit status... 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting status of 45ec78a919d6220521b55ca63582d365664b85c1 to FAILURE with url https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/ and message: 'FAILURE 15:12:19 13289 tests run, 67 skipped, 0 failed.' 15:12:19 Using context: JDK 8 and Scala 2.12 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Finished: ABORTED
> {noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)