You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Jeff Zhang (JIRA)" <ji...@apache.org> on 2015/05/29 13:16:18 UTC
[jira] [Comment Edited] (TEZ-2391) TestVertexImpl timing out at
times on jenkins builds
[ https://issues.apache.org/jira/browse/TEZ-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564591#comment-14564591 ]
Jeff Zhang edited comment on TEZ-2391 at 5/29/15 11:15 AM:
-----------------------------------------------------------
e.g. TestVertexImpl#testInputInitializerEvents fails very frequently on ubuntu.
This is a bug in TestStateChangeNotifier.
TestStateChangeNotifier#addedEventToQueue runs in dispatcher thread while TestStateChangeNotifier#processedEventFromQueue runs in state change notifier event handling thread. It is not guaranteed that addedEventToQueue is invoked before processedEventFromQueue. So if processedEventFromQueue is invoked before addedEventToQueue, this would cause wait there. Upload one patch to fix this issue. [~hitesh] [~bikassaha] Please help review it.
* sleep in TestStateChangeNotifier#processedEventFromQueue until there's available events
* invoke updateTracker.stop() after dispatcher is stopped. otherwise there will still be hang issue in the teardown method. e.g. state change notifier event handling thread is stopped before all the events are processed TestStateChangeNotifier#processedEventFromQueue, in this case the dispatcher thread will hang in TestStateChangeNotifier#addedEventToQueue.
Besides that, I found another issue in TestVertexImpl, sometimes I will meet the following issue.
{code}
testInputInitializerVertexStateUpdates(org.apache.tez.dag.app.dag.impl.TestVertexImpl)
java.lang.AssertionError: expected:<3> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at org.apache.tez.dag.app.dag.impl.TestVertexImpl.testInputInitializerVertexStateUpdates(TestVertexImpl.java:3887)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
was (Author: zjffdu):
e.g. TestVertexImpl#testInputInitializerEvents fails very frequently on ubuntu.
This is a bug in TestStateChangeNotifier.
TestStateChangeNotifier#addedEventToQueue runs in dispatcher thread while TestStateChangeNotifier#processedEventFromQueue runs in state change notifier event handling thread. It is not guaranteed that addedEventToQueue is invoked before processedEventFromQueue. So if processedEventFromQueue is invoked before addedEventToQueue, this would cause wait there. Upload one patch to fix this issue.
* sleep in TestStateChangeNotifier#processedEventFromQueue until there's available events
* invoke updateTracker.stop() after dispatcher is stopped. otherwise there will still be hang issue in the teardown method. e.g. state change notifier event handling thread is stopped before all the events are processed TestStateChangeNotifier#processedEventFromQueue, in this case the dispatcher thread will hang in TestStateChangeNotifier#addedEventToQueue.
Besides that, I found another issue in TestVertexImpl, sometimes I will meet the following issue.
{code}
testInputInitializerVertexStateUpdates(org.apache.tez.dag.app.dag.impl.TestVertexImpl)
java.lang.AssertionError: expected:<3> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at org.apache.tez.dag.app.dag.impl.TestVertexImpl.testInputInitializerVertexStateUpdates(TestVertexImpl.java:3887)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
> TestVertexImpl timing out at times on jenkins builds
> -----------------------------------------------------
>
> Key: TEZ-2391
> URL: https://issues.apache.org/jira/browse/TEZ-2391
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Hitesh Shah
> Assignee: Mit Desai
> Priority: Blocker
> Attachments: TEZ-2391-1.patch, TEZ-2391.patch, TestVertexImpl-output.txt
>
>
> For example, https://builds.apache.org/job/Tez-Build/1028/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)