You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2015/04/12 00:03:12 UTC
[jira] [Created] (TEZ-2310) AM Deadlock in VertexImpl
Daniel Dai created TEZ-2310:
-------------------------------
Summary: AM Deadlock in VertexImpl
Key: TEZ-2310
URL: https://issues.apache.org/jira/browse/TEZ-2310
Project: Apache Tez
Issue Type: Bug
Reporter: Daniel Dai
Fix For: 0.7.0
See the following deadlock in testing:
Thread#1:
{code}
Daemon Thread [App Shared Pool - #3] (Suspended)
owns: VertexManager$VertexManagerPluginContextImpl (id=327)
owns: ShuffleVertexManager (id=328)
owns: VertexManager (id=329)
waiting for: VertexManager$VertexManagerPluginContextImpl (id=326)
VertexManager$VertexManagerPluginContextImpl.onStateUpdated(VertexStateUpdate) line: 344
StateChangeNotifier$ListenerContainer.sendStateUpdate(VertexStateUpdate) line: 138
StateChangeNotifier$ListenerContainer.access$100(StateChangeNotifier$ListenerContainer, VertexStateUpdate) line: 122
StateChangeNotifier.sendStateUpdate(TezVertexID, VertexStateUpdate) line: 116
StateChangeNotifier.stateChanged(TezVertexID, VertexStateUpdate) line: 106
VertexImpl.maybeSendConfiguredEvent() line: 3385
VertexImpl.doneReconfiguringVertex() line: 1634
VertexManager$VertexManagerPluginContextImpl.doneReconfiguringVertex() line: 339
ShuffleVertexManager.schedulePendingTasks(int) line: 561
ShuffleVertexManager.schedulePendingTasks() line: 620
ShuffleVertexManager.handleVertexStateUpdate(VertexStateUpdate) line: 731
ShuffleVertexManager.onVertexStateUpdated(VertexStateUpdate) line: 744
VertexManager$VertexManagerEventOnVertexStateUpdate.invoke() line: 527
VertexManager$VertexManagerEvent$1.run() line: 612
VertexManager$VertexManagerEvent$1.run() line: 607
AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method]
Subject.doAs(Subject, PrivilegedExceptionAction<T>) line: 415
UserGroupInformation.doAs(PrivilegedExceptionAction<T>) line: 1548
VertexManager$VertexManagerEventOnVertexStateUpdate(VertexManager$VertexManagerEvent).call() line: 607
VertexManager$VertexManagerEventOnVertexStateUpdate(VertexManager$VertexManagerEvent).call() line: 596
ListenableFutureTask<V>(FutureTask<V>).run() line: 262
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
{code}
Thread #2
{code}
Daemon Thread [App Shared Pool - #2] (Suspended)
owns: VertexManager$VertexManagerPluginContextImpl (id=326)
owns: PigGraceShuffleVertexManager (id=344)
owns: VertexManager (id=345)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 186
ReentrantReadWriteLock$NonfairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt() line: 834
ReentrantReadWriteLock$NonfairSync(AbstractQueuedSynchronizer).doAcquireShared(int) line: 964
ReentrantReadWriteLock$NonfairSync(AbstractQueuedSynchronizer).acquireShared(int) line: 1282
ReentrantReadWriteLock$ReadLock.lock() line: 731
VertexImpl.getTotalTasks() line: 952
VertexManager$VertexManagerPluginContextImpl.getVertexNumTasks(String) line: 162
PigGraceShuffleVertexManager(ShuffleVertexManager).updateSourceTaskCount() line: 435
PigGraceShuffleVertexManager(ShuffleVertexManager).onVertexStarted(Map<String,List<Integer>>) line: 353
VertexManager$VertexManagerEventOnVertexStarted.invoke() line: 541
VertexManager$VertexManagerEvent$1.run() line: 612
VertexManager$VertexManagerEvent$1.run() line: 607
AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method]
Subject.doAs(Subject, PrivilegedExceptionAction<T>) line: 415
UserGroupInformation.doAs(PrivilegedExceptionAction<T>) line: 1548
VertexManager$VertexManagerEventOnVertexStarted(VertexManager$VertexManagerEvent).call() line: 607
VertexManager$VertexManagerEventOnVertexStarted(VertexManager$VertexManagerEvent).call() line: 596
ListenableFutureTask<V>(FutureTask<V>).run() line: 262
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
{code}
What happens is thread #1 holding a writeLock (VertexImpl:1628) and enter into a synchronized block (ShuffleVertexManager.onVertexStateUpdated), in the mean time, thread #2 already in the synchronized block (ShuffleVertexManager.onVertexStarted) and try to get a readLock(VertexImpl:952). Holding a lock and then enter a synchronized block might be dangerous.
I attach a patch which avoiding that and then deadlock goes away. Not sure if that is the right fix or if any other patterns like this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)