[jira] [Created] (TWILL-105) Race condition causing deadlock in ZKDecoratorService

["Terence Yim (JIRA)" <ji...@apache.org>]

Terence Yim created TWILL-105:
---------------------------------

             Summary: Race condition causing deadlock in ZKDecoratorService
                 Key: TWILL-105
                 URL: https://issues.apache.org/jira/browse/TWILL-105
             Project: Apache Twill
          Issue Type: Bug
          Components: core
    Affects Versions: 0.3.0-incubating, 0.4.0-incubating
            Reporter: Terence Yim
            Assignee: Terence Yim
            Priority: Blocker


In the ZKDecoratorService there are race condition that when starting/stopping the service. Here is the deadlock detected when starting service. Similar one was observed during stopping.

Found one Java-level deadlock:
=============================
" STARTING":
  waiting for ownable synchronizer 7dc0b0070, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "main"
"main":
  waiting to lock monitor 7fb91a80e718 (object 7dc0b0198, a java.util.concurrent.ConcurrentLinkedQueue),
  which is held by " STARTING"

Java stack information for the threads listed above:
===================================================
" STARTING":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <7dc0b0070> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at com.google.common.util.concurrent.AbstractService.notifyStarted(AbstractService.java:235)
at org.apache.twill.internal.ZKServiceDecorator.access$2400(ZKServiceDecorator.java:69)
at org.apache.twill.internal.ZKServiceDecorator$DecoratedServiceListener.running(ZKServiceDecorator.java:408)
at com.google.common.util.concurrent.AbstractService$3$1.run(AbstractService.java:397)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
at com.google.common.util.concurrent.AbstractService$ListenerExecutorPair.execute(AbstractService.java:470)
at com.google.common.util.concurrent.AbstractService$3.run(AbstractService.java:395)
at com.google.common.util.concurrent.AbstractService.executeListeners(AbstractService.java:369)
- locked <7dc0b0198> (a java.util.concurrent.ConcurrentLinkedQueue)
at com.google.common.util.concurrent.AbstractService.notifyStarted(AbstractService.java:255)
at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:44)
at java.lang.Thread.run(Thread.java:695)
"main":
at com.google.common.util.concurrent.AbstractService.executeListeners(AbstractService.java:368)
- waiting to lock <7dc0b0198> (a java.util.concurrent.ConcurrentLinkedQueue)
at com.google.common.util.concurrent.AbstractService.start(AbstractService.java:176)
at com.google.common.util.concurrent.AbstractIdleService.start(AbstractIdleService.java:102)
at org.apache.twill.internal.ZKServiceDecorator$1$1.run(ZKServiceDecorator.java:133)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:160)
at org.apache.twill.internal.ZKServiceDecorator$1.onSuccess(ZKServiceDecorator.java:126)
at org.apache.twill.internal.ZKServiceDecorator$1.onSuccess(ZKServiceDecorator.java:114)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:799)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
at org.apache.twill.internal.zookeeper.SettableOperationFuture$1.run(SettableOperationFuture.java:56)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:160)
at org.apache.twill.internal.zookeeper.SettableOperationFuture.addListener(SettableOperationFuture.java:53)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:809)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:749)
at org.apache.twill.internal.ZKServiceDecorator.doStart(ZKServiceDecorator.java:114)
at com.google.common.util.concurrent.AbstractService.start(AbstractService.java:170)
at org.apache.twill.internal.state.ZKServiceDecoratorTest.testStateTransition(ZKServiceDecoratorTest.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

Found 1 deadlock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)