You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2014/09/05 22:40:29 UTC
[jira] [Commented] (BROOKLYN-66) Deadlock in group.addMember() when
unmanaging other member of group
[ https://issues.apache.org/jira/browse/BROOKLYN-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123536#comment-14123536 ]
ASF GitHub Bot commented on BROOKLYN-66:
----------------------------------------
GitHub user aledsage opened a pull request:
https://github.com/apache/incubator-brooklyn/pull/151
BROOKLYN-66: fix deadlock (and various others)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aledsage/incubator-brooklyn fix/deadlock
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-brooklyn/pull/151.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #151
----
commit 7ea0a6578bde8bad35da175593cbd714bb749ce5
Author: Aled Sage <al...@gmail.com>
Date: 2014-09-05T18:08:01Z
BROOKLYN-66: Fix deadlock in LocalEntityManager
commit 112193a9b84c932e992e6a6cef7414abf48e10bf
Author: Aled Sage <al...@gmail.com>
Date: 2014-09-05T18:22:32Z
Fix concurrency issue stopping cluster
- when resizing to zero, we shrink by currentSize. If someone
concurrently stops an entity, then don’t worry that we were only
able to stop (currentSize-1) entities. Just continue.
commit ccb92a88b0b82f1b2a486debd5965dc301168159
Author: Aled Sage <al...@gmail.com>
Date: 2014-09-05T18:55:18Z
Fix failing ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember
commit a49280957f5515de54ac6961eec522f16e04130a
Author: Aled Sage <al...@gmail.com>
Date: 2014-09-05T19:02:12Z
Change EntityAdjuncts.findWithUniqueTag to tryFindWithUniqueTag
- Make it consistent with guava naming
commit ac567c16ed39a14deca22075f0ec29314c2fe8b6
Author: Aled Sage <al...@gmail.com>
Date: 2014-09-05T19:10:32Z
Move some tests form AbstractEntityLegacyTest to EntitySpecTest
commit 0447fbbd13d4f6fbdc2e3f5aefa6f8ec8c95c65d
Author: Aled Sage <al...@gmail.com>
Date: 2014-09-05T19:14:02Z
Simplify HighAvailabilityManagerSplitBrainTest
commit 2503de7bb6c993b5dc964d78004262ff2d784916
Author: Aled Sage <al...@gmail.com>
Date: 2014-09-05T19:17:36Z
CatalogLibrariesDo: minor tidy
----
> Deadlock in group.addMember() when unmanaging other member of group
> -------------------------------------------------------------------
>
> Key: BROOKLYN-66
> URL: https://issues.apache.org/jira/browse/BROOKLYN-66
> Project: Brooklyn
> Issue Type: Bug
> Affects Versions: 0.7.0-M1
> Reporter: Aled Sage
>
> When running `ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember` lots of times, I hit this deadlock.
> In brief...
> When method is synchronized, hit deadlock:
> 1. thread called unmanage() on a member of a group, so we got the lock on LocalEntityManager
> and called group.removeMember; this ties to synchronize on AbstractGroupImpl.members
> 2. another thread was doing AbstractGroupImpl.addMember, which is synchronized on AbstractGroupImpl.members;
> it tries to call Entities.manage(child) which calls LocalEntityManager.getEntity(), which is
> synchronized on this.
> We MUST NOT call alien code from within the management framework while holding locks.
> The AbstractGroup.removeMember is effectively alien because a user could override it, and because
> it is entity specific. Therefore this should be fixed in LocalEntityManager.
> Found one Java-level deadlock:
> =============================
> "brooklyn-execmanager-Lol69DXC-10":
> waiting to lock monitor 7f9774943fc0 (object 7f3303f70, a brooklyn.management.internal.LocalEntityManager),
> which is held by "brooklyn-execmanager-Lol69DXC-2"
> "brooklyn-execmanager-Lol69DXC-2":
> waiting to lock monitor 7f977a88b5f0 (object 7f36e75d0, a brooklyn.util.collections.SetFromLiveMap),
> which is held by "brooklyn-execmanager-Lol69DXC-10"
> Java stack information for the threads listed above:
> ===================================================
> "brooklyn-execmanager-Lol69DXC-10":
> at brooklyn.management.internal.LocalEntityManager.getEntity(LocalEntityManager.java:198)
> - waiting to lock <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
> at brooklyn.management.internal.LocalEntityManager.isManaged(LocalEntityManager.java:207)
> at brooklyn.management.internal.LocalEntityManager.manage(LocalEntityManager.java:235)
> at brooklyn.entity.basic.Entities.manage(Entities.java:768)
> at brooklyn.entity.basic.AbstractGroupImpl.addMember(AbstractGroupImpl.java:132)
> - locked <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
> at com.sun.proxy.$Proxy57.addMember(Unknown Source)
> at brooklyn.entity.group.DynamicClusterImpl.quarantineFailedNodes(DynamicClusterImpl.java:683)
> at brooklyn.entity.group.DynamicClusterImpl.addInEachLocation(DynamicClusterImpl.java:670)
> at brooklyn.entity.group.DynamicClusterImpl.addInSingleLocation(DynamicClusterImpl.java:615)
> at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:484)
> - locked <7f34800e8> (a [Ljava.lang.Object;)
> at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:474)
> - locked <7f34800e8> (a [Ljava.lang.Object;)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
> at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
> at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
> at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
> at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
> at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
> at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
> at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
> at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:149)
> at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
> at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
> at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
> at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> "brooklyn-execmanager-Lol69DXC-2":
> at brooklyn.entity.basic.AbstractGroupImpl.removeMember(AbstractGroupImpl.java:148)
> - waiting to lock <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
> at com.sun.proxy.$Proxy57.removeMember(Unknown Source)
> at brooklyn.management.internal.LocalEntityManager.unmanageNonRecursive(LocalEntityManager.java:408)
> - locked <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
> at brooklyn.management.internal.LocalEntityManager.unmanage(LocalEntityManager.java:284)
> at brooklyn.entity.basic.Entities.unmcaanage(Entities.java:851)
> at brooklyn.entity.group.DynamicClusterImpl.discardNode(DynamicClusterImpl.java:836)
> at brooklyn.entity.group.DynamicClusterImpl.shrink(DynamicClusterImpl.java:609)
> at brooklyn.entity.group.DynamicClusterImpl.stop(DynamicClusterImpl.java:393)
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
> at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
> at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
> at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
> at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
> at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
> at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
> at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
> at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:102)
> at com.sun.proxy.$Proxy56.stop(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:166)
> at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
> at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
> at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
> at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Found 1 deadlock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)