You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/01/04 21:31:00 UTC
[jira] [Commented] (GEODE-8696) Startup of JMX Manager may hang
during crash of other members
[ https://issues.apache.org/jira/browse/GEODE-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258511#comment-17258511 ]
ASF GitHub Bot commented on GEODE-8696:
---------------------------------------
kirklund commented on pull request #5728:
URL: https://github.com/apache/geode/pull/5728#issuecomment-754230887
JmxServerReconnectDistributedTest hit a java-level deadlock which also shows that DUnit RMI calls don't have a timeout. It hangs forever until the DistributedTest job times out.
```
Found one Java-level deadlock:
=============================
"Management Task1":
waiting to lock monitor 0x00007fa59ba23f28 (object 0x00000007a10a3b98, a org.apache.geode.management.internal.FederatingManager),
which is held by "RMI TCP Connection(8)-169.254.70.169"
"RMI TCP Connection(8)-169.254.70.169":
waiting to lock monitor 0x00007fa52eefecb8 (object 0x00000007a1077650, a java.lang.Object),
which is held by "Management Task1"
Java stack information for the threads listed above:
===================================================
"Management Task1":
at org.apache.geode.management.internal.FederatingManager.isRunning(FederatingManager.java:186)
- waiting to lock <0x00000007a10a3b98> (a org.apache.geode.management.internal.FederatingManager)
at org.apache.geode.management.internal.SystemManagementService.isManager(SystemManagementService.java:348)
at org.apache.geode.management.internal.LocalManager.doManagementTask(LocalManager.java:310)
- locked <0x00000007a1077650> (a java.lang.Object)
at org.apache.geode.management.internal.LocalManager.access$100(LocalManager.java:60)
at org.apache.geode.management.internal.LocalManager$ManagementTask.run(LocalManager.java:373)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RMI TCP Connection(8)-169.254.70.169":
at org.apache.geode.management.internal.LocalManager.unMarkForFederation(LocalManager.java:194)
- waiting to lock <0x00000007a1077650> (a java.lang.Object)
at org.apache.geode.management.internal.SystemManagementService.unregisterMBean(SystemManagementService.java:343)
at org.apache.geode.management.internal.beans.DistributedSystemBridge.removeRegion(DistributedSystemBridge.java:1508)
- locked <0x00000007a1045360> (a java.util.concurrent.ConcurrentHashMap)
at org.apache.geode.management.internal.beans.MBeanAggregator$RegionHandler.handleProxyRemoval(MBeanAggregator.java:324)
at org.apache.geode.management.internal.beans.MBeanAggregator.afterRemoveProxy(MBeanAggregator.java:168)
at org.apache.geode.management.internal.SystemManagementService.afterRemoveProxy(SystemManagementService.java:615)
at org.apache.geode.management.internal.MBeanProxyFactory.removeProxy(MBeanProxyFactory.java:169)
at org.apache.geode.management.internal.MBeanProxyFactory.removeAllProxies(MBeanProxyFactory.java:144)
at org.apache.geode.management.internal.FederatingManager.removeMemberArtifacts(FederatingManager.java:516)
at org.apache.geode.management.internal.FederatingManager.stopManagingActivity(FederatingManager.java:349)
at org.apache.geode.management.internal.FederatingManager.stopManager(FederatingManager.java:181)
- locked <0x00000007a10a3b98> (a org.apache.geode.management.internal.FederatingManager)
at org.apache.geode.management.internal.SystemManagementService.close(SystemManagementService.java:206)
- locked <0x00000007a0ad18b0> (a java.util.HashMap)
at org.apache.geode.management.internal.beans.ManagementAdapter.handleCacheRemoval(ManagementAdapter.java:717)
at org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:131)
at org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2086)
at org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:643)
at org.apache.geode.internal.cache.GemFireCacheImpl.doClose(GemFireCacheImpl.java:2192)
- locked <0x00000007a07d27f0> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2151)
at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1559)
- locked <0x00000007a07d27f0> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1257)
at org.apache.geode.test.dunit.Disconnect.disconnectFromDS(Disconnect.java:44)
at org.apache.geode.test.dunit.rules.DistributedRule$TearDown.tearDownInVM(DistributedRule.java:237)
at org.apache.geode.test.dunit.rules.DistributedRule$TearDown.lambda$doTearDown$bb17a952$1(DistributedRule.java:224)
at org.apache.geode.test.dunit.rules.DistributedRule$TearDown$$Lambda$361/1785950851.run(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$24/803288407.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Startup of JMX Manager may hang during crash of other members
> -------------------------------------------------------------
>
> Key: GEODE-8696
> URL: https://issues.apache.org/jira/browse/GEODE-8696
> Project: Geode
> Issue Type: Bug
> Components: jmx, management
> Reporter: Kirk Lund
> Assignee: Kirk Lund
> Priority: Major
> Labels: GeodeOperationAPI, pull-request-available
>
> The fix for GEODE-7400 removed final from the executorService field to introduce a Supplier<ExecutorService>. I think this hang was caused by adding synchronized to FederatingManager.executeTask instead of making executorService a volatile field.
> _vm_3_thr_3_client2_host2_21145_ hung while synchronized on *0x00000000f6316520*:
> {noformat}
> "vm_3_thr_3_client2_host2_21145" #56 daemon prio=5 os_prio=0 tid=0x00007f1854002000 nid=0x5326 waiting on condition [0x00007f18520e2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f6364030> (a java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:244)
> at org.apache.geode.management.internal.FederatingManager.startManagingActivity(FederatingManager.java:256)
> at org.apache.geode.management.internal.FederatingManager.startManager(FederatingManager.java:121)
> - locked <0x00000000f6316520> (a org.apache.geode.management.internal.FederatingManager)
> at org.apache.geode.management.internal.SystemManagementService.startManager(SystemManagementService.java:373)
> - locked <0x00000000fed91a70> (a java.util.HashMap)
> at org.apache.geode.management.internal.beans.ManagementAdapter.handleCacheCreation(ManagementAdapter.java:197)
> at org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:127)
> at org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2089)
> at org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:643)
> at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1363)
> at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
> - locked <0x00000000e11065f0> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
> - locked <0x00000000e1101220> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
> at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:158)
> - locked <0x00000000e1101220> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:142)
> at hydra.CacheVersionHelper.configureAndCreateCache(CacheVersionHelper.java:51)
> at hydra.CacheHelper.createCacheWithHttpService(CacheHelper.java:127)
> - locked <0x00000000fecab060> (a java.lang.Class for hydra.CacheHelper)
> at hydra.CacheHelper.createCache(CacheHelper.java:87)
> at splitBrain.NetworkPartitionTest.initialize(NetworkPartitionTest.java:293)
> at splitBrain.NetworkPartitionTest.initializeInstance(NetworkPartitionTest.java:228)
> - locked <0x00000000e14a2db0> (a java.lang.Class for splitBrain.NetworkPartitionTest)
> at splitBrain.NetworkPartitionTest.HydraTask_initialize(NetworkPartitionTest.java:203)
> - locked <0x00000000e14a2db0> (a java.lang.Class for splitBrain.NetworkPartitionTest)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at hydra.MethExecutor.execute(MethExecutor.java:173)
> at hydra.MethExecutor.execute(MethExecutor.java:141)
> at hydra.TestTask.execute(TestTask.java:197)
> at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)
> {noformat}
> _vm_3_thr_3_client2_host2_21145_ is waiting for _FederatingManager1_ to complete:
> {noformat}
> "FederatingManager1" #66 daemon prio=5 os_prio=0 tid=0x00007f1874327800 nid=0x5331 waiting on condition [0x00007f18515d9000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f63b2590> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
> at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)
> at org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:639)
> at org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:620)
> at org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:534)
> at org.apache.geode.internal.cache.StateFlushOperation.flush(StateFlushOperation.java:244)
> at org.apache.geode.internal.cache.InitialImageOperation.getFromOne(InitialImageOperation.java:432)
> at org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1236)
> at org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1082)
> at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:2971)
> at org.apache.geode.internal.cache.InternalCacheForClientAccess.createInternalRegion(InternalCacheForClientAccess.java:255)
> at org.apache.geode.management.internal.FederatingManager.addMemberArtifacts(FederatingManager.java:448)
> - locked <0x00000000fec1a480> (a org.apache.geode.distributed.internal.membership.InternalDistributedMember)
> at org.apache.geode.management.internal.FederatingManager$GIITask.call(FederatingManager.java:560)
> at org.apache.geode.management.internal.FederatingManager$GIITask.call(FederatingManager.java:550)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> _DM-MemberEventInvoker_ is preventing _FederatingManager1_ from completing its call to {{ReplyProcessor21.basicWait}} while waiting to lock *0x00000000f6316520* which is held by _vm_3_thr_3_client2_host2_21145_:
> {noformat}
> "DM-MemberEventInvoker" #28 daemon prio=5 os_prio=0 tid=0x00007f1860356000 nid=0x5313 waiting for monitor entry [0x00007f18531f2000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.management.internal.FederatingManager.executeTask(FederatingManager.java:230)
> - waiting to lock <0x00000000f6316520> (a org.apache.geode.management.internal.FederatingManager)
> at org.apache.geode.management.internal.FederatingManager.removeMember(FederatingManager.java:160)
> at org.apache.geode.management.internal.ManagementMembershipListener.memberDeparted(ManagementMembershipListener.java:57)
> at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberCrashedEvent.handleEvent(ClusterDistributionManager.java:2519)
> at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:2424)
> at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:2413)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:1401)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.access$200(ClusterDistributionManager.java:108)
> at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:1433)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)