You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Dinesh Akhand <di...@amdocs.com> on 2018/10/23 13:32:04 UTC
Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Hi team,
Recently we see JVM stuck , in stack trace I can see below method having problem
As per document
From link :
https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheListener.html
this warning:
WARNING: To avoid risk of deadlock, do not invoke CacheFactory.getAnyInstance() from within any callback methods. Instead use EntryEvent.getRegion().getCache() or RegionEvent.getRegion().getCache()
What is the best solution to avoid it.
Function Execution Processor1" #247 daemon prio=10 os_prio=0 tid=0x00007f5798268000 nid=0x3ff5 waiting for monitor entry [0x00007f576adf0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x0000000699feafa0> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
at org.apache.geode.management.internal.cli.functions.GetRegionsFunction.execute(GetRegionsFunction.java:44)
at org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:185)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:662)
at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1108)
at java.lang.Thread.run(Thread.java:745
"P2P message reader for 10.218.110.61(sbimgapp16-server1:65602)<v7>:1026 shared ordered uid=139 port=62033" #403 daemon prio=10 os_prio=0 tid=0x00007f1ad4114800 nid=0xda7c waiting for monitor entry [0x0000
7f1a28fcc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021df685a8> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
at amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ebbe630> (a org.apache.commons.pool.impl.GenericKeyedObjectPool)
at amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x000000021f874d08> (a amdocs.imdg.model.Customer)
at org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at org.apache.geode.internal.InternalDataSerializer.basicRead
Thanks,
Dinesh Akhand
“Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
RE: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Dinesh Akhand <di...@amdocs.com>.
In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a send of a primary put from another server. As part of that put operation, its delivering the event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated. This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the cache initialization is complete.
Regarding above command, First time this happed when JVM is getting up and Function start executing on that same JVM , but JVM is not initialized till now.
So avoiding such issue, I am trying to add the check with new method is geode.
Which tells user function that IsJVMinitializationCompleted then only function execute should work.
Thanks,
Dinesh Akhand
-----Original Message-----
From: Barry Oglesby <bo...@pivotal.io>
Sent: Thursday, October 25, 2018 11:09 PM
To: user@geode.apache.org
Cc: dev@geode.apache.org
Subject: Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
The main thread has locked the CacheFactory (0x000000021de38d70) and is initializing the cache. At this time it is registering a partitioned region by getting a distributed lock. It is waiting for a reply to this request from a remote member:
"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on condition [0x00007f8563ca2000]
java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000001c9d4b2e8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858)
at
org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242)
at
org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537)
at
org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626)
at
org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883)
at
org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318)
at
org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987)
- locked <0x000000021dd35258> (a
org.apache.geode.internal.cache.PartitionedRegion)
at
org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157)
at
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004)
at
org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512)
at
org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
at
org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758)
- locked <0x000000021de39cf0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
at
org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860)
at
org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772)
at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699)
at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231)
In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a send of a primary put from another server. As part of that put operation, its delivering the event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated. This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the cache initialization is complete.
"P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024
shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0
tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry [0x00007f84d0f9a000]
java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at
amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at
amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at
amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at
amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ed39f00> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at
amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at
amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at
amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at
amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x00000002469dd138> (a amdocs.imdg.model.Customer) at
org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at
org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)
at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)
at
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)
at
org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)
- locked <0x000000021e7b50c8> (a
org.apache.geode.internal.cache.VMCachedDeserializable)
at
org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975)
at
org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown
Source)
at
org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007)
at
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970)
at
amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971)
at
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928)
at
org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786)
at
org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689)
at java.lang.Thread.run(Thread.java:745)
Then, there are 9 more P2P message reader threads waiting for the GenericKeyedObjectPool (0x000000021ed39f00) held by this thread.
I'm not sure whether or not this is a deadlock. It would be interesting to see additional thread dumps for this server to see if the main thread is processing or stuck and also thread dumps for the other server(s) to see specifically what the thread processing the DLockRequestMessage is doing (thats the remote thread that the main thread above is waiting for).
Would it be possible to create the VSDCountersManager in the main thread before the P2P message reader thread attempts to do it (like in a Function init)?
Thanks,
Barry Oglesby
On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <ab...@pivotal.io> wrote:
> Sorry I don’t think I understand your question. Did you encounter a
> problem with getAnyInstance()?
>
> Anthony
>
>
> > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <di...@amdocs.com> wrote:
> >
> > No dead local was there , reader thread are waiting mostly.
> > PFA whole stack strace.
> >
> > Thanks,
> > Dinesh Akhand
>
>
“Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
RE: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Dinesh Akhand <di...@amdocs.com>.
In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a send of a primary put from another server. As part of that put operation, its delivering the event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated. This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the cache initialization is complete.
Regarding above command, First time this happed when JVM is getting up and Function start executing on that same JVM , but JVM is not initialized till now.
So avoiding such issue, I am trying to add the check with new method is geode.
Which tells user function that IsJVMinitializationCompleted then only function execute should work.
Thanks,
Dinesh Akhand
-----Original Message-----
From: Barry Oglesby <bo...@pivotal.io>
Sent: Thursday, October 25, 2018 11:09 PM
To: user@geode.apache.org
Cc: dev@geode.apache.org
Subject: Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
The main thread has locked the CacheFactory (0x000000021de38d70) and is initializing the cache. At this time it is registering a partitioned region by getting a distributed lock. It is waiting for a reply to this request from a remote member:
"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on condition [0x00007f8563ca2000]
java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000001c9d4b2e8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858)
at
org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242)
at
org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537)
at
org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626)
at
org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883)
at
org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318)
at
org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987)
- locked <0x000000021dd35258> (a
org.apache.geode.internal.cache.PartitionedRegion)
at
org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157)
at
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004)
at
org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512)
at
org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
at
org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758)
- locked <0x000000021de39cf0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
at
org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860)
at
org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772)
at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699)
at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231)
In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a send of a primary put from another server. As part of that put operation, its delivering the event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated. This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the cache initialization is complete.
"P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024
shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0
tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry [0x00007f84d0f9a000]
java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at
amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at
amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at
amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at
amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ed39f00> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at
amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at
amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at
amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at
amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x00000002469dd138> (a amdocs.imdg.model.Customer) at
org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at
org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)
at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)
at
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)
at
org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)
- locked <0x000000021e7b50c8> (a
org.apache.geode.internal.cache.VMCachedDeserializable)
at
org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975)
at
org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown
Source)
at
org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007)
at
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970)
at
amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971)
at
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928)
at
org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786)
at
org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689)
at java.lang.Thread.run(Thread.java:745)
Then, there are 9 more P2P message reader threads waiting for the GenericKeyedObjectPool (0x000000021ed39f00) held by this thread.
I'm not sure whether or not this is a deadlock. It would be interesting to see additional thread dumps for this server to see if the main thread is processing or stuck and also thread dumps for the other server(s) to see specifically what the thread processing the DLockRequestMessage is doing (thats the remote thread that the main thread above is waiting for).
Would it be possible to create the VSDCountersManager in the main thread before the P2P message reader thread attempts to do it (like in a Function init)?
Thanks,
Barry Oglesby
On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <ab...@pivotal.io> wrote:
> Sorry I don’t think I understand your question. Did you encounter a
> problem with getAnyInstance()?
>
> Anthony
>
>
> > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <di...@amdocs.com> wrote:
> >
> > No dead local was there , reader thread are waiting mostly.
> > PFA whole stack strace.
> >
> > Thanks,
> > Dinesh Akhand
>
>
“Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Barry Oglesby <bo...@pivotal.io>.
The main thread has locked the CacheFactory (0x000000021de38d70) and is
initializing the cache. At this time it is registering a partitioned region
by getting a distributed lock. It is waiting for a reply to this request
from a remote member:
"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on
condition [0x00007f8563ca2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000001c9d4b2e8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858)
at
org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242)
at
org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537)
at
org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626)
at
org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883)
at
org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318)
at
org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987)
- locked <0x000000021dd35258> (a
org.apache.geode.internal.cache.PartitionedRegion)
at
org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157)
at
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004)
at
org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512)
at
org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
at
org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758)
- locked <0x000000021de39cf0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
at
org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860)
at
org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772)
at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699)
at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231)
In the meantime, a PutMessage is being processed by a P2P message reader
thread. This is a send of a primary put from another server. As part of
that put operation, its delivering the event to a gateway sender which
ultimately causes a VSDCountersManager to be instantiated. This is blocked
waiting for the CacheFactory. The CacheFactory won't be available until the
cache initialization is complete.
"P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024
shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0
tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry
[0x00007f84d0f9a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at
amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at
amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at
amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at
amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ed39f00> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at
amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at
amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at
amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at
amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x00000002469dd138> (a amdocs.imdg.model.Customer)
at
org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at
org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)
at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)
at
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)
at
org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)
- locked <0x000000021e7b50c8> (a
org.apache.geode.internal.cache.VMCachedDeserializable)
at
org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975)
at
org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown
Source)
at
org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007)
at
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970)
at
amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971)
at
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928)
at
org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786)
at
org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689)
at java.lang.Thread.run(Thread.java:745)
Then, there are 9 more P2P message reader threads waiting for the
GenericKeyedObjectPool (0x000000021ed39f00) held by this thread.
I'm not sure whether or not this is a deadlock. It would be interesting to
see additional thread dumps for this server to see if the main thread is
processing or stuck and also thread dumps for the other server(s) to see
specifically what the thread processing the DLockRequestMessage is doing
(thats the remote thread that the main thread above is waiting for).
Would it be possible to create the VSDCountersManager in the main thread
before the P2P message reader thread attempts to do it (like in a Function
init)?
Thanks,
Barry Oglesby
On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <ab...@pivotal.io> wrote:
> Sorry I don’t think I understand your question. Did you encounter a
> problem with getAnyInstance()?
>
> Anthony
>
>
> > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <di...@amdocs.com> wrote:
> >
> > No dead local was there , reader thread are waiting mostly.
> > PFA whole stack strace.
> >
> > Thanks,
> > Dinesh Akhand
>
>
Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Barry Oglesby <bo...@pivotal.io>.
The main thread has locked the CacheFactory (0x000000021de38d70) and is
initializing the cache. At this time it is registering a partitioned region
by getting a distributed lock. It is waiting for a reply to this request
from a remote member:
"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on
condition [0x00007f8563ca2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000001c9d4b2e8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858)
at
org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242)
at
org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537)
at
org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626)
at
org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883)
at
org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318)
at
org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987)
- locked <0x000000021dd35258> (a
org.apache.geode.internal.cache.PartitionedRegion)
at
org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157)
at
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004)
at
org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512)
at
org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
at
org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758)
- locked <0x000000021de39cf0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
at
org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860)
at
org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772)
at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699)
at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231)
In the meantime, a PutMessage is being processed by a P2P message reader
thread. This is a send of a primary put from another server. As part of
that put operation, its delivering the event to a gateway sender which
ultimately causes a VSDCountersManager to be instantiated. This is blocked
waiting for the CacheFactory. The CacheFactory won't be available until the
cache initialization is complete.
"P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024
shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0
tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry
[0x00007f84d0f9a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at
amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at
amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at
amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at
amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ed39f00> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at
amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at
amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at
amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at
amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x00000002469dd138> (a amdocs.imdg.model.Customer)
at
org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at
org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)
at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)
at
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)
at
org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)
- locked <0x000000021e7b50c8> (a
org.apache.geode.internal.cache.VMCachedDeserializable)
at
org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975)
at
org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown
Source)
at
org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007)
at
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970)
at
amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971)
at
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928)
at
org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786)
at
org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689)
at java.lang.Thread.run(Thread.java:745)
Then, there are 9 more P2P message reader threads waiting for the
GenericKeyedObjectPool (0x000000021ed39f00) held by this thread.
I'm not sure whether or not this is a deadlock. It would be interesting to
see additional thread dumps for this server to see if the main thread is
processing or stuck and also thread dumps for the other server(s) to see
specifically what the thread processing the DLockRequestMessage is doing
(thats the remote thread that the main thread above is waiting for).
Would it be possible to create the VSDCountersManager in the main thread
before the P2P message reader thread attempts to do it (like in a Function
init)?
Thanks,
Barry Oglesby
On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <ab...@pivotal.io> wrote:
> Sorry I don’t think I understand your question. Did you encounter a
> problem with getAnyInstance()?
>
> Anthony
>
>
> > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <di...@amdocs.com> wrote:
> >
> > No dead local was there , reader thread are waiting mostly.
> > PFA whole stack strace.
> >
> > Thanks,
> > Dinesh Akhand
>
>
Re: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Anthony Baker <ab...@pivotal.io>.
Sorry I don’t think I understand your question. Did you encounter a problem with getAnyInstance()?
Anthony
> On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <di...@amdocs.com> wrote:
>
> No dead local was there , reader thread are waiting mostly.
> PFA whole stack strace.
>
> Thanks,
> Dinesh Akhand
Re: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Anthony Baker <ab...@pivotal.io>.
Sorry I don’t think I understand your question. Did you encounter a problem with getAnyInstance()?
Anthony
> On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <di...@amdocs.com> wrote:
>
> No dead local was there , reader thread are waiting mostly.
> PFA whole stack strace.
>
> Thanks,
> Dinesh Akhand
RE: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Dinesh Akhand <di...@amdocs.com>.
No dead local was there , reader thread are waiting mostly.
PFA whole stack strace.
Thanks,
Dinesh Akhand
-----Original Message-----
From: Anthony Baker <ab...@pivotal.io>
Sent: Tuesday, October 23, 2018 7:34 PM
To: dev@geode.apache.org
Cc: user@geode.apache.org
Subject: Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Dinesh, have you analyzed the full thread dump to see if there is a deadlock? I can’t tell just from these 2 threads if there is a deadlock.
Anthony
> On Oct 23, 2018, at 6:32 AM, Dinesh Akhand <di...@amdocs.com> wrote:
>
> Hi team,
>
> Recently we see JVM stuck , in stack trace I can see below method
> having problem
>
> As per document
>
> From link :
>
>
> https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cach
> e/CacheListener.html
>
> this warning:
>
> WARNING: To avoid risk of deadlock, do not invoke
> CacheFactory.getAnyInstance() from within any callback methods.
> Instead use EntryEvent.getRegion().getCache() or
> RegionEvent.getRegion().getCache()
> What is the best solution to avoid it.
>
> Function Execution Processor1" #247 daemon prio=10 os_prio=0 tid=0x00007f5798268000 nid=0x3ff5 waiting for monitor entry [0x00007f576adf0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x0000000699feafa0> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at org.apache.geode.management.internal.cli.functions.GetRegionsFunction.execute(GetRegionsFunction.java:44)
> at org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:185)
> at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
> at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:662)
> at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1108)
> at java.lang.Thread.run(Thread.java:745
>
> "P2P message reader for
> 10.218.110.61(sbimgapp16-server1:65602)<v7>:1026 shared ordered uid=139 port=62033" #403 daemon prio=10 os_prio=0 tid=0x00007f1ad4114800 nid=0xda7c waiting for monitor entry [0x0000 7f1a28fcc000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x000000021df685a8> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
> at amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
> at amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
> at amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
> at amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
> at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
> - locked <0x000000021ebbe630> (a org.apache.commons.pool.impl.GenericKeyedObjectPool)
> at amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
> at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
> at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
> at amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
> at amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
> at amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
> at amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
> at amdocs.imdg.model.Customer.fromData(Customer.java:696)
> - locked <0x000000021f874d08> (a amdocs.imdg.model.Customer)
> at org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
> at org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
> at org.apache.geode.internal.InternalDataSerializer.basicRead
>
>
>
> Thanks,
> Dinesh Akhand
>
> “Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
“Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
RE: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Dinesh Akhand <di...@amdocs.com>.
No dead local was there , reader thread are waiting mostly.
PFA whole stack strace.
Thanks,
Dinesh Akhand
-----Original Message-----
From: Anthony Baker <ab...@pivotal.io>
Sent: Tuesday, October 23, 2018 7:34 PM
To: dev@geode.apache.org
Cc: user@geode.apache.org
Subject: Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Dinesh, have you analyzed the full thread dump to see if there is a deadlock? I can’t tell just from these 2 threads if there is a deadlock.
Anthony
> On Oct 23, 2018, at 6:32 AM, Dinesh Akhand <di...@amdocs.com> wrote:
>
> Hi team,
>
> Recently we see JVM stuck , in stack trace I can see below method
> having problem
>
> As per document
>
> From link :
>
>
> https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cach
> e/CacheListener.html
>
> this warning:
>
> WARNING: To avoid risk of deadlock, do not invoke
> CacheFactory.getAnyInstance() from within any callback methods.
> Instead use EntryEvent.getRegion().getCache() or
> RegionEvent.getRegion().getCache()
> What is the best solution to avoid it.
>
> Function Execution Processor1" #247 daemon prio=10 os_prio=0 tid=0x00007f5798268000 nid=0x3ff5 waiting for monitor entry [0x00007f576adf0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x0000000699feafa0> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at org.apache.geode.management.internal.cli.functions.GetRegionsFunction.execute(GetRegionsFunction.java:44)
> at org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:185)
> at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
> at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:662)
> at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1108)
> at java.lang.Thread.run(Thread.java:745
>
> "P2P message reader for
> 10.218.110.61(sbimgapp16-server1:65602)<v7>:1026 shared ordered uid=139 port=62033" #403 daemon prio=10 os_prio=0 tid=0x00007f1ad4114800 nid=0xda7c waiting for monitor entry [0x0000 7f1a28fcc000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x000000021df685a8> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
> at amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
> at amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
> at amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
> at amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
> at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
> - locked <0x000000021ebbe630> (a org.apache.commons.pool.impl.GenericKeyedObjectPool)
> at amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
> at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
> at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
> at amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
> at amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
> at amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
> at amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
> at amdocs.imdg.model.Customer.fromData(Customer.java:696)
> - locked <0x000000021f874d08> (a amdocs.imdg.model.Customer)
> at org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
> at org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
> at org.apache.geode.internal.InternalDataSerializer.basicRead
>
>
>
> Thanks,
> Dinesh Akhand
>
> “Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
“Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
Re: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Anthony Baker <ab...@pivotal.io>.
Dinesh, have you analyzed the full thread dump to see if there is a deadlock? I can’t tell just from these 2 threads if there is a deadlock.
Anthony
> On Oct 23, 2018, at 6:32 AM, Dinesh Akhand <di...@amdocs.com> wrote:
>
> Hi team,
>
> Recently we see JVM stuck , in stack trace I can see below method having problem
>
> As per document
>
> From link :
>
> https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheListener.html
>
> this warning:
>
> WARNING: To avoid risk of deadlock, do not invoke CacheFactory.getAnyInstance() from within any callback methods. Instead use EntryEvent.getRegion().getCache() or RegionEvent.getRegion().getCache()
> What is the best solution to avoid it.
>
> Function Execution Processor1" #247 daemon prio=10 os_prio=0 tid=0x00007f5798268000 nid=0x3ff5 waiting for monitor entry [0x00007f576adf0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x0000000699feafa0> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at org.apache.geode.management.internal.cli.functions.GetRegionsFunction.execute(GetRegionsFunction.java:44)
> at org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:185)
> at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
> at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:662)
> at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1108)
> at java.lang.Thread.run(Thread.java:745
>
> "P2P message reader for 10.218.110.61(sbimgapp16-server1:65602)<v7>:1026 shared ordered uid=139 port=62033" #403 daemon prio=10 os_prio=0 tid=0x00007f1ad4114800 nid=0xda7c waiting for monitor entry [0x0000
> 7f1a28fcc000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x000000021df685a8> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
> at amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
> at amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
> at amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
> at amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
> at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
> - locked <0x000000021ebbe630> (a org.apache.commons.pool.impl.GenericKeyedObjectPool)
> at amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
> at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
> at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
> at amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
> at amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
> at amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
> at amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
> at amdocs.imdg.model.Customer.fromData(Customer.java:696)
> - locked <0x000000021f874d08> (a amdocs.imdg.model.Customer)
> at org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
> at org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
> at org.apache.geode.internal.InternalDataSerializer.basicRead
>
>
>
> Thanks,
> Dinesh Akhand
>
> “Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.
Re: Thread block on
org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Posted by Anthony Baker <ab...@pivotal.io>.
Dinesh, have you analyzed the full thread dump to see if there is a deadlock? I can’t tell just from these 2 threads if there is a deadlock.
Anthony
> On Oct 23, 2018, at 6:32 AM, Dinesh Akhand <di...@amdocs.com> wrote:
>
> Hi team,
>
> Recently we see JVM stuck , in stack trace I can see below method having problem
>
> As per document
>
> From link :
>
> https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheListener.html
>
> this warning:
>
> WARNING: To avoid risk of deadlock, do not invoke CacheFactory.getAnyInstance() from within any callback methods. Instead use EntryEvent.getRegion().getCache() or RegionEvent.getRegion().getCache()
> What is the best solution to avoid it.
>
> Function Execution Processor1" #247 daemon prio=10 os_prio=0 tid=0x00007f5798268000 nid=0x3ff5 waiting for monitor entry [0x00007f576adf0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x0000000699feafa0> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at org.apache.geode.management.internal.cli.functions.GetRegionsFunction.execute(GetRegionsFunction.java:44)
> at org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:185)
> at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
> at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:662)
> at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1108)
> at java.lang.Thread.run(Thread.java:745
>
> "P2P message reader for 10.218.110.61(sbimgapp16-server1:65602)<v7>:1026 shared ordered uid=139 port=62033" #403 daemon prio=10 os_prio=0 tid=0x00007f1ad4114800 nid=0xda7c waiting for monitor entry [0x0000
> 7f1a28fcc000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
> - waiting to lock <0x000000021df685a8> (a java.lang.Class for org.apache.geode.cache.CacheFactory)
> at amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
> at amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
> at amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
> at amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
> at amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
> at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
> - locked <0x000000021ebbe630> (a org.apache.commons.pool.impl.GenericKeyedObjectPool)
> at amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
> at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
> at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
> at amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
> at amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
> at amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
> at amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
> at amdocs.imdg.model.Customer.fromData(Customer.java:696)
> - locked <0x000000021f874d08> (a amdocs.imdg.model.Customer)
> at org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
> at org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
> at org.apache.geode.internal.InternalDataSerializer.basicRead
>
>
>
> Thanks,
> Dinesh Akhand
>
> “Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.