RE: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)

[Dinesh Akhand <di...@amdocs.com>]

In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a send of a primary put from another server. As part of that put operation, its delivering the event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated. This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the cache initialization is complete.

Regarding above command,  First time this happed when JVM is getting up and Function start executing on that same JVM , but JVM is not initialized till now.
So avoiding such issue, I am trying to add the check with new method is geode.

Which tells user function that IsJVMinitializationCompleted then only function execute should work.

Thanks,
Dinesh Akhand

-----Original Message-----
From: Barry Oglesby <bo...@pivotal.io> 
Sent: Thursday, October 25, 2018 11:09 PM
To: user@geode.apache.org
Cc: dev@geode.apache.org
Subject: Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)

The main thread has locked the CacheFactory (0x000000021de38d70) and is initializing the cache. At this time it is registering a partitioned region by getting a distributed lock. It is waiting for a reply to this request from a remote member:

"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on condition [0x00007f8563ca2000]
   java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000001c9d4b2e8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858)
at
org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242)
at
org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537)
at
org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626)
at
org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883)
at
org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318)
at
org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987)
- locked <0x000000021dd35258> (a
org.apache.geode.internal.cache.PartitionedRegion)
at
org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157)
at
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004)
at
org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512)
at
org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
at
org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758)
- locked <0x000000021de39cf0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
at
org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860)
at
org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772)
at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699)
at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231)

In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a send of a primary put from another server. As part of that put operation, its delivering the event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated. This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the cache initialization is complete.

"P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024
shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0
tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry [0x00007f84d0f9a000]
   java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at
amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at
amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at
amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at
amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ed39f00> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at
amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at
amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at
amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at
amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x00000002469dd138> (a amdocs.imdg.model.Customer) at
org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at
org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)
at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)
at
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)
at
org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)
- locked <0x000000021e7b50c8> (a
org.apache.geode.internal.cache.VMCachedDeserializable)
at
org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975)
at
org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown
Source)
at
org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007)
at
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970)
at
amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971)
at
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928)
at
org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786)
at
org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689)
at java.lang.Thread.run(Thread.java:745)

Then, there are 9 more P2P message reader threads waiting for the GenericKeyedObjectPool (0x000000021ed39f00) held by this thread.

I'm not sure whether or not this is a deadlock. It would be interesting to see additional thread dumps for this server to see if the main thread is processing or stuck and also thread dumps for the other server(s) to see specifically what the thread processing the DLockRequestMessage is doing (thats the remote thread that the main thread above is waiting for).

Would it be possible to create the VSDCountersManager in the main thread before the P2P message reader thread attempts to do it (like in a Function init)?

Thanks,
Barry Oglesby



On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <ab...@pivotal.io> wrote:

> Sorry I don’t think I understand your question.  Did you encounter a 
> problem with getAnyInstance()?
>
> Anthony
>
>
> > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <di...@amdocs.com> wrote:
> >
> > No dead local was there , reader thread are waiting mostly.
> > PFA whole stack strace.
> >
> > Thanks,
> > Dinesh Akhand
>
>
“Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”.