[jira] [Commented] (GEODE-7537) hang in gii/rebalance of AEQ in recycled server (with persistence)

["Eric Shu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GEODE-7537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989062#comment-16989062 ] 

Eric Shu commented on GEODE-7537:
---------------------------------

During startup to recover from disk, GII is being performed. There could be sometimes that GatewaySenderQueueEntrySynchronization is needed during GII:

"P2P message reader for rs-GEM-2778-1346a0i32xlarge-hydra-client-16(bridgegemfire4_host1_12432:12432)<ec><v1>:41006 shared unordered uid=20 port=49952" #60 daemon prio=10 os_prio=0 tid=0x00007f196800c800 nid=0x3b98 waiting for monitor entry [0x00007f19ed5ce000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.geode.internal.cache.CacheFactoryStatics.getAnyInstance(CacheFactoryStatics.java:85)
        - waiting to lock <0x00000000e03ed128> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
        at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:396)
        at org.apache.geode.internal.cache.wan.GatewaySenderQueueEntrySynchronizationOperation$GatewaySenderQueueEntrySynchronizationReplyProcessor.getCache(GatewaySenderQueueEntrySynchronizationOperation.java:171)
        at org.apache.geode.internal.cache.wan.GatewaySenderQueueEntrySynchronizationOperation$GatewaySenderQueueEntrySynchronizationReplyProcessor.putSynchronizationEvents(GatewaySenderQueueEntrySynchronizationOperation.java:165)
        at org.apache.geode.internal.cache.wan.GatewaySenderQueueEntrySynchronizationOperation$GatewaySenderQueueEntrySynchronizationReplyProcessor.process(GatewaySenderQueueEntrySynchronizationOperation.java:152)
        at org.apache.geode.distributed.internal.ReplyMessage.process(ReplyMessage.java:213)
        at org.apache.geode.distributed.internal.ReplyMessage.dmProcess(ReplyMessage.java:197)
        at org.apache.geode.distributed.internal.ReplyMessage.process(ReplyMessage.java:190)
        at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:372)
        at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:427)
        at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2057)
        at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1831)
        at org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$74/261348144.messageReceived(Unknown Source)
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:999)
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership.handleOrDeferMessage(GMSMembership.java:929)
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership.processMessage(GMSMembership.java:1284)
        at org.apache.geode.distributed.internal.DistributionImpl$MyDCReceiver.messageReceived(DistributionImpl.java:820)
        at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:706)
        at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:703)
        at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3393)
        at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:3132)
        at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2927)
        at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1752)
        at org.apache.geode.internal.tcp.Connection.run(Connection.java:1584)
        at java.lang.Thread.run(Thread.java:748)

However, as the cache is still initializing, it holds the lock needed.
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000e132db50> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
        at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:802)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:779)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:865)
        at org.apache.geode.internal.cache.persistence.MembershipViewRequest$MembershipViewRequestReplyProcessor.getResult(MembershipViewRequest.java:195)
        at org.apache.geode.internal.cache.persistence.MembershipViewRequest.send(MembershipViewRequest.java:80)
        at org.apache.geode.internal.cache.persistence.PersistenceAdvisorImpl.updateMembershipView(PersistenceAdvisorImpl.java:248)
        at org.apache.geode.internal.cache.BucketPersistenceAdvisor.updateMembershipView(BucketPersistenceAdvisor.java:185)
        at org.apache.geode.internal.cache.BucketPersistenceAdvisor.updateMembershipView(BucketPersistenceAdvisor.java:249)
        at org.apache.geode.internal.cache.BucketPersistenceAdvisor.initializeMembershipView(BucketPersistenceAdvisor.java:210)
        at org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDisk(ProxyBucketRegion.java:491)
        at org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDiskRecursively(ProxyBucketRegion.java:406)
        at org.apache.geode.internal.cache.PRHARedundancyProvider.recoverPersistentBuckets(PRHARedundancyProvider.java:1661)
        at org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:1069)
        - locked <0x00000000e0a29c90> (a org.apache.geode.internal.cache.PartitionedRegion)
        at org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1179)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3040)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.createRegion(LuceneIndexImpl.java:274)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createRegion(LuceneIndexForPartitionedRegion.java:184)
        at org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createLuceneListenersAndFileChunkRegions(LuceneIndexForPartitionedRegion.java:121)
        at org.apache.geode.cache.lucene.internal.LuceneIndexImpl.initialize(LuceneIndexImpl.java:150)
        at org.apache.geode.cache.lucene.internal.LuceneServiceImpl.afterDataRegionCreated(LuceneServiceImpl.java:366)
        at org.apache.geode.cache.lucene.internal.LuceneRegionListener.afterCreate(LuceneRegionListener.java:117)
        at org.apache.geode.internal.cache.GemFireCacheImpl.invokeRegionAfter(GemFireCacheImpl.java:3116)
        at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3088)
        at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2928)
        at org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:237)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:654)
        at org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:588)
        at org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
        at org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4296)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1392)
        at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1226)
        at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
        - locked <0x00000000e03f5378> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
        - locked <0x00000000e03ed128> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
        at org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
        at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)


> hang in gii/rebalance of AEQ in recycled server (with persistence)
> ------------------------------------------------------------------
>
>                 Key: GEODE-7537
>                 URL: https://issues.apache.org/jira/browse/GEODE-7537
>             Project: Geode
>          Issue Type: Bug
>          Components: lucene
>            Reporter: Mark Hanson
>            Assignee: Eric Shu
>            Priority: Major
>
> Actively being investigated...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)