You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Eric Shu (JIRA)" <ji...@apache.org> on 2018/10/17 16:46:00 UTC
[jira] [Created] (GEODE-5888) A removeAll/putAll operation could
remove lockObject held by another thread if region is closing
Eric Shu created GEODE-5888:
-------------------------------
Summary: A removeAll/putAll operation could remove lockObject held by another thread if region is closing
Key: GEODE-5888
URL: https://issues.apache.org/jira/browse/GEODE-5888
Project: Geode
Issue Type: Bug
Components: regions
Reporter: Eric Shu
A bulk operation needs to hold lockObjects in BucketRegion.waitUntilLocked. However, it may hit RegionDestroyedException instead, and it will release lockObjects it does not hold.
org.apache.geode.cache.RegionDestroyedException: Partitioned Region @21554698 [path='/partitionedRegion'; dataPolicy=PERSISTENT_PARTITION; prId=1; isDestroyed=true; isClosed=true; retryTimeout=3600000; serialNumber=7; partition attributes=PartitionAttributes@2144398013[redundantCopies=0;localMaxMemory=205;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM rs-GEM-1352-1526a0i3large-hydra-client-34(dataStoregemfire4_host1_19090:19090)<ec><v23>:1028]
at org.apache.geode.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:7396)
at org.apache.geode.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2745)
at org.apache.geode.internal.cache.BucketRegion.waitUntilLocked(BucketRegion.java:493)
at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.doLocalRemoveAll(RemoveAllPRMessage.java:437)
at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.operateOnPartitionedRegion(RemoveAllPRMessage.java:317)
at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:325)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:367)
at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:424)
at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2873)
at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:2553)
at org.apache.geode.distributed.internal.ClusterDistributionManager.access$1300(ClusterDistributionManager.java:109)
at org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.messageReceived(ClusterDistributionManager.java:3411)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1102)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1022)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:399)
at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:722)
at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:824)
at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3898)
at org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3508)
at org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1814)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Once the exception is hit, it will remove the lockObjects hold by another thread performing another operation.
at org.apache.geode.internal.cache.BucketRegion.removeAndNotifyKeys(BucketRegion.java:454)
at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.doLocalRemoveAll(RemoveAllPRMessage.java:534)
at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.operateOnPartitionedRegion(RemoveAllPRMessage.java:317)
at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:325)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:367)
at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:424)
at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2873)
at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:2553)
at org.apache.geode.distributed.internal.ClusterDistributionManager.access$1300(ClusterDistributionManager.java:109)
at org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.messageReceived(ClusterDistributionManager.java:3411)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1102)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1022)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:399)
at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:722)
at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:824)
at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3898)
at org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3508)
at org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1814)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This causes a race condition that a new operation could come in and work on a region entry that the other thread is working on but the lockObject of the region entry was removed by the thread hitting the RegionDestroyedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)