You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Eric Shu (JIRA)" <ji...@apache.org> on 2018/10/17 16:46:00 UTC
[jira] [Updated] (GEODE-5888) A removeAll/putAll operation could
remove lockObject held by another thread if region is closing
[ https://issues.apache.org/jira/browse/GEODE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Shu updated GEODE-5888:
----------------------------
Affects Version/s: 1.1.0
> A removeAll/putAll operation could remove lockObject held by another thread if region is closing
> ------------------------------------------------------------------------------------------------
>
> Key: GEODE-5888
> URL: https://issues.apache.org/jira/browse/GEODE-5888
> Project: Geode
> Issue Type: Bug
> Components: regions
> Affects Versions: 1.1.0
> Reporter: Eric Shu
> Priority: Major
>
> A bulk operation needs to hold lockObjects in BucketRegion.waitUntilLocked. However, it may hit RegionDestroyedException instead, and it will release lockObjects it does not hold.
> org.apache.geode.cache.RegionDestroyedException: Partitioned Region @21554698 [path='/partitionedRegion'; dataPolicy=PERSISTENT_PARTITION; prId=1; isDestroyed=true; isClosed=true; retryTimeout=3600000; serialNumber=7; partition attributes=PartitionAttributes@2144398013[redundantCopies=0;localMaxMemory=205;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM rs-GEM-1352-1526a0i3large-hydra-client-34(dataStoregemfire4_host1_19090:19090)<ec><v23>:1028]
> at org.apache.geode.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:7396)
> at org.apache.geode.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2745)
> at org.apache.geode.internal.cache.BucketRegion.waitUntilLocked(BucketRegion.java:493)
> at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.doLocalRemoveAll(RemoveAllPRMessage.java:437)
> at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.operateOnPartitionedRegion(RemoveAllPRMessage.java:317)
> at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:325)
> at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:367)
> at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:424)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2873)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:2553)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.access$1300(ClusterDistributionManager.java:109)
> at org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.messageReceived(ClusterDistributionManager.java:3411)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1102)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1022)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:399)
> at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:722)
> at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:824)
> at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3898)
> at org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3508)
> at org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1814)
> at org.apache.geode.internal.tcp.Connection.run(Connection.java:1676)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Once the exception is hit, it will remove the lockObjects hold by another thread performing another operation.
> at org.apache.geode.internal.cache.BucketRegion.removeAndNotifyKeys(BucketRegion.java:454)
> at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.doLocalRemoveAll(RemoveAllPRMessage.java:534)
> at org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.operateOnPartitionedRegion(RemoveAllPRMessage.java:317)
> at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:325)
> at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:367)
> at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:424)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2873)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:2553)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.access$1300(ClusterDistributionManager.java:109)
> at org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.messageReceived(ClusterDistributionManager.java:3411)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1102)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1022)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:399)
> at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:722)
> at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:824)
> at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3898)
> at org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3508)
> at org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1814)
> at org.apache.geode.internal.tcp.Connection.run(Connection.java:1676)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> This causes a race condition that a new operation could come in and work on a region entry that the other thread is working on but the lockObject of the region entry was removed by the thread hitting the RegionDestroyedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)