You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Anilkumar Gingade (JIRA)" <ji...@apache.org> on 2019/04/01 18:31:00 UTC
[jira] [Created] (GEODE-6581) deadlock between tombstone gc and
region initial image processing threads
Anilkumar Gingade created GEODE-6581:
----------------------------------------
Summary: deadlock between tombstone gc and region initial image processing threads
Key: GEODE-6581
URL: https://issues.apache.org/jira/browse/GEODE-6581
Project: Geode
Issue Type: Bug
Components: regions
Reporter: Anilkumar Gingade
There is a potential for tombstoneGC thread to be dead-locked with initial image processing thread due to the order of locking with RegionEntry and RegionSize lock (below stack dump).
This is similar to GEODE-6526 except here the removeTombstone is called from initial image put opertation.
{noformat}
Found one Java-level deadlock:
=============================
"Pooled Message Processor 90":
waiting to lock monitor 0x00007fa780005978 (object 0x00000000fa19a890, a java.lang.Object),
which is held by "Pooled Message Processor 14"
"Pooled Message Processor 14":
waiting to lock monitor 0x00007fa7940edb68 (object 0x00000000fa633db0, a org.apache.geode.internal.cache.entries.VersionedThinDiskRegionEntryHeapStringKey2),
which is held by "Pooled High Priority Message Processor 10"
"Pooled High Priority Message Processor 10":
waiting to lock monitor 0x00007fa7940edab8 (object 0x00000000fbdaf610, a java.lang.String),
which is held by "Pooled Message Processor 14"
Java stack information for the threads listed above:
===================================================
"Pooled Message Processor 90":
at org.apache.geode.internal.cache.TombstoneService.gcTombstones(TombstoneService.java:209)
- waiting to lock <0x00000000fa19a890> (a java.lang.Object)
at org.apache.geode.internal.cache.LocalRegion.expireTombstones(LocalRegion.java:3293)
at org.apache.geode.internal.cache.DistributedTombstoneOperation$TombstoneMessage.operateOnRegion(DistributedTombstoneOperation.java:169)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1191)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1091)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:378)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:444)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:1121)
at org.apache.geode.distributed.internal.ClusterDistributionManager.access$000(ClusterDistributionManager.java:109)
at org.apache.geode.distributed.internal.ClusterDistributionManager$4$1.run(ClusterDistributionManager.java:791)
at java.lang.Thread.run(Thread.java:748)
"Pooled Message Processor 14":
at org.apache.geode.internal.cache.AbstractRegionMap.removeTombstone(AbstractRegionMap.java:3322)
- waiting to lock <0x00000000fa633db0> (a org.apache.geode.internal.cache.entries.VersionedThinDiskRegionEntryHeapStringKey2)
- locked <0x00000000fbdaf610> (a java.lang.String)
at org.apache.geode.internal.cache.TombstoneService.gcTombstones(TombstoneService.java:259)
- locked <0x00000000fa19a890> (a java.lang.Object)
at org.apache.geode.internal.cache.LocalRegion.expireTombstones(LocalRegion.java:3293)
at org.apache.geode.internal.cache.DistributedTombstoneOperation$TombstoneMessage.operateOnRegion(DistributedTombstoneOperation.java:169)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1191)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1091)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:378)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:444)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:1121)
at org.apache.geode.distributed.internal.ClusterDistributionManager.access$000(ClusterDistributionManager.java:109)
at org.apache.geode.distributed.internal.ClusterDistributionManager$4$1.run(ClusterDistributionManager.java:791)
at java.lang.Thread.run(Thread.java:748)
"Pooled High Priority Message Processor 10":
at org.apache.geode.internal.cache.AbstractRegionMap.removeTombstone(AbstractRegionMap.java:3321)
- waiting to lock <0x00000000fbdaf610> (a java.lang.String)
at org.apache.geode.internal.cache.AbstractRegionMap.initialImagePut(AbstractRegionMap.java:954)
- locked <0x00000000fa633db0> (a org.apache.geode.internal.cache.entries.VersionedThinDiskRegionEntryHeapStringKey2)
- locked <0x00000000fbde7e18> (a org.apache.geode.internal.cache.entries.VersionedThinDiskRegionEntryHeapStringKey2)
at org.apache.geode.internal.cache.InitialImageOperation.processChunk(InitialImageOperation.java:933)
- locked <0x00000000fa633db0> (a org.apache.geode.internal.cache.entries.VersionedThinDiskRegionEntryHeapStringKey2)
at org.apache.geode.internal.cache.InitialImageOperation$ImageProcessor.process(InitialImageOperation.java:1309)
at org.apache.geode.distributed.internal.ReplyMessage.process(ReplyMessage.java:213)
at org.apache.geode.internal.cache.InitialImageOperation$ImageReplyMessage.process(InitialImageOperation.java:2789)
at org.apache.geode.distributed.internal.ReplyMessage.dmProcess(ReplyMessage.java:193)
at org.apache.geode.distributed.internal.ReplyMessage.process(ReplyMessage.java:186)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:378)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:444)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:1121)
at org.apache.geode.distributed.internal.ClusterDistributionManager.access$000(ClusterDistributionManager.java:109)
at org.apache.geode.distributed.internal.ClusterDistributionManager$5$1.run(ClusterDistributionManager.java:832)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)