You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Barry Oglesby <bo...@pivotal.io> on 2016/03/16 21:52:45 UTC
Review Request 44929: GEODE-1067: The dispatcher now handles
IllegalStateException by retrying batch without the released event
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44929/
-----------------------------------------------------------
Review request for geode, anilkumar gingade and Dan Smith.
Repository: geode
Description
-------
GEODE-1067: The dispatcher now handles IllegalStateException by retrying batch without the released event
Diffs
-----
geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/AbstractGatewaySenderEventProcessor.java 51b125a4a51cbfde3141404dae7084e0432b341b
geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventImpl.java d8922f874830831fff573f8f1a6a9ed09fbcfde8
geode-wan/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventRemoteDispatcher.java 22dff3dc898cfc89c3ed1752f6cb352717140263
Diff: https://reviews.apache.org/r/44929/diff/
Testing
-------
- repeatedly ran ParallelGatewaySenderOperation_2_OffHeapDUnitTest (along fixes for GEODE-670 and GEODE-1095)
- running precheckin
Thanks,
Barry Oglesby
Re: Review Request 44929: GEODE-1067: The dispatcher now handles
IllegalStateException by retrying batch without the released event
Posted by Barry Oglesby <bo...@pivotal.io>.
> On March 17, 2016, 5:06 p.m., Dan Smith wrote:
> > What are the reasons why the serialized value is not available? Could this change cause us to drop events without realizing it due to bugs in getting the off heap value?
The data PR is locally destroyed which in turn locally destroys the queue PR. This clears the AbstractRegionMap which iterates the entries and releases them:
[warn 2016/03/17 12:37:45.423 PDT <RMI TCP Connection(1)-192.168.2.7> tid=0x13] AAA GatewaySenderEventImpl.release released key: 5
java.lang.Exception
at com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventImpl.release(GatewaySenderEventImpl.java:1259)
at com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventImpl.release(GatewaySenderEventImpl.java:1265)
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap$Segment.clear(CustomEntryConcurrentHashMap.java:1114)
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.clear(CustomEntryConcurrentHashMap.java:1976)
at com.gemstone.gemfire.internal.cache.AbstractRegionMap._mapClear(AbstractRegionMap.java:421)
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.clear(AbstractRegionMap.java:467)
at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.clear(AbstractLRURegionMap.java:678)
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.close(AbstractRegionMap.java:432)
at com.gemstone.gemfire.internal.cache.LocalRegion.closeEntries(LocalRegion.java:2979)
at com.gemstone.gemfire.internal.cache.AbstractBucketRegionQueue.access$001(AbstractBucketRegionQueue.java:52)
at com.gemstone.gemfire.internal.cache.AbstractBucketRegionQueue$1.run(AbstractBucketRegionQueue.java:503)
at com.gemstone.gemfire.internal.offheap.OffHeapRegionEntryHelper.doWithOffHeapClear(OffHeapRegionEntryHelper.java:409)
at com.gemstone.gemfire.internal.cache.AbstractBucketRegionQueue.closeEntries(AbstractBucketRegionQueue.java:500)
at com.gemstone.gemfire.internal.cache.BucketRegionQueue.access$001(BucketRegionQueue.java:61)
at com.gemstone.gemfire.internal.cache.BucketRegionQueue$2.run(BucketRegionQueue.java:227)
at com.gemstone.gemfire.internal.offheap.OffHeapRegionEntryHelper.doWithOffHeapClear(OffHeapRegionEntryHelper.java:409)
at com.gemstone.gemfire.internal.cache.BucketRegionQueue.closeEntries(BucketRegionQueue.java:224)
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.basicDestroy(DiskStoreImpl.java:2673)
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.endDestroyRegion(DiskStoreImpl.java:3273)
at com.gemstone.gemfire.internal.cache.AbstractDiskRegion.endDestroy(AbstractDiskRegion.java:564)
at com.gemstone.gemfire.internal.cache.AbstractDiskRegion.beginDestroy(AbstractDiskRegion.java:581)
at com.gemstone.gemfire.internal.cache.PartitionedRegion.closePartitionedRegion(PartitionedRegion.java:7694)
at com.gemstone.gemfire.internal.cache.PartitionedRegion.postDestroyRegion(PartitionedRegion.java:8362)
at com.gemstone.gemfire.internal.cache.LocalRegion.recursiveDestroyRegion(LocalRegion.java:2948)
at com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6886)
at com.gemstone.gemfire.internal.cache.PartitionedRegion.destroyParallelGatewaySenderRegion(PartitionedRegion.java:7901)
at com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6871)
at com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6834)
at com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7984)
at com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7945)
at com.gemstone.gemfire.internal.cache.AbstractRegion.localDestroyRegion(AbstractRegion.java:354)
at com.gemstone.gemfire.internal.cache.wan.WANTestBase.localDestroyRegion(WANTestBase.java:3212)
There are a few things that could be looked at:
After a GatewaySenderEventImpl is peeked from the queue in ParallelGatewaySenderQueue.peek, makeHeapCopyIfOffHeap is invoked on it. That call doesn't really do anything.
Basically the test in GatewaySenderEventImpl.makeHeapCopyIfOffHeap (below) is false because hasRefCount is false so 'this' is returned (no copy is made). Should hasRefCount return true here? If it did return true, then a copy would be made, and I think this issue would be resolved.
if (v instanceof StoredObject && ((StoredObject) v).hasRefCount()) {
Also, the queue PR is being destroyed with cacheWrite=false (set in PartitionedRegion.localDestroyRegion):
[warn 2016/03/17 12:31:38.057 PDT <RMI TCP Connection(1)-192.168.2.7> tid=0x13] AAA PartitionedRegion.destroyParallelGatewaySenderRegion destroying queue region /ln2_PARALLEL_GATEWAY_SENDER_QUEUE; cacheWrite: false
java.lang.Exception
at com.gemstone.gemfire.internal.cache.PartitionedRegion.destroyParallelGatewaySenderRegion(PartitionedRegion.java:7880)
at com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6871)
at com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6834)
at com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7984)
at com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7945)
at com.gemstone.gemfire.internal.cache.AbstractRegion.localDestroyRegion(AbstractRegion.java:354)
at com.gemstone.gemfire.internal.cache.wan.WANTestBase.localDestroyRegion(WANTestBase.java:3212)
at com.gemstone.gemfire.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOperation_2_DUnitTest.lambda$testParallelGatewaySenders_MultiplNode_UserPR_localDestroy_RecreateXXX$76295ecb$3(ConcurrentParallelGatewaySenderOperation_2_DUnitTest.java:339)
at com.gemstone.gemfire.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOperation_2_DUnitTest$$Lambda$31/936369667.run(Unknown Source)
If cacheWrite were true, then the destroyParallelGatewaySenderRegion would wait for the queue to drain before destroying the queue PR. There are other uses of cacheWrite in this method though, so I'm not sure what the consequences of changing this to true are.
if (cacheWrite && parallelQueueRegion.size() != 0 && keepWaiting) {
continue;
}
- Barry
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44929/#review124048
-----------------------------------------------------------
On March 16, 2016, 8:52 p.m., Barry Oglesby wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44929/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 8:52 p.m.)
>
>
> Review request for geode, anilkumar gingade and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> GEODE-1067: The dispatcher now handles IllegalStateException by retrying batch without the released event
>
>
> Diffs
> -----
>
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/AbstractGatewaySenderEventProcessor.java 51b125a4a51cbfde3141404dae7084e0432b341b
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventImpl.java d8922f874830831fff573f8f1a6a9ed09fbcfde8
> geode-wan/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventRemoteDispatcher.java 22dff3dc898cfc89c3ed1752f6cb352717140263
>
> Diff: https://reviews.apache.org/r/44929/diff/
>
>
> Testing
> -------
>
> - repeatedly ran ParallelGatewaySenderOperation_2_OffHeapDUnitTest (along fixes for GEODE-670 and GEODE-1095)
> - running precheckin
>
>
> Thanks,
>
> Barry Oglesby
>
>
Re: Review Request 44929: GEODE-1067: The dispatcher now handles
IllegalStateException by retrying batch without the released event
Posted by Dan Smith <ds...@pivotal.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44929/#review124048
-----------------------------------------------------------
What are the reasons why the serialized value is not available? Could this change cause us to drop events without realizing it due to bugs in getting the off heap value?
- Dan Smith
On March 16, 2016, 8:52 p.m., Barry Oglesby wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44929/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 8:52 p.m.)
>
>
> Review request for geode, anilkumar gingade and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> GEODE-1067: The dispatcher now handles IllegalStateException by retrying batch without the released event
>
>
> Diffs
> -----
>
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/AbstractGatewaySenderEventProcessor.java 51b125a4a51cbfde3141404dae7084e0432b341b
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventImpl.java d8922f874830831fff573f8f1a6a9ed09fbcfde8
> geode-wan/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventRemoteDispatcher.java 22dff3dc898cfc89c3ed1752f6cb352717140263
>
> Diff: https://reviews.apache.org/r/44929/diff/
>
>
> Testing
> -------
>
> - repeatedly ran ParallelGatewaySenderOperation_2_OffHeapDUnitTest (along fixes for GEODE-670 and GEODE-1095)
> - running precheckin
>
>
> Thanks,
>
> Barry Oglesby
>
>
Re: Review Request 44929: GEODE-1067: The dispatcher now handles
IllegalStateException by retrying batch without the released event
Posted by Dan Smith <ds...@pivotal.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44929/#review124092
-----------------------------------------------------------
Ship it!
Ship It!
- Dan Smith
On March 16, 2016, 8:52 p.m., Barry Oglesby wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44929/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 8:52 p.m.)
>
>
> Review request for geode, anilkumar gingade and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> GEODE-1067: The dispatcher now handles IllegalStateException by retrying batch without the released event
>
>
> Diffs
> -----
>
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/AbstractGatewaySenderEventProcessor.java 51b125a4a51cbfde3141404dae7084e0432b341b
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventImpl.java d8922f874830831fff573f8f1a6a9ed09fbcfde8
> geode-wan/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventRemoteDispatcher.java 22dff3dc898cfc89c3ed1752f6cb352717140263
>
> Diff: https://reviews.apache.org/r/44929/diff/
>
>
> Testing
> -------
>
> - repeatedly ran ParallelGatewaySenderOperation_2_OffHeapDUnitTest (along fixes for GEODE-670 and GEODE-1095)
> - running precheckin
>
>
> Thanks,
>
> Barry Oglesby
>
>