You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Kirk Lund (Jira)" <ji...@apache.org> on 2020/04/15 18:03:00 UTC
[jira] [Comment Edited] (GEODE-7503) Creating cache after closing
cache may throw DistributedSystemDisconnectedException
[ https://issues.apache.org/jira/browse/GEODE-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982984#comment-16982984 ]
Kirk Lund edited comment on GEODE-7503 at 4/15/20, 6:02 PM:
------------------------------------------------------------
One known cause in dunit tests is when creation of a persistent region fails and DiskStoreImpl.lambda$handleDiskAccessException forks a new Thread to close the Cache which succeeds in closing the Cache before the main test thread (which is invoking Cache.close()) closes it. The main test thread then early-outs because the DiskStore thread is already closing the Cache. The main test thread then tries to create a Cache which collides with the DiskStore thread which is still closing the Cache and is about to disconnect the DistributedSystem.
{noformat}
java.lang.Throwable: KIRK GemFireCacheImpl closed 632230948
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2365)
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1917)
at org.apache.geode.internal.cache.DiskStoreImpl.lambda$handleDiskAccessException$2(DiskStoreImpl.java:3380)
at java.lang.Thread.run(Thread.java:748)
{noformat}
{noformat}
java.lang.Throwable: KIRK InternalDistributedSystem closed 306674056
at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1637)
at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1225)
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2351)
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1917)
at org.apache.geode.internal.cache.DiskStoreImpl.lambda$handleDiskAccessException$2(DiskStoreImpl.java:3380)
at java.lang.Thread.run(Thread.java:748)
{noformat}
{noformat}
java.lang.Throwable: KIRK DiskStoreImpl closing cache 1555793073
at org.apache.geode.internal.cache.DiskStoreImpl.handleDiskAccessException(DiskStoreImpl.java:3376)
at org.apache.geode.internal.cache.PartitionedRegion.createAndValidatePersistentConfig(PartitionedRegion.java:956)
at org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:999)
at org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1179)
at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3043)
at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2931)
at org.apache.geode.internal.cache.GemFireCacheImpl.createRegion(GemFireCacheImpl.java:2918)
at org.apache.geode.cache.RegionFactory.create(RegionFactory.java:755)
at org.apache.geode.cache.CacheFactoryRecreateRegressionTest.createCacheAndColocatedPRs(CacheFactoryRecreateRegressionTest.java:109)
at org.apache.geode.cache.CacheFactoryRecreateRegressionTest.lambda$recreateDoesNotThrowDistributedSystemDisconnectedException$0(CacheFactoryRecreateRegressionTest.java:62)
at org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:62)
at org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:786)
at org.assertj.core.api.Assertions.catchThrowable(Assertions.java:1200)
at org.apache.geode.cache.CacheFactoryRecreateRegressionTest.recreateDoesNotThrowDistributedSystemDisconnectedException(CacheFactoryRecreateRegressionTest.java:62)
{noformat}
was (Author: klund):
This bug appears to be caused when creation of a persistent region fails and DiskStoreImpl.lambda$handleDiskAccessException forks a new Thread to close the Cache which succeeds in closing the Cache before the main test thread closes it. The main test thread then early outs because the DiskStore thread is already closing the Cache. The main test thread then tries to create a Cache which collides with the DiskStore thread which is still closing the Cache and DistributedSystem.
{noformat}
java.lang.Throwable: KIRK GemFireCacheImpl closed 632230948
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2365)
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1917)
at org.apache.geode.internal.cache.DiskStoreImpl.lambda$handleDiskAccessException$2(DiskStoreImpl.java:3380)
at java.lang.Thread.run(Thread.java:748)
{noformat}
{noformat}
java.lang.Throwable: KIRK InternalDistributedSystem closed 306674056
at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1637)
at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1225)
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2351)
at org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1917)
at org.apache.geode.internal.cache.DiskStoreImpl.lambda$handleDiskAccessException$2(DiskStoreImpl.java:3380)
at java.lang.Thread.run(Thread.java:748)
{noformat}
{noformat}
java.lang.Throwable: KIRK DiskStoreImpl closing cache 1555793073
at org.apache.geode.internal.cache.DiskStoreImpl.handleDiskAccessException(DiskStoreImpl.java:3376)
at org.apache.geode.internal.cache.PartitionedRegion.createAndValidatePersistentConfig(PartitionedRegion.java:956)
at org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:999)
at org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1179)
at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3043)
at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2931)
at org.apache.geode.internal.cache.GemFireCacheImpl.createRegion(GemFireCacheImpl.java:2918)
at org.apache.geode.cache.RegionFactory.create(RegionFactory.java:755)
at org.apache.geode.cache.CacheFactoryRecreateRegressionTest.createCacheAndColocatedPRs(CacheFactoryRecreateRegressionTest.java:109)
at org.apache.geode.cache.CacheFactoryRecreateRegressionTest.lambda$recreateDoesNotThrowDistributedSystemDisconnectedException$0(CacheFactoryRecreateRegressionTest.java:62)
at org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:62)
at org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:786)
at org.assertj.core.api.Assertions.catchThrowable(Assertions.java:1200)
at org.apache.geode.cache.CacheFactoryRecreateRegressionTest.recreateDoesNotThrowDistributedSystemDisconnectedException(CacheFactoryRecreateRegressionTest.java:62)
{noformat}
> Creating cache after closing cache may throw DistributedSystemDisconnectedException
> -----------------------------------------------------------------------------------
>
> Key: GEODE-7503
> URL: https://issues.apache.org/jira/browse/GEODE-7503
> Project: Geode
> Issue Type: Bug
> Components: core
> Reporter: Kirk Lund
> Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> This problem is caused by a race condition and may reproduce if you attempt to create a Cache using the CacheFactory immediately after more than one thread invokes Cache.close().
> {noformat}
> org.apache.geode.distributed.DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected.
> at org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:945)
> at org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1665)
> at org.apache.geode.internal.cache.GemFireCacheImpl.<init>(GemFireCacheImpl.java:791)
> at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:187)
> at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:158)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:142)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)