You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "xiaojian zhou (JIRA)" <ji...@apache.org> on 2016/02/28 19:09:18 UTC
[jira] [Resolved] (GEODE-774) WAN Cluster outgage due to
DiskAccessException
[ https://issues.apache.org/jira/browse/GEODE-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
xiaojian zhou resolved GEODE-774.
---------------------------------
Resolution: Fixed
Assignee: xiaojian zhou
Fix Version/s: 1.0.0-incubating.M1
1.0.0-incubating
> WAN Cluster outgage due to DiskAccessException
> ----------------------------------------------
>
> Key: GEODE-774
> URL: https://issues.apache.org/jira/browse/GEODE-774
> Project: Geode
> Issue Type: Bug
> Reporter: xiaojian zhou
> Assignee: xiaojian zhou
> Fix For: 1.0.0-incubating, 1.0.0-incubating.M1
>
>
> {noformat}
> This is the 52550 issue we simulated and reproduced in house.
> CLIENT vm_4_thr_9_bridge_1_2_w2-2013-lin-04_6672
> TASK[0] newWan.WANTest.HydraTask_doOps
> ERROR util.TestException: com.gemstone.gemfire.cache.CacheClosedException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
> at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1464)
> at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:91)
> at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:8118)
> at com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2994)
> at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet$KeysSetIterator.next(PartitionedRegion.java:6544)
> at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6638)
> at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6631)
> at java.util.Collections$UnmodifiableCollection.toArray(Collections.java:1033)
> at java.util.ArrayList.addAll(ArrayList.java:577)
> at newWan.WANOperationsClient.getExistingKeys(WANOperationsClient.java:792)
> at newWan.WANOperationsClient.putAll(WANOperationsClient.java:463)
> at newWan.WANOperationsClient.doOps(WANOperationsClient.java:1419)
> at newWan.WANTest.doOps(WANTest.java:1650)
> at newWan.WANTest.HydraTask_doOps(WANTest.java:1639)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at hydra.MethExecutor.execute(MethExecutor.java:198)
> at hydra.MethExecutor.execute(MethExecutor.java:162)
> at hydra.TestTask.execute(TestTask.java:195)
> at hydra.RemoteTestModule$1.run(RemoteTestModule.java:216)
> Caused by: com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1812)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
> at com.gemstone.gemfire.internal.cache.Oplog.create(Oplog.java:3553)
> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.create(PersistentOplogSet.java:189)
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:751)
> at com.gemstone.gemfire.internal.cache.DiskRegion.put(DiskRegion.java:381)
> at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeBytesToDisk(DiskEntry.java:874)
> at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:867)
> at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:857)
> at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.doAsyncFlush(DiskEntry.java:1742)
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1768)
> ... 1 more
> [CIRCULAR REFERENCE:com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)]
> To reproduce it, run following bt test:
> rollingupgrade/newWan/parallel/garminHCT.conf
> bridgeHostsPerSite=2
> bridgeThreadsPerVM=2
> bridgeVMsPerHost=4
> clientMem=2g
> edgeHostsPerSite=2
> edgeThreadsPerVM=5
> edgeVMsPerHost=5
> locatorHostsPerSite=2
> locatorThreadsPerVM=1
> locatorVMsPerHost=1
> resultWaitSec=1200
> serverMem=12g
> wanSites=2
> with local.conf:
> newWan.WANOperationsClientPrms-randModValueForDebug = true;
> hydra.HostPrms-hostNames =
> fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf
> fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf
> fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf
> ;
> hydra.HostPrms-resourceDirBaseMapFileName = /home/xzhou/bin/hostdirs.prop;
> hydra.RegionPrms-cacheListeners = none;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)