You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "xiaojian zhou (JIRA)" <ji...@apache.org> on 2016/01/09 03:02:39 UTC
[jira] [Created] (GEODE-774) WAN Cluster outgage due to
DiskAccessException
xiaojian zhou created GEODE-774:
-----------------------------------
Summary: WAN Cluster outgage due to DiskAccessException
Key: GEODE-774
URL: https://issues.apache.org/jira/browse/GEODE-774
Project: Geode
Issue Type: Bug
Reporter: xiaojian zhou
{noformat}
This is the garmin issue we simulated and reproduced in house.
CLIENT vm_4_thr_9_bridge_1_2_w2-2013-lin-04_6672
TASK[0] newWan.WANTest.HydraTask_doOps
ERROR util.TestException: com.gemstone.gemfire.cache.CacheClosedException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1464)
at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:91)
at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:8118)
at com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2994)
at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet$KeysSetIterator.next(PartitionedRegion.java:6544)
at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6638)
at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6631)
at java.util.Collections$UnmodifiableCollection.toArray(Collections.java:1033)
at java.util.ArrayList.addAll(ArrayList.java:577)
at newWan.WANOperationsClient.getExistingKeys(WANOperationsClient.java:792)
at newWan.WANOperationsClient.putAll(WANOperationsClient.java:463)
at newWan.WANOperationsClient.doOps(WANOperationsClient.java:1419)
at newWan.WANTest.doOps(WANTest.java:1650)
at newWan.WANTest.HydraTask_doOps(WANTest.java:1639)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at hydra.MethExecutor.execute(MethExecutor.java:198)
at hydra.MethExecutor.execute(MethExecutor.java:162)
at hydra.TestTask.execute(TestTask.java:195)
at hydra.RemoteTestModule$1.run(RemoteTestModule.java:216)
Caused by: com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1812)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
at com.gemstone.gemfire.internal.cache.Oplog.create(Oplog.java:3553)
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.create(PersistentOplogSet.java:189)
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:751)
at com.gemstone.gemfire.internal.cache.DiskRegion.put(DiskRegion.java:381)
at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeBytesToDisk(DiskEntry.java:874)
at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:867)
at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:857)
at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.doAsyncFlush(DiskEntry.java:1742)
at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1768)
... 1 more
[CIRCULAR REFERENCE:com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)]
To reproduce it, run following bt test:
rollingupgrade/newWan/parallel/garminHCT.conf
bridgeHostsPerSite=2
bridgeThreadsPerVM=2
bridgeVMsPerHost=4
clientMem=2g
edgeHostsPerSite=2
edgeThreadsPerVM=5
edgeVMsPerHost=5
locatorHostsPerSite=2
locatorThreadsPerVM=1
locatorVMsPerHost=1
resultWaitSec=1200
serverMem=12g
wanSites=2
with local.conf:
newWan.WANOperationsClientPrms-randModValueForDebug = true;
hydra.HostPrms-hostNames =
fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf
fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf
fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf
;
hydra.HostPrms-resourceDirBaseMapFileName = /home/xzhou/bin/hostdirs.prop;
hydra.RegionPrms-cacheListeners = none;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)