You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Kapil Goyal <go...@vmware.com> on 2016/10/14 01:46:12 UTC
GemFire persisted data corruption - how to debug?
Hi Folks,
I am doing some crash testing with a single cache node of GemFire, where I power off the VM where cache is running and then bring it back up. Upon restart, GemFire refuses to come up with this error:
Caused by: java.lang.NullPointerException
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.getDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverRegionsThatAreReady(PersistentOplogSet.java:367) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverRegionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
It hints at GemFire data on disk being corrupted, so I used 'gfsh' to verify:
gfsh>validate offline-disk-store --name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
Validating nsxDiskStore
/nsx_sys/ArrayListIDPriorityModel: entryCount=0
/nsx_sys/Crl: entryCount=0
/nsx_sys/Certificate: entryCount=1
......
Error in validating disk store nsxDiskStore is : null
This confirms that the disk-store is corrupted, but doesn't give any more information to debug this further. How do I go about debugging this? Have you seen this before and are there any fixes/workarounds available?
Thanks
Kapil
Re: GemFire persisted data corruption - how to debug?
Posted by Kapil Goyal <go...@vmware.com>.
Hi Michael,
Thanks for the suggestion and I do agree with FT and HA part. It’s important for us to work with a single node as of now and use GemFire as a persistent store, somewhat as a replacement of a traditional database server such as mysql.
Regards
Kapil
From: Michael Stolz <ms...@pivotal.io>>
Reply-To: "user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>" <us...@geode.incubator.apache.org>>
Date: Friday, October 14, 2016 at 2:00 PM
To: "user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>" <us...@geode.incubator.apache.org>>
Subject: Re: GemFire persisted data corruption - how to debug?
It is highly unusual to use Geode with just a single cache node. A big part of the value of an In-Memory Data Grid is that it can provide fault-tolerance and high-availability for your data. Please consider running at least 3 nodes in your tests as that would be the minimum real-world configuration that Geode would likely be used in.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: 631-835-4771
On Sat, Oct 15, 2016 at 4:59 AM, Kapil Goyal <go...@vmware.com>> wrote:
Thanks Anthony.
We have already enabled synchronous disk writes to minimize data loss in the event of crash.
From: Anthony Baker <ab...@pivotal.io>>
Reply-To: <us...@geode.incubator.apache.org>>
Date: Thursday, October 13, 2016 at 8:31 PM
To: <us...@geode.incubator.apache.org>>
Subject: Re: GemFire persisted data corruption - how to debug?
Hi Kapil,
Geode (by default) writes data synchronously to other cluster members. If a node crashes like in your test, the update is preserved by the cluster even in the absence of persistence. Synchronous disk writes can be turned on (see [1]) but many users prefer to avoid the fsync performance penalty.
Anthony
[1] https://cwiki.apache.org/confluence/display/GEODE/Native+Disk+Persistence<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Native-2BDisk-2BPersistence&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=jI39iGhJSMsophpPzgwoqWd6xw05tZ5QPHjlmT5c7Tw&m=6Xqiqt_blRT_CsKoO_DzK7RypEVblo86aOPVKMAKqQo&s=hnJ3DPIaTd3dcYRSz3CxWwcJ0cAr6OmhA7SLX2A99iM&e=>
On Oct 13, 2016, at 6:46 PM, Kapil Goyal <go...@vmware.com>> wrote:
Hi Folks,
I am doing some crash testing with a single cache node of GemFire, where I power off the VM where cache is running and then bring it back up. Upon restart, GemFire refuses to come up with this error:
Caused by: java.lang.NullPointerException
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.getDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverRegionsThatAreReady(PersistentOplogSet.java:367) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverRegionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
It hints at GemFire data on disk being corrupted, so I used 'gfsh' to verify:
gfsh>validate offline-disk-store --name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
Validating nsxDiskStore
/nsx_sys/ArrayListIDPriorityModel: entryCount=0
/nsx_sys/Crl: entryCount=0
/nsx_sys/Certificate: entryCount=1
……
Error in validating disk store nsxDiskStore is : null
This confirms that the disk-store is corrupted, but doesn't give any more information to debug this further. How do I go about debugging this? Have you seen this before and are there any fixes/workarounds available?
Thanks
Kapil
Re: GemFire persisted data corruption - how to debug?
Posted by Michael Stolz <ms...@pivotal.io>.
It is highly unusual to use Geode with just a single cache node. A big part
of the value of an In-Memory Data Grid is that it can provide
fault-tolerance and high-availability for your data. Please consider
running at least 3 nodes in your tests as that would be the minimum
real-world configuration that Geode would likely be used in.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: 631-835-4771
On Sat, Oct 15, 2016 at 4:59 AM, Kapil Goyal <go...@vmware.com> wrote:
> Thanks Anthony.
>
> We have already enabled synchronous disk writes to minimize data loss in
> the event of crash.
>
> From: Anthony Baker <ab...@pivotal.io>
> Reply-To: <us...@geode.incubator.apache.org>
> Date: Thursday, October 13, 2016 at 8:31 PM
> To: <us...@geode.incubator.apache.org>
> Subject: Re: GemFire persisted data corruption - how to debug?
>
> Hi Kapil,
>
> Geode (by default) writes data synchronously to other cluster members. If
> a node crashes like in your test, the update is preserved by the cluster
> even in the absence of persistence. Synchronous disk writes can be turned
> on (see [1]) but many users prefer to avoid the fsync performance penalty.
>
> Anthony
>
> [1] https://cwiki.apache.org/confluence/display/GEODE/
> Native+Disk+Persistence
>
> On Oct 13, 2016, at 6:46 PM, Kapil Goyal <go...@vmware.com> wrote:
>
> Hi Folks,
>
> I am doing some crash testing with a single cache node of GemFire, where I
> power off the VM where cache is running and then bring it back up. Upon
> restart, GemFire refuses to come up with this error:
>
> Caused by: java.lang.NullPointerException
> at com.gemstone.gemfire.internal.util.concurrent.
> CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$
> HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.util.concurrent.
> CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.AbstractRegionMap.
> getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.
> getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.
> getDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.
> recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.
> recoverRegionsThatAreReady(PersistentOplogSet.java:367)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.
> recoverRegionsThatAreReady(DiskStoreImpl.java:2065)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.
> initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.
> doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.
> create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.
> createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425)
> ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.
> create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.
> loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
> at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306)
> ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
> at org.springframework.data.gemfire.CacheFactoryBean.
> getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.
> RELEASE.jar:1.5.2.RELEASE]
>
> It hints at GemFire data on disk being corrupted, so I used 'gfsh' to
> verify:
>
> gfsh>validate offline-disk-store --name=nsxDiskStore
> --disk-dirs=/common/nsxapi/data/self
>
> Validating nsxDiskStore
> /nsx_sys/ArrayListIDPriorityModel: entryCount=0
> /nsx_sys/Crl: entryCount=0
> /nsx_sys/Certificate: entryCount=1
> ……
> Error in validating disk store nsxDiskStore is : null
>
> This confirms that the disk-store is corrupted, but doesn't give any more
> information to debug this further. How do I go about debugging this? Have
> you seen this before and are there any fixes/workarounds available?
>
> Thanks
> Kapil
>
>
>
Re: GemFire persisted data corruption - how to debug?
Posted by Kapil Goyal <go...@vmware.com>.
Thanks Anthony.
We have already enabled synchronous disk writes to minimize data loss in the event of crash.
From: Anthony Baker <ab...@pivotal.io>>
Reply-To: <us...@geode.incubator.apache.org>>
Date: Thursday, October 13, 2016 at 8:31 PM
To: <us...@geode.incubator.apache.org>>
Subject: Re: GemFire persisted data corruption - how to debug?
Hi Kapil,
Geode (by default) writes data synchronously to other cluster members. If a node crashes like in your test, the update is preserved by the cluster even in the absence of persistence. Synchronous disk writes can be turned on (see [1]) but many users prefer to avoid the fsync performance penalty.
Anthony
[1] https://cwiki.apache.org/confluence/display/GEODE/Native+Disk+Persistence
On Oct 13, 2016, at 6:46 PM, Kapil Goyal <go...@vmware.com>> wrote:
Hi Folks,
I am doing some crash testing with a single cache node of GemFire, where I power off the VM where cache is running and then bring it back up. Upon restart, GemFire refuses to come up with this error:
Caused by: java.lang.NullPointerException
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.getDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverRegionsThatAreReady(PersistentOplogSet.java:367) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverRegionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
It hints at GemFire data on disk being corrupted, so I used 'gfsh' to verify:
gfsh>validate offline-disk-store --name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
Validating nsxDiskStore
/nsx_sys/ArrayListIDPriorityModel: entryCount=0
/nsx_sys/Crl: entryCount=0
/nsx_sys/Certificate: entryCount=1
......
Error in validating disk store nsxDiskStore is : null
This confirms that the disk-store is corrupted, but doesn't give any more information to debug this further. How do I go about debugging this? Have you seen this before and are there any fixes/workarounds available?
Thanks
Kapil
Re: GemFire persisted data corruption - how to debug?
Posted by Anthony Baker <ab...@pivotal.io>.
Hi Kapil,
Geode (by default) writes data synchronously to other cluster members. If a node crashes like in your test, the update is preserved by the cluster even in the absence of persistence. Synchronous disk writes can be turned on (see [1]) but many users prefer to avoid the fsync performance penalty.
Anthony
[1] https://cwiki.apache.org/confluence/display/GEODE/Native+Disk+Persistence
> On Oct 13, 2016, at 6:46 PM, Kapil Goyal <go...@vmware.com> wrote:
>
> Hi Folks,
>
> I am doing some crash testing with a single cache node of GemFire, where I power off the VM where cache is running and then bring it back up. Upon restart, GemFire refuses to come up with this error:
>
> Caused by: java.lang.NullPointerException
> at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.getDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverRegionsThatAreReady(PersistentOplogSet.java:367) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverRegionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
> at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
> at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
> at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
>
> It hints at GemFire data on disk being corrupted, so I used 'gfsh' to verify:
>
> gfsh>validate offline-disk-store --name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
>
> Validating nsxDiskStore
> /nsx_sys/ArrayListIDPriorityModel: entryCount=0
> /nsx_sys/Crl: entryCount=0
> /nsx_sys/Certificate: entryCount=1
> ……
> Error in validating disk store nsxDiskStore is : null
>
> This confirms that the disk-store is corrupted, but doesn't give any more information to debug this further. How do I go about debugging this? Have you seen this before and are there any fixes/workarounds available?
>
> Thanks
> Kapil
Re: GemFire persisted data corruption - how to debug?
Posted by Kapil Goyal <go...@vmware.com>.
Hi Dan,
Thanks for providing more information about the inner workings of disk-store.
We enabled sync-writes right from the start and it was enabled when VM was powered off.
I was able to enable trace logging, but couldn't make out much from the logs I got. I also tried attaching a debugger to the validation code and it just helped me reached the same conclusion as you - a key is null.
Regards
Kapil
From: Dan Smith <ds...@pivotal.io>>
Reply-To: "user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>" <us...@geode.incubator.apache.org>>
Date: Friday, October 14, 2016 at 1:47 PM
To: "user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>" <us...@geode.incubator.apache.org>>
Subject: Re: GemFire persisted data corruption - how to debug?
Hi Kapil,
Hmm, that's not good. It looks like it somehow ending up with a key that is null. Did you have synchronous writes enabled before powering off the VM?
With or without synchronous writes enabled, this is not expected or a known issue. Every write to the disk store files records an end of record marker, so an incomplete write or something like that shouldn't be able to cause this.
One way to get more debugging information is to enable trace logging when you validate that offline disk store. But this prints out a lot of debugging information, so it might be hard to follow what's going on:
gfsh validate offline-disk-store --J=-Dgemfire.log-level=trace --name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
-Dan
On Fri, Oct 14, 2016 at 10:59 AM, Anilkumar Gingade <ag...@pivotal.io>> wrote:
Kapil,
This is related to GemFire 8.2. If its a critical issue, please reach-out to GemFire support so that it will be tracked and addressed in time.
Can you see this with Geode versions, this helps to eliminate/identify the version where the issues i happening.
Is it possible for you to share your .xml and persistent files?
Thanks,
-Anil.
On Thu, Oct 13, 2016 at 6:46 PM, Kapil Goyal <go...@vmware.com>> wrote:
Hi Folks,
I am doing some crash testing with a single cache node of GemFire, where I power off the VM where cache is running and then bring it back up. Upon restart, GemFire refuses to come up with this error:
Caused by: java.lang.NullPointerException
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.ge<https://urldefense.proofpoint.com/v2/url?u=http-3A__cache.PlaceHolderDiskRegion.ge&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=jI39iGhJSMsophpPzgwoqWd6xw05tZ5QPHjlmT5c7Tw&m=GpPVQWlXbmUVFsVG6xvwkhIlnb_jNflOci9phO_ttGI&s=THtATbPW4Cvv38hMtEGr6rJ-7I8DwSgDlOmS-E5QeX4&e=>tDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverRegionsThatAreReady(PersistentOplogSet.java:367) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverRegionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306) ~[spring-data-gemfire-1.5.2.RE<https://urldefense.proofpoint.com/v2/url?u=http-3A__spring-2Ddata-2Dgemfire-2D1.5.2.RE&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=jI39iGhJSMsophpPzgwoqWd6xw05tZ5QPHjlmT5c7Tw&m=GpPVQWlXbmUVFsVG6xvwkhIlnb_jNflOci9phO_ttGI&s=IqCtRW2UuGyEtMekG7SU-R0MfTT8HC5276GcMSh0HgM&e=>LEASE.jar:1.5.2.RELEASE]
at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.RE<https://urldefense.proofpoint.com/v2/url?u=http-3A__spring-2Ddata-2Dgemfire-2D1.5.2.RE&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=jI39iGhJSMsophpPzgwoqWd6xw05tZ5QPHjlmT5c7Tw&m=GpPVQWlXbmUVFsVG6xvwkhIlnb_jNflOci9phO_ttGI&s=IqCtRW2UuGyEtMekG7SU-R0MfTT8HC5276GcMSh0HgM&e=>LEASE.jar:1.5.2.RELEASE]
It hints at GemFire data on disk being corrupted, so I used 'gfsh' to verify:
gfsh>validate offline-disk-store --name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
Validating nsxDiskStore
/nsx_sys/ArrayListIDPriorityModel: entryCount=0
/nsx_sys/Crl: entryCount=0
/nsx_sys/Certificate: entryCount=1
......
Error in validating disk store nsxDiskStore is : null
This confirms that the disk-store is corrupted, but doesn't give any more information to debug this further. How do I go about debugging this? Have you seen this before and are there any fixes/workarounds available?
Thanks
Kapil
Re: GemFire persisted data corruption - how to debug?
Posted by Dan Smith <ds...@pivotal.io>.
Hi Kapil,
Hmm, that's not good. It looks like it somehow ending up with a key that is
null. Did you have synchronous writes enabled before powering off the VM?
With or without synchronous writes enabled, this is not expected or a known
issue. Every write to the disk store files records an end of record marker,
so an incomplete write or something like that shouldn't be able to cause
this.
One way to get more debugging information is to enable trace logging when
you validate that offline disk store. But this prints out a lot of
debugging information, so it might be hard to follow what's going on:
gfsh validate offline-disk-store --J=-Dgemfire.log-level=trace
--name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
-Dan
On Fri, Oct 14, 2016 at 10:59 AM, Anilkumar Gingade <ag...@pivotal.io>
wrote:
> Kapil,
>
> This is related to GemFire 8.2. If its a critical issue, please reach-out
> to GemFire support so that it will be tracked and addressed in time.
>
> Can you see this with Geode versions, this helps to eliminate/identify the
> version where the issues i happening.
>
> Is it possible for you to share your .xml and persistent files?
>
> Thanks,
> -Anil.
>
>
>
>
> On Thu, Oct 13, 2016 at 6:46 PM, Kapil Goyal <go...@vmware.com> wrote:
>
>> Hi Folks,
>>
>>
>>
>> I am doing some crash testing with a single cache node of GemFire, where
>> I power off the VM where cache is running and then bring it back up. Upon
>> restart, GemFire refuses to come up with this error:
>>
>>
>>
>> Caused by: java.lang.NullPointerException
>>
>> at com.gemstone.gemfire.internal.util.concurrent.CustomEntryCon
>> currentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$Hash
>> RegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.util.concurrent.CustomEntryCon
>> currentHashMap.get(CustomEntryConcurrentHashMap.java:1447)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.get
>> Entry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.ge
>> tDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recov
>> erOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recov
>> erRegionsThatAreReady(PersistentOplogSet.java:367)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverReg
>> ionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initialize
>> IfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialR
>> ecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.cre
>> ate(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.c
>> reateDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.
>> create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCac
>> heXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
>>
>> at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306)
>> ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
>>
>> at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455)
>> ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
>>
>>
>>
>> It hints at GemFire data on disk being corrupted, so I used 'gfsh' to
>> verify:
>>
>>
>>
>> gfsh>validate offline-disk-store --name=nsxDiskStore
>> --disk-dirs=/common/nsxapi/data/self
>>
>>
>>
>> Validating nsxDiskStore
>>
>> /nsx_sys/ArrayListIDPriorityModel: entryCount=0
>>
>> /nsx_sys/Crl: entryCount=0
>>
>> /nsx_sys/Certificate: entryCount=1
>>
>> ……
>>
>> Error in validating disk store nsxDiskStore is : null
>>
>>
>>
>> This confirms that the disk-store is corrupted, but doesn't give any more
>> information to debug this further. How do I go about debugging this? Have
>> you seen this before and are there any fixes/workarounds available?
>>
>>
>>
>> Thanks
>>
>> Kapil
>>
>
>
Re: GemFire persisted data corruption - how to debug?
Posted by Eric Shu <es...@pivotal.io>.
AFAIK we have not seen this issue before. Will take a look at this.
Regards,
Eric
On Fri, Oct 14, 2016 at 1:03 PM, Kapil Goyal <go...@vmware.com> wrote:
> Thanks Anil.
>
> Is this a known issue and is there any documentation on it? Does this
> issue not impact Geode?
>
> Regards
> Kapil
>
> From: Anilkumar Gingade <ag...@pivotal.io>
> Reply-To: "user@geode.incubator.apache.org" <user@geode.incubator.apache.
> org>
> Date: Friday, October 14, 2016 at 10:59 AM
> To: "user@geode.incubator.apache.org" <us...@geode.incubator.apache.org>
> Subject: Re: GemFire persisted data corruption - how to debug?
>
> Kapil,
>
> This is related to GemFire 8.2. If its a critical issue, please reach-out
> to GemFire support so that it will be tracked and addressed in time.
>
> Can you see this with Geode versions, this helps to eliminate/identify the
> version where the issues i happening.
>
> Is it possible for you to share your .xml and persistent files?
>
> Thanks,
> -Anil.
>
>
>
>
> On Thu, Oct 13, 2016 at 6:46 PM, Kapil Goyal <go...@vmware.com> wrote:
>
>> Hi Folks,
>>
>>
>>
>> I am doing some crash testing with a single cache node of GemFire, where
>> I power off the VM where cache is running and then bring it back up. Upon
>> restart, GemFire refuses to come up with this error:
>>
>>
>>
>> Caused by: java.lang.NullPointerException
>>
>> at com.gemstone.gemfire.internal.util.concurrent.CustomEntryCon
>> currentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$Hash
>> RegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.util.concurrent.CustomEntryCon
>> currentHashMap.get(CustomEntryConcurrentHashMap.java:1447)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.get
>> Entry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.ge
>> tDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recov
>> erOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recov
>> erRegionsThatAreReady(PersistentOplogSet.java:367)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverReg
>> ionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initialize
>> IfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialR
>> ecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.cre
>> ate(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.c
>> reateDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425)
>> ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.
>> create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
>>
>> at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCac
>> heXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
>>
>> at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306)
>> ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
>>
>> at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455)
>> ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
>>
>>
>>
>> It hints at GemFire data on disk being corrupted, so I used 'gfsh' to
>> verify:
>>
>>
>>
>> gfsh>validate offline-disk-store --name=nsxDiskStore
>> --disk-dirs=/common/nsxapi/data/self
>>
>>
>>
>> Validating nsxDiskStore
>>
>> /nsx_sys/ArrayListIDPriorityModel: entryCount=0
>>
>> /nsx_sys/Crl: entryCount=0
>>
>> /nsx_sys/Certificate: entryCount=1
>>
>> ……
>>
>> Error in validating disk store nsxDiskStore is : null
>>
>>
>>
>> This confirms that the disk-store is corrupted, but doesn't give any more
>> information to debug this further. How do I go about debugging this? Have
>> you seen this before and are there any fixes/workarounds available?
>>
>>
>>
>> Thanks
>>
>> Kapil
>>
>
>
Re: GemFire persisted data corruption - how to debug?
Posted by Kapil Goyal <go...@vmware.com>.
Thanks Anil.
Is this a known issue and is there any documentation on it? Does this issue not impact Geode?
Regards
Kapil
From: Anilkumar Gingade <ag...@pivotal.io>>
Reply-To: "user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>" <us...@geode.incubator.apache.org>>
Date: Friday, October 14, 2016 at 10:59 AM
To: "user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>" <us...@geode.incubator.apache.org>>
Subject: Re: GemFire persisted data corruption - how to debug?
Kapil,
This is related to GemFire 8.2. If its a critical issue, please reach-out to GemFire support so that it will be tracked and addressed in time.
Can you see this with Geode versions, this helps to eliminate/identify the version where the issues i happening.
Is it possible for you to share your .xml and persistent files?
Thanks,
-Anil.
On Thu, Oct 13, 2016 at 6:46 PM, Kapil Goyal <go...@vmware.com>> wrote:
Hi Folks,
I am doing some crash testing with a single cache node of GemFire, where I power off the VM where cache is running and then bring it back up. Upon restart, GemFire refuses to come up with this error:
Caused by: java.lang.NullPointerException
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.getDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.PersistentOplogSet.recoverRegionsThatAreReady(PersistentOplogSet.java:367) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.recoverRegionsThatAreReady(DiskStoreImpl.java:2065) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreImpl.doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
It hints at GemFire data on disk being corrupted, so I used 'gfsh' to verify:
gfsh>validate offline-disk-store --name=nsxDiskStore --disk-dirs=/common/nsxapi/data/self
Validating nsxDiskStore
/nsx_sys/ArrayListIDPriorityModel: entryCount=0
/nsx_sys/Crl: entryCount=0
/nsx_sys/Certificate: entryCount=1
......
Error in validating disk store nsxDiskStore is : null
This confirms that the disk-store is corrupted, but doesn't give any more information to debug this further. How do I go about debugging this? Have you seen this before and are there any fixes/workarounds available?
Thanks
Kapil
Re: GemFire persisted data corruption - how to debug?
Posted by Anilkumar Gingade <ag...@pivotal.io>.
Kapil,
This is related to GemFire 8.2. If its a critical issue, please reach-out
to GemFire support so that it will be tracked and addressed in time.
Can you see this with Geode versions, this helps to eliminate/identify the
version where the issues i happening.
Is it possible for you to share your .xml and persistent files?
Thanks,
-Anil.
On Thu, Oct 13, 2016 at 6:46 PM, Kapil Goyal <go...@vmware.com> wrote:
> Hi Folks,
>
>
>
> I am doing some crash testing with a single cache node of GemFire, where I
> power off the VM where cache is running and then bring it back up. Upon
> restart, GemFire refuses to come up with this error:
>
>
>
> Caused by: java.lang.NullPointerException
>
> at com.gemstone.gemfire.internal.util.concurrent.
> CustomEntryConcurrentHashMap.keyHash(CustomEntryConcurrentHashMap.java:228)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.AbstractRegionEntry$
> HashRegionEntryCreator.keyHashCode(AbstractRegionEntry.java:934)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.util.concurrent.
> CustomEntryConcurrentHashMap.get(CustomEntryConcurrentHashMap.java:1447)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.AbstractRegionMap.
> getEntry(AbstractRegionMap.java:368) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.
> getEntry(AbstractLRURegionMap.java:47) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.PlaceHolderDiskRegion.
> getDiskEntry(PlaceHolderDiskRegion.java:93) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.Oplog.readModifyEntry(Oplog.java:2779)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.Oplog.readCrf(Oplog.java:1957)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.Oplog.recoverCrf(Oplog.java:2270)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.
> recoverOplogs(PersistentOplogSet.java:459) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.PersistentOplogSet.
> recoverRegionsThatAreReady(PersistentOplogSet.java:367)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.
> recoverRegionsThatAreReady(DiskStoreImpl.java:2065)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.
> initializeIfNeeded(DiskStoreImpl.java:2052) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.DiskStoreImpl.
> doInitialRecovery(DiskStoreImpl.java:2057) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.DiskStoreFactoryImpl.
> create(DiskStoreFactoryImpl.java:135) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.
> createDiskStore(CacheCreation.java:650) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:425)
> ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.
> create(CacheXmlParser.java:331) ~[gemfire-8.2.0.2.jar:?]
>
> at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.
> loadCacheXml(GemFireCacheImpl.java:4248) ~[gemfire-8.2.0.2.jar:?]
>
> at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:306)
> ~[spring-data-gemfire-1.5.2.RELEASE.jar:1.5.2.RELEASE]
>
> at org.springframework.data.gemfire.CacheFactoryBean.
> getObject(CacheFactoryBean.java:455) ~[spring-data-gemfire-1.5.2.
> RELEASE.jar:1.5.2.RELEASE]
>
>
>
> It hints at GemFire data on disk being corrupted, so I used 'gfsh' to
> verify:
>
>
>
> gfsh>validate offline-disk-store --name=nsxDiskStore
> --disk-dirs=/common/nsxapi/data/self
>
>
>
> Validating nsxDiskStore
>
> /nsx_sys/ArrayListIDPriorityModel: entryCount=0
>
> /nsx_sys/Crl: entryCount=0
>
> /nsx_sys/Certificate: entryCount=1
>
> ……
>
> Error in validating disk store nsxDiskStore is : null
>
>
>
> This confirms that the disk-store is corrupted, but doesn't give any more
> information to debug this further. How do I go about debugging this? Have
> you seen this before and are there any fixes/workarounds available?
>
>
>
> Thanks
>
> Kapil
>