You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by "Thacker, Dharam" <dh...@jpmorgan.com> on 2017/05/03 10:35:48 UTC
IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Hi Team,
Could you guide us with below exception?
How did we get that?
Step1: Our region is already loaded with 1M+ records [Region<String,PdxInstance> -- PDXInstance is JsonString using JsonFormatter]
Step2: When client is instructed about bulk updates in database, we calculate diff of keys using (cache - database) to remove stale deleted entries from region
Step3: To do the same, we run below loop,
for(Object key : cacheKeys ){
template.remove(key);
}
Region def:
<geode:replicated-region id="Event"
cache-ref="geodeCache"
scope="distributed-ack"
key-constraint="java.lang.String"
value-constraint="org.apache.geode.pdx.PdxInstance"
shortcut="REPLICATE_PERSISTENT_OVERFLOW"
persistent="true"
disk-synchronous="false"
disk-store-ref="event_disk_store">
<geode:cache-loader ref="eventCacheMissDbLoader"/>
</geode:replicated-region>
<geode:index id="event_uuid_indx" expression="i.uuid" from="/Event i" cache-ref="geodeCache"/>
<geode:index id="event_intField_indx" expression="i.intField" from="/Event i" cache-ref="geodeCache" type="FUNCTIONAL"/>
Which results into below exception >>
java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString cannot be cast to java.lang.Integer
at java.lang.Integer.compareTo(Integer.java:52)
at org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
... 23 common frames omitted
Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
at org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
at org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
... 17 common frames omitted
Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException: org.apache.geode.cache.query.internal.index.IMQException
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
at org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
at org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
at org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
at org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
at org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
at org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
... 1 common frames omitted
Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote server on XXXXXX: : While performing a remote destroy
at org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
at org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
at org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
at org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
at org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
Regards,
Dharam
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
RE: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by "Thacker, Dharam" <dh...@jpmorgan.com>.
Modification/Update is also done from client side in the same refresh() method where modification follows deletion!
getGemfireTemplate().putAll(result);
Here result is a map which contains Key -> pdxInstance.
1. Let say I have json {uuid: “a-b-c-d”, intField: 21} [Region has entry with Key = “abcd” --> value = PDXInstance]
2. New update {uuid: “a-b-c-d”,intField: 22} [Region entry with Key=”abcd” would be replaced by new PDXInstance now]
It may happen that oldentry is missing intField on which index was designed and now newentry contains intField in which case index would be added and vice versa for deletion.
Regards,
Dharam
From: Jason Huynh [mailto:jhuynh@pivotal.io]
Sent: Wednesday, May 03, 2017 11:18 PM
To: user@geode.apache.org
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
How are you updating the 8 records from the database? Is it being done directly on the server? Is it an actual region.put(key, JSONFormatter.format(newString))?
Is there any chance that the object in the cache is being modified underneath the index?
I am not quite sure how the values were indexed properly and then now unable to be removed unless there is something occurring that is changing the value into an integer from a pdxstring.
On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Huynh,
Entries are being put using GemfirreTemplate from client side.
gemfireTemplate.putAll(map) [Where map --> Map<String, PdxInsatnce> And PdxInstance --> Jsonformatter.fromjson(jsonString)]
Yes you are right. They are being modified before deletion. Let me describe those steps.
1. Let’s say cache has 10 records and database has 8 records.
2. Calculating delta is costly as well as emptying region and reloading is also not valid option for us due to realtime use case of the same region data
3. So we first update all 8 records from database into cache which will replace “Value” part of region if key matches
4. Then question comes about those extra 2 records which still exists into cache but not in database
5. So we calculate key different of (cacheKeys – databaseKeys) [A MINUS B] and delete unwanted keys from cache
Regards,
Dharam
From: Jason Huynh [mailto:jhuynh@pivotal.io<ma...@pivotal.io>]
Sent: Wednesday, May 03, 2017 10:53 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
Hi Dharam,
How are the entries being put into the region? Are they put through a client or is it done on the server itself?
Does the put command look similar to :
region.put(key, JSONFormatter.fromJSON(someJsonString))?
Somehow the object that is being removed is now returning an Integer instead of a PdxString for it's index value (either intField or uuid, depending on which index is failing). I am not sure exactly how that would happen at this point, but it will help to know exactly how these objects are put and how they are updated and removed. Are they being modified at any point?
What version are you currently using? I'm going to guess 1.0?
Is there an actual domain object for these objects that some values are being deserialized to on the server?
-Jason Huynh
On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Team,
Could you guide us with below exception?
How did we get that?
Step1: Our region is already loaded with 1M+ records [Region<String,PdxInstance> -- PDXInstance is JsonString using JsonFormatter]
Step2: When client is instructed about bulk updates in database, we calculate diff of keys using (cache - database) to remove stale deleted entries from region
Step3: To do the same, we run below loop,
for(Object key : cacheKeys ){
template.remove(key);
}
Region def:
<geode:replicated-region id="Event"
cache-ref="geodeCache"
scope="distributed-ack"
key-constraint="java.lang.String"
value-constraint="org.apache.geode.pdx.PdxInstance"
shortcut="REPLICATE_PERSISTENT_OVERFLOW"
persistent="true"
disk-synchronous="false"
disk-store-ref="event_disk_store">
<geode:cache-loader ref="eventCacheMissDbLoader"/>
</geode:replicated-region>
<geode:index id="event_uuid_indx" expression="i.uuid" from="/Event i" cache-ref="geodeCache"/>
<geode:index id="event_intField_indx" expression="i.intField" from="/Event i" cache-ref="geodeCache" type="FUNCTIONAL"/>
Which results into below exception >>
java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString cannot be cast to java.lang.Integer
at java.lang.Integer.compareTo(Integer.java:52)
at org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
... 23 common frames omitted
Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
at org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
at org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
... 17 common frames omitted
Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException: org.apache.geode.cache.query.internal.index.IMQException
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
at org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
at org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
at org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
at org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
at org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
at org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
... 1 common frames omitted
Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote server on XXXXXX: : While performing a remote destroy
at org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
at org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
at org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
at org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
at org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
Regards,
Dharam
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by Jason Huynh <jh...@pivotal.io>.
In this case, the type seems to be either string or int. Which is most
likely why you get the ClassCastException with respect to the Integer
compareTo PdxString.
On Fri, May 5, 2017 at 11:35 AM Anilkumar Gingade <ag...@pivotal.io>
wrote:
> >> 1.Would null be ok even if index is defined on that field?
> Yes. Index handles null values.
>
> >> 2.Can I still support such scenarios with multiple pdxtypes for similar
> objects in 1 region?
> Yes...Region supports heterogeneous objects...And indexing works as long
> as the name and type are same. If the given name is not found in the
> object, it doesn't index it.
>
> -Anil.
>
>
> On Fri, May 5, 2017 at 4:12 AM, Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
>> Thanks Anil and Huyah! I tried to reproduce this issue in my local and
>> found below ones with pdxTypes.
>>
>>
>>
>> 1. For the first time, data goes to cache as expected and I can
>> see multiple types for field cusip as shown below.
>>
>> 2. I took my server offline
>>
>> 3. I attempted to restart my server where it tried to read data
>> from disk store (PDX as well as Region – 2 separate files and directories)
>>
>> 4. And it results into below.
>>
>>
>>
>> There is potential bug in source side as well where datatype is not
>> consistent which I can attempt to fix and ‘Undefined’ is for null cusip!
>>
>> 1. Would null be ok even if index is defined on that field?
>>
>> 2. Can I still support such scenarios with multiple pdxtypes for
>> similar objects in 1 region?
>>
>>
>>
>>
>>
>> Thanks & Regards,
>>
>> Dharam
>>
>>
>>
>> *From:* Anilkumar Gingade [mailto:agingade@pivotal.io]
>> *Sent:* Friday, May 05, 2017 2:35 AM
>> *To:* user@geode.apache.org
>> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
>> from Compact Map Range Index
>>
>>
>>
>> Dharam,
>>
>>
>>
>> Is it possible to send a reproducible test/example?
>>
>>
>>
>> -Anil.
>>
>>
>>
>>
>>
>> On Thu, May 4, 2017 at 10:37 AM, Jason Huynh <jh...@pivotal.io> wrote:
>>
>> Hi Dharam,
>>
>>
>>
>> Based on your stack trace from the first email, it looks like you were
>> creating a CompactRangeIndex. You are correct, if index maintenance is
>> asynchronous the regular RangeIndex will be used but it does not look to be
>> the case for your original stack trace.
>>
>>
>>
>> The regular RangeIndex takes a bit more memory and we do not support
>> overflow regions with range indexes, this is because the range index stores
>> a copy of the value in the index structure (it would not make sense to
>> overflow the region but keep a copy of the value in memory for the index).
>>
>>
>>
>> I am still not sure how the system got into the state to throw the
>> original exception without something weird going on with the region value.
>> Whether it was modified beneath the index or some weird deserialization
>> occurring during index maintenance.
>>
>>
>>
>> -Jason
>>
>>
>>
>> On Wed, May 3, 2017 at 11:31 PM Thacker, Dharam <
>> dharam.thacker@jpmorgan.com> wrote:
>>
>> I assume that below statements would create compact indexes.
>>
>> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
>> i"* cache-ref=*"geodeCache"*/>
>>
>> <geode:index id=*"event_intField_indx"* expression=*"i.intField"* from=*"/Event
>> i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>>
>>
>>
>> Does that mean that index maintenance mode must be selected as
>> synchronous? [Which is not in my case]
>>
>> Is my understanding on FUNCTIONAL index correct?
>>
>> What are the prerequisites for asynchronous index mode?
>>
>>
>>
>> Thanks & Regards,
>>
>> Dharam
>>
>>
>>
>> *From:* Thacker, Dharam
>> *Sent:* Wednesday, May 03, 2017 11:26 PM
>> *To:* 'user@geode.apache.org'
>> *Subject:* RE: IndexMaintenanceException while remove old keys correctly
>> from Compact Map Range Index
>>
>>
>>
>> Modification/Update is also done from client side in the same refresh()
>> method where modification follows deletion!
>>
>> getGemfireTemplate().putAll(result);
>>
>>
>>
>> Here result is a map which contains Key -> pdxInstance.
>>
>> 1. Let say I have json {uuid: “a-b-c-d”, intField: 21} [Region
>> has entry with Key = “abcd” à value = PDXInstance]
>>
>> 2. New update {uuid: “a-b-c-d”,intField: 22} [Region entry with
>> Key=”abcd” would be replaced by new PDXInstance now]
>>
>>
>>
>> It may happen that oldentry is missing intField on which index was
>> designed and now newentry contains intField in which case index would be
>> added and vice versa for deletion.
>>
>>
>>
>> Regards,
>>
>> Dharam
>>
>>
>>
>> *From:* Jason Huynh [mailto:jhuynh@pivotal.io <jh...@pivotal.io>]
>> *Sent:* Wednesday, May 03, 2017 11:18 PM
>>
>>
>> *To:* user@geode.apache.org
>> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
>> from Compact Map Range Index
>>
>>
>>
>> How are you updating the 8 records from the database? Is it being done
>> directly on the server? Is it an actual region.put(key,
>> JSONFormatter.format(newString))?
>>
>>
>>
>> Is there any chance that the object in the cache is being modified
>> underneath the index?
>>
>>
>>
>> I am not quite sure how the values were indexed properly and then now
>> unable to be removed unless there is something occurring that is changing
>> the value into an integer from a pdxstring.
>>
>>
>>
>>
>>
>> On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <
>> dharam.thacker@jpmorgan.com> wrote:
>>
>> Hi Huynh,
>>
>>
>>
>> Entries are being put using GemfirreTemplate from client side.
>>
>> gemfireTemplate.putAll(map) [Where map à Map<String, PdxInsatnce> And
>> PdxInstance à Jsonformatter.fromjson(jsonString)]
>>
>>
>>
>> Yes you are right. They are being modified before deletion. Let me
>> describe those steps.
>>
>>
>>
>> 1. Let’s say cache has 10 records and database has 8 records.
>>
>> 2. Calculating delta is costly as well as emptying region and
>> reloading is also not valid option for us due to realtime use case of the
>> same region data
>>
>> 3. So we first update all 8 records from database into cache which
>> will replace “Value” part of region if key matches
>>
>> 4. Then question comes about those extra 2 records which still
>> exists into cache but not in database
>>
>> 5. So we calculate key different of (cacheKeys – databaseKeys) [A
>> MINUS B] and delete unwanted keys from cache
>>
>>
>>
>> Regards,
>>
>> Dharam
>>
>>
>>
>> *From:* Jason Huynh [mailto:jhuynh@pivotal.io]
>> *Sent:* Wednesday, May 03, 2017 10:53 PM
>> *To:* user@geode.apache.org
>> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
>> from Compact Map Range Index
>>
>>
>>
>> Hi Dharam,
>>
>>
>>
>> How are the entries being put into the region? Are they put through a
>> client or is it done on the server itself?
>>
>>
>>
>> Does the put command look similar to :
>>
>> region.put(key, JSONFormatter.*fromJSON*(someJsonString))?
>>
>>
>>
>> Somehow the object that is being removed is now returning an Integer
>> instead of a PdxString for it's index value (either intField or uuid,
>> depending on which index is failing). I am not sure exactly how that would
>> happen at this point, but it will help to know exactly how these objects
>> are put and how they are updated and removed. Are they being modified at
>> any point?
>>
>>
>>
>> What version are you currently using? I'm going to guess 1.0?
>>
>>
>>
>> Is there an actual domain object for these objects that some values are
>> being deserialized to on the server?
>>
>>
>>
>> -Jason Huynh
>>
>>
>>
>>
>>
>>
>>
>> On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <
>> dharam.thacker@jpmorgan.com> wrote:
>>
>> Hi Team,
>>
>>
>>
>> Could you guide us with below exception?
>>
>>
>>
>> *How did we get that?*
>>
>>
>>
>> Step1: Our region is already loaded with 1M+ records
>> [Region<String,PdxInstance> -- PDXInstance is JsonString using
>> JsonFormatter]
>>
>> Step2: When client is instructed about bulk updates in database, we
>> calculate diff of keys using (cache - database) to remove stale deleted
>> entries from region
>>
>> Step3: To do the same, we run below loop,
>>
>>
>>
>>
>>
>> for(Object key : cacheKeys ){
>>
>> template.remove(key);
>>
>> }
>>
>>
>>
>> *Region def:*
>>
>>
>>
>> <geode:replicated-region id=*"Event"*
>>
>> cache-ref=*"geodeCache"*
>>
>> scope=*"distributed-ack"*
>>
>> key-constraint=*"java.lang.String"*
>>
>> value-constraint=
>> *"org.apache.geode.pdx.PdxInstance"*
>>
>> shortcut=*"REPLICATE_PERSISTENT_OVERFLOW"*
>>
>> persistent=*"true"*
>>
>> disk-synchronous=*"false"*
>>
>> disk-store-ref=*"event_disk_store"*>
>>
>> <geode:cache-loader ref=*"eventCacheMissDbLoader"*/>
>>
>> </geode:replicated-region>
>>
>>
>>
>> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
>> i"* cache-ref=*"geodeCache"*/>
>>
>> <geode:index id=*"event_intField_indx"* expression=*"i.intField"*
>> from=*"/Event i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>>
>>
>>
>> Which results into below exception >>
>>
>>
>>
>> java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString
>> cannot be cast to java.lang.Integer
>>
>> at java.lang.Integer.compareTo(Integer.java:52)
>>
>> at
>> org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
>>
>> at
>> java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
>>
>> at
>> java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
>>
>> at
>> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
>>
>> at
>> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
>>
>> at
>> org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
>>
>> at
>> org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
>>
>> at
>> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
>>
>> at
>> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
>>
>> at
>> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
>>
>> at
>> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
>>
>> ... 23 common frames omitted
>>
>> Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
>>
>> at
>> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
>>
>> at
>> org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
>>
>> at
>> org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
>>
>> at
>> org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
>>
>> at
>> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
>>
>> at
>> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
>>
>> at
>> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
>>
>> ... 17 common frames omitted
>>
>> Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException:
>> org.apache.geode.cache.query.internal.index.IMQException
>>
>> at
>> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
>>
>> at
>> org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
>>
>> at
>> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
>>
>> at
>> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
>>
>> at
>> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
>>
>> at
>> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
>>
>> at
>> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
>>
>> at
>> org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
>>
>> at
>> org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
>>
>> at
>> org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
>>
>> at
>> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
>>
>> at
>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>>
>> at
>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
>>
>> at
>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at
>> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
>>
>> ... 1 common frames omitted
>>
>> Wrapped by: org.apache.geode.cache.client.ServerOperationException:
>> remote server on XXXXXX: : While performing a remote destroy
>>
>> at
>> org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
>>
>> at
>> org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
>>
>> at
>> org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
>>
>> at
>> org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
>>
>> at
>> org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
>>
>> at
>> org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
>>
>> at
>> org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
>>
>> at
>> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
>>
>> at
>> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
>>
>> at
>> org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
>>
>> at
>> org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
>>
>>
>>
>> Regards,
>>
>> Dharam
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>
>
Re: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by Anilkumar Gingade <ag...@pivotal.io>.
>> 1.Would null be ok even if index is defined on that field?
Yes. Index handles null values.
>> 2.Can I still support such scenarios with multiple pdxtypes for similar
objects in 1 region?
Yes...Region supports heterogeneous objects...And indexing works as long as
the name and type are same. If the given name is not found in the object,
it doesn't index it.
-Anil.
On Fri, May 5, 2017 at 4:12 AM, Thacker, Dharam <dharam.thacker@jpmorgan.com
> wrote:
> Thanks Anil and Huyah! I tried to reproduce this issue in my local and
> found below ones with pdxTypes.
>
>
>
> 1. For the first time, data goes to cache as expected and I can see
> multiple types for field cusip as shown below.
>
> 2. I took my server offline
>
> 3. I attempted to restart my server where it tried to read data
> from disk store (PDX as well as Region – 2 separate files and directories)
>
> 4. And it results into below.
>
>
>
> There is potential bug in source side as well where datatype is not
> consistent which I can attempt to fix and ‘Undefined’ is for null cusip!
>
> 1. Would null be ok even if index is defined on that field?
>
> 2. Can I still support such scenarios with multiple pdxtypes for
> similar objects in 1 region?
>
>
>
>
>
> Thanks & Regards,
>
> Dharam
>
>
>
> *From:* Anilkumar Gingade [mailto:agingade@pivotal.io]
> *Sent:* Friday, May 05, 2017 2:35 AM
> *To:* user@geode.apache.org
> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> Dharam,
>
>
>
> Is it possible to send a reproducible test/example?
>
>
>
> -Anil.
>
>
>
>
>
> On Thu, May 4, 2017 at 10:37 AM, Jason Huynh <jh...@pivotal.io> wrote:
>
> Hi Dharam,
>
>
>
> Based on your stack trace from the first email, it looks like you were
> creating a CompactRangeIndex. You are correct, if index maintenance is
> asynchronous the regular RangeIndex will be used but it does not look to be
> the case for your original stack trace.
>
>
>
> The regular RangeIndex takes a bit more memory and we do not support
> overflow regions with range indexes, this is because the range index stores
> a copy of the value in the index structure (it would not make sense to
> overflow the region but keep a copy of the value in memory for the index).
>
>
>
> I am still not sure how the system got into the state to throw the
> original exception without something weird going on with the region value.
> Whether it was modified beneath the index or some weird deserialization
> occurring during index maintenance.
>
>
>
> -Jason
>
>
>
> On Wed, May 3, 2017 at 11:31 PM Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
> I assume that below statements would create compact indexes.
>
> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
> i"* cache-ref=*"geodeCache"*/>
>
> <geode:index id=*"event_intField_indx"* expression=*"i.intField"* from=*"/Event
> i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>
>
>
> Does that mean that index maintenance mode must be selected as
> synchronous? [Which is not in my case]
>
> Is my understanding on FUNCTIONAL index correct?
>
> What are the prerequisites for asynchronous index mode?
>
>
>
> Thanks & Regards,
>
> Dharam
>
>
>
> *From:* Thacker, Dharam
> *Sent:* Wednesday, May 03, 2017 11:26 PM
> *To:* 'user@geode.apache.org'
> *Subject:* RE: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> Modification/Update is also done from client side in the same refresh()
> method where modification follows deletion!
>
> getGemfireTemplate().putAll(result);
>
>
>
> Here result is a map which contains Key -> pdxInstance.
>
> 1. Let say I have json {uuid: “a-b-c-d”, intField: 21} [Region has
> entry with Key = “abcd” à value = PDXInstance]
>
> 2. New update {uuid: “a-b-c-d”,intField: 22} [Region entry with
> Key=”abcd” would be replaced by new PDXInstance now]
>
>
>
> It may happen that oldentry is missing intField on which index was
> designed and now newentry contains intField in which case index would be
> added and vice versa for deletion.
>
>
>
> Regards,
>
> Dharam
>
>
>
> *From:* Jason Huynh [mailto:jhuynh@pivotal.io <jh...@pivotal.io>]
> *Sent:* Wednesday, May 03, 2017 11:18 PM
>
>
> *To:* user@geode.apache.org
> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> How are you updating the 8 records from the database? Is it being done
> directly on the server? Is it an actual region.put(key,
> JSONFormatter.format(newString))?
>
>
>
> Is there any chance that the object in the cache is being modified
> underneath the index?
>
>
>
> I am not quite sure how the values were indexed properly and then now
> unable to be removed unless there is something occurring that is changing
> the value into an integer from a pdxstring.
>
>
>
>
>
> On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
> Hi Huynh,
>
>
>
> Entries are being put using GemfirreTemplate from client side.
>
> gemfireTemplate.putAll(map) [Where map à Map<String, PdxInsatnce> And
> PdxInstance à Jsonformatter.fromjson(jsonString)]
>
>
>
> Yes you are right. They are being modified before deletion. Let me
> describe those steps.
>
>
>
> 1. Let’s say cache has 10 records and database has 8 records.
>
> 2. Calculating delta is costly as well as emptying region and
> reloading is also not valid option for us due to realtime use case of the
> same region data
>
> 3. So we first update all 8 records from database into cache which
> will replace “Value” part of region if key matches
>
> 4. Then question comes about those extra 2 records which still
> exists into cache but not in database
>
> 5. So we calculate key different of (cacheKeys – databaseKeys) [A
> MINUS B] and delete unwanted keys from cache
>
>
>
> Regards,
>
> Dharam
>
>
>
> *From:* Jason Huynh [mailto:jhuynh@pivotal.io]
> *Sent:* Wednesday, May 03, 2017 10:53 PM
> *To:* user@geode.apache.org
> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> Hi Dharam,
>
>
>
> How are the entries being put into the region? Are they put through a
> client or is it done on the server itself?
>
>
>
> Does the put command look similar to :
>
> region.put(key, JSONFormatter.*fromJSON*(someJsonString))?
>
>
>
> Somehow the object that is being removed is now returning an Integer
> instead of a PdxString for it's index value (either intField or uuid,
> depending on which index is failing). I am not sure exactly how that would
> happen at this point, but it will help to know exactly how these objects
> are put and how they are updated and removed. Are they being modified at
> any point?
>
>
>
> What version are you currently using? I'm going to guess 1.0?
>
>
>
> Is there an actual domain object for these objects that some values are
> being deserialized to on the server?
>
>
>
> -Jason Huynh
>
>
>
>
>
>
>
> On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
> Hi Team,
>
>
>
> Could you guide us with below exception?
>
>
>
> *How did we get that?*
>
>
>
> Step1: Our region is already loaded with 1M+ records
> [Region<String,PdxInstance> -- PDXInstance is JsonString using
> JsonFormatter]
>
> Step2: When client is instructed about bulk updates in database, we
> calculate diff of keys using (cache - database) to remove stale deleted
> entries from region
>
> Step3: To do the same, we run below loop,
>
>
>
>
>
> for(Object key : cacheKeys ){
>
> template.remove(key);
>
> }
>
>
>
> *Region def:*
>
>
>
> <geode:replicated-region id=*"Event"*
>
> cache-ref=*"geodeCache"*
>
> scope=*"distributed-ack"*
>
> key-constraint=*"java.lang.String"*
>
> value-constraint=*"org.apache.geode.pdx.PdxInstance"*
>
> shortcut=*"REPLICATE_PERSISTENT_OVERFLOW"*
>
> persistent=*"true"*
>
> disk-synchronous=*"false"*
>
> disk-store-ref=*"event_disk_store"*>
>
> <geode:cache-loader ref=*"eventCacheMissDbLoader"*/>
>
> </geode:replicated-region>
>
>
>
> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
> i"* cache-ref=*"geodeCache"*/>
>
> <geode:index id=*"event_intField_indx"* expression=*"i.intField"*
> from=*"/Event i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>
>
>
> Which results into below exception >>
>
>
>
> java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString
> cannot be cast to java.lang.Integer
>
> at java.lang.Integer.compareTo(Integer.java:52)
>
> at org.apache.geode.cache.query.internal.types.
> ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
>
> at java.util.concurrent.ConcurrentSkipListMap.cpr(
> ConcurrentSkipListMap.java:655)
>
> at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(
> ConcurrentSkipListMap.java:682)
>
> at java.util.concurrent.ConcurrentSkipListMap.doGet(
> ConcurrentSkipListMap.java:781)
>
> at java.util.concurrent.ConcurrentSkipListMap.get(
> ConcurrentSkipListMap.java:1546)
>
> at org.apache.geode.cache.query.internal.index.MemoryIndexStore.
> basicRemoveMapping(MemoryIndexStore.java:308)
>
> at org.apache.geode.cache.query.internal.index.MemoryIndexStore.
> removeMapping(MemoryIndexStore.java:286)
>
> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
> IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
>
> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
> IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
>
> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
> IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
>
> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
> IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
>
> ... 23 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
>
> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
> IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
>
> at org.apache.geode.cache.query.internal.index.CompactRangeIndex.
> removeMapping(CompactRangeIndex.java:167)
>
> at org.apache.geode.cache.query.internal.index.AbstractIndex.
> removeIndexMapping(AbstractIndex.java:511)
>
> at org.apache.geode.cache.query.internal.index.IndexManager.
> processAction(IndexManager.java:1111)
>
> at org.apache.geode.cache.query.internal.index.IndexManager.
> updateIndexes(IndexManager.java:967)
>
> at org.apache.geode.cache.query.internal.index.IndexManager.
> updateIndexes(IndexManager.java:941)
>
> at org.apache.geode.internal.cache.AbstractRegionEntry.
> destroy(AbstractRegionEntry.java:815)
>
> ... 17 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException:
> org.apache.geode.cache.query.internal.index.IMQException
>
> at org.apache.geode.internal.cache.AbstractRegionEntry.
> destroy(AbstractRegionEntry.java:820)
>
> at org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(
> AbstractRegionMap.java:3038)
>
> at org.apache.geode.internal.cache.AbstractRegionMap.
> destroy(AbstractRegionMap.java:1386)
>
> at org.apache.geode.internal.cache.LocalRegion.mapDestroy(
> LocalRegion.java:7019)
>
> at org.apache.geode.internal.cache.LocalRegion.mapDestroy(
> LocalRegion.java:6991)
>
> at org.apache.geode.internal.cache.LocalRegionDataView.
> destroyExistingEntry(LocalRegionDataView.java:55)
>
> at org.apache.geode.internal.cache.LocalRegion.
> basicDestroy(LocalRegion.java:6956)
>
> at org.apache.geode.internal.cache.DistributedRegion.basicDestroy(
> DistributedRegion.java:1738)
>
> at org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(
> LocalRegion.java:5801)
>
> at org.apache.geode.internal.cache.tier.sockets.command.
> Destroy65.cmdExecute(Destroy65.java:232)
>
> at org.apache.geode.internal.cache.tier.sockets.
> BaseCommand.execute(BaseCommand.java:147)
>
> at org.apache.geode.internal.cache.tier.sockets.
> ServerConnection.doNormalMsg(ServerConnection.java:783)
>
> at org.apache.geode.internal.cache.tier.sockets.
> ServerConnection.doOneMessage(ServerConnection.java:913)
>
> at org.apache.geode.internal.cache.tier.sockets.
> ServerConnection.run(ServerConnection.java:1143)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at org.apache.geode.internal.cache.tier.sockets.
> AcceptorImpl$1$1.run(AcceptorImpl.java:546)
>
> ... 1 common frames omitted
>
> Wrapped by: org.apache.geode.cache.client.ServerOperationException:
> remote server on XXXXXX: : While performing a remote destroy
>
> at org.apache.geode.cache.client.internal.AbstractOp.
> processAck(AbstractOp.java:263)
>
> at org.apache.geode.cache.client.internal.DestroyOp$
> DestroyOpImpl.processResponse(DestroyOp.java:201)
>
> at org.apache.geode.cache.client.internal.AbstractOp.
> attemptReadResponse(AbstractOp.java:176)
>
> at org.apache.geode.cache.client.internal.AbstractOp.attempt(
> AbstractOp.java:388)
>
> at org.apache.geode.cache.client.internal.ConnectionImpl.
> execute(ConnectionImpl.java:272)
>
> at org.apache.geode.cache.client.internal.pooling.
> PooledConnection.execute(PooledConnection.java:328)
>
> at org.apache.geode.cache.client.internal.OpExecutorImpl.
> executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
>
> at org.apache.geode.cache.client.internal.OpExecutorImpl.
> execute(OpExecutorImpl.java:155)
>
> at org.apache.geode.cache.client.internal.OpExecutorImpl.
> execute(OpExecutorImpl.java:110)
>
> at org.apache.geode.cache.client.internal.PoolImpl.execute(
> PoolImpl.java:697)
>
> at org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
>
>
>
>
> Regards,
>
> Dharam
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
RE: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by "Thacker, Dharam" <dh...@jpmorgan.com>.
Thanks Anil and Huyah! I tried to reproduce this issue in my local and found below ones with pdxTypes.
1. For the first time, data goes to cache as expected and I can see multiple types for field cusip as shown below.
2. I took my server offline
3. I attempted to restart my server where it tried to read data from disk store (PDX as well as Region – 2 separate files and directories)
4. And it results into below.
There is potential bug in source side as well where datatype is not consistent which I can attempt to fix and ‘Undefined’ is for null cusip!
1. Would null be ok even if index is defined on that field?
2. Can I still support such scenarios with multiple pdxtypes for similar objects in 1 region?
[cid:image001.png@01D2C5BE.8FAEEE70]
Thanks & Regards,
Dharam
From: Anilkumar Gingade [mailto:agingade@pivotal.io]
Sent: Friday, May 05, 2017 2:35 AM
To: user@geode.apache.org
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
Dharam,
Is it possible to send a reproducible test/example?
-Anil.
On Thu, May 4, 2017 at 10:37 AM, Jason Huynh <jh...@pivotal.io>> wrote:
Hi Dharam,
Based on your stack trace from the first email, it looks like you were creating a CompactRangeIndex. You are correct, if index maintenance is asynchronous the regular RangeIndex will be used but it does not look to be the case for your original stack trace.
The regular RangeIndex takes a bit more memory and we do not support overflow regions with range indexes, this is because the range index stores a copy of the value in the index structure (it would not make sense to overflow the region but keep a copy of the value in memory for the index).
I am still not sure how the system got into the state to throw the original exception without something weird going on with the region value. Whether it was modified beneath the index or some weird deserialization occurring during index maintenance.
-Jason
On Wed, May 3, 2017 at 11:31 PM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
I assume that below statements would create compact indexes.
<geode:index id="event_uuid_indx" expression="i.uuid" from="/Event i" cache-ref="geodeCache"/>
<geode:index id="event_intField_indx" expression="i.intField" from="/Event i" cache-ref="geodeCache" type="FUNCTIONAL"/>
Does that mean that index maintenance mode must be selected as synchronous? [Which is not in my case]
Is my understanding on FUNCTIONAL index correct?
What are the prerequisites for asynchronous index mode?
Thanks & Regards,
Dharam
From: Thacker, Dharam
Sent: Wednesday, May 03, 2017 11:26 PM
To: 'user@geode.apache.org<ma...@geode.apache.org>'
Subject: RE: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
Modification/Update is also done from client side in the same refresh() method where modification follows deletion!
getGemfireTemplate().putAll(result);
Here result is a map which contains Key -> pdxInstance.
1. Let say I have json {uuid: “a-b-c-d”, intField: 21} [Region has entry with Key = “abcd” --> value = PDXInstance]
2. New update {uuid: “a-b-c-d”,intField: 22} [Region entry with Key=”abcd” would be replaced by new PDXInstance now]
It may happen that oldentry is missing intField on which index was designed and now newentry contains intField in which case index would be added and vice versa for deletion.
Regards,
Dharam
From: Jason Huynh [mailto:jhuynh@pivotal.io]
Sent: Wednesday, May 03, 2017 11:18 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
How are you updating the 8 records from the database? Is it being done directly on the server? Is it an actual region.put(key, JSONFormatter.format(newString))?
Is there any chance that the object in the cache is being modified underneath the index?
I am not quite sure how the values were indexed properly and then now unable to be removed unless there is something occurring that is changing the value into an integer from a pdxstring.
On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Huynh,
Entries are being put using GemfirreTemplate from client side.
gemfireTemplate.putAll(map) [Where map --> Map<String, PdxInsatnce> And PdxInstance --> Jsonformatter.fromjson(jsonString)]
Yes you are right. They are being modified before deletion. Let me describe those steps.
1. Let’s say cache has 10 records and database has 8 records.
2. Calculating delta is costly as well as emptying region and reloading is also not valid option for us due to realtime use case of the same region data
3. So we first update all 8 records from database into cache which will replace “Value” part of region if key matches
4. Then question comes about those extra 2 records which still exists into cache but not in database
5. So we calculate key different of (cacheKeys – databaseKeys) [A MINUS B] and delete unwanted keys from cache
Regards,
Dharam
From: Jason Huynh [mailto:jhuynh@pivotal.io<ma...@pivotal.io>]
Sent: Wednesday, May 03, 2017 10:53 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
Hi Dharam,
How are the entries being put into the region? Are they put through a client or is it done on the server itself?
Does the put command look similar to :
region.put(key, JSONFormatter.fromJSON(someJsonString))?
Somehow the object that is being removed is now returning an Integer instead of a PdxString for it's index value (either intField or uuid, depending on which index is failing). I am not sure exactly how that would happen at this point, but it will help to know exactly how these objects are put and how they are updated and removed. Are they being modified at any point?
What version are you currently using? I'm going to guess 1.0?
Is there an actual domain object for these objects that some values are being deserialized to on the server?
-Jason Huynh
On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Team,
Could you guide us with below exception?
How did we get that?
Step1: Our region is already loaded with 1M+ records [Region<String,PdxInstance> -- PDXInstance is JsonString using JsonFormatter]
Step2: When client is instructed about bulk updates in database, we calculate diff of keys using (cache - database) to remove stale deleted entries from region
Step3: To do the same, we run below loop,
for(Object key : cacheKeys ){
template.remove(key);
}
Region def:
<geode:replicated-region id="Event"
cache-ref="geodeCache"
scope="distributed-ack"
key-constraint="java.lang.String"
value-constraint="org.apache.geode.pdx.PdxInstance"
shortcut="REPLICATE_PERSISTENT_OVERFLOW"
persistent="true"
disk-synchronous="false"
disk-store-ref="event_disk_store">
<geode:cache-loader ref="eventCacheMissDbLoader"/>
</geode:replicated-region>
<geode:index id="event_uuid_indx" expression="i.uuid" from="/Event i" cache-ref="geodeCache"/>
<geode:index id="event_intField_indx" expression="i.intField" from="/Event i" cache-ref="geodeCache" type="FUNCTIONAL"/>
Which results into below exception >>
java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString cannot be cast to java.lang.Integer
at java.lang.Integer.compareTo(Integer.java:52)
at org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
... 23 common frames omitted
Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
at org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
at org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
... 17 common frames omitted
Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException: org.apache.geode.cache.query.internal.index.IMQException
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
at org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
at org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
at org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
at org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
at org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
at org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
... 1 common frames omitted
Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote server on XXXXXX: : While performing a remote destroy
at org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
at org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
at org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
at org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
at org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
Regards,
Dharam
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by Anilkumar Gingade <ag...@pivotal.io>.
Dharam,
Is it possible to send a reproducible test/example?
-Anil.
On Thu, May 4, 2017 at 10:37 AM, Jason Huynh <jh...@pivotal.io> wrote:
> Hi Dharam,
>
> Based on your stack trace from the first email, it looks like you were
> creating a CompactRangeIndex. You are correct, if index maintenance is
> asynchronous the regular RangeIndex will be used but it does not look to be
> the case for your original stack trace.
>
> The regular RangeIndex takes a bit more memory and we do not support
> overflow regions with range indexes, this is because the range index stores
> a copy of the value in the index structure (it would not make sense to
> overflow the region but keep a copy of the value in memory for the index).
>
> I am still not sure how the system got into the state to throw the
> original exception without something weird going on with the region value.
> Whether it was modified beneath the index or some weird deserialization
> occurring during index maintenance.
>
> -Jason
>
> On Wed, May 3, 2017 at 11:31 PM Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
>> I assume that below statements would create compact indexes.
>>
>> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
>> i"* cache-ref=*"geodeCache"*/>
>>
>> <geode:index id=*"event_intField_indx"* expression=*"i.intField"* from=*"/Event
>> i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>>
>>
>>
>> Does that mean that index maintenance mode must be selected as
>> synchronous? [Which is not in my case]
>>
>> Is my understanding on FUNCTIONAL index correct?
>>
>> What are the prerequisites for asynchronous index mode?
>>
>>
>>
>> Thanks & Regards,
>>
>> Dharam
>>
>>
>>
>> *From:* Thacker, Dharam
>> *Sent:* Wednesday, May 03, 2017 11:26 PM
>> *To:* 'user@geode.apache.org'
>> *Subject:* RE: IndexMaintenanceException while remove old keys correctly
>> from Compact Map Range Index
>>
>>
>>
>> Modification/Update is also done from client side in the same refresh()
>> method where modification follows deletion!
>>
>> getGemfireTemplate().putAll(result);
>>
>>
>>
>> Here result is a map which contains Key -> pdxInstance.
>>
>> 1. Let say I have json {uuid: “a-b-c-d”, intField: 21} [Region
>> has entry with Key = “abcd” à value = PDXInstance]
>>
>> 2. New update {uuid: “a-b-c-d”,intField: 22} [Region entry with
>> Key=”abcd” would be replaced by new PDXInstance now]
>>
>>
>>
>> It may happen that oldentry is missing intField on which index was
>> designed and now newentry contains intField in which case index would be
>> added and vice versa for deletion.
>>
>>
>>
>> Regards,
>>
>> Dharam
>>
>>
>>
>> *From:* Jason Huynh [mailto:jhuynh@pivotal.io <jh...@pivotal.io>]
>> *Sent:* Wednesday, May 03, 2017 11:18 PM
>>
>>
>> *To:* user@geode.apache.org
>> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
>> from Compact Map Range Index
>>
>>
>>
>> How are you updating the 8 records from the database? Is it being done
>> directly on the server? Is it an actual region.put(key,
>> JSONFormatter.format(newString))?
>>
>>
>>
>> Is there any chance that the object in the cache is being modified
>> underneath the index?
>>
>>
>>
>> I am not quite sure how the values were indexed properly and then now
>> unable to be removed unless there is something occurring that is changing
>> the value into an integer from a pdxstring.
>>
>>
>>
>>
>>
>> On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <
>> dharam.thacker@jpmorgan.com> wrote:
>>
>> Hi Huynh,
>>
>>
>>
>> Entries are being put using GemfirreTemplate from client side.
>>
>> gemfireTemplate.putAll(map) [Where map à Map<String, PdxInsatnce> And
>> PdxInstance à Jsonformatter.fromjson(jsonString)]
>>
>>
>>
>> Yes you are right. They are being modified before deletion. Let me
>> describe those steps.
>>
>>
>>
>> 1. Let’s say cache has 10 records and database has 8 records.
>>
>> 2. Calculating delta is costly as well as emptying region and
>> reloading is also not valid option for us due to realtime use case of the
>> same region data
>>
>> 3. So we first update all 8 records from database into cache which
>> will replace “Value” part of region if key matches
>>
>> 4. Then question comes about those extra 2 records which still
>> exists into cache but not in database
>>
>> 5. So we calculate key different of (cacheKeys – databaseKeys) [A
>> MINUS B] and delete unwanted keys from cache
>>
>>
>>
>> Regards,
>>
>> Dharam
>>
>>
>>
>> *From:* Jason Huynh [mailto:jhuynh@pivotal.io]
>> *Sent:* Wednesday, May 03, 2017 10:53 PM
>> *To:* user@geode.apache.org
>> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
>> from Compact Map Range Index
>>
>>
>>
>> Hi Dharam,
>>
>>
>>
>> How are the entries being put into the region? Are they put through a
>> client or is it done on the server itself?
>>
>>
>>
>> Does the put command look similar to :
>>
>> region.put(key, JSONFormatter.*fromJSON*(someJsonString))?
>>
>>
>>
>> Somehow the object that is being removed is now returning an Integer
>> instead of a PdxString for it's index value (either intField or uuid,
>> depending on which index is failing). I am not sure exactly how that would
>> happen at this point, but it will help to know exactly how these objects
>> are put and how they are updated and removed. Are they being modified at
>> any point?
>>
>>
>>
>> What version are you currently using? I'm going to guess 1.0?
>>
>>
>>
>> Is there an actual domain object for these objects that some values are
>> being deserialized to on the server?
>>
>>
>>
>> -Jason Huynh
>>
>>
>>
>>
>>
>>
>>
>> On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <
>> dharam.thacker@jpmorgan.com> wrote:
>>
>> Hi Team,
>>
>>
>>
>> Could you guide us with below exception?
>>
>>
>>
>> *How did we get that?*
>>
>>
>>
>> Step1: Our region is already loaded with 1M+ records
>> [Region<String,PdxInstance> -- PDXInstance is JsonString using
>> JsonFormatter]
>>
>> Step2: When client is instructed about bulk updates in database, we
>> calculate diff of keys using (cache - database) to remove stale deleted
>> entries from region
>>
>> Step3: To do the same, we run below loop,
>>
>>
>>
>>
>>
>> for(Object key : cacheKeys ){
>>
>> template.remove(key);
>>
>> }
>>
>>
>>
>> *Region def:*
>>
>>
>>
>> <geode:replicated-region id=*"Event"*
>>
>> cache-ref=*"geodeCache"*
>>
>> scope=*"distributed-ack"*
>>
>> key-constraint=*"java.lang.String"*
>>
>> value-constraint=
>> *"org.apache.geode.pdx.PdxInstance"*
>>
>> shortcut=*"REPLICATE_PERSISTENT_OVERFLOW"*
>>
>> persistent=*"true"*
>>
>> disk-synchronous=*"false"*
>>
>> disk-store-ref=*"event_disk_store"*>
>>
>> <geode:cache-loader ref=*"eventCacheMissDbLoader"*/>
>>
>> </geode:replicated-region>
>>
>>
>>
>> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
>> i"* cache-ref=*"geodeCache"*/>
>>
>> <geode:index id=*"event_intField_indx"* expression=*"i.intField"*
>> from=*"/Event i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>>
>>
>>
>> Which results into below exception >>
>>
>>
>>
>> java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString
>> cannot be cast to java.lang.Integer
>>
>> at java.lang.Integer.compareTo(Integer.java:52)
>>
>> at org.apache.geode.cache.query.internal.types.
>> ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
>>
>> at java.util.concurrent.ConcurrentSkipListMap.cpr(
>> ConcurrentSkipListMap.java:655)
>>
>> at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(
>> ConcurrentSkipListMap.java:682)
>>
>> at java.util.concurrent.ConcurrentSkipListMap.doGet(
>> ConcurrentSkipListMap.java:781)
>>
>> at java.util.concurrent.ConcurrentSkipListMap.get(
>> ConcurrentSkipListMap.java:1546)
>>
>> at org.apache.geode.cache.query.internal.index.MemoryIndexStore.
>> basicRemoveMapping(MemoryIndexStore.java:308)
>>
>> at org.apache.geode.cache.query.internal.index.MemoryIndexStore.
>> removeMapping(MemoryIndexStore.java:286)
>>
>> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
>> IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
>>
>> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
>> IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
>>
>> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
>> IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
>>
>> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
>> IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
>>
>> ... 23 common frames omitted
>>
>> Wrapped by: org.apache.geode.cache.query.internal.index.IMQException:
>> null
>>
>> at org.apache.geode.cache.query.internal.index.CompactRangeIndex$
>> IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
>>
>> at org.apache.geode.cache.query.internal.index.CompactRangeIndex.
>> removeMapping(CompactRangeIndex.java:167)
>>
>> at org.apache.geode.cache.query.internal.index.AbstractIndex.
>> removeIndexMapping(AbstractIndex.java:511)
>>
>> at org.apache.geode.cache.query.internal.index.IndexManager.
>> processAction(IndexManager.java:1111)
>>
>> at org.apache.geode.cache.query.internal.index.IndexManager.
>> updateIndexes(IndexManager.java:967)
>>
>> at org.apache.geode.cache.query.internal.index.IndexManager.
>> updateIndexes(IndexManager.java:941)
>>
>> at org.apache.geode.internal.cache.AbstractRegionEntry.
>> destroy(AbstractRegionEntry.java:815)
>>
>> ... 17 common frames omitted
>>
>> Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException:
>> org.apache.geode.cache.query.internal.index.IMQException
>>
>> at org.apache.geode.internal.cache.AbstractRegionEntry.
>> destroy(AbstractRegionEntry.java:820)
>>
>> at org.apache.geode.internal.cache.AbstractRegionMap.
>> destroyEntry(AbstractRegionMap.java:3038)
>>
>> at org.apache.geode.internal.cache.AbstractRegionMap.
>> destroy(AbstractRegionMap.java:1386)
>>
>> at org.apache.geode.internal.cache.LocalRegion.mapDestroy(
>> LocalRegion.java:7019)
>>
>> at org.apache.geode.internal.cache.LocalRegion.mapDestroy(
>> LocalRegion.java:6991)
>>
>> at org.apache.geode.internal.cache.LocalRegionDataView.
>> destroyExistingEntry(LocalRegionDataView.java:55)
>>
>> at org.apache.geode.internal.cache.LocalRegion.
>> basicDestroy(LocalRegion.java:6956)
>>
>> at org.apache.geode.internal.cache.DistributedRegion.
>> basicDestroy(DistributedRegion.java:1738)
>>
>> at org.apache.geode.internal.cache.LocalRegion.
>> basicBridgeDestroy(LocalRegion.java:5801)
>>
>> at org.apache.geode.internal.cache.tier.sockets.command.
>> Destroy65.cmdExecute(Destroy65.java:232)
>>
>> at org.apache.geode.internal.cache.tier.sockets.
>> BaseCommand.execute(BaseCommand.java:147)
>>
>> at org.apache.geode.internal.cache.tier.sockets.
>> ServerConnection.doNormalMsg(ServerConnection.java:783)
>>
>> at org.apache.geode.internal.cache.tier.sockets.
>> ServerConnection.doOneMessage(ServerConnection.java:913)
>>
>> at org.apache.geode.internal.cache.tier.sockets.
>> ServerConnection.run(ServerConnection.java:1143)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>
>> at org.apache.geode.internal.cache.tier.sockets.
>> AcceptorImpl$1$1.run(AcceptorImpl.java:546)
>>
>> ... 1 common frames omitted
>>
>> Wrapped by: org.apache.geode.cache.client.ServerOperationException:
>> remote server on XXXXXX: : While performing a remote destroy
>>
>> at org.apache.geode.cache.client.internal.AbstractOp.
>> processAck(AbstractOp.java:263)
>>
>> at org.apache.geode.cache.client.internal.DestroyOp$
>> DestroyOpImpl.processResponse(DestroyOp.java:201)
>>
>> at org.apache.geode.cache.client.internal.AbstractOp.
>> attemptReadResponse(AbstractOp.java:176)
>>
>> at org.apache.geode.cache.client.internal.AbstractOp.attempt(
>> AbstractOp.java:388)
>>
>> at org.apache.geode.cache.client.internal.ConnectionImpl.
>> execute(ConnectionImpl.java:272)
>>
>> at org.apache.geode.cache.client.internal.pooling.
>> PooledConnection.execute(PooledConnection.java:328)
>>
>> at org.apache.geode.cache.client.internal.OpExecutorImpl.
>> executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
>>
>> at org.apache.geode.cache.client.internal.OpExecutorImpl.
>> execute(OpExecutorImpl.java:155)
>>
>> at org.apache.geode.cache.client.internal.OpExecutorImpl.
>> execute(OpExecutorImpl.java:110)
>>
>> at org.apache.geode.cache.client.internal.PoolImpl.execute(
>> PoolImpl.java:697)
>>
>> at org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
>>
>>
>>
>>
>> Regards,
>>
>> Dharam
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>
Re: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by Jason Huynh <jh...@pivotal.io>.
Hi Dharam,
Based on your stack trace from the first email, it looks like you were
creating a CompactRangeIndex. You are correct, if index maintenance is
asynchronous the regular RangeIndex will be used but it does not look to be
the case for your original stack trace.
The regular RangeIndex takes a bit more memory and we do not support
overflow regions with range indexes, this is because the range index stores
a copy of the value in the index structure (it would not make sense to
overflow the region but keep a copy of the value in memory for the index).
I am still not sure how the system got into the state to throw the original
exception without something weird going on with the region value. Whether
it was modified beneath the index or some weird deserialization occurring
during index maintenance.
-Jason
On Wed, May 3, 2017 at 11:31 PM Thacker, Dharam <dh...@jpmorgan.com>
wrote:
> I assume that below statements would create compact indexes.
>
> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
> i"* cache-ref=*"geodeCache"*/>
>
> <geode:index id=*"event_intField_indx"* expression=*"i.intField"* from=*"/Event
> i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>
>
>
> Does that mean that index maintenance mode must be selected as
> synchronous? [Which is not in my case]
>
> Is my understanding on FUNCTIONAL index correct?
>
> What are the prerequisites for asynchronous index mode?
>
>
>
> Thanks & Regards,
>
> Dharam
>
>
>
> *From:* Thacker, Dharam
> *Sent:* Wednesday, May 03, 2017 11:26 PM
> *To:* 'user@geode.apache.org'
> *Subject:* RE: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> Modification/Update is also done from client side in the same refresh()
> method where modification follows deletion!
>
> getGemfireTemplate().putAll(result);
>
>
>
> Here result is a map which contains Key -> pdxInstance.
>
> 1. Let say I have json {uuid: “a-b-c-d”, intField: 21} [Region has
> entry with Key = “abcd” à value = PDXInstance]
>
> 2. New update {uuid: “a-b-c-d”,intField: 22} [Region entry with
> Key=”abcd” would be replaced by new PDXInstance now]
>
>
>
> It may happen that oldentry is missing intField on which index was
> designed and now newentry contains intField in which case index would be
> added and vice versa for deletion.
>
>
>
> Regards,
>
> Dharam
>
>
>
> *From:* Jason Huynh [mailto:jhuynh@pivotal.io <jh...@pivotal.io>]
> *Sent:* Wednesday, May 03, 2017 11:18 PM
>
>
> *To:* user@geode.apache.org
> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> How are you updating the 8 records from the database? Is it being done
> directly on the server? Is it an actual region.put(key,
> JSONFormatter.format(newString))?
>
>
>
> Is there any chance that the object in the cache is being modified
> underneath the index?
>
>
>
> I am not quite sure how the values were indexed properly and then now
> unable to be removed unless there is something occurring that is changing
> the value into an integer from a pdxstring.
>
>
>
>
>
> On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
> Hi Huynh,
>
>
>
> Entries are being put using GemfirreTemplate from client side.
>
> gemfireTemplate.putAll(map) [Where map à Map<String, PdxInsatnce> And
> PdxInstance à Jsonformatter.fromjson(jsonString)]
>
>
>
> Yes you are right. They are being modified before deletion. Let me
> describe those steps.
>
>
>
> 1. Let’s say cache has 10 records and database has 8 records.
>
> 2. Calculating delta is costly as well as emptying region and
> reloading is also not valid option for us due to realtime use case of the
> same region data
>
> 3. So we first update all 8 records from database into cache which
> will replace “Value” part of region if key matches
>
> 4. Then question comes about those extra 2 records which still
> exists into cache but not in database
>
> 5. So we calculate key different of (cacheKeys – databaseKeys) [A
> MINUS B] and delete unwanted keys from cache
>
>
>
> Regards,
>
> Dharam
>
>
>
> *From:* Jason Huynh [mailto:jhuynh@pivotal.io]
> *Sent:* Wednesday, May 03, 2017 10:53 PM
> *To:* user@geode.apache.org
> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> Hi Dharam,
>
>
>
> How are the entries being put into the region? Are they put through a
> client or is it done on the server itself?
>
>
>
> Does the put command look similar to :
>
> region.put(key, JSONFormatter.*fromJSON*(someJsonString))?
>
>
>
> Somehow the object that is being removed is now returning an Integer
> instead of a PdxString for it's index value (either intField or uuid,
> depending on which index is failing). I am not sure exactly how that would
> happen at this point, but it will help to know exactly how these objects
> are put and how they are updated and removed. Are they being modified at
> any point?
>
>
>
> What version are you currently using? I'm going to guess 1.0?
>
>
>
> Is there an actual domain object for these objects that some values are
> being deserialized to on the server?
>
>
>
> -Jason Huynh
>
>
>
>
>
>
>
> On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
> Hi Team,
>
>
>
> Could you guide us with below exception?
>
>
>
> *How did we get that?*
>
>
>
> Step1: Our region is already loaded with 1M+ records
> [Region<String,PdxInstance> -- PDXInstance is JsonString using
> JsonFormatter]
>
> Step2: When client is instructed about bulk updates in database, we
> calculate diff of keys using (cache - database) to remove stale deleted
> entries from region
>
> Step3: To do the same, we run below loop,
>
>
>
>
>
> for(Object key : cacheKeys ){
>
> template.remove(key);
>
> }
>
>
>
> *Region def:*
>
>
>
> <geode:replicated-region id=*"Event"*
>
> cache-ref=*"geodeCache"*
>
> scope=*"distributed-ack"*
>
> key-constraint=*"java.lang.String"*
>
> value-constraint=*"org.apache.geode.pdx.PdxInstance"*
>
> shortcut=*"REPLICATE_PERSISTENT_OVERFLOW"*
>
> persistent=*"true"*
>
> disk-synchronous=*"false"*
>
> disk-store-ref=*"event_disk_store"*>
>
> <geode:cache-loader ref=*"eventCacheMissDbLoader"*/>
>
> </geode:replicated-region>
>
>
>
> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
> i"* cache-ref=*"geodeCache"*/>
>
> <geode:index id=*"event_intField_indx"* expression=*"i.intField"*
> from=*"/Event i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>
>
>
> Which results into below exception >>
>
>
>
> java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString
> cannot be cast to java.lang.Integer
>
> at java.lang.Integer.compareTo(Integer.java:52)
>
> at
> org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
>
> at
> org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
>
> at
> org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
>
> ... 23 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
>
> at
> org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
>
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
>
> ... 17 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException:
> org.apache.geode.cache.query.internal.index.IMQException
>
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
>
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
>
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
>
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
>
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
>
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
>
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
>
> at
> org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
>
> at
> org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
>
> at
> org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
>
> at
> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at
> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
>
> ... 1 common frames omitted
>
> Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote
> server on XXXXXX: : While performing a remote destroy
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
>
> at
> org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
>
> at
> org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
>
> at
> org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
>
> at
> org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
>
> at
> org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
>
>
>
> Regards,
>
> Dharam
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
RE: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by "Thacker, Dharam" <dh...@jpmorgan.com>.
I assume that below statements would create compact indexes.
<geode:index id="event_uuid_indx" expression="i.uuid" from="/Event i" cache-ref="geodeCache"/>
<geode:index id="event_intField_indx" expression="i.intField" from="/Event i" cache-ref="geodeCache" type="FUNCTIONAL"/>
Does that mean that index maintenance mode must be selected as synchronous? [Which is not in my case]
Is my understanding on FUNCTIONAL index correct?
What are the prerequisites for asynchronous index mode?
Thanks & Regards,
Dharam
From: Thacker, Dharam
Sent: Wednesday, May 03, 2017 11:26 PM
To: 'user@geode.apache.org'
Subject: RE: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
Modification/Update is also done from client side in the same refresh() method where modification follows deletion!
getGemfireTemplate().putAll(result);
Here result is a map which contains Key -> pdxInstance.
1. Let say I have json {uuid: “a-b-c-d”, intField: 21} [Region has entry with Key = “abcd” --> value = PDXInstance]
2. New update {uuid: “a-b-c-d”,intField: 22} [Region entry with Key=”abcd” would be replaced by new PDXInstance now]
It may happen that oldentry is missing intField on which index was designed and now newentry contains intField in which case index would be added and vice versa for deletion.
Regards,
Dharam
From: Jason Huynh [mailto:jhuynh@pivotal.io]
Sent: Wednesday, May 03, 2017 11:18 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
How are you updating the 8 records from the database? Is it being done directly on the server? Is it an actual region.put(key, JSONFormatter.format(newString))?
Is there any chance that the object in the cache is being modified underneath the index?
I am not quite sure how the values were indexed properly and then now unable to be removed unless there is something occurring that is changing the value into an integer from a pdxstring.
On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Huynh,
Entries are being put using GemfirreTemplate from client side.
gemfireTemplate.putAll(map) [Where map --> Map<String, PdxInsatnce> And PdxInstance --> Jsonformatter.fromjson(jsonString)]
Yes you are right. They are being modified before deletion. Let me describe those steps.
1. Let’s say cache has 10 records and database has 8 records.
2. Calculating delta is costly as well as emptying region and reloading is also not valid option for us due to realtime use case of the same region data
3. So we first update all 8 records from database into cache which will replace “Value” part of region if key matches
4. Then question comes about those extra 2 records which still exists into cache but not in database
5. So we calculate key different of (cacheKeys – databaseKeys) [A MINUS B] and delete unwanted keys from cache
Regards,
Dharam
From: Jason Huynh [mailto:jhuynh@pivotal.io<ma...@pivotal.io>]
Sent: Wednesday, May 03, 2017 10:53 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
Hi Dharam,
How are the entries being put into the region? Are they put through a client or is it done on the server itself?
Does the put command look similar to :
region.put(key, JSONFormatter.fromJSON(someJsonString))?
Somehow the object that is being removed is now returning an Integer instead of a PdxString for it's index value (either intField or uuid, depending on which index is failing). I am not sure exactly how that would happen at this point, but it will help to know exactly how these objects are put and how they are updated and removed. Are they being modified at any point?
What version are you currently using? I'm going to guess 1.0?
Is there an actual domain object for these objects that some values are being deserialized to on the server?
-Jason Huynh
On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Team,
Could you guide us with below exception?
How did we get that?
Step1: Our region is already loaded with 1M+ records [Region<String,PdxInstance> -- PDXInstance is JsonString using JsonFormatter]
Step2: When client is instructed about bulk updates in database, we calculate diff of keys using (cache - database) to remove stale deleted entries from region
Step3: To do the same, we run below loop,
for(Object key : cacheKeys ){
template.remove(key);
}
Region def:
<geode:replicated-region id="Event"
cache-ref="geodeCache"
scope="distributed-ack"
key-constraint="java.lang.String"
value-constraint="org.apache.geode.pdx.PdxInstance"
shortcut="REPLICATE_PERSISTENT_OVERFLOW"
persistent="true"
disk-synchronous="false"
disk-store-ref="event_disk_store">
<geode:cache-loader ref="eventCacheMissDbLoader"/>
</geode:replicated-region>
<geode:index id="event_uuid_indx" expression="i.uuid" from="/Event i" cache-ref="geodeCache"/>
<geode:index id="event_intField_indx" expression="i.intField" from="/Event i" cache-ref="geodeCache" type="FUNCTIONAL"/>
Which results into below exception >>
java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString cannot be cast to java.lang.Integer
at java.lang.Integer.compareTo(Integer.java:52)
at org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
... 23 common frames omitted
Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
at org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
at org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
... 17 common frames omitted
Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException: org.apache.geode.cache.query.internal.index.IMQException
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
at org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
at org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
at org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
at org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
at org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
at org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
... 1 common frames omitted
Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote server on XXXXXX: : While performing a remote destroy
at org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
at org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
at org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
at org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
at org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
Regards,
Dharam
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by Jason Huynh <jh...@pivotal.io>.
How are you updating the 8 records from the database? Is it being done
directly on the server? Is it an actual region.put(key,
JSONFormatter.format(newString))?
Is there any chance that the object in the cache is being modified
underneath the index?
I am not quite sure how the values were indexed properly and then now
unable to be removed unless there is something occurring that is changing
the value into an integer from a pdxstring.
On Wed, May 3, 2017 at 10:36 AM Thacker, Dharam <dh...@jpmorgan.com>
wrote:
> Hi Huynh,
>
>
>
> Entries are being put using GemfirreTemplate from client side.
>
> gemfireTemplate.putAll(map) [Where map à Map<String, PdxInsatnce> And
> PdxInstance à Jsonformatter.fromjson(jsonString)]
>
>
>
> Yes you are right. They are being modified before deletion. Let me
> describe those steps.
>
>
>
> 1. Let’s say cache has 10 records and database has 8 records.
>
> 2. Calculating delta is costly as well as emptying region and
> reloading is also not valid option for us due to realtime use case of the
> same region data
>
> 3. So we first update all 8 records from database into cache which
> will replace “Value” part of region if key matches
>
> 4. Then question comes about those extra 2 records which still
> exists into cache but not in database
>
> 5. So we calculate key different of (cacheKeys – databaseKeys) [A
> MINUS B] and delete unwanted keys from cache
>
>
>
> Regards,
>
> Dharam
>
>
>
> *From:* Jason Huynh [mailto:jhuynh@pivotal.io]
> *Sent:* Wednesday, May 03, 2017 10:53 PM
> *To:* user@geode.apache.org
> *Subject:* Re: IndexMaintenanceException while remove old keys correctly
> from Compact Map Range Index
>
>
>
> Hi Dharam,
>
>
>
> How are the entries being put into the region? Are they put through a
> client or is it done on the server itself?
>
>
>
> Does the put command look similar to :
>
> region.put(key, JSONFormatter.*fromJSON*(someJsonString))?
>
>
>
> Somehow the object that is being removed is now returning an Integer
> instead of a PdxString for it's index value (either intField or uuid,
> depending on which index is failing). I am not sure exactly how that would
> happen at this point, but it will help to know exactly how these objects
> are put and how they are updated and removed. Are they being modified at
> any point?
>
>
>
> What version are you currently using? I'm going to guess 1.0?
>
>
>
> Is there an actual domain object for these objects that some values are
> being deserialized to on the server?
>
>
>
> -Jason Huynh
>
>
>
>
>
>
>
> On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
> Hi Team,
>
>
>
> Could you guide us with below exception?
>
>
>
> *How did we get that?*
>
>
>
> Step1: Our region is already loaded with 1M+ records
> [Region<String,PdxInstance> -- PDXInstance is JsonString using
> JsonFormatter]
>
> Step2: When client is instructed about bulk updates in database, we
> calculate diff of keys using (cache - database) to remove stale deleted
> entries from region
>
> Step3: To do the same, we run below loop,
>
>
>
>
>
> for(Object key : cacheKeys ){
>
> template.remove(key);
>
> }
>
>
>
> *Region def:*
>
>
>
> <geode:replicated-region id=*"Event"*
>
> cache-ref=*"geodeCache"*
>
> scope=*"distributed-ack"*
>
> key-constraint=*"java.lang.String"*
>
> value-constraint=*"org.apache.geode.pdx.PdxInstance"*
>
> shortcut=*"REPLICATE_PERSISTENT_OVERFLOW"*
>
> persistent=*"true"*
>
> disk-synchronous=*"false"*
>
> disk-store-ref=*"event_disk_store"*>
>
> <geode:cache-loader ref=*"eventCacheMissDbLoader"*/>
>
> </geode:replicated-region>
>
>
>
> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
> i"* cache-ref=*"geodeCache"*/>
>
> <geode:index id=*"event_intField_indx"* expression=*"i.intField"*
> from=*"/Event i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>
>
>
> Which results into below exception >>
>
>
>
> java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString
> cannot be cast to java.lang.Integer
>
> at java.lang.Integer.compareTo(Integer.java:52)
>
> at
> org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
>
> at
> org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
>
> at
> org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
>
> ... 23 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
>
> at
> org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
>
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
>
> ... 17 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException:
> org.apache.geode.cache.query.internal.index.IMQException
>
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
>
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
>
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
>
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
>
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
>
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
>
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
>
> at
> org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
>
> at
> org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
>
> at
> org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
>
> at
> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at
> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
>
> ... 1 common frames omitted
>
> Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote
> server on XXXXXX: : While performing a remote destroy
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
>
> at
> org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
>
> at
> org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
>
> at
> org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
>
> at
> org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
>
> at
> org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
>
>
>
> Regards,
>
> Dharam
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
RE: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by "Thacker, Dharam" <dh...@jpmorgan.com>.
Hi Huynh,
Entries are being put using GemfirreTemplate from client side.
gemfireTemplate.putAll(map) [Where map --> Map<String, PdxInsatnce> And PdxInstance --> Jsonformatter.fromjson(jsonString)]
Yes you are right. They are being modified before deletion. Let me describe those steps.
1. Let’s say cache has 10 records and database has 8 records.
2. Calculating delta is costly as well as emptying region and reloading is also not valid option for us due to realtime use case of the same region data
3. So we first update all 8 records from database into cache which will replace “Value” part of region if key matches
4. Then question comes about those extra 2 records which still exists into cache but not in database
5. So we calculate key different of (cacheKeys – databaseKeys) [A MINUS B] and delete unwanted keys from cache
Regards,
Dharam
From: Jason Huynh [mailto:jhuynh@pivotal.io]
Sent: Wednesday, May 03, 2017 10:53 PM
To: user@geode.apache.org
Subject: Re: IndexMaintenanceException while remove old keys correctly from Compact Map Range Index
Hi Dharam,
How are the entries being put into the region? Are they put through a client or is it done on the server itself?
Does the put command look similar to :
region.put(key, JSONFormatter.fromJSON(someJsonString))?
Somehow the object that is being removed is now returning an Integer instead of a PdxString for it's index value (either intField or uuid, depending on which index is failing). I am not sure exactly how that would happen at this point, but it will help to know exactly how these objects are put and how they are updated and removed. Are they being modified at any point?
What version are you currently using? I'm going to guess 1.0?
Is there an actual domain object for these objects that some values are being deserialized to on the server?
-Jason Huynh
On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Team,
Could you guide us with below exception?
How did we get that?
Step1: Our region is already loaded with 1M+ records [Region<String,PdxInstance> -- PDXInstance is JsonString using JsonFormatter]
Step2: When client is instructed about bulk updates in database, we calculate diff of keys using (cache - database) to remove stale deleted entries from region
Step3: To do the same, we run below loop,
for(Object key : cacheKeys ){
template.remove(key);
}
Region def:
<geode:replicated-region id="Event"
cache-ref="geodeCache"
scope="distributed-ack"
key-constraint="java.lang.String"
value-constraint="org.apache.geode.pdx.PdxInstance"
shortcut="REPLICATE_PERSISTENT_OVERFLOW"
persistent="true"
disk-synchronous="false"
disk-store-ref="event_disk_store">
<geode:cache-loader ref="eventCacheMissDbLoader"/>
</geode:replicated-region>
<geode:index id="event_uuid_indx" expression="i.uuid" from="/Event i" cache-ref="geodeCache"/>
<geode:index id="event_intField_indx" expression="i.intField" from="/Event i" cache-ref="geodeCache" type="FUNCTIONAL"/>
Which results into below exception >>
java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString cannot be cast to java.lang.Integer
at java.lang.Integer.compareTo(Integer.java:52)
at org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
at org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
... 23 common frames omitted
Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
at org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
at org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
at org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
... 17 common frames omitted
Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException: org.apache.geode.cache.query.internal.index.IMQException
at org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
at org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
at org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
at org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
at org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
at org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
at org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
... 1 common frames omitted
Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote server on XXXXXX: : While performing a remote destroy
at org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
at org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
at org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
at org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
at org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
Regards,
Dharam
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: IndexMaintenanceException while remove old keys correctly from
Compact Map Range Index
Posted by Jason Huynh <jh...@pivotal.io>.
Hi Dharam,
How are the entries being put into the region? Are they put through a
client or is it done on the server itself?
Does the put command look similar to :
region.put(key, JSONFormatter.fromJSON(someJsonString))?
Somehow the object that is being removed is now returning an Integer
instead of a PdxString for it's index value (either intField or uuid,
depending on which index is failing). I am not sure exactly how that would
happen at this point, but it will help to know exactly how these objects
are put and how they are updated and removed. Are they being modified at
any point?
What version are you currently using? I'm going to guess 1.0?
Is there an actual domain object for these objects that some values are
being deserialized to on the server?
-Jason Huynh
On Wed, May 3, 2017 at 3:36 AM Thacker, Dharam <dh...@jpmorgan.com>
wrote:
> Hi Team,
>
>
>
> Could you guide us with below exception?
>
>
>
> *How did we get that?*
>
>
>
> Step1: Our region is already loaded with 1M+ records
> [Region<String,PdxInstance> -- PDXInstance is JsonString using
> JsonFormatter]
>
> Step2: When client is instructed about bulk updates in database, we
> calculate diff of keys using (cache - database) to remove stale deleted
> entries from region
>
> Step3: To do the same, we run below loop,
>
>
>
>
>
> for(Object key : cacheKeys ){
>
> template.remove(key);
>
> }
>
>
>
> *Region def:*
>
>
>
> <geode:replicated-region id=*"Event"*
>
> cache-ref=*"geodeCache"*
>
> scope=*"distributed-ack"*
>
> key-constraint=*"java.lang.String"*
>
> value-constraint=*"org.apache.geode.pdx.PdxInstance"*
>
> shortcut=*"REPLICATE_PERSISTENT_OVERFLOW"*
>
> persistent=*"true"*
>
> disk-synchronous=*"false"*
>
> disk-store-ref=*"event_disk_store"*>
>
> <geode:cache-loader ref=*"eventCacheMissDbLoader"*/>
>
> </geode:replicated-region>
>
>
>
> <geode:index id=*"event_uuid_indx"* expression=*"i.uuid"* from=*"/Event
> i"* cache-ref=*"geodeCache"*/>
>
> <geode:index id=*"event_intField_indx"* expression=*"i.intField"*
> from=*"/Event i"* cache-ref=*"geodeCache"* type=*"FUNCTIONAL"*/>
>
>
>
> Which results into below exception >>
>
>
>
> java.lang.ClassCastException: org.apache.geode.pdx.internal.PdxString
> cannot be cast to java.lang.Integer
>
> at java.lang.Integer.compareTo(Integer.java:52)
>
> at
> org.apache.geode.cache.query.internal.types.ExtendedNumericComparator.compare(ExtendedNumericComparator.java:49)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781)
>
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
>
> at
> org.apache.geode.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:308)
>
> at
> org.apache.geode.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:286)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1695)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1627)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1637)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1477)
>
> ... 23 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.internal.index.IMQException: null
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1491)
>
> at
> org.apache.geode.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:167)
>
> at
> org.apache.geode.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:511)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1111)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:967)
>
> at
> org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:941)
>
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:815)
>
> ... 17 common frames omitted
>
> Wrapped by: org.apache.geode.cache.query.IndexMaintenanceException:
> org.apache.geode.cache.query.internal.index.IMQException
>
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:820)
>
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3038)
>
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1386)
>
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:7019)
>
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6991)
>
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
>
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6956)
>
> at
> org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1738)
>
> at
> org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5801)
>
> at
> org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:232)
>
> at
> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:147)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1143)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at
> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
>
> ... 1 common frames omitted
>
> Wrapped by: org.apache.geode.cache.client.ServerOperationException: remote
> server on XXXXXX: : While performing a remote destroy
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.processAck(AbstractOp.java:263)
>
> at
> org.apache.geode.cache.client.internal.DestroyOp$DestroyOpImpl.processResponse(DestroyOp.java:201)
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:176)
>
> at
> org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
>
> at
> org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
>
> at
> org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
>
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:110)
>
> at
> org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:697)
>
> at
> org.apache.geode.cache.client.internal.DestroyOp.execute(DestroyOp.java:93)
>
>
>
> Regards,
>
> Dharam
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>