You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "kimec.ethome.sk" <ki...@ethome.sk> on 2021/05/17 06:10:10 UTC
concurrency issues in entry processor
Greetings,
we have recently run into a concurrency issue in an entry processor.
Suppose we have a cache that is IgniteCache<Long, HashMap<String,
String>>.
Entries in this cache are added and modified solely via EntryProcessors.
Now suppose that the cache contains a value for key 1L - a map with
several key value pairs.
Next suppose, we execute the same entry processor for the key 1L TWICE
at the very same time on the same value.
Up until now my understanding was that since the HashMap is stored
offheap, each entry processor would get it's own deserialized copy of
the HashMap for modification and Ignite would then orchestrate the
update of the value (serializing/deserializing the value to offheap
reagion).
However, what we see in our test case is that both entry processor
"executions" receive the same deserialized HashMap and thus the
modification of the HashMap in both processors at the same time causes
java.util.ConcurrentModificationException.
I would expect this kind of behaviour in case the HashMap was stored on
regular Java heap but if the value is deserialized from offheap region?
Is my logic correct? I am trying to wrap my head around this but does
this mean that any complex datastractures stored in must require
separate locking?
Thanks!
Kamil
Re: concurrency issues in entry processor
Posted by Kamil Misuth <ki...@ethome.sk>.
Hi Semyon,
I have tracked the issue down in our code base and in the end it wasn't
Ignite's fault.
Sorry for the false alarm.
Kamil
On 5/19/21 9:43 AM, Kamil Misuth wrote:
> Hi,
>
> yes. I am preparing a reproducer but this is happening in a production
> grade app deployed in our testing environment. I will send you the
> reproducer once its' ready. Hopefully it won't take long.
>
> Kamil
>
> On 5/19/21 7:48 AM, Данилов Семён wrote:
>> Hello!
>>
>> Ignite indeed has such optimizations but I don't see them enabled in
>> your configuration so perhaps it's a bug.
>> I still have trouble reproducing your issue, so could you please send
>> a minimal reproducer?
>>
>> Kind regards, Semyon.
>>
>> 19.05.2021, 06:12, "Kamil Misuth" <ki...@ethome.sk>:
>>> Hi Semyon,
>>>
>>> we are using Ignite 2.8.1. That being said, could Ignite really be
>>> using
>>> some behind the scenes optimization which allows for a value
>>> instance to
>>> be re-used between multiple EntryProcessor runs?
>>>
>>> Kamil
>>>
>>> On 5/17/21 11:45 AM, Данилов Семён wrote:
>>>> I tried reproducing your issue but with no avail. What version of
>>>> Apache Ignite do you use?
>>>>
>>>> Kind regards, Semyon.
>>>>
>>>> 17.05.2021, 11:03, "kimec.ethome.sk" <ki...@ethome.sk>:
>>>>> Hi Semyon,
>>>>>
>>>>> the cache configuration is:
>>>>>
>>>>> CacheConfiguration<Long, HashMap<String, String>> cc = new
>>>>> CacheConfiguration<>();
>>>>> cc.setName("fancyCache");
>>>>> cc.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>>>> cc.setCacheMode(CacheMode.PARTITIONED);
>>>>> cc.setBackups(1);
>>>>> cc.setRebalanceMode(CacheRebalanceMode.SYNC);
>>>>> cc.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>>>>> cc.setReadFromBackup(false);
>>>>>
>>>>> Kamil
>>>>>
>>>>> On 2021-05-17 09:06, Данилов Семён wrote:
>>>>>> Hello, Kamil!
>>>>>>
>>>>>> Could you please provide your cache configuration?
>>>>>>
>>>>>> Kind regards, Semyon.
>>>>>>
>>>>>> 17.05.2021, 09:10, "kimec.ethome.sk" <ki...@ethome.sk>:
>>>>>>> Greetings,
>>>>>>>
>>>>>>> we have recently run into a concurrency issue in an entry
>>>>>>> processor.
>>>>>>>
>>>>>>> Suppose we have a cache that is IgniteCache<Long,
>>>>>>> HashMap<String,
>>>>>>> String>>.
>>>>>>> Entries in this cache are added and modified solely via
>>>>>>> EntryProcessors.
>>>>>>>
>>>>>>> Now suppose that the cache contains a value for key 1L - a
>>>>>>> map with
>>>>>>> several key value pairs.
>>>>>>> Next suppose, we execute the same entry processor for the
>>>>>>> key 1L TWICE
>>>>>>> at the very same time on the same value.
>>>>>>>
>>>>>>> Up until now my understanding was that since the HashMap is
>>>>>>> stored
>>>>>>> offheap, each entry processor would get it's own
>>>>>>> deserialized copy of
>>>>>>> the HashMap for modification and Ignite would then
>>>>>>> orchestrate the
>>>>>>> update of the value (serializing/deserializing the value to
>>>>>>> offheap
>>>>>>> reagion).
>>>>>>>
>>>>>>> However, what we see in our test case is that both entry
>>>>>>> processor
>>>>>>> "executions" receive the same deserialized HashMap and thus the
>>>>>>> modification of the HashMap in both processors at the same
>>>>>>> time causes
>>>>>>> java.util.ConcurrentModificationException.
>>>>>>>
>>>>>>> I would expect this kind of behaviour in case the HashMap
>>>>>>> was stored
>>>>>>> on
>>>>>>> regular Java heap but if the value is deserialized from offheap
>>>>>>> region?
>>>>>>> Is my logic correct? I am trying to wrap my head around this
>>>>>>> but does
>>>>>>> this mean that any complex datastractures stored in must
>>>>>>> require
>>>>>>> separate locking?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Kamil
Re: concurrency issues in entry processor
Posted by Kamil Misuth <ki...@ethome.sk>.
Hi,
yes. I am preparing a reproducer but this is happening in a production
grade app deployed in our testing environment. I will send you the
reproducer once its' ready. Hopefully it won't take long.
Kamil
On 5/19/21 7:48 AM, Данилов Семён wrote:
> Hello!
>
> Ignite indeed has such optimizations but I don't see them enabled in your configuration so perhaps it's a bug.
> I still have trouble reproducing your issue, so could you please send a minimal reproducer?
>
> Kind regards, Semyon.
>
> 19.05.2021, 06:12, "Kamil Misuth" <ki...@ethome.sk>:
>> Hi Semyon,
>>
>> we are using Ignite 2.8.1. That being said, could Ignite really be using
>> some behind the scenes optimization which allows for a value instance to
>> be re-used between multiple EntryProcessor runs?
>>
>> Kamil
>>
>> On 5/17/21 11:45 AM, Данилов Семён wrote:
>>> I tried reproducing your issue but with no avail. What version of Apache Ignite do you use?
>>>
>>> Kind regards, Semyon.
>>>
>>> 17.05.2021, 11:03, "kimec.ethome.sk" <ki...@ethome.sk>:
>>>> Hi Semyon,
>>>>
>>>> the cache configuration is:
>>>>
>>>> CacheConfiguration<Long, HashMap<String, String>> cc = new
>>>> CacheConfiguration<>();
>>>> cc.setName("fancyCache");
>>>> cc.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>>> cc.setCacheMode(CacheMode.PARTITIONED);
>>>> cc.setBackups(1);
>>>> cc.setRebalanceMode(CacheRebalanceMode.SYNC);
>>>> cc.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>>>> cc.setReadFromBackup(false);
>>>>
>>>> Kamil
>>>>
>>>> On 2021-05-17 09:06, Данилов Семён wrote:
>>>>> Hello, Kamil!
>>>>>
>>>>> Could you please provide your cache configuration?
>>>>>
>>>>> Kind regards, Semyon.
>>>>>
>>>>> 17.05.2021, 09:10, "kimec.ethome.sk" <ki...@ethome.sk>:
>>>>>> Greetings,
>>>>>>
>>>>>> we have recently run into a concurrency issue in an entry processor.
>>>>>>
>>>>>> Suppose we have a cache that is IgniteCache<Long, HashMap<String,
>>>>>> String>>.
>>>>>> Entries in this cache are added and modified solely via
>>>>>> EntryProcessors.
>>>>>>
>>>>>> Now suppose that the cache contains a value for key 1L - a map with
>>>>>> several key value pairs.
>>>>>> Next suppose, we execute the same entry processor for the key 1L TWICE
>>>>>> at the very same time on the same value.
>>>>>>
>>>>>> Up until now my understanding was that since the HashMap is stored
>>>>>> offheap, each entry processor would get it's own deserialized copy of
>>>>>> the HashMap for modification and Ignite would then orchestrate the
>>>>>> update of the value (serializing/deserializing the value to offheap
>>>>>> reagion).
>>>>>>
>>>>>> However, what we see in our test case is that both entry processor
>>>>>> "executions" receive the same deserialized HashMap and thus the
>>>>>> modification of the HashMap in both processors at the same time causes
>>>>>> java.util.ConcurrentModificationException.
>>>>>>
>>>>>> I would expect this kind of behaviour in case the HashMap was stored
>>>>>> on
>>>>>> regular Java heap but if the value is deserialized from offheap
>>>>>> region?
>>>>>> Is my logic correct? I am trying to wrap my head around this but does
>>>>>> this mean that any complex datastractures stored in must require
>>>>>> separate locking?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Kamil
Re: concurrency issues in entry processor
Posted by Данилов Семён <sa...@yandex.ru>.
Hello!
Ignite indeed has such optimizations but I don't see them enabled in your configuration so perhaps it's a bug.
I still have trouble reproducing your issue, so could you please send a minimal reproducer?
Kind regards, Semyon.
19.05.2021, 06:12, "Kamil Misuth" <ki...@ethome.sk>:
> Hi Semyon,
>
> we are using Ignite 2.8.1. That being said, could Ignite really be using
> some behind the scenes optimization which allows for a value instance to
> be re-used between multiple EntryProcessor runs?
>
> Kamil
>
> On 5/17/21 11:45 AM, Данилов Семён wrote:
>> I tried reproducing your issue but with no avail. What version of Apache Ignite do you use?
>>
>> Kind regards, Semyon.
>>
>> 17.05.2021, 11:03, "kimec.ethome.sk" <ki...@ethome.sk>:
>>> Hi Semyon,
>>>
>>> the cache configuration is:
>>>
>>> CacheConfiguration<Long, HashMap<String, String>> cc = new
>>> CacheConfiguration<>();
>>> cc.setName("fancyCache");
>>> cc.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>> cc.setCacheMode(CacheMode.PARTITIONED);
>>> cc.setBackups(1);
>>> cc.setRebalanceMode(CacheRebalanceMode.SYNC);
>>> cc.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>>> cc.setReadFromBackup(false);
>>>
>>> Kamil
>>>
>>> On 2021-05-17 09:06, Данилов Семён wrote:
>>>> Hello, Kamil!
>>>>
>>>> Could you please provide your cache configuration?
>>>>
>>>> Kind regards, Semyon.
>>>>
>>>> 17.05.2021, 09:10, "kimec.ethome.sk" <ki...@ethome.sk>:
>>>>> Greetings,
>>>>>
>>>>> we have recently run into a concurrency issue in an entry processor.
>>>>>
>>>>> Suppose we have a cache that is IgniteCache<Long, HashMap<String,
>>>>> String>>.
>>>>> Entries in this cache are added and modified solely via
>>>>> EntryProcessors.
>>>>>
>>>>> Now suppose that the cache contains a value for key 1L - a map with
>>>>> several key value pairs.
>>>>> Next suppose, we execute the same entry processor for the key 1L TWICE
>>>>> at the very same time on the same value.
>>>>>
>>>>> Up until now my understanding was that since the HashMap is stored
>>>>> offheap, each entry processor would get it's own deserialized copy of
>>>>> the HashMap for modification and Ignite would then orchestrate the
>>>>> update of the value (serializing/deserializing the value to offheap
>>>>> reagion).
>>>>>
>>>>> However, what we see in our test case is that both entry processor
>>>>> "executions" receive the same deserialized HashMap and thus the
>>>>> modification of the HashMap in both processors at the same time causes
>>>>> java.util.ConcurrentModificationException.
>>>>>
>>>>> I would expect this kind of behaviour in case the HashMap was stored
>>>>> on
>>>>> regular Java heap but if the value is deserialized from offheap
>>>>> region?
>>>>> Is my logic correct? I am trying to wrap my head around this but does
>>>>> this mean that any complex datastractures stored in must require
>>>>> separate locking?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Kamil
Re: concurrency issues in entry processor
Posted by Kamil Misuth <ki...@ethome.sk>.
Hi Semyon,
we are using Ignite 2.8.1. That being said, could Ignite really be using
some behind the scenes optimization which allows for a value instance to
be re-used between multiple EntryProcessor runs?
Kamil
On 5/17/21 11:45 AM, Данилов Семён wrote:
> I tried reproducing your issue but with no avail. What version of Apache Ignite do you use?
>
> Kind regards, Semyon.
>
> 17.05.2021, 11:03, "kimec.ethome.sk" <ki...@ethome.sk>:
>> Hi Semyon,
>>
>> the cache configuration is:
>>
>> CacheConfiguration<Long, HashMap<String, String>> cc = new
>> CacheConfiguration<>();
>> cc.setName("fancyCache");
>> cc.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>> cc.setCacheMode(CacheMode.PARTITIONED);
>> cc.setBackups(1);
>> cc.setRebalanceMode(CacheRebalanceMode.SYNC);
>> cc.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>> cc.setReadFromBackup(false);
>>
>> Kamil
>>
>> On 2021-05-17 09:06, Данилов Семён wrote:
>>> Hello, Kamil!
>>>
>>> Could you please provide your cache configuration?
>>>
>>> Kind regards, Semyon.
>>>
>>> 17.05.2021, 09:10, "kimec.ethome.sk" <ki...@ethome.sk>:
>>>> Greetings,
>>>>
>>>> we have recently run into a concurrency issue in an entry processor.
>>>>
>>>> Suppose we have a cache that is IgniteCache<Long, HashMap<String,
>>>> String>>.
>>>> Entries in this cache are added and modified solely via
>>>> EntryProcessors.
>>>>
>>>> Now suppose that the cache contains a value for key 1L - a map with
>>>> several key value pairs.
>>>> Next suppose, we execute the same entry processor for the key 1L TWICE
>>>> at the very same time on the same value.
>>>>
>>>> Up until now my understanding was that since the HashMap is stored
>>>> offheap, each entry processor would get it's own deserialized copy of
>>>> the HashMap for modification and Ignite would then orchestrate the
>>>> update of the value (serializing/deserializing the value to offheap
>>>> reagion).
>>>>
>>>> However, what we see in our test case is that both entry processor
>>>> "executions" receive the same deserialized HashMap and thus the
>>>> modification of the HashMap in both processors at the same time causes
>>>> java.util.ConcurrentModificationException.
>>>>
>>>> I would expect this kind of behaviour in case the HashMap was stored
>>>> on
>>>> regular Java heap but if the value is deserialized from offheap
>>>> region?
>>>> Is my logic correct? I am trying to wrap my head around this but does
>>>> this mean that any complex datastractures stored in must require
>>>> separate locking?
>>>>
>>>> Thanks!
>>>>
>>>> Kamil
Re: concurrency issues in entry processor
Posted by Данилов Семён <sa...@yandex.ru>.
I tried reproducing your issue but with no avail. What version of Apache Ignite do you use?
Kind regards, Semyon.
17.05.2021, 11:03, "kimec.ethome.sk" <ki...@ethome.sk>:
> Hi Semyon,
>
> the cache configuration is:
>
> CacheConfiguration<Long, HashMap<String, String>> cc = new
> CacheConfiguration<>();
> cc.setName("fancyCache");
> cc.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cc.setCacheMode(CacheMode.PARTITIONED);
> cc.setBackups(1);
> cc.setRebalanceMode(CacheRebalanceMode.SYNC);
> cc.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
> cc.setReadFromBackup(false);
>
> Kamil
>
> On 2021-05-17 09:06, Данилов Семён wrote:
>> Hello, Kamil!
>>
>> Could you please provide your cache configuration?
>>
>> Kind regards, Semyon.
>>
>> 17.05.2021, 09:10, "kimec.ethome.sk" <ki...@ethome.sk>:
>>> Greetings,
>>>
>>> we have recently run into a concurrency issue in an entry processor.
>>>
>>> Suppose we have a cache that is IgniteCache<Long, HashMap<String,
>>> String>>.
>>> Entries in this cache are added and modified solely via
>>> EntryProcessors.
>>>
>>> Now suppose that the cache contains a value for key 1L - a map with
>>> several key value pairs.
>>> Next suppose, we execute the same entry processor for the key 1L TWICE
>>> at the very same time on the same value.
>>>
>>> Up until now my understanding was that since the HashMap is stored
>>> offheap, each entry processor would get it's own deserialized copy of
>>> the HashMap for modification and Ignite would then orchestrate the
>>> update of the value (serializing/deserializing the value to offheap
>>> reagion).
>>>
>>> However, what we see in our test case is that both entry processor
>>> "executions" receive the same deserialized HashMap and thus the
>>> modification of the HashMap in both processors at the same time causes
>>> java.util.ConcurrentModificationException.
>>>
>>> I would expect this kind of behaviour in case the HashMap was stored
>>> on
>>> regular Java heap but if the value is deserialized from offheap
>>> region?
>>> Is my logic correct? I am trying to wrap my head around this but does
>>> this mean that any complex datastractures stored in must require
>>> separate locking?
>>>
>>> Thanks!
>>>
>>> Kamil
Re: concurrency issues in entry processor
Posted by "kimec.ethome.sk" <ki...@ethome.sk>.
Hi Semyon,
the cache configuration is:
CacheConfiguration<Long, HashMap<String, String>> cc = new
CacheConfiguration<>();
cc.setName("fancyCache");
cc.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cc.setCacheMode(CacheMode.PARTITIONED);
cc.setBackups(1);
cc.setRebalanceMode(CacheRebalanceMode.SYNC);
cc.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
cc.setReadFromBackup(false);
Kamil
On 2021-05-17 09:06, Данилов Семён wrote:
> Hello, Kamil!
>
> Could you please provide your cache configuration?
>
> Kind regards, Semyon.
>
> 17.05.2021, 09:10, "kimec.ethome.sk" <ki...@ethome.sk>:
>> Greetings,
>>
>> we have recently run into a concurrency issue in an entry processor.
>>
>> Suppose we have a cache that is IgniteCache<Long, HashMap<String,
>> String>>.
>> Entries in this cache are added and modified solely via
>> EntryProcessors.
>>
>> Now suppose that the cache contains a value for key 1L - a map with
>> several key value pairs.
>> Next suppose, we execute the same entry processor for the key 1L TWICE
>> at the very same time on the same value.
>>
>> Up until now my understanding was that since the HashMap is stored
>> offheap, each entry processor would get it's own deserialized copy of
>> the HashMap for modification and Ignite would then orchestrate the
>> update of the value (serializing/deserializing the value to offheap
>> reagion).
>>
>> However, what we see in our test case is that both entry processor
>> "executions" receive the same deserialized HashMap and thus the
>> modification of the HashMap in both processors at the same time causes
>> java.util.ConcurrentModificationException.
>>
>> I would expect this kind of behaviour in case the HashMap was stored
>> on
>> regular Java heap but if the value is deserialized from offheap
>> region?
>> Is my logic correct? I am trying to wrap my head around this but does
>> this mean that any complex datastractures stored in must require
>> separate locking?
>>
>> Thanks!
>>
>> Kamil
Re: concurrency issues in entry processor
Posted by Данилов Семён <sa...@yandex.ru>.
Hello, Kamil!
Could you please provide your cache configuration?
Kind regards, Semyon.
17.05.2021, 09:10, "kimec.ethome.sk" <ki...@ethome.sk>:
> Greetings,
>
> we have recently run into a concurrency issue in an entry processor.
>
> Suppose we have a cache that is IgniteCache<Long, HashMap<String,
> String>>.
> Entries in this cache are added and modified solely via EntryProcessors.
>
> Now suppose that the cache contains a value for key 1L - a map with
> several key value pairs.
> Next suppose, we execute the same entry processor for the key 1L TWICE
> at the very same time on the same value.
>
> Up until now my understanding was that since the HashMap is stored
> offheap, each entry processor would get it's own deserialized copy of
> the HashMap for modification and Ignite would then orchestrate the
> update of the value (serializing/deserializing the value to offheap
> reagion).
>
> However, what we see in our test case is that both entry processor
> "executions" receive the same deserialized HashMap and thus the
> modification of the HashMap in both processors at the same time causes
> java.util.ConcurrentModificationException.
>
> I would expect this kind of behaviour in case the HashMap was stored on
> regular Java heap but if the value is deserialized from offheap region?
> Is my logic correct? I am trying to wrap my head around this but does
> this mean that any complex datastractures stored in must require
> separate locking?
>
> Thanks!
>
> Kamil