You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Prasad Bhalerao <pr...@gmail.com> on 2019/01/22 13:37:25 UTC

cache.removeAll cache operation is slow

Hi,

I am removing around 2 million entries from cache using cache.removeAll api.
I removing entries in a batch of 100k.  To remove 2 million entries ignite
takes around 218 seconds.
This cache is transactiona and write synchronization mode is full sync.

Is there any way to improve the performance of removeAll operation?

Thanks,
Prasad

Re: cache.removeAll cache operation is slow

Posted by Prasad Bhalerao <pr...@gmail.com>.
The data I am removing from cache is colocated on the same node. So I think
exchange will happen only for updating the backup copies. Please correct me
if I am wrong.

I believe that remove all operation first acquires the lock on all input
keys and then removes the data. If I go for single remove, will it not
acquire lock per key per remove operation. Will it not slow down?

Will try your suggestions.



On Tue 22 Jan, 2019, 7:22 PM Denis Mekhanikov <dmekhanikov@gmail.com wrote:

> Prasad,
>
> When you run a transaction, that involves many entries, the whole key set
> is sent between nodes multiple times.
> It also generates a lot of garbage, which cannot be released until the
> transaction is committed.
> It's better to remove values in small batches.
> So, try changing the batch size. It's possible, that singular removes will
> work faster.
>
> Denis
>
> вт, 22 янв. 2019 г. в 16:37, Prasad Bhalerao <prasadbhalerao1983@gmail.com
> >:
>
>> Hi,
>>
>> I am removing around 2 million entries from cache using cache.removeAll
>> api.
>> I removing entries in a batch of 100k.  To remove 2 million
>> entries ignite takes around 218 seconds.
>> This cache is transactiona and write synchronization mode is full sync.
>>
>> Is there any way to improve the performance of removeAll operation?
>>
>> Thanks,
>> Prasad
>>
>>
>>

Re: cache.removeAll cache operation is slow

Posted by Denis Mekhanikov <dm...@gmail.com>.
Prasad,

When you run a transaction, that involves many entries, the whole key set
is sent between nodes multiple times.
It also generates a lot of garbage, which cannot be released until the
transaction is committed.
It's better to remove values in small batches.
So, try changing the batch size. It's possible, that singular removes will
work faster.

Denis

вт, 22 янв. 2019 г. в 16:37, Prasad Bhalerao <pr...@gmail.com>:

> Hi,
>
> I am removing around 2 million entries from cache using cache.removeAll
> api.
> I removing entries in a batch of 100k.  To remove 2 million entries ignite
> takes around 218 seconds.
> This cache is transactiona and write synchronization mode is full sync.
>
> Is there any way to improve the performance of removeAll operation?
>
> Thanks,
> Prasad
>
>
>