You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Justin Ji <bi...@foxmail.com> on 2018/10/09 12:47:25 UTC

Update all data in cache spend a long time

Hi - 

I have a cache with more than 1 million records, when I update the whole
records in it, it spends a long time(almost 6 minutes).

Here is the SQL:
"update " + IgniteTableKey.T_DEVICE_ONLINE_STATUS.getCode()+ " set
isOnline=0, mqttTime=" + System.currentTimeMillis() / 1000 + " where 1=1";

And the Cache configuration is :

CacheConfiguration<K, V> cacheCfg = new CacheConfiguration<>();
        cacheCfg.setName(cacheName);
        cacheCfg.setCacheMode(CacheMode.PARTITIONED);
        cacheCfg.setBackups(1);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
       
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DeviceStatusCacheStore.class));
        cacheCfg.setWriteThrough(true);
        cacheCfg.setWriteBehindEnabled(true);
        cacheCfg.setReadThrough(true);
        cacheCfg.setWriteBehindFlushThreadCount(4);
        cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
        cacheCfg.setWriteBehindFlushSize(409600);
        cacheCfg.setWriteBehindBatchSize(1024);
        cacheCfg.setStoreKeepBinary(true);
        cfg.setCacheConfiguration(cacheCfg);

So I want to know is there a way that can speed up the execution？



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Update all data in cache spend a long time

Posted by Justin Ji <bi...@foxmail.com>.
Thank you, I will try queryParallelism.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Update all data in cache spend a long time

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

1. I don't think you should use LIMIT because it is very hard to make sure
that all entries are updated exactly once. It is better to partition data
by values of some field.
2. It is hard to say outright, did you try queryParallelism?

Regards,
-- 
Ilya Kasnacheev


вт, 9 окт. 2018 г. в 16:27, Justin Ji <bi...@foxmail.com>:

> Ilya -
>
> Thank for your reply and point out the problem may exist in my code, I will
> try to split the operation into smaller ones.
>
> And I still have two problems:
>
> 1. Can I split the operation with LIMIT?
> 2. Where is time spent in the operation?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Update all data in cache spend a long time

Posted by Justin Ji <bi...@foxmail.com>.
Ilya - 

Thank for your reply and point out the problem may exist in my code, I will
try to split the operation into smaller ones.

And I still have two problems:

1. Can I split the operation with LIMIT?
2. Where is time spent in the operation?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Update all data in cache spend a long time

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

Have you tried setting queryParallelism on this cache to some high value
(such as your number of CPU cores)?

Note that this query will likely cause lifting all data in cache to heap at
once, which is probably not what you expect. My recommendation is also to
try splitting this operation into smaller ones, see if they finish faster.

Regards,
-- 
Ilya Kasnacheev


вт, 9 окт. 2018 г. в 15:47, Justin Ji <bi...@foxmail.com>:

> Hi -
>
> I have a cache with more than 1 million records, when I update the whole
> records in it, it spends a long time(almost 6 minutes).
>
> Here is the SQL:
> "update " + IgniteTableKey.T_DEVICE_ONLINE_STATUS.getCode()+ " set
> isOnline=0, mqttTime=" + System.currentTimeMillis() / 1000 + " where 1=1";
>
> And the Cache configuration is :
>
> CacheConfiguration<K, V> cacheCfg = new CacheConfiguration<>();
>         cacheCfg.setName(cacheName);
>         cacheCfg.setCacheMode(CacheMode.PARTITIONED);
>         cacheCfg.setBackups(1);
>         cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>
>
> cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DeviceStatusCacheStore.class));
>         cacheCfg.setWriteThrough(true);
>         cacheCfg.setWriteBehindEnabled(true);
>         cacheCfg.setReadThrough(true);
>         cacheCfg.setWriteBehindFlushThreadCount(4);
>         cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
>         cacheCfg.setWriteBehindFlushSize(409600);
>         cacheCfg.setWriteBehindBatchSize(1024);
>         cacheCfg.setStoreKeepBinary(true);
>         cfg.setCacheConfiguration(cacheCfg);
>
> So I want to know is there a way that can speed up the execution？
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>