You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ctranxuan <ce...@streamdata.io> on 2017/04/11 09:53:06 UTC
Lots of cache creation become slow
Hi,
We are trying for test purposes to create lots of cache on a machine.
Something like:
for (int i = 0; i < 10000; i++) {
IgniteCache<Long, Snapshot> cache;
cache = ignite.<Long, Snapshot>getOrCreateCache("cache-" +
i).withAsync();
LOGGER.info("starting to read cache #" + cache.getName());
}
What we are noticing is that after ~200 caches created, the cache creation
becomes slower and slower. For instance, when reaching the 2000th cache, it
takes between 2-3 seconds. For instance, here some logs:
09:34:27.857 starting to read cache #cache-2087
09:34:29.621 starting to read cache #cache-2088
09:34:31.450 starting to read cache #cache-2089
09:34:33.127 starting to read cache #cache-2090
That's may be a naive way of doing that and may be a naive question: but is
it normal that the more cache is created, the more time the creation takes?
The program is run on a 12GB RAM machine with 4 CPUs with a Java 8
(1.8.0_121).
Thanks in advance for the answers!
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Lots of cache creation become slow
Posted by ctranxuan <ce...@streamdata.io>.
Well, that was a new design assumption for the architecture of our PoC.
At the beginning, we wanted some "multi-tenant" caches with 1 continuous
query for each. Then, we've tried to switch with one cache and 1 continous
query for each key.
But, you're right. We have already started to challenge our PoC with 1
continous query for the whole cache.
Thanks for the advice!
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875p12427.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Lots of cache creation become slow
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Cédric,
Can you clarify why not create 1 continuous query and listen to all the
changes for all the keys?
D.
On Thu, Apr 13, 2017 at 8:00 AM, ctranxuan <ce...@streamdata.io>
wrote:
> Well, actually we were interesting in having continuous queries listening
> multi-tenant caches.
>
> This was the postulate for the architecture of a PoC project. Based on this
> discussion, we are switching to another architecture postulate where we
> have
> one cache with thousands continuous queries listening the changes of
> thousands keys of the cache (basically 1 continuous query per key).
>
> So, at the beginning, we were investigating how many caches / continuous
> queries could be supported by a node. May be, it's not the right way to
> evaluate this?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875p11955.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Lots of cache creation become slow
Posted by ctranxuan <ce...@streamdata.io>.
Well, actually we were interesting in having continuous queries listening
multi-tenant caches.
This was the postulate for the architecture of a PoC project. Based on this
discussion, we are switching to another architecture postulate where we have
one cache with thousands continuous queries listening the changes of
thousands keys of the cache (basically 1 continuous query per key).
So, at the beginning, we were investigating how many caches / continuous
queries could be supported by a node. May be, it's not the right way to
evaluate this?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875p11955.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Lots of cache creation become slow
Posted by Alexey Kuznetsov <ak...@apache.org>.
Cédric,
Just, curious, what for do you need 2000 caches on single node? :)
On Wed, Apr 12, 2017 at 8:47 PM, ctranxuan <ce...@streamdata.io>
wrote:
> Thanks a lot for the answer!
> We'll try to run tests with these hints.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875p11906.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Alexey Kuznetsov
Re: Lots of cache creation become slow
Posted by ctranxuan <ce...@streamdata.io>.
Thanks a lot for the answer!
We'll try to run tests with these hints.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875p11906.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Lots of cache creation become slow
Posted by Andrey Gura <ag...@apache.org>.
Creation of each cache requires creation and initialization of
internal data structures that leads to increased pressure to GC. Could
you enable GC logs and look at result. I think you will find long GC
pauses.
In order to reduce memory consumption by created caches on creation
stage we can do the following:
- Decrease cache start size (see cacheConfiguration.setStartSize(),
default value is 1 500 000). It safe but can lead to some performance
penalty during dynamic increasing cache size;
- For atomic caches we can decrease size of deferred delete queue size
via JVM parameter (-DIGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE=<your
value>, default value is 200 000 entries per partition). It's static
configuration and can't be changed at runtime;
- Partitions number also affect memory consumption and cache creation
time. But it affect also data distribution so should be configured
carefully. Caches with default configuration uses
RendevousAffinityFunction ith 1024 partitions. This parameter can't be
changed after cache creation and start.
Does it work for you?
On Tue, Apr 11, 2017 at 12:53 PM, ctranxuan
<ce...@streamdata.io> wrote:
> Hi,
> We are trying for test purposes to create lots of cache on a machine.
> Something like:
>
> for (int i = 0; i < 10000; i++) {
> IgniteCache<Long, Snapshot> cache;
> cache = ignite.<Long, Snapshot>getOrCreateCache("cache-" +
> i).withAsync();
> LOGGER.info("starting to read cache #" + cache.getName());
> }
>
> What we are noticing is that after ~200 caches created, the cache creation
> becomes slower and slower. For instance, when reaching the 2000th cache, it
> takes between 2-3 seconds. For instance, here some logs:
>
> 09:34:27.857 starting to read cache #cache-2087
> 09:34:29.621 starting to read cache #cache-2088
> 09:34:31.450 starting to read cache #cache-2089
> 09:34:33.127 starting to read cache #cache-2090
>
> That's may be a naive way of doing that and may be a naive question: but is
> it normal that the more cache is created, the more time the creation takes?
>
> The program is run on a 12GB RAM machine with 4 CPUs with a Java 8
> (1.8.0_121).
>
> Thanks in advance for the answers!
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Lots-of-cache-creation-become-slow-tp11875.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.