You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Justin Ji <bi...@foxmail.com> on 2018/11/28 12:48:24 UTC
Ignite cache.getAll takes a long time
Hi all
I have a cluster with 3 server nodes which have 4 CPU cores and 8Gb memory.
When applications call cache.getAll(set), it often takes a long time almost
500ms(about 0.6%), even the set size is less than 3.
And I do not think it was caused by the network, because the network latency
between application and server is less than 2 milliseconds.
The CPU of Ignite servers were between 3%~5% and the QPS are about 1000 read
and 1000 write.
Here is my code segment:
Cache:
CacheConfiguration<String, BinaryObject> cacheCfg =
DefaultIgniteConfiguration.getCacheConfiguration(IgniteCacheKey.DATA_POINT.getCode());
cacheCfg.setBackups(2);
cacheCfg.setDataRegionName(Constants.THREE_GB_REGION);
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
cacheCfg.setWriteThrough(true);
cacheCfg.setWriteBehindEnabled(true);
cacheCfg.setWriteBehindFlushThreadCount(2);
//每15秒刷新一次
cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
cacheCfg.setWriteBehindFlushSize(409600);
cacheCfg.setWriteBehindBatchSize(1024);
cacheCfg.setStoreKeepBinary(true);
cacheCfg.setQueryParallelism(4);
//2M
cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
getAll:
public Map<String, BinaryObject> getAll(Set<String> keys) {
long total = totalRead.incrementAndGet();
long startTime = System.currentTimeMillis();
Map<String, BinaryObject> all = igniteCache.getAll(keys);
long currentTime = System.currentTimeMillis();
long took = currentTime - startTime;
if (took > 500) {
readGreaterThan500.incrementAndGet();
logger.warn("read complete, cost:{}ms", took);
}
if (total % 10000 == 0) {
double percent = 1000.0 * readGreaterThan500.get() / total;
logger.info("percent that more 500ms:{}‰",
readGreaterThan500.get(), total, format.format(percent));
}
return all;
}
putAllAsync:
public IgniteFuture<Void> putAllAsync(Map<String, DpCache> map) {
long total = totalWrite.incrementAndGet();
long startTime = System.currentTimeMillis();
Map<String, BinaryObject> objectMap = Maps.newHashMap();
for (Map.Entry<String, DpCache> entry : map.entrySet()) {
objectMap.put(entry.getKey(),
ignite.binary().toBinary(entry.getValue()));
}
IgniteFuture<Void> future = igniteCache.putAllAsync(objectMap);
long currentTime = System.currentTimeMillis();
long took = currentTime - startTime;
if (took > 500) {
writeGreaterThan500.incrementAndGet();
logger.warn("write complete, cost:{}ms", took);
}
if (total % 10000 == 0) {
double percent = 1000.0 * writeGreaterThan500.get() / total;
logger.info("percent that more than 500ms :{}‰",
writeGreaterThan500.get(), total,
format.format(percent));
}
return future;
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite cache.getAll takes a long time
Posted by Stanislav Lukyanov <st...@gmail.com>.
I guess it could be caused by https://issues.apache.org/jira/browse/IGNITE-5003 which is mentioned in that thread.
Also, make sure that your cache store code doesn’t cause you troubles – that you don’t open a new connection every time,
don’t have unnecessary blocking, etc.
Stan
From: Justin Ji
Sent: 3 декабря 2018 г. 13:10
To: user@ignite.apache.org
Subject: RE: Ignite cache.getAll takes a long time
Stan -
Thank for your reply!
Yes, the getAll and putAll(async) executed in parallel(amount of operations
executed at the same time).
But I think it may be caused by the write-behind, when I disabled the
write-behind the timeout disappeared, and I enabled the write-behind the
timeout appeared.
It is a little similar to
http://apache-ignite-users.70518.x6.nabble.com/write-behind-performance-impacting-main-thread-Write-behind-buffer-is-never-full-td17940.html
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite cache.getAll takes a long time
Posted by Justin Ji <bi...@foxmail.com>.
Stan -
Thank for your reply!
Yes, the getAll and putAll(async) executed in parallel(amount of operations
executed at the same time).
But I think it may be caused by the write-behind, when I disabled the
write-behind the timeout disappeared, and I enabled the write-behind the
timeout appeared.
It is a little similar to
http://apache-ignite-users.70518.x6.nabble.com/write-behind-performance-impacting-main-thread-Write-behind-buffer-is-never-full-td17940.html
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite cache.getAll takes a long time
Posted by Stanislav Lukyanov <st...@gmail.com>.
Hi,
Start by looking at what’s going on in the cluster when you see these long reads.
Collecting a JFR record or a GC log would be nice.
Also, do you see that with concurrent reads and writes, i.e. do you have getAll and putAll executing in parallel?
Perhaps there is some sort of contention between them.
Finally, try to find correlation between the key sets you get high latency for. Perhaps there is something special about these keys.
Stan
From: Justin Ji
Sent: 29 ноября 2018 г. 5:24
To: user@ignite.apache.org
Subject: Re: Ignite cache.getAll takes a long time
I use the default thread pool settings, can it improve the performance after
I increase the system thread pool?
Another question, should I add the configuration in server nodes?
Looking forward to your reply
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite cache.getAll takes a long time
Posted by Justin Ji <bi...@foxmail.com>.
I use the default thread pool settings, can it improve the performance after
I increase the system thread pool?
Another question, should I add the configuration in server nodes?
Looking forward to your reply
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/