You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by 胡永亮/Bob <hu...@neusoft.com> on 2016/09/23 09:27:26 UTC
A very slow code of using entryprocessor, why?
hi, everyone
The code and related log are the following content, how can I do to increase the performance? Thank you very much.
Code:
IgniteCache<String, Map<Long, List<baselink>>> mapMatchingData = IgniteManager.getInstance().getMapMatchingData();
Iterator<Cache.Entry<String, Map<Long, List<baselink>>>> it1 = mapMatchingData.iterator();
while(it1.hasNext()){
Cache.Entry<String, Map<Long, List<baselink>>> cEntry = it1.next();
final String car_key = cEntry.getKey();
log.info("car_key:"+car_key);
mapMatchingData.invoke(car_key, new EntryProcessor<String, Map<Long, List<baselink>>, Void>() {
@Override
public Void process(MutableEntry<String, Map<Long, List<baselink>>> entry, Object... args) {
Map<Long, List<baselink>> localMap = entry.getValue();
Iterator<Entry<Long, List<baselink>>> entryIter = localMap.entrySet().iterator();
while (entryIter.hasNext()) {
//以当前时间戳为标准，删除过期数据
Entry<Long, List<baselink>> timestd = entryIter.next();
Long stdtime = timestd.getKey();
log.info("stdtime:"+stdtime);
//删除过期数据，根据数据保存时间
if(Math.abs(m_timeStamp_sec-stdtime)>keepDataTime)//过期数据删除
{
log.info("remove stdtime:"+stdtime);
entryIter.remove();
}
}
if(localMap.size()==0){
log.info("remove car_key:"+car_key);
entry.remove();
}else{
entry.setValue(localMap);
}
return null;
}});
}
Some logs are the following:
17:20:25.157 INFO [Timer-0] (DataCollection.java:146) -car_key:15701311774
17:20:25.525 INFO [Timer-0] (DataCollection.java:146) -car_key:13370126402
17:20:25.895 INFO [Timer-0] (DataCollection.java:146) -car_key:13466436318
17:20:26.265 INFO [Timer-0] (DataCollection.java:146) -car_key:13331093473
17:20:26.634 INFO [Timer-0] (DataCollection.java:146) -car_key:13311577054
17:20:27.003 INFO [Timer-0] (DataCollection.java:146) -car_key:1000096559
17:20:27.370 INFO [Timer-0] (DataCollection.java:146) -car_key:13381083402
17:20:27.739 INFO [Timer-0] (DataCollection.java:146) -car_key:1000094139
17:20:28.107 INFO [Timer-0] (DataCollection.java:146) -car_key:13651216432
17:20:28.477 INFO [Timer-0] (DataCollection.java:146) -car_key:13331159406
17:20:28.844 INFO [Timer-0] (DataCollection.java:146) -car_key:13488631105
17:20:29.212 INFO [Timer-0] (DataCollection.java:146) -car_key:13488631347
17:20:29.579 INFO [Timer-0] (DataCollection.java:146) -car_key:13370103413
17:20:29.975 INFO [Timer-0] (DataCollection.java:146) -car_key:13311403662
17:20:30.343 INFO [Timer-0] (DataCollection.java:146) -car_key:13301156741
17:20:30.711 INFO [Timer-0] (DataCollection.java:146) -car_key:13311426410
17:20:31.079 INFO [Timer-0] (DataCollection.java:146) -car_key:1000099948
17:20:31.450 INFO [Timer-0] (DataCollection.java:146) -car_key:1000098617
17:20:31.819 INFO [Timer-0] (DataCollection.java:146) -car_key:1000096439
17:20:32.187 INFO [Timer-0] (DataCollection.java:146) -car_key:13301277620
17:20:32.566 INFO [Timer-0] (DataCollection.java:146) -car_key:13311403541
17:20:32.934 INFO [Timer-0] (DataCollection.java:146) -car_key:13439317371
17:20:33.302 INFO [Timer-0] (DataCollection.java:146) -car_key:13466435348
17:20:33.670 INFO [Timer-0] (DataCollection.java:146) -car_key:13331159405
17:20:34.038 INFO [Timer-0] (DataCollection.java:146) -car_key:13381095984
17:20:34.409 INFO [Timer-0] (DataCollection.java:146) -car_key:13488631346
Bob
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: A very slow code of using entryprocessor, why?
Posted by 胡永亮/Bob <hu...@neusoft.com>.
Hi, Val
According to your suggestion, I have a deeper understanding about this.
I will have a try.
Val, Thank you very much.
Bob
From: vkulichenko
Date: 2016-09-24 06:06
To: user@ignite.apache.org
Subject: Re: A very slow code of using entryprocessor, why?
Hi Bob,
Each invoke is a synchronous network operation, so it's speed depends mostly
on network latency. However, it looks like you're processing the whole data
set. In this I would recommend to broadcast a closure using Compute Grid [1]
and within the closure iterate through the local data using
IgniteCache.localEntries method. This way the network traffic will be
minimal and you will parallelize the execution.
[1] https://apacheignite.readme.io/docs/compute-grid
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/A-very-slow-code-of-using-entryprocessor-why-tp7902p7923.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: A very slow code of using entryprocessor, why?
Posted by vkulichenko <va...@gmail.com>.
Hi Bob,
Each invoke is a synchronous network operation, so it's speed depends mostly
on network latency. However, it looks like you're processing the whole data
set. In this I would recommend to broadcast a closure using Compute Grid [1]
and within the closure iterate through the local data using
IgniteCache.localEntries method. This way the network traffic will be
minimal and you will parallelize the execution.
[1] https://apacheignite.readme.io/docs/compute-grid
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/A-very-slow-code-of-using-entryprocessor-why-tp7902p7923.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.