You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by 胡永亮/Bob <hu...@neusoft.com> on 2016/09/12 09:33:16 UTC
Increase Ignite instances can't increase the speed of compute
Hi, everyone:
I am using Ignite for computing and cache.
I use the same input data and the same compute logic.
When my ignite cluster's node is 2 in 2 machines, the total cost time is 38s.
But, when I increase the Ignite cluster nodes to 3 in 3 machines, the cost time is 32s/51s/41s
4 instances in 4 machines, the cost time is 32s/40s.
The compute speed can't change faster, what may the reason be?
Thanks.
Bob
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: Increase Ignite instances can't increase the speed of compute
Posted by 胡永亮/Bob <hu...@neusoft.com>.
Hi, Val
Thank you very much, I think this is very helpful for me.
I will change my compute code, and have a try.
Bob
From: vkulichenko
Date: 2016-09-23 07:00
To: user@ignite.apache.org
Subject: Re: Re: Increase Ignite instances can't increase the speed of compute
Hi Bob,
Most likely these are distributed cache operations and since they are all
synchronous, you wait for each of them one by one. You can try to batch your
updates to improve performance or use data streamer [1]. It would be even
better if you used affinity collocation [2] and made sure that all the
updates are local and network is not used during the computation.
[1] https://apacheignite.readme.io/docs/data-streamers
[2] https://apacheignite.readme.io/docs/affinity-collocation
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Increase-Ignite-instances-can-t-increase-the-speed-of-compute-tp7670p7891.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: Increase Ignite instances can't increase the speed of
compute
Posted by vkulichenko <va...@gmail.com>.
Hi Bob,
Most likely these are distributed cache operations and since they are all
synchronous, you wait for each of them one by one. You can try to batch your
updates to improve performance or use data streamer [1]. It would be even
better if you used affinity collocation [2] and made sure that all the
updates are local and network is not used during the computation.
[1] https://apacheignite.readme.io/docs/data-streamers
[2] https://apacheignite.readme.io/docs/affinity-collocation
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Increase-Ignite-instances-can-t-increase-the-speed-of-compute-tp7670p7891.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Re: Increase Ignite instances can't increase the speed of compute
Posted by 胡永亮/Bob <hu...@neusoft.com>.
hi,
I found one point which cost too much time in one compute job.
For example, one job cost 35s total, the code of writing result data to ignite cache cost 29s.
My ignite cache which is used to save the result data is:
Cache 'MapMatchingData':
+==========================================================================================================+
| Name | Value |
+==========================================================================================================+
| Mode | PARTITIONED |
| Atomicity Mode | ATOMIC |
| Atomic Write Ordering Mode | PRIMARY |
| Statistic Enabled | off |
| Management Enabled | off |
| Time To Live Eager Flag | true |
| Write Synchronization Mode | FULL_ASYNC |
| Swap Enabled | off |
| Invalidate | off |
| Start Size | 1500000 |
| Affinity Function | o.a.i.cache.affinity.rendezvous.RendezvousAffinityFunction |
| Affinity Backups | 0 |
| Affinity Partitions | 1024 |
| Affinity Exclude Neighbors | false |
| Affinity Mapper | o.a.i.i.processors.cache.CacheDefaultBinaryAffinityKeyMapper |
| Rebalance Mode | ASYNC |
| Rebalance Batch Size | 524288 |
| Rebalance Thread Pool size | 2 |
| Rebalance Timeout | 10000 |
| Rebalance Delay | 0 |
| Time Between Rebalance Messages | 0 |
| Eviction Policy Enabled | off |
| Eviction Policy | <n/a> |
| Eviction Policy Max Size | <n/a> |
| Eviction Filter | <n/a> |
| Eviction Key Buffer Size | 1024 |
| Eviction Synchronized | off |
| Eviction Overflow Ratio | 10.0 |
| Synchronous Eviction Timeout | 10000 |
| Synchronous Eviction Concurrency Level | 4 |
| Near Cache Enabled | off |
| Near Start Size | 0 |
| Near Eviction Policy | <n/a> |
| Near Eviction Policy Max Size | <n/a> |
| Default Lock Timeout | 0 |
| Metadata type count | 0 |
| Cache Interceptor | <n/a> |
| Store Enabled | off |
| Store Class | <n/a> |
| Store Factory Class | |
| Store Keep Binary | false |
| Store Read Through | off |
| Store Write Through | off |
| Write-Behind Enabled | off |
| Write-Behind Flush Size | 10240 |
| Write-Behind Frequency | 5000 |
| Write-Behind Flush Threads Count | 1 |
| Write-Behind Batch Size | 512 |
| Concurrent Asynchronous Operations Number | 500000 |
| Memory Mode | ONHEAP_TIERED |
| Off-Heap Size | <n/a> |
| Loader Factory Class Name | <n/a> |
| Writer Factory Class Name | <n/a> |
| Expiry Policy Factory Class Name | javax.cache.configuration.FactoryBuilder$SingletonFactory |
| Query Execution Time Threshold | 3000 |
| Query Schema Name | |
| Query Escaped Names | off |
| Query Onheap Cache Size | 10240 |
| Query SQL functions | <n/a> |
| Query Indexed Types | <n/a> |
+----------------------------------------------------------------------------------------------------------+
My code which write result data to ignite cache is the following, this code cost too much time than computing code.
//Batch to write to Ignite
Iterator<Entry<String, Map<Long, List<baselink>>>> it = local_writeCache.entrySet().iterator();
int writeNum = 0;
while(it.hasNext()){
Entry<String, Map<Long, List<baselink>>> entry = it.next();
String carKey = entry.getKey();
final Map<Long, List<baselink>> value = entry.getValue();
writeNum += value.size();
long startTime = System.currentTimeMillis();
if(!mapMatchingData.containsKey(carKey)){
mapMatchingData.put(carKey, value);
}else{
mapMatchingData.invoke(carKey, new EntryProcessor<String, Map<Long, List<baselink>>, Void>() {
@Override
public Void process(MutableEntry<String, Map<Long, List<baselink>>> entry, Object... args) {
Map<Long, List<baselink>> map = entry.getValue();
map.putAll(value);
entry.setValue(map);
return null;
}
});
}
I analyzed the log, I found not every writing task of compute job cost so much time, someone cost only 1s to write result to ignite cache: MapMatchingData.
So, I think writing result cost very much time, due to the lock of MapMatchingData for concurrent writing.
But, I want to know how to change this status, to shorten the cost time of writing ignite cache.
Thanks.
Bob
From: Vladislav Pyatkov
Date: 2016-09-14 20:18
To: user@ignite.apache.org
Subject: Re: Re: Increase Ignite instances can't increase the speed of compute
Hello,
I have not saw shortcomings in your code, but me was not clear which size of data moved between nodes or are data collect?
Can you create working example?
So I was able to reproduce the problem.
On Tue, Sep 13, 2016 at 12:06 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
Only 118 jobs for this test.
Bob
From: Taras Ledkov
Date: 2016-09-13 14:52
To: user@ignite.apache.org
Subject: Re: Re: Increase Ignite instances can't increase the speed of compute
Hi,
How many MatchingJobs do you submit?
On Tue, Sep 13, 2016 at 12:29 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
Hello, Vladislav
The following is some code.
...
IgniteCompute compute = ignite.compute();//.withAsync();
compute.run(new MatchingJob(m_mapReadyDataPara));
}
private static class MatchingJob implements IgniteRunnable{
private Map<String, Map<Long, std>> m_mapReadyData;
private IgniteCache<String, Map<Long, List<baselink>>> mapMatchingData; //This is a cache in Ignite cluster.
...
public void run() {
...
Iterator<Entry<String, Map<Long, std>>> entryKeyIterator1 = m_mapReadyData.entrySet().iterator(); //m_mapReadyData is the input data, its size is 5000 for every job now.
Map<String, Map<Long, List<baselink>>> local_writeCache = new HashMap<String, Map<Long, List<baselink>>>();
...
//Then the job read detail data from m_mapReadyData, and compute.
while (entryKeyIterator1.hasNext()) {
Entry<String, Map<Long, std>> eKey1 = entryKeyIterator1.next();
String carKey = eKey1.getKey();
Map<Long, std> value1 = eKey1.getValue();
//local node cache
Map<Long, List<baselink>> existMapbaselink = local_mapMatchingData.get(carKey);
if(existMapbaselink == null){
existMapbaselink = mapMatchingData.get(carKey); //Read data to compute with it from Ignite cache. This data's size is 154M for PARTITIONED mode.
if(existMapbaselink != null)
local_mapMatchingData.putIfAbsent(carKey, existMapbaselink);
}
//some compute logic code
mapbaselink = local_writeCache.get(carKey);
if(mapbaselink == null){
mapbaselink = new TreeMap<Long, List<baselink>>();
}
mapbaselink.put(stdtime, ListBaseLink);
local_writeCache.put(carKey, mapbaselink);
}
//batch to write data into Ignite.
Iterator<Entry<String, Map<Long, List<baselink>>>> it = local_writeCache.entrySet().iterator();
while(it.hasNext()){
Entry<String, Map<Long, List<baselink>>> entry = it.next();
String carKey = entry.getKey();
final Map<Long, List<baselink>> value = entry.getValue();
if(!mapMatchingData.containsKey(carKey)){
mapMatchingData.put(carKey, value);
}else{
mapMatchingData.invoke(carKey, new EntryProcessor<String, Map<Long, List<baselink>>, Void>() {
@Override
public Void process(MutableEntry<String, Map<Long, List<baselink>>> entry, Object... args) {
Map<Long, List<baselink>> map = entry.getValue();
map.putAll(value);
entry.setValue(map);
return null;
}
});
}
}
bob
From: Vladislav Pyatkov
Date: 2016-09-12 18:37
To: user@ignite.apache.org
Subject: Re: Increase Ignite instances can't increase the speed of compute
Hello,
I don't understand, what do you try to measure, without code.
Size of calculation task, size of data moved into network have importance.
Could you please provide code example?
On Mon, Sep 12, 2016 at 12:33 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
Hi, everyone:
I am using Ignite for computing and cache.
I use the same input data and the same compute logic.
When my ignite cluster's node is 2 in 2 machines, the total cost time is 38s.
But, when I increase the Ignite cluster nodes to 3 in 3 machines, the cost time is 32s/51s/41s
4 instances in 4 machines, the cost time is 32s/40s.
The compute speed can't change faster, what may the reason be?
Thanks.
Bob
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
--
Vladislav Pyatkov
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
--
Vladislav Pyatkov
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: Increase Ignite instances can't increase the speed of compute
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hello,
I have not saw shortcomings in your code, but me was not clear which size
of data moved between nodes or are data collect?
Can you create working example?
So I was able to reproduce the problem.
On Tue, Sep 13, 2016 at 12:06 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
>
> Only 118 jobs for this test.
>
> ------------------------------
> Bob
>
>
> *From:* Taras Ledkov <tl...@gridgain.com>
> *Date:* 2016-09-13 14:52
> *To:* user@ignite.apache.org
> *Subject:* Re: Re: Increase Ignite instances can't increase the speed of
> compute
> Hi,
>
> How many MatchingJobs do you submit?
>
>
> On Tue, Sep 13, 2016 at 12:29 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
>
>> Hello, Vladislav <vl...@gmail.com>
>>
>> The following is some code.
>>
>> ...
>> IgniteCompute compute = ignite.compute();//.withAsync();
>>
>> compute.run(new MatchingJob(m_mapReadyDataPara));
>> }
>>
>> private static class MatchingJob implements IgniteRunnable{
>>
>> private Map<String, Map<Long, std>> m_mapReadyData;
>> private IgniteCache<String, Map<Long, List<baselink>>> mapMatchingData;
>> *//This is a cache in Ignite cluster.*
>> ...
>>
>> public void run() {
>> ...
>> Iterator<Entry<String, Map<Long, std>>> entryKeyIterator1
>> = m_mapReadyData.entrySet().iterator(); *//**m_mapReadyData is the
>> input data, its size is 5000 for every job now.*
>> Map<String, Map<Long, List<baselink>>> local_writeCache =
>> new HashMap<String, Map<Long, List<baselink>>>();
>> ...
>> *//Then the job read detail data from m_mapReadyData, and compute.*
>> while (entryKeyIterator1.hasNext()) {
>> Entry<String, Map<Long, std>> eKey1 = entryK
>> eyIterator1.next();
>> String carKey = eKey1.getKey();
>> Map<Long, std> value1 = eKey1.getValue();
>>
>> * //local node cache*
>> Map<Long, List<baselink>> existMapbaselink =
>> local_mapMatchingData.get(carKey);
>> if(existMapbaselink == null){
>> existMapbaselink = mapMatchingData.get(carKey); * //Read
>> data to compute with it from Ignite cache. This data's size is 154M for *
>> PARTITIONED *mode.*
>> if(existMapbaselink != null)
>> local_mapMatchingData.putIfAbsent(ca
>> rKey, existMapbaselink);
>> }
>>
>> * //some compute logic code*
>>
>> mapbaselink = local_writeCache.get(carKey);
>> if(mapbaselink == null){
>> mapbaselink = new TreeMap<Long, List<baselink>>();
>> }
>> mapbaselink.put(stdtime, ListBaseLink);
>> local_writeCache.put(carKey, mapbaselink);
>> }
>>
>> *//batch to write data into Ignite.*
>> Iterator<Entry<String, Map<Long, List<baselink>>
>> >> it = local_writeCache.entrySet().iterator();
>> while(it.hasNext()){
>> Entry<String, Map<Long, List<baselink>>> ent
>> ry = it.next();
>> String carKey = entry.getKey();
>> final Map<Long, List<baselink>> value = entry.getValue();
>>
>> if(!mapMatchingData.containsKey(carKey)){
>> mapMatchingData.put(carKey, value);
>> }else{
>> mapMatchingData.invoke(carKey, new
>> EntryProcessor<String, Map<Long, List<baselink>>, Void>() {
>> @Override
>> public Void process(MutableEntry<Strin
>> g, Map<Long, List<baselink>>> entry, Object... args) {
>> Map<Long, List<baselink>> map = en
>> try.getValue();
>> map.putAll(value);
>> entry.setValue(map);
>> return null;
>> }
>> });
>> }
>>
>> }
>>
>>
>> ------------------------------
>> bob
>>
>>
>> *From:* Vladislav Pyatkov <vl...@gmail.com>
>> *Date:* 2016-09-12 18:37
>> *To:* user@ignite.apache.org
>> *Subject:* Re: Increase Ignite instances can't increase the speed of
>> compute
>> Hello,
>>
>> I don't understand, what do you try to measure, without code.
>> Size of calculation task, size of data moved into network have importance.
>>
>> Could you please provide code example?
>>
>> On Mon, Sep 12, 2016 at 12:33 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
>>
>>> Hi, everyone:
>>>
>>> I am using Ignite for computing and cache.
>>>
>>> I use the same input data and the same compute logic.
>>> When my ignite cluster's node is 2 in 2 machines, the total cost
>>> time is 38s.
>>>
>>> But, when I increase the Ignite cluster nodes to 3 in 3 machines,
>>> the cost time is 32s/51s/41s
>>> 4 instances in 4 machines, the cost time is 32s/40s.
>>>
>>> The compute speed can't change faster, what may the reason be?
>>>
>>> Thanks.
>>>
>>> Bob
>>>
>>> ------------------------------------------------------------
>>> ---------------------------------------
>>> Confidentiality Notice: The information contained in this e-mail and any
>>> accompanying attachment(s)
>>> is intended only for the use of the intended recipient and may be
>>> confidential and/or privileged of
>>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
>>> reader of this communication is
>>> not the intended recipient, unauthorized use, forwarding, printing,
>>> storing, disclosure or copying
>>> is strictly prohibited, and may be unlawful.If you have received this
>>> communication in error,please
>>> immediately notify the sender by return e-mail, and delete the original
>>> message and all copies from
>>> your system. Thank you.
>>> ------------------------------------------------------------
>>> ---------------------------------------
>>>
>>
>>
>>
>> --
>> Vladislav Pyatkov
>>
>>
>> ------------------------------------------------------------
>> ---------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any
>> accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be
>> confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
>> reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing,
>> storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this
>> communication in error,please
>> immediately notify the sender by return e-mail, and delete the original
>> message and all copies from
>> your system. Thank you.
>> ------------------------------------------------------------
>> ---------------------------------------
>>
>
>
> ------------------------------------------------------------
> ---------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
> ------------------------------------------------------------
> ---------------------------------------
>
--
Vladislav Pyatkov
Re: Re: Increase Ignite instances can't increase the speed of compute
Posted by 胡永亮/Bob <hu...@neusoft.com>.
Only 118 jobs for this test.
Bob
From: Taras Ledkov
Date: 2016-09-13 14:52
To: user@ignite.apache.org
Subject: Re: Re: Increase Ignite instances can't increase the speed of compute
Hi,
How many MatchingJobs do you submit?
On Tue, Sep 13, 2016 at 12:29 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
Hello, Vladislav
The following is some code.
...
IgniteCompute compute = ignite.compute();//.withAsync();
compute.run(new MatchingJob(m_mapReadyDataPara));
}
private static class MatchingJob implements IgniteRunnable{
private Map<String, Map<Long, std>> m_mapReadyData;
private IgniteCache<String, Map<Long, List<baselink>>> mapMatchingData; //This is a cache in Ignite cluster.
...
public void run() {
...
Iterator<Entry<String, Map<Long, std>>> entryKeyIterator1 = m_mapReadyData.entrySet().iterator(); //m_mapReadyData is the input data, its size is 5000 for every job now.
Map<String, Map<Long, List<baselink>>> local_writeCache = new HashMap<String, Map<Long, List<baselink>>>();
...
//Then the job read detail data from m_mapReadyData, and compute.
while (entryKeyIterator1.hasNext()) {
Entry<String, Map<Long, std>> eKey1 = entryKeyIterator1.next();
String carKey = eKey1.getKey();
Map<Long, std> value1 = eKey1.getValue();
//local node cache
Map<Long, List<baselink>> existMapbaselink = local_mapMatchingData.get(carKey);
if(existMapbaselink == null){
existMapbaselink = mapMatchingData.get(carKey); //Read data to compute with it from Ignite cache. This data's size is 154M for PARTITIONED mode.
if(existMapbaselink != null)
local_mapMatchingData.putIfAbsent(carKey, existMapbaselink);
}
//some compute logic code
mapbaselink = local_writeCache.get(carKey);
if(mapbaselink == null){
mapbaselink = new TreeMap<Long, List<baselink>>();
}
mapbaselink.put(stdtime, ListBaseLink);
local_writeCache.put(carKey, mapbaselink);
}
//batch to write data into Ignite.
Iterator<Entry<String, Map<Long, List<baselink>>>> it = local_writeCache.entrySet().iterator();
while(it.hasNext()){
Entry<String, Map<Long, List<baselink>>> entry = it.next();
String carKey = entry.getKey();
final Map<Long, List<baselink>> value = entry.getValue();
if(!mapMatchingData.containsKey(carKey)){
mapMatchingData.put(carKey, value);
}else{
mapMatchingData.invoke(carKey, new EntryProcessor<String, Map<Long, List<baselink>>, Void>() {
@Override
public Void process(MutableEntry<String, Map<Long, List<baselink>>> entry, Object... args) {
Map<Long, List<baselink>> map = entry.getValue();
map.putAll(value);
entry.setValue(map);
return null;
}
});
}
}
bob
From: Vladislav Pyatkov
Date: 2016-09-12 18:37
To: user@ignite.apache.org
Subject: Re: Increase Ignite instances can't increase the speed of compute
Hello,
I don't understand, what do you try to measure, without code.
Size of calculation task, size of data moved into network have importance.
Could you please provide code example?
On Mon, Sep 12, 2016 at 12:33 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
Hi, everyone:
I am using Ignite for computing and cache.
I use the same input data and the same compute logic.
When my ignite cluster's node is 2 in 2 machines, the total cost time is 38s.
But, when I increase the Ignite cluster nodes to 3 in 3 machines, the cost time is 32s/51s/41s
4 instances in 4 machines, the cost time is 32s/40s.
The compute speed can't change faster, what may the reason be?
Thanks.
Bob
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
--
Vladislav Pyatkov
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: Increase Ignite instances can't increase the speed of compute
Posted by Taras Ledkov <tl...@gridgain.com>.
Hi,
How many MatchingJobs do you submit?
On Tue, Sep 13, 2016 at 12:29 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
> Hello, Vladislav <vl...@gmail.com>
>
> The following is some code.
>
> ...
> IgniteCompute compute = ignite.compute();//.withAsync();
>
> compute.run(new MatchingJob(m_mapReadyDataPara));
> }
>
> private static class MatchingJob implements IgniteRunnable{
>
> private Map<String, Map<Long, std>> m_mapReadyData;
> private IgniteCache<String, Map<Long, List<baselink>>> mapMatchingData;
> *//This is a cache in Ignite cluster.*
> ...
>
> public void run() {
> ...
> Iterator<Entry<String, Map<Long, std>>> entryKeyIterator1 = m_
> mapReadyData.entrySet().iterator(); *//**m_mapReadyData is the input
> data, its size is 5000 for every job now.*
> Map<String, Map<Long, List<baselink>>> local_writeCache =
> new HashMap<String, Map<Long, List<baselink>>>();
> ...
> *//Then the job read detail data from m_mapReadyData, and compute.*
> while (entryKeyIterator1.hasNext()) {
> Entry<String, Map<Long, std>> eKey1 =
> entryKeyIterator1.next();
> String carKey = eKey1.getKey();
> Map<Long, std> value1 = eKey1.getValue();
>
> * //local node cache*
> Map<Long, List<baselink>> existMapbaselink = local_
> mapMatchingData.get(carKey);
> if(existMapbaselink == null){
> existMapbaselink = mapMatchingData.get(carKey); * //Read
> data to compute with it from Ignite cache. This data's size is 154M for *
> PARTITIONED *mode.*
> if(existMapbaselink != null)
> local_mapMatchingData.putIfAbsent(
> carKey, existMapbaselink);
> }
>
> * //some compute logic code*
>
> mapbaselink = local_writeCache.get(carKey);
> if(mapbaselink == null){
> mapbaselink = new TreeMap<Long, List<baselink>>();
> }
> mapbaselink.put(stdtime, ListBaseLink);
> local_writeCache.put(carKey, mapbaselink);
> }
>
> *//batch to write data into Ignite.*
> Iterator<Entry<String, Map<Long, List<baselink>>>> it = local_
> writeCache.entrySet().iterator();
> while(it.hasNext()){
> Entry<String, Map<Long, List<baselink>>>
> entry = it.next();
> String carKey = entry.getKey();
> final Map<Long, List<baselink>> value = entry.getValue();
>
> if(!mapMatchingData.containsKey(carKey)){
> mapMatchingData.put(carKey, value);
> }else{
> mapMatchingData.invoke(carKey,
> new EntryProcessor<String, Map<Long, List<baselink>>, Void>() {
> @Override
> public Void process(MutableEntry<
> String, Map<Long, List<baselink>>> entry, Object... args) {
> Map<Long, List<baselink>> map =
> entry.getValue();
> map.putAll(value);
> entry.setValue(map);
> return null;
> }
> });
> }
>
> }
>
>
> ------------------------------
> bob
>
>
> *From:* Vladislav Pyatkov <vl...@gmail.com>
> *Date:* 2016-09-12 18:37
> *To:* user@ignite.apache.org
> *Subject:* Re: Increase Ignite instances can't increase the speed of
> compute
> Hello,
>
> I don't understand, what do you try to measure, without code.
> Size of calculation task, size of data moved into network have importance.
>
> Could you please provide code example?
>
> On Mon, Sep 12, 2016 at 12:33 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
>
>> Hi, everyone:
>>
>> I am using Ignite for computing and cache.
>>
>> I use the same input data and the same compute logic.
>> When my ignite cluster's node is 2 in 2 machines, the total cost time
>> is 38s.
>>
>> But, when I increase the Ignite cluster nodes to 3 in 3 machines, the
>> cost time is 32s/51s/41s
>> 4 instances in 4 machines, the cost time is 32s/40s.
>>
>> The compute speed can't change faster, what may the reason be?
>>
>> Thanks.
>>
>> Bob
>>
>> ------------------------------------------------------------
>> ---------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any
>> accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be
>> confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
>> reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing,
>> storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this
>> communication in error,please
>> immediately notify the sender by return e-mail, and delete the original
>> message and all copies from
>> your system. Thank you.
>> ------------------------------------------------------------
>> ---------------------------------------
>>
>
>
>
> --
> Vladislav Pyatkov
>
>
> ------------------------------------------------------------
> ---------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
> ------------------------------------------------------------
> ---------------------------------------
>
Re: Re: Increase Ignite instances can't increase the speed of compute
Posted by 胡永亮/Bob <hu...@neusoft.com>.
Hello, Vladislav
The following is some code.
...
IgniteCompute compute = ignite.compute();//.withAsync();
compute.run(new MatchingJob(m_mapReadyDataPara));
}
private static class MatchingJob implements IgniteRunnable{
private Map<String, Map<Long, std>> m_mapReadyData;
private IgniteCache<String, Map<Long, List<baselink>>> mapMatchingData; //This is a cache in Ignite cluster.
...
public void run() {
...
Iterator<Entry<String, Map<Long, std>>> entryKeyIterator1 = m_mapReadyData.entrySet().iterator(); //m_mapReadyData is the input data, its size is 5000 for every job now.
Map<String, Map<Long, List<baselink>>> local_writeCache = new HashMap<String, Map<Long, List<baselink>>>();
...
//Then the job read detail data from m_mapReadyData, and compute.
while (entryKeyIterator1.hasNext()) {
Entry<String, Map<Long, std>> eKey1 = entryKeyIterator1.next();
String carKey = eKey1.getKey();
Map<Long, std> value1 = eKey1.getValue();
//local node cache
Map<Long, List<baselink>> existMapbaselink = local_mapMatchingData.get(carKey);
if(existMapbaselink == null){
existMapbaselink = mapMatchingData.get(carKey); //Read data to compute with it from Ignite cache. This data's size is 154M for PARTITIONED mode.
if(existMapbaselink != null)
local_mapMatchingData.putIfAbsent(carKey, existMapbaselink);
}
//some compute logic code
mapbaselink = local_writeCache.get(carKey);
if(mapbaselink == null){
mapbaselink = new TreeMap<Long, List<baselink>>();
}
mapbaselink.put(stdtime, ListBaseLink);
local_writeCache.put(carKey, mapbaselink);
}
//batch to write data into Ignite.
Iterator<Entry<String, Map<Long, List<baselink>>>> it = local_writeCache.entrySet().iterator();
while(it.hasNext()){
Entry<String, Map<Long, List<baselink>>> entry = it.next();
String carKey = entry.getKey();
final Map<Long, List<baselink>> value = entry.getValue();
if(!mapMatchingData.containsKey(carKey)){
mapMatchingData.put(carKey, value);
}else{
mapMatchingData.invoke(carKey, new EntryProcessor<String, Map<Long, List<baselink>>, Void>() {
@Override
public Void process(MutableEntry<String, Map<Long, List<baselink>>> entry, Object... args) {
Map<Long, List<baselink>> map = entry.getValue();
map.putAll(value);
entry.setValue(map);
return null;
}
});
}
}
bob
From: Vladislav Pyatkov
Date: 2016-09-12 18:37
To: user@ignite.apache.org
Subject: Re: Increase Ignite instances can't increase the speed of compute
Hello,
I don't understand, what do you try to measure, without code.
Size of calculation task, size of data moved into network have importance.
Could you please provide code example?
On Mon, Sep 12, 2016 at 12:33 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
Hi, everyone:
I am using Ignite for computing and cache.
I use the same input data and the same compute logic.
When my ignite cluster's node is 2 in 2 machines, the total cost time is 38s.
But, when I increase the Ignite cluster nodes to 3 in 3 machines, the cost time is 32s/51s/41s
4 instances in 4 machines, the cost time is 32s/40s.
The compute speed can't change faster, what may the reason be?
Thanks.
Bob
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
--
Vladislav Pyatkov
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Increase Ignite instances can't increase the speed of compute
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hello,
I don't understand, what do you try to measure, without code.
Size of calculation task, size of data moved into network have importance.
Could you please provide code example?
On Mon, Sep 12, 2016 at 12:33 PM, 胡永亮/Bob <hu...@neusoft.com> wrote:
> Hi, everyone:
>
> I am using Ignite for computing and cache.
>
> I use the same input data and the same compute logic.
> When my ignite cluster's node is 2 in 2 machines, the total cost time
> is 38s.
>
> But, when I increase the Ignite cluster nodes to 3 in 3 machines, the
> cost time is 32s/51s/41s
> 4 instances in 4 machines, the cost time is 32s/40s.
>
> The compute speed can't change faster, what may the reason be?
>
> Thanks.
>
> Bob
>
> ------------------------------------------------------------
> ---------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
> ------------------------------------------------------------
> ---------------------------------------
>
--
Vladislav Pyatkov