You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by babu prasad <ba...@gmail.com> on 2016/01/13 05:27:34 UTC
Cache.put latencies high
Hi,
I have configured 2 ignite servers with a heap size of 8G each.
Running with backups=1 and primary_sync mode.
Ignite servers are being used as a write behind cache for my Aurora
database.
I am trying to run a load test with 3 clients talking to the remote cache
in the 2 ignite servers.
All the hosts are in the same availability zone.
My clients do a simple put and I calculate time taken for put on the client
side.
long startTime = System.currentTimeMillis();
cache.put(k, c1);
long elapsedTime = System.currentTimeMillis() - startTime;
System.out.println("Total elapsed timein milliseconds: " + elapsedTime);
Here is the latency from the last few requests:
Total elapsed timein milliseconds: 31
Total elapsed timein milliseconds: 29
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 28
Total elapsed timein milliseconds: 28
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 29
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 28
Total elapsed timein milliseconds: 28
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 28
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 29
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 29
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 28
Total elapsed timein milliseconds: 26
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 27
Total elapsed timein milliseconds: 29
Not sure what is going on here. I am pretty sure I am doing something wrong
here.
Thanks!
Re: Cache.put latencies high
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
If you're interested in the cache loading rather than in pure benchmarking
then I would suggest you using IgniteDataStreamer [1] as one of the fastest
way of a cache preloading.
Next if you're not going to use transaction then you can switch a cache to
ATOMIC mode. It will work faster.
Finally, it's always better to use cache.putAll() rather than cache.put().
However, if you're interested in pure benchmarking then you can refer to the
details of configs and benchmarks of Yardstick (official benchmarking
toolkit used by Ignite).
[1] https://apacheignite.readme.io/docs/data-loading#ignitedatastreamer
[2] https://github.com/apacheignite/yardstick-ignite
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-put-latencies-high-tp2524p2611.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Cache.put latencies high
Posted by Babu Prasad <ba...@gmail.com>.
All nodes are in same az. 1-2 ms ping times.
Sent from my iPhone
> On Jan 17, 2016, at 10:52 PM, Jörn Franke <jo...@gmail.com> wrote:
>
> Are you using AWS ? What is the ping time between the nodes?
>
>> On 18 Jan 2016, at 06:48, Babu Prasad <ba...@gmail.com> wrote:
>>
>> I did simple sequential puts to the cache. The latencies kept spiking intermittently to 30ms or higher.
>> The test took about 30 minutes to load 1M records. I am using the s3 ip finder for discovery.
>> I would expect 1-2 ms at max putting to a cache per request, but 30 ms seems a little higher.
>> Are there best practices I should follow to tune the server and the client for better latency?
>>
>> Sent from my iPhone
>>
>>> On Jan 12, 2016, at 11:59 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>>>
>>> This is usually a problem with one-threaded benchmarks.
>>>
>>> 1. You should add a warm up step, i.e. have your system work for about a minute before starting measuring.
>>>
>>> 2. You should decide whether your application will be single-threaded or multi-threaded. If it is multi-threaded, then your test should also be multi-threaded.
>>>
>>> 3. And finally you should verify that network works well in your environment. Do you have 10G ethernet?
>>>
>>> D.
>>>
>>>> On Tue, Jan 12, 2016 at 8:27 PM, babu prasad <ba...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I have configured 2 ignite servers with a heap size of 8G each.
>>>> Running with backups=1 and primary_sync mode.
>>>>
>>>> Ignite servers are being used as a write behind cache for my Aurora database.
>>>>
>>>> I am trying to run a load test with 3 clients talking to the remote cache in the 2 ignite servers.
>>>> All the hosts are in the same availability zone.
>>>>
>>>> My clients do a simple put and I calculate time taken for put on the client side.
>>>>
>>>> long startTime = System.currentTimeMillis();
>>>> cache.put(k, c1);
>>>> long elapsedTime = System.currentTimeMillis() - startTime;
>>>> System.out.println("Total elapsed timein milliseconds: " + elapsedTime);
>>>>
>>>> Here is the latency from the last few requests:
>>>> Total elapsed timein milliseconds: 31
>>>> Total elapsed timein milliseconds: 29
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 28
>>>> Total elapsed timein milliseconds: 28
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 29
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 28
>>>> Total elapsed timein milliseconds: 28
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 28
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 29
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 29
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 28
>>>> Total elapsed timein milliseconds: 26
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 27
>>>> Total elapsed timein milliseconds: 29
>>>>
>>>> Not sure what is going on here. I am pretty sure I am doing something wrong here.
>>>>
>>>>
>>>> Thanks!
>>>
Re: Cache.put latencies high
Posted by Jörn Franke <jo...@gmail.com>.
Are you using AWS ? What is the ping time between the nodes?
> On 18 Jan 2016, at 06:48, Babu Prasad <ba...@gmail.com> wrote:
>
> I did simple sequential puts to the cache. The latencies kept spiking intermittently to 30ms or higher.
> The test took about 30 minutes to load 1M records. I am using the s3 ip finder for discovery.
> I would expect 1-2 ms at max putting to a cache per request, but 30 ms seems a little higher.
> Are there best practices I should follow to tune the server and the client for better latency?
>
> Sent from my iPhone
>
>> On Jan 12, 2016, at 11:59 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>>
>> This is usually a problem with one-threaded benchmarks.
>>
>> 1. You should add a warm up step, i.e. have your system work for about a minute before starting measuring.
>>
>> 2. You should decide whether your application will be single-threaded or multi-threaded. If it is multi-threaded, then your test should also be multi-threaded.
>>
>> 3. And finally you should verify that network works well in your environment. Do you have 10G ethernet?
>>
>> D.
>>
>>> On Tue, Jan 12, 2016 at 8:27 PM, babu prasad <ba...@gmail.com> wrote:
>>> Hi,
>>>
>>> I have configured 2 ignite servers with a heap size of 8G each.
>>> Running with backups=1 and primary_sync mode.
>>>
>>> Ignite servers are being used as a write behind cache for my Aurora database.
>>>
>>> I am trying to run a load test with 3 clients talking to the remote cache in the 2 ignite servers.
>>> All the hosts are in the same availability zone.
>>>
>>> My clients do a simple put and I calculate time taken for put on the client side.
>>>
>>> long startTime = System.currentTimeMillis();
>>> cache.put(k, c1);
>>> long elapsedTime = System.currentTimeMillis() - startTime;
>>> System.out.println("Total elapsed timein milliseconds: " + elapsedTime);
>>>
>>> Here is the latency from the last few requests:
>>> Total elapsed timein milliseconds: 31
>>> Total elapsed timein milliseconds: 29
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 28
>>> Total elapsed timein milliseconds: 28
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 29
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 28
>>> Total elapsed timein milliseconds: 28
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 28
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 29
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 29
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 28
>>> Total elapsed timein milliseconds: 26
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 27
>>> Total elapsed timein milliseconds: 29
>>>
>>> Not sure what is going on here. I am pretty sure I am doing something wrong here.
>>>
>>>
>>> Thanks!
>>
Re: Cache.put latencies high
Posted by Babu Prasad <ba...@gmail.com>.
I did simple sequential puts to the cache. The latencies kept spiking intermittently to 30ms or higher.
The test took about 30 minutes to load 1M records. I am using the s3 ip finder for discovery.
I would expect 1-2 ms at max putting to a cache per request, but 30 ms seems a little higher.
Are there best practices I should follow to tune the server and the client for better latency?
Sent from my iPhone
> On Jan 12, 2016, at 11:59 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> This is usually a problem with one-threaded benchmarks.
>
> 1. You should add a warm up step, i.e. have your system work for about a minute before starting measuring.
>
> 2. You should decide whether your application will be single-threaded or multi-threaded. If it is multi-threaded, then your test should also be multi-threaded.
>
> 3. And finally you should verify that network works well in your environment. Do you have 10G ethernet?
>
> D.
>
>> On Tue, Jan 12, 2016 at 8:27 PM, babu prasad <ba...@gmail.com> wrote:
>> Hi,
>>
>> I have configured 2 ignite servers with a heap size of 8G each.
>> Running with backups=1 and primary_sync mode.
>>
>> Ignite servers are being used as a write behind cache for my Aurora database.
>>
>> I am trying to run a load test with 3 clients talking to the remote cache in the 2 ignite servers.
>> All the hosts are in the same availability zone.
>>
>> My clients do a simple put and I calculate time taken for put on the client side.
>>
>> long startTime = System.currentTimeMillis();
>> cache.put(k, c1);
>> long elapsedTime = System.currentTimeMillis() - startTime;
>> System.out.println("Total elapsed timein milliseconds: " + elapsedTime);
>>
>> Here is the latency from the last few requests:
>> Total elapsed timein milliseconds: 31
>> Total elapsed timein milliseconds: 29
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 28
>> Total elapsed timein milliseconds: 28
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 29
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 28
>> Total elapsed timein milliseconds: 28
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 28
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 29
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 29
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 28
>> Total elapsed timein milliseconds: 26
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 27
>> Total elapsed timein milliseconds: 29
>>
>> Not sure what is going on here. I am pretty sure I am doing something wrong here.
>>
>>
>> Thanks!
>
Re: Cache.put latencies high
Posted by Dmitriy Setrakyan <ds...@apache.org>.
This is usually a problem with one-threaded benchmarks.
1. You should add a warm up step, i.e. have your system work for about a
minute before starting measuring.
2. You should decide whether your application will be single-threaded or
multi-threaded. If it is multi-threaded, then your test should also be
multi-threaded.
3. And finally you should verify that network works well in your
environment. Do you have 10G ethernet?
D.
On Tue, Jan 12, 2016 at 8:27 PM, babu prasad <ba...@gmail.com> wrote:
> Hi,
>
> I have configured 2 ignite servers with a heap size of 8G each.
> Running with backups=1 and primary_sync mode.
>
> Ignite servers are being used as a write behind cache for my Aurora
> database.
>
> I am trying to run a load test with 3 clients talking to the remote cache
> in the 2 ignite servers.
> All the hosts are in the same availability zone.
>
> My clients do a simple put and I calculate time taken for put on the
> client side.
>
> long startTime = System.currentTimeMillis();
> cache.put(k, c1);
> long elapsedTime = System.currentTimeMillis() - startTime;
> System.out.println("Total elapsed timein milliseconds: " + elapsedTime);
>
> Here is the latency from the last few requests:
> Total elapsed timein milliseconds: 31
> Total elapsed timein milliseconds: 29
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 28
> Total elapsed timein milliseconds: 28
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 29
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 28
> Total elapsed timein milliseconds: 28
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 28
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 29
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 29
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 28
> Total elapsed timein milliseconds: 26
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 27
> Total elapsed timein milliseconds: 29
>
> Not sure what is going on here. I am pretty sure I am doing something
> wrong here.
>
>
> Thanks!
>
>