You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "Coleman, JohnSteven (Agoda)" <Jo...@agoda.com> on 2019/04/19 05:19:51 UTC

cache update slow

Hi,

I'm inserting and then updating 250 cache entries using a couple of threads. So a total of 500 cache puts. The initial 250 writes take a little under 500ms. So each initial put is taking approx. 2ms. While that's a little faster than writing to database, I would expect RAM write to be much faster, i.e. sub millisecond.

Any ideas why cache write is not so fast and how to improve it?

John

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

Re: cache update slow

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

I don't think that we publish results.

Please take look at our benchmarking approach at
https://apacheignite.readme.io/docs/perfomance-benchmarking

Regards,
-- 
Ilya Kasnacheev


пн, 29 апр. 2019 г. в 05:57, Coleman, JohnSteven (Agoda) <
JohnSteven.Coleman@agoda.com>:

> Hi,
>
>
>
> Thanks for that observation. I increased cache test to 100,000 entries and
> the average write time is far better at around 23K wps. It seems like a lot
> of latency on the first few hundred writes.
>
> Do you have any benchmarks published?
>
> John
>
>
>
> *From:* Ilya Kasnacheev <il...@gmail.com>
> *Sent:* Friday, April 26, 2019 7:29 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: cache update slow
>
>
>
> Email received from outside the company. If in doubt don't click links nor
> open attachments!
> ------------------------------
>
> Hello!
>
>
>
> I think that comparing steady state benchmarks of multi-million operations
> versus 500 operations is misleading.
>
>
>
> 500 operations is probably not enough to gain full benefits from e.g. JIT.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> пт, 26 апр. 2019 г. в 12:20, Coleman, JohnSteven (Agoda) <
> JohnSteven.Coleman@agoda.com>:
>
> Hi,
>
> Yes, comparing to DMA is apples and oranges comparison, but gives an idea
> of the relative gap in performance.
>
> A better comparison would be to an alike product such as NCache. They
> claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd
> have to compare using my scenario for a valid comparison. But this is more
> like the kind of gap in performance I'd expect vs DMA. But then NCache
> product is also quite different from ignite, so what to say?
>
> regards,
> John
>
> http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html
>
> -----Original Message-----
> From: Maxim.Pudov <pu...@gmail.com>
> Sent: Friday, April 26, 2019 3:17 PM
> To: user@ignite.apache.org
> Subject: RE: cache update slow
>
> Email received from outside the company. If in doubt don't click links nor
> open attachments!
> ________________________________
>
> Glad you met your requirements. I think it is not fair to compare Ignite
> with direct memory access, so I can't really say whether this is a good
> result or not. In your case .net process starts a java process and
> communicates with it via JNI [1]. Also Ignite stores cache data off-heap,
> which requires serialisation [2].
>
> [1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
> [2] https://apacheignite.readme.io/docs/durable-memory
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
> ________________________________
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>
>

RE: cache update slow

Posted by "Coleman, JohnSteven (Agoda)" <Jo...@agoda.com>.
Hi,

Thanks for that observation. I increased cache test to 100,000 entries and the average write time is far better at around 23K wps. It seems like a lot of latency on the first few hundred writes.

Do you have any benchmarks published?

John

From: Ilya Kasnacheev <il...@gmail.com>
Sent: Friday, April 26, 2019 7:29 PM
To: user@ignite.apache.org
Subject: Re: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________
Hello!

I think that comparing steady state benchmarks of multi-million operations versus 500 operations is misleading.

500 operations is probably not enough to gain full benefits from e.g. JIT.

Regards,
--
Ilya Kasnacheev


пт, 26 апр. 2019 г. в 12:20, Coleman, JohnSteven (Agoda) <Jo...@agoda.com>>:
Hi,

Yes, comparing to DMA is apples and oranges comparison, but gives an idea of the relative gap in performance.

A better comparison would be to an alike product such as NCache. They claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd have to compare using my scenario for a valid comparison. But this is more like the kind of gap in performance I'd expect vs DMA. But then NCache product is also quite different from ignite, so what to say?

regards,
John

http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html

-----Original Message-----
From: Maxim.Pudov <pu...@gmail.com>>
Sent: Friday, April 26, 2019 3:17 PM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Glad you met your requirements. I think it is not fair to compare Ignite with direct memory access, so I can't really say whether this is a good result or not. In your case .net process starts a java process and communicates with it via JNI [1]. Also Ignite stores cache data off-heap, which requires serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

Re: cache update slow

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

I think that comparing steady state benchmarks of multi-million operations
versus 500 operations is misleading.

500 operations is probably not enough to gain full benefits from e.g. JIT.

Regards,
-- 
Ilya Kasnacheev


пт, 26 апр. 2019 г. в 12:20, Coleman, JohnSteven (Agoda) <
JohnSteven.Coleman@agoda.com>:

> Hi,
>
> Yes, comparing to DMA is apples and oranges comparison, but gives an idea
> of the relative gap in performance.
>
> A better comparison would be to an alike product such as NCache. They
> claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd
> have to compare using my scenario for a valid comparison. But this is more
> like the kind of gap in performance I'd expect vs DMA. But then NCache
> product is also quite different from ignite, so what to say?
>
> regards,
> John
>
> http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html
>
> -----Original Message-----
> From: Maxim.Pudov <pu...@gmail.com>
> Sent: Friday, April 26, 2019 3:17 PM
> To: user@ignite.apache.org
> Subject: RE: cache update slow
>
> Email received from outside the company. If in doubt don't click links nor
> open attachments!
> ________________________________
>
> Glad you met your requirements. I think it is not fair to compare Ignite
> with direct memory access, so I can't really say whether this is a good
> result or not. In your case .net process starts a java process and
> communicates with it via JNI [1]. Also Ignite stores cache data off-heap,
> which requires serialisation [2].
>
> [1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
> [2] https://apacheignite.readme.io/docs/durable-memory
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
> ________________________________
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>

RE: cache update slow

Posted by "Coleman, JohnSteven (Agoda)" <Jo...@agoda.com>.
Hi,

Yes, comparing to DMA is apples and oranges comparison, but gives an idea of the relative gap in performance.

A better comparison would be to an alike product such as NCache. They claims 20K wps*, thus 20 times faster than my ignite results, but obvs I'd have to compare using my scenario for a valid comparison. But this is more like the kind of gap in performance I'd expect vs DMA. But then NCache product is also quite different from ignite, so what to say?

regards,
John

http://www.alachisoft.com/ncache/ncache-performance-benchmarks.html

-----Original Message-----
From: Maxim.Pudov <pu...@gmail.com>
Sent: Friday, April 26, 2019 3:17 PM
To: user@ignite.apache.org
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Glad you met your requirements. I think it is not fair to compare Ignite with direct memory access, so I can't really say whether this is a good result or not. In your case .net process starts a java process and communicates with it via JNI [1]. Also Ignite stores cache data off-heap, which requires serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

RE: cache update slow

Posted by "Maxim.Pudov" <pu...@gmail.com>.
Glad you met your requirements. I think it is not fair to compare Ignite with
direct memory access, so I can't really say whether this is a good result or
not. In your case .net process starts a java process and communicates with
it via JNI [1]. Also Ignite stores cache data off-heap, which requires
serialisation [2].

[1] https://apacheignite-net.readme.io/docs#section-ignite-and-ignitenet
[2] https://apacheignite.readme.io/docs/durable-memory



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: cache update slow

Posted by "Coleman, JohnSteven (Agoda)" <Jo...@agoda.com>.
Hi,

Thanks for the tip. I implemented with data streamer and observe a significant improvement. However it still takes >1ms per cache entry addition which is fast enough for my requirements, but still >500 times slower than DMA. Is this largely a factor of network overhead (even though I use localhost cache), or the underlying caching mechanics?

Regards,
John

-----Original Message-----
From: Maxim.Pudov <pu...@gmail.com>
Sent: Tuesday, April 23, 2019 8:12 PM
To: user@ignite.apache.org
Subject: RE: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Thanks for sharing your code. I didn't realise you use .NET. Check out how you can benefit from data streamer in .NET [1]. It was designed to populate your cache faster, so it could help you to improve performance.

[1] https://apacheignite-net.readme.io/docs/data-streamers



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

RE: cache update slow

Posted by "Maxim.Pudov" <pu...@gmail.com>.
Thanks for sharing your code. I didn't realise you use .NET. Check out how
you can benefit from data streamer in .NET [1]. It was designed to populate
your cache faster, so it could help you to improve performance.

[1] https://apacheignite-net.readme.io/docs/data-streamers



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: cache update slow

Posted by "Coleman, JohnSteven (Agoda)" <Jo...@agoda.com>.
Hi,

I'm just using the cache created by running a local process, see below. This is just for a POC and works, but I do need to have high performance.

John

          using (var ignite = Ignition.Start())
            {
                QueryEntity[] entities =
                {
                    new QueryEntity(typeof(int), typeof(SimpleTransaction))
                };
                var cfg = new CacheConfiguration
                {
                    OnheapCacheEnabled = true,
                    ReadThrough = false,
                    WriteThrough = false,
                    KeepBinaryInStore = false,
                    QueryEntities = entities,
                    Name = "SimpleTransactions"
                };
                ignite.AddCacheConfiguration(cfg);
                ICache<int, SimpleTransaction> cache = ignite.GetOrCreateCache<int, SimpleTransaction>(cfg);


-----Original Message-----
From: Maxim.Pudov <pu...@gmail.com>
Sent: Friday, April 19, 2019 3:37 PM
To: user@ignite.apache.org
Subject: Re: cache update slow

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Hi, the execution time depends on the configuration of your cache and your cluster. How many node do you have? What is your cache configuration? Have you tried Ignite data streamer [1] instead of cache.put(K,V)?

[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

Re: cache update slow

Posted by "Maxim.Pudov" <pu...@gmail.com>.
Hi, the execution time depends on the configuration of your cache and your
cluster. How many node do you have? What is your cache configuration? Have
you tried Ignite data streamer [1] instead of cache.put(K,V)?

[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/