You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by jjimeno <jj...@omp.com> on 2021/01/27 11:07:01 UTC
Performance of Ignite as key-value datastore. C++ Thin Client.
Hi everyone,
For our project, we have the next requirements:
- One single cache <Guid, BinaryBlob>
- Ability to lock a list of cache entries.
- Large transactions. A typical one is to commit 1.2 million keys (a single
PutAll call) with a total size of around 600MB.
- Persistence
In our proof of concept, we've got everything implemented and running:
- One server node 2.9.1. Native persistence is enabled for default data
region.
- One client application using one Ignite C++ Thin Client to connect to the
server node.
- Both, server and client, are in the same machine by now.
With this scenario, we're currently evaluating Ignite vs RocksDB. We would
really like to choose Ignite because of its scalability, but we are facing a
problem related to its performance:
In Ignite, one single transaction commit of 1.2 million keys and 600MB takes
around 70 seconds to complete, while RocksDB takes no more than 12 seconds.
Moreover, if a second local node is added to the cluster, the application is
not even able of completing the transaction (it stops after 10 minutes)
Default data region's page size has been modified up to 16KB. Persistence
has been enabled.
Cache is PARTITIONED with TRANSACTIONAL atomicity mode.
Because of the requirement about locking keys, performed transaction is
PESSIMISTIC + READ_COMMITTED.
The rest of the configuration values are the default ones (No backup,
PRIMARY_SYNC, no OnHeapCache, etc)
So, my questions are:
- Taking the requirements into account, is Ignite a good option?
- It's those time values that one might expect?
- If not, any advice to improve them?
Configuration files for both server nodes have been attached. Thanks
everyone in advance for your help and time,
first-node.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t3059/first-node.xml>
second-node.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t3059/second-node.xml>
Josemari
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin
Client.
Posted by jjimeno <jj...@omp.com>.
Hello,
I understand these deadlocks when there are more than one PuAll at the same
time, but in this case there is only one, and it’s always sorted.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Pavel Tupitsyn <pt...@apache.org>.
> how a deadlock is possible if keys are no repeated
Deadlock is possible when two putAll operations are executed
at the same time on the same keys, but key order is different in the
provided map.
> LinkedHashMap is used for putAll operation
> but I'm using a c++ std::map
Unfortunately, this warning is misleading when a non-Java client is used.
Make sure that the order of entries is always the same when doing a putAll
operation.
If your current test does not perform multiple operations in parallel, this
is not an issue.
On Wed, Jan 27, 2021 at 6:40 PM jjimeno <jj...@omp.com> wrote:
> Hello! Ok... thanks Ilya!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin
Client.
Posted by jjimeno <jj...@omp.com>.
Hello! Ok... thanks Ilya!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It seems to me that you should never use messageQueueLimit - if you ever
saturate your network it will break down your cluster.
Regards,
--
Ilya Kasnacheev
ср, 27 янв. 2021 г. в 15:55, Zhenya Stanilovsky <ar...@mail.ru>:
> 1. Yes i know about this message, don`t pay attention (just remove it). Ilya is this param is safe to use ?
> 2. plz rerun your nodes with -DIGNITE_QUIET=false jvm param (there are would be more informative logs).
>
> 3 You have long running tx in your logs (IGNITE_QUIET will help to detect
> why it hangs) you can configure default timeout [1]
> 4 If tx will hang one more — plz attach new logs and up this thread.
>
> [1]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/TransactionConfiguration.html#setDefaultTxTimeout-long-
>
>
>
>
>
>
>
> Hi Zhenya,
>
> Thanks for your quick response
>
> 1. If it is not set, the following message appears on node's startup:
> [13:22:33] Message queue limit is set to 0 which may lead to potential
> OOMEs
> when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
> message queues growth on sender and receiver sides.
> 2. No improvement.
> 3. I've got the same question :) Here are the cluster logs until it
> crashes: ignite-9f92ab96.log
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t3059/ignite-9f92ab96.log
> >
> 4. Yes, I'm aware about that since I reported it... but there is only one
> transaction in the test
> 5. Yes, 4Gb is large enough. There is only one single transaction of 600MB
> 6. Yes, in fact, that's why I modified the page size
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>
>
>
>
Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin
Client.
Posted by jjimeno <jj...@omp.com>.
2. I've found this:
[16:02:23,836][WARNING][client-connector-#139][GridDhtColocatedCache] <vds>
Unordered map java.util.LinkedHashMap is used for putAll operation on cache
vds. This can lead to a distribut
ed deadlock. Switch to a sorted map like TreeMap instead.
... but I'm using a c++ std::map. It's hard to think that it is being
mapped to a unordered map in Java and how a deadlock is possible if keys are
no repeated. I've modified my code to use Put instead of PutAll and the
transaction finishes... in 312sec
There is a way of fixing this?
3. I think the fix to the problem is not increasing a timeout but in
reducing the commit time. More than 300 sec for a transaction of 600MB is
not what I would expect from Ignite
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Zhenya Stanilovsky <ar...@mail.ru>.
1. Yes i know about this message, don`t pay attention (just remove it). Ilya is this param is safe to use ?
2. plz rerun your nodes with -DIGNITE_QUIET=false jvm param (there are would be more informative logs).
3 You have long running tx in your logs (IGNITE_QUIET will help to detect why it hangs) you can configure default timeout [1]
4 If tx will hang one more — plz attach new logs and up this thread.
[1] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/TransactionConfiguration.html#setDefaultTxTimeout-long-
>
>>
>>>
>>>>Hi Zhenya,
>>>>
>>>>Thanks for your quick response
>>>>
>>>>1. If it is not set, the following message appears on node's startup:
>>>>[13:22:33] Message queue limit is set to 0 which may lead to potential OOMEs
>>>>when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
>>>>message queues growth on sender and receiver sides.
>>>>2. No improvement.
>>>>3. I've got the same question :) Here are the cluster logs until it
>>>>crashes: ignite-9f92ab96.log
>>>>< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/ignite-9f92ab96.log >
>>>>4. Yes, I'm aware about that since I reported it... but there is only one
>>>>transaction in the test
>>>>5. Yes, 4Gb is large enough. There is only one single transaction of 600MB
>>>>6. Yes, in fact, that's why I modified the page size
>>>>
>>>>
>>>>
>>>>--
>>>>Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>>
>>>
>>>
Re: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by jjimeno <jj...@omp.com>.
Hi Zhenya,
Thanks for your quick response
1. If it is not set, the following message appears on node's startup:
[13:22:33] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
2. No improvement.
3. I've got the same question :) Here are the cluster logs until it
crashes: ignite-9f92ab96.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t3059/ignite-9f92ab96.log>
4. Yes, I'm aware about that since I reported it... but there is only one
transaction in the test
5. Yes, 4Gb is large enough. There is only one single transaction of 600MB
6. Yes, in fact, that's why I modified the page size
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Zhenya Stanilovsky <ar...@mail.ru>.
hi jjimeno.
* I doubt about « messageQueueLimit » correctness, plz remove it and try once more.
* 16k per page is questionable, i suggest to try with default.
* why there is no progress with 2 node? can you append somehow logs from 2 nodes with transactions degradation?
* Take into account [1]
* You can collide with page replacements, is 4 Gb are ok for all your data ?
* Are you reading performance tricks ? [2]
[1] https://issues.apache.org/jira/browse/IGNITE-13997
[2] https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/general-perf-tips
>Hi everyone,
>
>For our project, we have the next requirements:
>
>- One single cache <Guid, BinaryBlob>
>- Ability to lock a list of cache entries.
>- Large transactions. A typical one is to commit 1.2 million keys (a single
>PutAll call) with a total size of around 600MB.
>- Persistence
>
>In our proof of concept, we've got everything implemented and running:
>
>- One server node 2.9.1. Native persistence is enabled for default data
>region.
>- One client application using one Ignite C++ Thin Client to connect to the
>server node.
>- Both, server and client, are in the same machine by now.
>
>With this scenario, we're currently evaluating Ignite vs RocksDB. We would
>really like to choose Ignite because of its scalability, but we are facing a
>problem related to its performance:
>
>In Ignite, one single transaction commit of 1.2 million keys and 600MB takes
>around 70 seconds to complete, while RocksDB takes no more than 12 seconds.
>Moreover, if a second local node is added to the cluster, the application is
>not even able of completing the transaction (it stops after 10 minutes)
>
>Default data region's page size has been modified up to 16KB. Persistence
>has been enabled.
>Cache is PARTITIONED with TRANSACTIONAL atomicity mode.
>Because of the requirement about locking keys, performed transaction is
>PESSIMISTIC + READ_COMMITTED.
>
>The rest of the configuration values are the default ones (No backup,
>PRIMARY_SYNC, no OnHeapCache, etc)
>
>So, my questions are:
>
>- Taking the requirements into account, is Ignite a good option?
>- It's those time values that one might expect?
>- If not, any advice to improve them?
>
>Configuration files for both server nodes have been attached. Thanks
>everyone in advance for your help and time,
>
>first-node.xml
>< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/first-node.xml >
>second-node.xml
>< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/second-node.xml >
>
>Josemari
>
>
>
>--
>Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re[4]: Performance of Ignite as key-value datastore. C++ Thin
Client.
Posted by jjimeno <jj...@omp.com>.
Thanks for updating!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re[4]: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
For the curious, the ticket is
https://issues.apache.org/jira/browse/IGNITE-14076
Regards,
--
Ilya Kasnacheev
пт, 29 янв. 2021 г. в 18:55, jjimeno <jj...@omp.com>:
> Ok... thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Re[4]: Performance of Ignite as key-value datastore. C++ Thin
Client.
Posted by jjimeno <jj...@omp.com>.
Ok... thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re[4]: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Zhenya Stanilovsky <ar...@mail.ru>.
I suppose you can estimate time of loading 1.2 million keys with 2 ignite nodes right now.
Just load 1200 keys 1000 times, i believe assumed time would be similar.
And i hope you understand the difference between lsm\btree and embeded\distributed system.
I can`t give you a fix date right now, may be next week will be clearer here.
thanks !
>First off, thanks to you and Zhenya for your support
>
>I'm afraid that's not easy, as key's data are unrelated.
>
>As we are in an evaluation phase, we would prefer to wait for the fix and
>perform single node benchmarking in the meanwhile... Instead of grouping
>keys, we might split the big transaction into several smaller ones, as
>Zhenya suggested, but that will not give us the real timings anyway.
>
>According to Zhenya's mail, the problem has been identified and it seems you
>can solve it. I know it's difficult, but.. could you give a rough fix date?
>
>
>
>--
>Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin
Client.
Posted by jjimeno <jj...@omp.com>.
First off, thanks to you and Zhenya for your support
I'm afraid that's not easy, as key's data are unrelated.
As we are in an evaluation phase, we would prefer to wait for the fix and
perform single node benchmarking in the meanwhile... Instead of grouping
keys, we might split the big transaction into several smaller ones, as
Zhenya suggested, but that will not give us the real timings anyway.
According to Zhenya's mail, the problem has been identified and it seems you
can solve it. I know it's difficult, but.. could you give a rough fix date?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Pavel Tupitsyn <pt...@apache.org>.
Josemari, can you please describe the use case in more detail?
Maybe it is possible to reduce the number of cache entries by combining the
data from multiple keys into one?
On Fri, Jan 29, 2021 at 7:01 AM Zhenya Stanilovsky <ar...@mail.ru>
wrote:
>
> I also confirm this issue, i will append additional info into ticket,
> seems we can fix it.
> jjimeno thanks for highlighting. The only work around for now — is somehow
> decrease tx enlisted keys.
>
>
> I am able to reproduce the problem - it occurs with any client, thick or
> thin.
>
> With one node the transaction completes in a reasonable time,
> but with two nodes it is orders of magnitude slower.
>
> We'll investigate and get back to you.
>
>
> On Thu, Jan 28, 2021 at 9:48 AM jjimeno <jjimeno@omp.com
> <//...@omp.com>> wrote:
>
> Hi again,
>
> As a test, I just disabled persistence in both nodes. The already
> mentioned
> transaction of 1.2 million keys and 600MB in size takes 298sec.
>
> Remember that for one single node and persistence enabled it takes 70sec,
> so
> just adding a second node makes the test more than 4 times slower.
>
> Is this, really, the performance that Ignite can offer? Please, don't take
> me wrong, I'm just asking, not criticizing. I want to be sure I'm not
> doing
> anything wrong and the timings I get are the expected ones...
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>
>
>
Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Zhenya Stanilovsky <ar...@mail.ru>.
I also confirm this issue, i will append additional info into ticket, seems we can fix it.
jjimeno thanks for highlighting. The only work around for now — is somehow decrease tx enlisted keys.
>I am able to reproduce the problem - it occurs with any client, thick or thin.
>
>With one node the transaction completes in a reasonable time,
>but with two nodes it is orders of magnitude slower.
>
>We'll investigate and get back to you.
>
>On Thu, Jan 28, 2021 at 9:48 AM jjimeno < jjimeno@omp.com > wrote:
>>Hi again,
>>
>>As a test, I just disabled persistence in both nodes. The already mentioned
>>transaction of 1.2 million keys and 600MB in size takes 298sec.
>>
>>Remember that for one single node and persistence enabled it takes 70sec, so
>>just adding a second node makes the test more than 4 times slower.
>>
>>Is this, really, the performance that Ignite can offer? Please, don't take
>>me wrong, I'm just asking, not criticizing. I want to be sure I'm not doing
>>anything wrong and the timings I get are the expected ones...
>>
>>Thanks!
>>
>>
>>
>>--
>>Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by Pavel Tupitsyn <pt...@apache.org>.
I am able to reproduce the problem - it occurs with any client, thick or
thin.
With one node the transaction completes in a reasonable time,
but with two nodes it is orders of magnitude slower.
We'll investigate and get back to you.
On Thu, Jan 28, 2021 at 9:48 AM jjimeno <jj...@omp.com> wrote:
> Hi again,
>
> As a test, I just disabled persistence in both nodes. The already
> mentioned
> transaction of 1.2 million keys and 600MB in size takes 298sec.
>
> Remember that for one single node and persistence enabled it takes 70sec,
> so
> just adding a second node makes the test more than 4 times slower.
>
> Is this, really, the performance that Ignite can offer? Please, don't take
> me wrong, I'm just asking, not criticizing. I want to be sure I'm not
> doing
> anything wrong and the timings I get are the expected ones...
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance of Ignite as key-value datastore. C++ Thin Client.
Posted by jjimeno <jj...@omp.com>.
Hi again,
As a test, I just disabled persistence in both nodes. The already mentioned
transaction of 1.2 million keys and 600MB in size takes 298sec.
Remember that for one single node and persistence enabled it takes 70sec, so
just adding a second node makes the test more than 4 times slower.
Is this, really, the performance that Ignite can offer? Please, don't take
me wrong, I'm just asking, not criticizing. I want to be sure I'm not doing
anything wrong and the timings I get are the expected ones...
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/