You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Biren Shah <Bi...@servicenow.com> on 2017/11/29 02:08:41 UTC
get api takes more time when adding more nodes to cluster
Hi,
On adding more nodes to the cluster, the get api slows down. I have a cache with 2 million entries. The cache gets loaded at the start of the cluster and does not change frequently. Here is the cache configuration. I had start the cluster with 2 nodes and added 2 more nodes later on.
CacheConfiguration<String, Entry> cacheConfig = new CacheConfiguration<>();
cacheConfig.setOnheapCacheEnabled(true);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setBackups(1);
cacheConfig.setCopyOnRead(false);
cacheConfig.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
cacheConfig.setStatisticsEnabled(true);
I have few questions.
1. Get operations on cache are not based on affinity. So, in that case which is better performant get or localpeek?
2. Cache is not initialized with keep binary option. We read most of the entries of the cache on regular basis. Shall I change it to keep the data in binary format? Will that help in bringing the get time down?
3. What configuration nobs I can change to investigate the issue further?
4. What is recommended cache configuration for such use case?
Each node is setup with 8GB on memory and can use upto 8 cores.
Thanks,
Biren Shah
Re: get api takes more time when adding more nodes to cluster
Posted by vkulichenko <va...@gmail.com>.
Biren,
I meant that you can have a standalone cluster and embed client node into
the application instead of server node. Making these caches replicated can
be also an option - in this case all reads will be local and fast.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: get api takes more time when adding more nodes to cluster
Posted by Biren Shah <Bi...@servicenow.com>.
I am not sure how client-server deployment would look in our case. Our application starts the ignite in server mode, initializes caches and later access those caches. To switch to client-server deployment, the application will start a node in server mode, initializes caches. Then the application will start another node in client mode on the same JVM? That does not to look right. We don’t want to separate out these functionalities into different applications/JVM.
I can make read heavy caches to be replicated. That will improve get operations on those caches. Ill try that out.
-Biren
On 11/29/17, 5:07 PM, "vkulichenko" <va...@gmail.com> wrote:
Biren,
If half of your operations became multiple times slower, why would you
expect throughput to increase? In case you don't use collocation, I would
recommend you to switch to client-server deployment. Initial performance
with two nodes and fully replicated cache can be slower than now, but you
will see how it scales out.
-Val
--
Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_&d=DwICAg&c=Zok6nrOF6Fe0JtVEqKh3FEeUbToa1PtNBZf6G01cvEQ&r=rbkF1xy5tYmkV8VMdTRVaIVhaXCNGxmyTB5plfGtWuY&m=q8E7KLmXbR4rdJjoAHbe2vqobCfu4moZki2hcqMjH6M&s=02J_1t0Q5IgCpjhUxcZsqyVJ3bOzlylkXdrdtB7Iw0Q&e=
Re: get api takes more time when adding more nodes to cluster
Posted by vkulichenko <va...@gmail.com>.
Biren,
If half of your operations became multiple times slower, why would you
expect throughput to increase? In case you don't use collocation, I would
recommend you to switch to client-server deployment. Initial performance
with two nodes and fully replicated cache can be slower than now, but you
will see how it scales out.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: get api takes more time when adding more nodes to cluster
Posted by Biren Shah <Bi...@servicenow.com>.
Well I am not expecting that by doubling the number of nodes, I will get 2x throughput. But it should be at some liner rate and definitely should not bring the throughput down. We have embedded the ignite in the application. On start of the application we start ignite in server mode. We initialize distributed caches and load data from persistent store. Then the application reads from and writes to caches.
-Biren
On 11/29/17, 3:52 PM, "vkulichenko" <va...@gmail.com> wrote:
Biren,
That's a wrong expectation because local in-memory read is drastically
faster than a network read. How do you choose a server node to read from?
What is overall use case?
-Val
--
Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_&d=DwICAg&c=Zok6nrOF6Fe0JtVEqKh3FEeUbToa1PtNBZf6G01cvEQ&r=rbkF1xy5tYmkV8VMdTRVaIVhaXCNGxmyTB5plfGtWuY&m=grCPTVl8FQVCiNf6dQwL5M3RP54gZKSIb-nKIoTtqh4&s=Zb6ay9er_YKcjzeqY9zeum4rf2rWCPnlYf9b_7wizXU&e=
Re: get api takes more time when adding more nodes to cluster
Posted by vkulichenko <va...@gmail.com>.
Biren,
That's a wrong expectation because local in-memory read is drastically
faster than a network read. How do you choose a server node to read from?
What is overall use case?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: get api takes more time when adding more nodes to cluster
Posted by Biren Shah <Bi...@servicenow.com>.
Hi Val,
We are doing read on server only. I understand the get will be slow in case of 4 nodes because its doing remote get for some cache entries. But it should not bring down overall throughput of an application. Just to give you some context, with 2 nodes the application can process 500K data points every minute. But when I add 2 more nodes to the cluster the throughput reduces to 400K data point. Expectation is that with total 4 nodes we should be able to process more data point.
-Biren
On 11/29/17, 2:52 PM, "vkulichenko" <va...@gmail.com> wrote:
Hi Biren,
Are you doing reads from a client or directly on server nodes? If the
letter, then I guess you just do not collocate properly. With two nodes and
one backup all data is available on both nodes, so any server side read
would be local. With four nodes some of them would be remote which is
obviously slower.
-Val
--
Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_&d=DwICAg&c=Zok6nrOF6Fe0JtVEqKh3FEeUbToa1PtNBZf6G01cvEQ&r=rbkF1xy5tYmkV8VMdTRVaIVhaXCNGxmyTB5plfGtWuY&m=mhGI_pnCKYuFMBxwVkhd6Ja-qVO_m4CWWvPSXjBzlU4&s=ZWb-NQgr5GT-noYFO2XXS6CMQoUtUE45zbWI-YqpuSA&e=
Re: get api takes more time when adding more nodes to cluster
Posted by vkulichenko <va...@gmail.com>.
Hi Biren,
Are you doing reads from a client or directly on server nodes? If the
letter, then I guess you just do not collocate properly. With two nodes and
one backup all data is available on both nodes, so any server side read
would be local. With four nodes some of them would be remote which is
obviously slower.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/