You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by amitpa <am...@nrifintech.com> on 2016/06/02 16:23:01 UTC
Ignite Write Behind performance
Hello,
I am struggling with improving ignite transaction performance. However this
is another problem
However if I am doing write behind and the writes are slow, I am seeing that
the over all performance of teh grid drops.
Is there any configuration that I can do to make ignite performance stable
even when the writes are being slow in write behind (due to slow DB)
Currently I have 10 threads in write behind mode.
What should be my system threadpool size assuming I have two caches which
participate in transactions and 2 nodes in 10G network.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by amitpa <am...@nrifintech.com>.
We use it to conservative 2 now in our current setting
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5549.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by bintisepaha <bi...@tudor.com>.
Thanks for that. we were setting flushsize to 0. how about flush thread
count?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5541.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by amitpa <am...@nrifintech.com>.
We use both. Which ever is faster gets called first ..
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5507.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by bintisepaha <bi...@tudor.com>.
Do you mind sharing your cache configuration for write behind? do you use
flush size or flush frequency?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5501.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by amitpa <am...@nrifintech.com>.
I did test this...For us I think Write behind gets called fine.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5475.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by bintisepaha <bi...@tudor.com>.
amitpa, We would be interested in learning how did this perform for you?
We have implemented spring txn to insert in database for write-behind.
Hoever, we see that sometiems write-behind is not even called for some
objects that we are certain were just updated in the cache. have you noticed
that?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5470.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by amitpa <am...@nrifintech.com>.
Thanks we are considering implementing JDBC batch inserts
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5418.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by Alexey Goncharuk <al...@gmail.com>.
Ignite _does_ use a separate thread pool to persist data in write-behind
mode, this is the essential difference between write-through and
write-behind. However, if your cache load rate is significantly higher than
the database insert rate, the write queue will grow faster than background
threads can flush data, so eventually you will get an OOME.
In this situation you can
- block and wait until there is some space in the write queue
- persist your own data synchronously
Ignite choses the second option and writes transaction data synchronously.
As Denis pointer out, you should tune write-behind parameters to make sure
that background threads are working at sufficient insert rate. Also make
sure that you use batched inserts in your CacheStore implementation.
Hope this helps,
AG
2016-06-03 10:26 GMT-07:00 amitpa <am...@nrifintech.com>:
> I understood this perfectly.
>
> What I mean is :-
>
> Shouldnt the write behind use a separete thread pool so that slow write
> behind process does not impact the whole grid ?
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5412.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Write Behind performance
Posted by amitpa <am...@nrifintech.com>.
I understood this perfectly.
What I mean is :-
Shouldnt the write behind use a separete thread pool so that slow write
behind process does not impact the whole grid ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5412.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by Denis Magda <dm...@gridgain.com>.
As I explained in the situation when the write-behind queue is full a Thread that updates a cache and propagates the changes into the write-behind store can be used to flush the write-behind queue since the queue is full. This situation can affect the performance especially if this happens too often.
—
Denis
> On Jun 3, 2016, at 11:30 AM, amitpa <am...@nrifintech.com> wrote:
>
> Danis,
>
> Thanks. I am trying that.
>
> However shouldnt the write behinds not impact the GRID performance at all.
> Since we are writing in write behin dmode to avoid the cost of slow disk IO.
> I understand that theres a limit, but shouldnt this thread pool be different
> if technically possible, allowing ignite to continue processing in case of
> slow write behind performance.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5400.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by amitpa <am...@nrifintech.com>.
Danis,
Thanks. I am trying that.
However shouldnt the write behinds not impact the GRID performance at all.
Since we are writing in write behin dmode to avoid the cost of slow disk IO.
I understand that theres a limit, but shouldnt this thread pool be different
if technically possible, allowing ignite to continue processing in case of
slow write behind performance.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385p5400.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Write Behind performance
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
Most likely your write-behind Threads can’t keep up with volume of data that is aggregated in the write-behind buffer and you get with the situation when from time to time writing to the DB is performed in the sync mode from system Thread(s).
I would suggest you playing with the following write-behind configuration to avoid the situation above:
- CacheConfiguration.setWriteBehindFlushSize - increase size of the write-behind buffer;
- CacheConfiguration.setWriteBehindFlushThreadCount;
- CacheConfiguration.setWriteBehindFlushFrequence - decrease this value triggering the flushing more frequently. Default value is 5 seconds.
—
Denis
> On Jun 2, 2016, at 7:23 PM, amitpa <am...@nrifintech.com> wrote:
>
> Hello,
>
> I am struggling with improving ignite transaction performance. However this
> is another problem
>
> However if I am doing write behind and the writes are slow, I am seeing that
> the over all performance of teh grid drops.
>
> Is there any configuration that I can do to make ignite performance stable
> even when the writes are being slow in write behind (due to slow DB)
>
> Currently I have 10 threads in write behind mode.
>
> What should be my system threadpool size assuming I have two caches which
> participate in transactions and 2 nodes in 10G network.
>
>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Write-Behind-performance-tp5385.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.