Inserts stalled by write-behind process

[bsmets <ba...@inventivegroup.com>]

We are currently using an ignite setup with caches with very large offheap
sizes (10GB+) and in these caches we are inserting entries of roughly 30 kB.
We flush the entries to disk every minute using write-behind.

What we observe is that after a while, the inserts of new entries into the
cache seem to become synchronized with the writing of entries to disk.
Considering our disk is much slower than writing in memory, this is
effectively throttling our inserts. 

We noticed that the WriteBehindTotalCriticalOverflowCount in the metrics is
pretty high as well, which leads us to believe that inserts are waiting for
entries to be written to disk. 

Is this correct, and if so, how can we increase the write-behind backlog?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Inserts-stalled-by-write-behind-process-tp3390.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Inserts stalled by write-behind process

[bintisepaha <bi...@tudor.com>]

Where do you see this property?
WriteBehindTotalCriticalOverflowCount 




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Inserts-stalled-by-write-behind-process-tp3390p5500.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Inserts stalled by write-behind process

[Dmitriy Setrakyan <ds...@apache.org>]

Agree with Valentin. One other thing you may wish to check is whether you
are doing a bulk insert to the database, or multiple individual inserts.
Bulk insert should perform better, of course.

D.

On Tue, Mar 8, 2016 at 4:50 PM, vkulichenko <va...@gmail.com>
wrote:

> Hi,
>
> You're right, write-behind store has a back pressure mechanism that starts
> to update DB synchronously if the queue is too long. Otherwise you will
> most
> likely eventually get out of memory error.
>
> The backlog size is controlled by
> CacheConfiguration.setWriteBehindFlushSize() property. When this size is
> reached, the store will start flushing to the database in the background.
> If
> it will detect that the flushing process is slower than cache updates
> (i.e.,
> the backlog continuous growing above the setting), it will switch to sync
> updates.
>
> Makes sense?
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Inserts-stalled-by-write-behind-process-tp3390p3399.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Inserts stalled by write-behind process

[vkulichenko <va...@gmail.com>]

Hi,

You're right, write-behind store has a back pressure mechanism that starts
to update DB synchronously if the queue is too long. Otherwise you will most
likely eventually get out of memory error.

The backlog size is controlled by
CacheConfiguration.setWriteBehindFlushSize() property. When this size is
reached, the store will start flushing to the database in the background. If
it will detect that the flushing process is slower than cache updates (i.e.,
the backlog continuous growing above the setting), it will switch to sync
updates.

Makes sense?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Inserts-stalled-by-write-behind-process-tp3390p3399.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.