You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by gigabot <mw...@gmail.com> on 2021/06/02 15:24:33 UTC

Native Persistence Tuning - Percentage kept in RAM

 When using native persistence, can I configure what percentage or amount f
data is kept in RAM? I have two caches, one I want kept mostly in RAM and
the other mostly on disk, so I want to configure their Data Regions
differently. It seems from testing that MaxSize only affects the total size
of disk+RAM.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Native Persistence Tuning - Percentage kept in RAM

Posted by Raymond Wilson <ra...@trimble.com>.
We encountered the same issue with a small data region (64Mb in our case).
If I recall correctly the failure was due to the checkpointing mechanism
not having enough pages to perform a checkpoint, rather than a critical
failure on a Put(). Adding more memory increases the number of pages (or
you can use a smaller page size).

Editorial note: Personally, I don't like this failure mode. While
increasing the available memory does 'make the problem go away', it feels
more a case of kicking the can down the road in that later increases in
load may bring about the same conditions. Modifying the checkpoint triggers
to checkpoint more often, or on a smaller fraction of dirty pages is
another approach that can be taken to mitigate the issue.


On Thu, Jun 3, 2021 at 5:02 AM Stephen Darlington <
stephen.darlington@gridgain.com> wrote:

> Your data region is too small. Ignite is an in-memory data grid, it’s not
> really designed to work with as little as 40Mb of memory.
>
> When I tried it on my machine, I get the following message:
>
> [17:42:09,246][SEVERE][checkpoint-runner-cpu-#80][] JVM will be halted
> immediately due to the failure: [failureCtx=FailureContext
> [type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException:
> Failed to find a page for eviction [segmentCapacity=1580, loaded=620,
> maxDirtyPages=465, dirtyPages=620, cpPages=0, pinnedInSegment=0,
> failedToPrepare=620]
> Out of memory in data region [name=default, initSize=20.0 MiB,
> maxSize=20.0 MiB, persistenceEnabled=true] Try the following:
>   ^-- Increase maximum off-heap memory size
> (DataRegionConfiguration.maxSize)
>
> Basically, it’s writing to the cluster faster than it can write the
> changes to disk. As it says, increase the amount of off-heap memory.
>
> If you’re getting the same error, you might be able to get it to work by
> feeding the data in more slowly.
>
> On 2 Jun 2021, at 16:44, gigabot <mw...@gmail.com> wrote:
>
> Hi I tried setting the MaxSize on a cache to a low number (40Mb) and it
> quickly ran in to an IgniteOutOfMemoryException, despite the fact there was
> plenty of space on disk. Why would it run out of RAM if it can persist data
> above the MaxSize the excess to disk?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>

-- 
<http://www.trimble.com/>
Raymond Wilson
Trimble Distinguished Engineer, Civil Construction Software (CCS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wilson@trimble.com

<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>

Re: Native Persistence Tuning - Percentage kept in RAM

Posted by Stephen Darlington <st...@gridgain.com>.
Your data region is too small. Ignite is an in-memory data grid, it’s not really designed to work with as little as 40Mb of memory.

When I tried it on my machine, I get the following message:

[17:42:09,246][SEVERE][checkpoint-runner-cpu-#80][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: Failed to find a page for eviction [segmentCapacity=1580, loaded=620, maxDirtyPages=465, dirtyPages=620, cpPages=0, pinnedInSegment=0, failedToPrepare=620]
Out of memory in data region [name=default, initSize=20.0 MiB, maxSize=20.0 MiB, persistenceEnabled=true] Try the following:
  ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)

Basically, it’s writing to the cluster faster than it can write the changes to disk. As it says, increase the amount of off-heap memory.

If you’re getting the same error, you might be able to get it to work by feeding the data in more slowly.

> On 2 Jun 2021, at 16:44, gigabot <mw...@gmail.com> wrote:
> 
> Hi I tried setting the MaxSize on a cache to a low number (40Mb) and it
> quickly ran in to an IgniteOutOfMemoryException, despite the fact there was
> plenty of space on disk. Why would it run out of RAM if it can persist data
> above the MaxSize the excess to disk?
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



Re: Native Persistence Tuning - Percentage kept in RAM

Posted by gigabot <mw...@gmail.com>.
Hi I tried setting the MaxSize on a cache to a low number (40Mb) and it
quickly ran in to an IgniteOutOfMemoryException, despite the fact there was
plenty of space on disk. Why would it run out of RAM if it can persist data
above the MaxSize the excess to disk?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Native Persistence Tuning - Percentage kept in RAM

Posted by Stephen Darlington <st...@gridgain.com>.
That’s how you do what you’re asking. MaxSize is the maximum size of the off-heap memory used.

> On 2 Jun 2021, at 16:34, gigabot <mw...@gmail.com> wrote:
> 
> Hi Stephen, I know how to configure separate data regions, but how do I
> configure what % of each is in RAM vs disk?
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



Re: Native Persistence Tuning - Percentage kept in RAM

Posted by gigabot <mw...@gmail.com>.
Hi Stephen, I know how to configure separate data regions, but how do I
configure what % of each is in RAM vs disk?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Native Persistence Tuning - Percentage kept in RAM

Posted by Stephen Darlington <st...@gridgain.com>.
Configure two different data regions: one for your “mostly in memory” cache; another for your “mostly on disk” cache.

> On 2 Jun 2021, at 16:24, gigabot <mw...@gmail.com> wrote:
> 
> 
> When using native persistence, can I configure what percentage or amount f
> data is kept in RAM? I have two caches, one I want kept mostly in RAM and
> the other mostly on disk, so I want to configure their Data Regions
> differently. It seems from testing that MaxSize only affects the total size
> of disk+RAM.
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/