Ignite Load Performance when memory exhausted

[njcstreet <nj...@gmail.com>]

Hi,

I am evaluating Apache Ignite for a use case where I know we may not have
enough RAM to store the entire database. For example we might store 30 days
of data, but only have enough memory for 15, and in fact most of the queries
will be against the last 5 days of data. I am doing some tests loading the
30 days of data (roughly 2 billion objects) into a test environment.

 

·         The test environment has 6 servers each with 128GB RAM, which I am
allocating roughly 70% to Ignite to allow the OS enough memory.

·         This is using Native Persistence in Ignite 2.8.1.

·         The disks I am using are not as fast as production would be (they
are not SSD) but they are still reasonably fast.

·         The object I am storing is reasonably simple, it has 25 properties
which are integers (2 of these have indexes on) and 4 which are floats, but
there are a lot of them (roughly 2 billion).

·         The cache the data is being loaded into is partitioned.

·         Data is loaded using IDataStreamer – there is a single loader
process which has 10 threads reading files, then submitting into a single
DataStreamer.

 

What I am observing is that:

 

·         Whilst the default region has RAM available, the inserts are quite
quick – even though the data is being written to disk (the WAL is enabled). 
I get on average 150K objects inserted per second.

·         Once the default region runs out of RAM, I see an error message
“WARN|org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl|Page
replacements started, pages will be rotated with disk, this will affect
storage performance (consider increasing
DataRegionConfiguration#setMaxSize).”

·         After this, inserts are much slower, going down to about 19K
objects per second.

 

Now I kind of understand what is happening here is that for every new page
in memory that needs to be allocated, Ignite is having to purge another page
from memory (this is implied by the error message). But what I don’t
understand is why the performance impact is so great (150K objects / sec
when memory is available vs 19K per sec when memory is exhausted). Before
and after the region ran out of RAM, the data was being written to disk.
After the region ran out of RAM, I would expect the only difference to be
that it needs to purge some data from RAM to make space for the data that is
loading – I wouldn’t have thought there should be any more disk activity
since through Ignite Persistence the data was already being written to disk.
Maybe I have something mis-configured, or I have misunderstood something?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Load Performance when memory exhausted

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

Until you got this message, you did not need to catch up with disk (except
for WAL writing, maybe), but once you see it, good days are over and you
need to do checkpoints with all pages which were updated recently.

There are things to tune here, but don't expect miracles.

https://apacheignite.readme.io/docs/durable-memory-tuning

Regards,
-- 
Ilya Kasnacheev


вт, 23 июн. 2020 г. в 18:25, njcstreet <nj...@gmail.com>:

> Hi,
>
> I am evaluating Apache Ignite for a use case where I know we may not have
> enough RAM to store the entire database. For example we might store 30 days
> of data, but only have enough memory for 15, and in fact most of the
> queries
> will be against the last 5 days of data. I am doing some tests loading the
> 30 days of data (roughly 2 billion objects) into a test environment.
>
>
>
> ·         The test environment has 6 servers each with 128GB RAM, which I
> am
> allocating roughly 70% to Ignite to allow the OS enough memory.
>
> ·         This is using Native Persistence in Ignite 2.8.1.
>
> ·         The disks I am using are not as fast as production would be (they
> are not SSD) but they are still reasonably fast.
>
> ·         The object I am storing is reasonably simple, it has 25
> properties
> which are integers (2 of these have indexes on) and 4 which are floats, but
> there are a lot of them (roughly 2 billion).
>
> ·         The cache the data is being loaded into is partitioned.
>
> ·         Data is loaded using IDataStreamer – there is a single loader
> process which has 10 threads reading files, then submitting into a single
> DataStreamer.
>
>
>
> What I am observing is that:
>
>
>
> ·         Whilst the default region has RAM available, the inserts are
> quite
> quick – even though the data is being written to disk (the WAL is
> enabled).
> I get on average 150K objects inserted per second.
>
> ·         Once the default region runs out of RAM, I see an error message
>
> “WARN|org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl|Page
> replacements started, pages will be rotated with disk, this will affect
> storage performance (consider increasing
> DataRegionConfiguration#setMaxSize).”
>
> ·         After this, inserts are much slower, going down to about 19K
> objects per second.
>
>
>
> Now I kind of understand what is happening here is that for every new page
> in memory that needs to be allocated, Ignite is having to purge another
> page
> from memory (this is implied by the error message). But what I don’t
> understand is why the performance impact is so great (150K objects / sec
> when memory is available vs 19K per sec when memory is exhausted). Before
> and after the region ran out of RAM, the data was being written to disk.
> After the region ran out of RAM, I would expect the only difference to be
> that it needs to purge some data from RAM to make space for the data that
> is
> loading – I wouldn’t have thought there should be any more disk activity
> since through Ignite Persistence the data was already being written to
> disk.
> Maybe I have something mis-configured, or I have misunderstood something?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>