You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Łukasz Dywicki <lu...@code-house.org> on 2023/01/26 23:58:28 UTC
Safe settings for ignite cache with external store
Dear all,
I come across use of Apache Ignite to cache results of expensive
computation operation.
Objectives are basic:
- Keep most of "hot" data in memory
- Offload cold part to cache store
- Keep memory utilization under control (evict entries as needed)
While it sounds basic, it doesn't seem to fit Ignite defaults.
What I am testing now is behavior with large objects which can grow up
to 10 mb (serialized) or 25 mb (json representation). Usually objects
will stay far below that threshold, but we can't make assumption on that.
I began testing various configurations of Ignite in order to facilitate
offloading of memory contents to database. So far I am stuck for two
days at Ignite/application itself running out of memory after processing
several of such large objects. While I know that storing 10 mb blob in
database is not the best idea, I have to test that behavior too.
By observing database contents I see that number of entries there grows,
but cache do not seem to be evicted. When I try to switch eviction, it
does require onheap to be switched on, and it still fails with LRU
eviction policy.
So far I ended up with a named cache and default region configured as below:
```
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setDataStorageConfiguration(new
DataStorageConfiguration()
.setDefaultDataRegionConfiguration(new DataRegionConfiguration()
.setPersistenceEnabled(false)
.setInitialSize(256L * 1024 * 1024)
.setMaxSize(512L * 1024 * 1024)
.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
.setSwapPath(null)
.setEvictionThreshold(0.75)
)
.setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
);
CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new
CacheConfiguration<>()
.setName(CACHE_NAME)
.setBackups(2)
.setAtomicityMode(CacheAtomicityMode.ATOMIC)
.setCacheStoreFactory(cacheJdbcBlobStoreFactory)
.setWriteThrough(true)
.setOnheapCacheEnabled(true)
.setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
.setReadThrough(true);
igniteConfiguration.setCacheConfiguration(
expensiveCache
);
```
What I observe is following - the cache keeps writing data into
database, but it does not remove old entries fast enough to prevent crash.
JVM parameters I use are fairly basic:
-Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
-XX:+DisableExplicitGC
The store mechanism is jdbc blob store. Exceptions I get happen to occur
in Ignite itself, processing (application code writing cache) or
communication thread used to feed cache. I collected one case here:
https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
The error message in linked gist advice to enable persistence (which I
did via cache store!), increase memory limit (which I don't want to do),
or enable eviction/expiry policy (which somehow miss behave).
To me it looks like self defense mechanisms Ignite has are being tricked
leading whole application to crash.
Can you please advise me which settings to tune and how in order to get
Ignite more stable under such load?
Kind regards,
Łukasz
Re: Safe settings for ignite cache with external store
Posted by Stephen Darlington <st...@gridgain.com>.
Glad to hear that you got it working.
> On 30 Jan 2023, at 23:05, Łukasz Dywicki <lu...@code-house.org> wrote:
>
> Hello Stephen,
> I've started tests with few angles, with larger and smaller memory setups, looked at Ignite unit tests and began looking for page eviction triggers. Once I found interesting code areas I realized that my test setup was not having proper page eviction set up. This lead me to further debugging which lead me piece of old code which was overriding persistence configuration, *effectively* reverting page eviction to default (no eviction).
> After clean up I successfully tested eviction with small caches (512MB max + direct buffer at 756MB). These worked stable under constant load, which proved the point that it is possible to have stable operation on fairly limited resources.
>
> Guilt goes to spring configuration pre/post-processing which went under radar till today. It was really specific to test application itself, not Ignite. I thank you for feedback, pointers and assistance!
>
> Kind regards,
> Łukasz
>
> On 30.01.2023 12:51, Stephen Darlington wrote:
>> I’m not saying you should test with 50Gb, but I am saying that 512Mb is too small. Maybe try a few gigs.
>> When you define eviction at a cache level, that is for the on-heap cache. Most people don’t need an on-heap cache.
>> You you define eviction at a data region level, that works on the off-heap data. As you observe, it works at a /page/ rather than a record level. (This is why a tiny region can be problematic. There are far few pages that can be chosen to evict.)
>>> On 30 Jan 2023, at 11:37, Łukasz Dywicki <lu...@code-house.org> wrote:
>>>
>>> Dear Jeremy and Stephen,
>>> Thank you for answers. I this issues I face is not bound exclusively to assigned memory but behavior of eviction policies. After all, without proper eviction or expiry policy, each amount of memory will eventually be exceeded. Small size of memory makes it simply easier to test. To be honest I do not see a point in testing eviction with 50GB of memory, if it cant be configured for 512MB. Expiry policy proved to work, but it will not preventing node from crash, as there always might be high write ratio causing memory to fill.
>>>
>>> Both LRU and FIFO eviction policies in default configuration do not take into account overall memory allocation, they work only with on heap caches. Unless memory size is explicitly specified they will rely on over-provisioned memory, or assume that memory assignment is right. This is possible only if cache entry have a predictable.
>>> Another point is that above eviction policies are dedicated to on-heap caches and are told to have no affect to off-heap region, why is so? By looking at sources I see CacheOffheapEvictionManager and GridCacheEvictionManager, but I can't grasp what enables eviction of entries from off heap memory managed by Ignite.
>>> Given that my use case is continuous operation with variety of entries having various sizes I can't assume on-heap cache with LRU nor FIFO, especially that they do not impact off heap memory. I also can't risk filling in memory managed by Ignite due to lack of clear eviction policy during peak load.
>>> As far I can understand from DataRegionConfiguration, its eviction mechanism works for whole pages, but I haven't seen this happening. Could it be due to fact that single cache entry in my test exceeds maximum page size (16MB)?
>>>
>>> Best regards,
>>> Łukasz
>>>
>>> On 30.01.2023 10:32, Stephen Darlington wrote:
>>>> Ignite is designed to work as a cluster with a lot of memory. A single node with 512Mb of memory just isn’t what most people are testing with. I’ve seen similar issues when people use tiny nodes. I don’t see anything immediately wrong with your configuration, but if it works with more memory, that’s your problem.
>>>>> On 28 Jan 2023, at 00:35, Łukasz Dywicki <lu...@code-house.org> wrote:
>>>>>
>>>>> Hello again,
>>>>> I've spent another day on this issue looking at configuration and with cluster larger than configured backup count ignite did the work.
>>>>> After that I reverted to testing of single node with minimal configuration and come to the point that the only way to keep Ignite survive load was setting ExpiryPolicy to very low value (10s).
>>>>>
>>>>> To me it seems that Ignite behavior is to preserve all entries in memory at any cost, even if there is a risk of running into OOM. Is that true?
>>>>> I would like to change this behavior and make sure that entries do not expire because of time as long as there is memory available to store them. They should be evicted by appropriate EvictionPolicy only if memory fills up.
>>>>> To me it looks like DataStoreSettings do not make any impact in this regard. At least setting page eviction to LRU do not change it.
>>>>>
>>>>> Please let me know if I am doing something wrong as I can not prove Ignite to be working stable. Even with such basic objectives I outlined in earlier mail.
>>>>>
>>>>> Kind regards,
>>>>> Łukasz
>>>>>
>>>>> On 27.01.2023 00:58, Łukasz Dywicki wrote:
>>>>>> Dear all,
>>>>>> I come across use of Apache Ignite to cache results of expensive computation operation.
>>>>>> Objectives are basic:
>>>>>> - Keep most of "hot" data in memory
>>>>>> - Offload cold part to cache store
>>>>>> - Keep memory utilization under control (evict entries as needed)
>>>>>> While it sounds basic, it doesn't seem to fit Ignite defaults.
>>>>>> What I am testing now is behavior with large objects which can grow up to 10 mb (serialized) or 25 mb (json representation). Usually objects will stay far below that threshold, but we can't make assumption on that.
>>>>>> I began testing various configurations of Ignite in order to facilitate offloading of memory contents to database. So far I am stuck for two days at Ignite/application itself running out of memory after processing several of such large objects. While I know that storing 10 mb blob in database is not the best idea, I have to test that behavior too.
>>>>>> By observing database contents I see that number of entries there grows, but cache do not seem to be evicted. When I try to switch eviction, it does require onheap to be switched on, and it still fails with LRU eviction policy.
>>>>>> So far I ended up with a named cache and default region configured as below:
>>>>>> ```
>>>>>> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>>>>>> igniteConfiguration.setDataStorageConfiguration(new DataStorageConfiguration()
>>>>>> .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
>>>>>> .setPersistenceEnabled(false)
>>>>>> .setInitialSize(256L * 1024 * 1024)
>>>>>> .setMaxSize(512L * 1024 * 1024)
>>>>>> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
>>>>>> .setSwapPath(null)
>>>>>> .setEvictionThreshold(0.75)
>>>>>> )
>>>>>> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
>>>>>> );
>>>>>> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new CacheConfiguration<>()
>>>>>> .setName(CACHE_NAME)
>>>>>> .setBackups(2)
>>>>>> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
>>>>>> .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
>>>>>> .setWriteThrough(true)
>>>>>> .setOnheapCacheEnabled(true)
>>>>>> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
>>>>>> .setReadThrough(true);
>>>>>> igniteConfiguration.setCacheConfiguration(
>>>>>> expensiveCache
>>>>>> );
>>>>>> ```
>>>>>> What I observe is following - the cache keeps writing data into database, but it does not remove old entries fast enough to prevent crash.
>>>>>> JVM parameters I use are fairly basic:
>>>>>> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>>>>>> The store mechanism is jdbc blob store. Exceptions I get happen to occur in Ignite itself, processing (application code writing cache) or communication thread used to feed cache. I collected one case here:
>>>>>> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
>>>>>> The error message in linked gist advice to enable persistence (which I did via cache store!), increase memory limit (which I don't want to do), or enable eviction/expiry policy (which somehow miss behave).
>>>>>> To me it looks like self defense mechanisms Ignite has are being tricked leading whole application to crash.
>>>>>> Can you please advise me which settings to tune and how in order to get Ignite more stable under such load?
>>>>>> Kind regards,
>>>>>> Łukasz
Re: Safe settings for ignite cache with external store
Posted by Łukasz Dywicki <lu...@code-house.org>.
Hello Stephen,
I've started tests with few angles, with larger and smaller memory
setups, looked at Ignite unit tests and began looking for page eviction
triggers. Once I found interesting code areas I realized that my test
setup was not having proper page eviction set up. This lead me to
further debugging which lead me piece of old code which was overriding
persistence configuration, *effectively* reverting page eviction to
default (no eviction).
After clean up I successfully tested eviction with small caches (512MB
max + direct buffer at 756MB). These worked stable under constant load,
which proved the point that it is possible to have stable operation on
fairly limited resources.
Guilt goes to spring configuration pre/post-processing which went under
radar till today. It was really specific to test application itself, not
Ignite. I thank you for feedback, pointers and assistance!
Kind regards,
Łukasz
On 30.01.2023 12:51, Stephen Darlington wrote:
> I’m not saying you should test with 50Gb, but I am saying that 512Mb is
> too small. Maybe try a few gigs.
>
> When you define eviction at a cache level, that is for the on-heap
> cache. Most people don’t need an on-heap cache.
>
> You you define eviction at a data region level, that works on the
> off-heap data. As you observe, it works at a /page/ rather than a record
> level. (This is why a tiny region can be problematic. There are far few
> pages that can be chosen to evict.)
>
>> On 30 Jan 2023, at 11:37, Łukasz Dywicki <lu...@code-house.org> wrote:
>>
>> Dear Jeremy and Stephen,
>> Thank you for answers. I this issues I face is not bound exclusively
>> to assigned memory but behavior of eviction policies. After all,
>> without proper eviction or expiry policy, each amount of memory will
>> eventually be exceeded. Small size of memory makes it simply easier to
>> test. To be honest I do not see a point in testing eviction with 50GB
>> of memory, if it cant be configured for 512MB. Expiry policy proved to
>> work, but it will not preventing node from crash, as there always
>> might be high write ratio causing memory to fill.
>>
>> Both LRU and FIFO eviction policies in default configuration do not
>> take into account overall memory allocation, they work only with on
>> heap caches. Unless memory size is explicitly specified they will rely
>> on over-provisioned memory, or assume that memory assignment is right.
>> This is possible only if cache entry have a predictable.
>> Another point is that above eviction policies are dedicated to on-heap
>> caches and are told to have no affect to off-heap region, why is so?
>> By looking at sources I see CacheOffheapEvictionManager and
>> GridCacheEvictionManager, but I can't grasp what enables eviction of
>> entries from off heap memory managed by Ignite.
>> Given that my use case is continuous operation with variety of entries
>> having various sizes I can't assume on-heap cache with LRU nor FIFO,
>> especially that they do not impact off heap memory. I also can't risk
>> filling in memory managed by Ignite due to lack of clear eviction
>> policy during peak load.
>> As far I can understand from DataRegionConfiguration, its eviction
>> mechanism works for whole pages, but I haven't seen this happening.
>> Could it be due to fact that single cache entry in my test exceeds
>> maximum page size (16MB)?
>>
>> Best regards,
>> Łukasz
>>
>> On 30.01.2023 10:32, Stephen Darlington wrote:
>>> Ignite is designed to work as a cluster with a lot of memory. A
>>> single node with 512Mb of memory just isn’t what most people are
>>> testing with. I’ve seen similar issues when people use tiny nodes. I
>>> don’t see anything immediately wrong with your configuration, but if
>>> it works with more memory, that’s your problem.
>>>> On 28 Jan 2023, at 00:35, Łukasz Dywicki <lu...@code-house.org> wrote:
>>>>
>>>> Hello again,
>>>> I've spent another day on this issue looking at configuration and
>>>> with cluster larger than configured backup count ignite did the work.
>>>> After that I reverted to testing of single node with minimal
>>>> configuration and come to the point that the only way to keep Ignite
>>>> survive load was setting ExpiryPolicy to very low value (10s).
>>>>
>>>> To me it seems that Ignite behavior is to preserve all entries in
>>>> memory at any cost, even if there is a risk of running into OOM. Is
>>>> that true?
>>>> I would like to change this behavior and make sure that entries do
>>>> not expire because of time as long as there is memory available to
>>>> store them. They should be evicted by appropriate EvictionPolicy
>>>> only if memory fills up.
>>>> To me it looks like DataStoreSettings do not make any impact in this
>>>> regard. At least setting page eviction to LRU do not change it.
>>>>
>>>> Please let me know if I am doing something wrong as I can not prove
>>>> Ignite to be working stable. Even with such basic objectives I
>>>> outlined in earlier mail.
>>>>
>>>> Kind regards,
>>>> Łukasz
>>>>
>>>> On 27.01.2023 00:58, Łukasz Dywicki wrote:
>>>>> Dear all,
>>>>> I come across use of Apache Ignite to cache results of expensive
>>>>> computation operation.
>>>>> Objectives are basic:
>>>>> - Keep most of "hot" data in memory
>>>>> - Offload cold part to cache store
>>>>> - Keep memory utilization under control (evict entries as needed)
>>>>> While it sounds basic, it doesn't seem to fit Ignite defaults.
>>>>> What I am testing now is behavior with large objects which can grow
>>>>> up to 10 mb (serialized) or 25 mb (json representation). Usually
>>>>> objects will stay far below that threshold, but we can't make
>>>>> assumption on that.
>>>>> I began testing various configurations of Ignite in order to
>>>>> facilitate offloading of memory contents to database. So far I am
>>>>> stuck for two days at Ignite/application itself running out of
>>>>> memory after processing several of such large objects. While I know
>>>>> that storing 10 mb blob in database is not the best idea, I have to
>>>>> test that behavior too.
>>>>> By observing database contents I see that number of entries there
>>>>> grows, but cache do not seem to be evicted. When I try to switch
>>>>> eviction, it does require onheap to be switched on, and it still
>>>>> fails with LRU eviction policy.
>>>>> So far I ended up with a named cache and default region configured
>>>>> as below:
>>>>> ```
>>>>> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>>>>> igniteConfiguration.setDataStorageConfiguration(new
>>>>> DataStorageConfiguration()
>>>>> .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
>>>>> .setPersistenceEnabled(false)
>>>>> .setInitialSize(256L * 1024 * 1024)
>>>>> .setMaxSize(512L * 1024 * 1024)
>>>>> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
>>>>> .setSwapPath(null)
>>>>> .setEvictionThreshold(0.75)
>>>>> )
>>>>> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
>>>>> );
>>>>> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache =
>>>>> new CacheConfiguration<>()
>>>>> .setName(CACHE_NAME)
>>>>> .setBackups(2)
>>>>> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
>>>>> .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
>>>>> .setWriteThrough(true)
>>>>> .setOnheapCacheEnabled(true)
>>>>> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
>>>>> .setReadThrough(true);
>>>>> igniteConfiguration.setCacheConfiguration(
>>>>> expensiveCache
>>>>> );
>>>>> ```
>>>>> What I observe is following - the cache keeps writing data into
>>>>> database, but it does not remove old entries fast enough to prevent
>>>>> crash.
>>>>> JVM parameters I use are fairly basic:
>>>>> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC
>>>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>>>>> The store mechanism is jdbc blob store. Exceptions I get happen to
>>>>> occur in Ignite itself, processing (application code writing cache)
>>>>> or communication thread used to feed cache. I collected one case here:
>>>>> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
>>>>> The error message in linked gist advice to enable persistence
>>>>> (which I did via cache store!), increase memory limit (which I
>>>>> don't want to do), or enable eviction/expiry policy (which somehow
>>>>> miss behave).
>>>>> To me it looks like self defense mechanisms Ignite has are being
>>>>> tricked leading whole application to crash.
>>>>> Can you please advise me which settings to tune and how in order to
>>>>> get Ignite more stable under such load?
>>>>> Kind regards,
>>>>> Łukasz
>
Re: Safe settings for ignite cache with external store
Posted by Stephen Darlington <st...@gridgain.com>.
I’m not saying you should test with 50Gb, but I am saying that 512Mb is too small. Maybe try a few gigs.
When you define eviction at a cache level, that is for the on-heap cache. Most people don’t need an on-heap cache.
You you define eviction at a data region level, that works on the off-heap data. As you observe, it works at a page rather than a record level. (This is why a tiny region can be problematic. There are far few pages that can be chosen to evict.)
> On 30 Jan 2023, at 11:37, Łukasz Dywicki <lu...@code-house.org> wrote:
>
> Dear Jeremy and Stephen,
> Thank you for answers. I this issues I face is not bound exclusively to assigned memory but behavior of eviction policies. After all, without proper eviction or expiry policy, each amount of memory will eventually be exceeded. Small size of memory makes it simply easier to test. To be honest I do not see a point in testing eviction with 50GB of memory, if it cant be configured for 512MB. Expiry policy proved to work, but it will not preventing node from crash, as there always might be high write ratio causing memory to fill.
>
> Both LRU and FIFO eviction policies in default configuration do not take into account overall memory allocation, they work only with on heap caches. Unless memory size is explicitly specified they will rely on over-provisioned memory, or assume that memory assignment is right. This is possible only if cache entry have a predictable.
> Another point is that above eviction policies are dedicated to on-heap caches and are told to have no affect to off-heap region, why is so? By looking at sources I see CacheOffheapEvictionManager and GridCacheEvictionManager, but I can't grasp what enables eviction of entries from off heap memory managed by Ignite.
> Given that my use case is continuous operation with variety of entries having various sizes I can't assume on-heap cache with LRU nor FIFO, especially that they do not impact off heap memory. I also can't risk filling in memory managed by Ignite due to lack of clear eviction policy during peak load.
> As far I can understand from DataRegionConfiguration, its eviction mechanism works for whole pages, but I haven't seen this happening. Could it be due to fact that single cache entry in my test exceeds maximum page size (16MB)?
>
> Best regards,
> Łukasz
>
> On 30.01.2023 10:32, Stephen Darlington wrote:
>> Ignite is designed to work as a cluster with a lot of memory. A single node with 512Mb of memory just isn’t what most people are testing with. I’ve seen similar issues when people use tiny nodes. I don’t see anything immediately wrong with your configuration, but if it works with more memory, that’s your problem.
>>> On 28 Jan 2023, at 00:35, Łukasz Dywicki <lu...@code-house.org> wrote:
>>>
>>> Hello again,
>>> I've spent another day on this issue looking at configuration and with cluster larger than configured backup count ignite did the work.
>>> After that I reverted to testing of single node with minimal configuration and come to the point that the only way to keep Ignite survive load was setting ExpiryPolicy to very low value (10s).
>>>
>>> To me it seems that Ignite behavior is to preserve all entries in memory at any cost, even if there is a risk of running into OOM. Is that true?
>>> I would like to change this behavior and make sure that entries do not expire because of time as long as there is memory available to store them. They should be evicted by appropriate EvictionPolicy only if memory fills up.
>>> To me it looks like DataStoreSettings do not make any impact in this regard. At least setting page eviction to LRU do not change it.
>>>
>>> Please let me know if I am doing something wrong as I can not prove Ignite to be working stable. Even with such basic objectives I outlined in earlier mail.
>>>
>>> Kind regards,
>>> Łukasz
>>>
>>> On 27.01.2023 00:58, Łukasz Dywicki wrote:
>>>> Dear all,
>>>> I come across use of Apache Ignite to cache results of expensive computation operation.
>>>> Objectives are basic:
>>>> - Keep most of "hot" data in memory
>>>> - Offload cold part to cache store
>>>> - Keep memory utilization under control (evict entries as needed)
>>>> While it sounds basic, it doesn't seem to fit Ignite defaults.
>>>> What I am testing now is behavior with large objects which can grow up to 10 mb (serialized) or 25 mb (json representation). Usually objects will stay far below that threshold, but we can't make assumption on that.
>>>> I began testing various configurations of Ignite in order to facilitate offloading of memory contents to database. So far I am stuck for two days at Ignite/application itself running out of memory after processing several of such large objects. While I know that storing 10 mb blob in database is not the best idea, I have to test that behavior too.
>>>> By observing database contents I see that number of entries there grows, but cache do not seem to be evicted. When I try to switch eviction, it does require onheap to be switched on, and it still fails with LRU eviction policy.
>>>> So far I ended up with a named cache and default region configured as below:
>>>> ```
>>>> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>>>> igniteConfiguration.setDataStorageConfiguration(new DataStorageConfiguration()
>>>> .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
>>>> .setPersistenceEnabled(false)
>>>> .setInitialSize(256L * 1024 * 1024)
>>>> .setMaxSize(512L * 1024 * 1024)
>>>> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
>>>> .setSwapPath(null)
>>>> .setEvictionThreshold(0.75)
>>>> )
>>>> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
>>>> );
>>>> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new CacheConfiguration<>()
>>>> .setName(CACHE_NAME)
>>>> .setBackups(2)
>>>> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
>>>> .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
>>>> .setWriteThrough(true)
>>>> .setOnheapCacheEnabled(true)
>>>> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
>>>> .setReadThrough(true);
>>>> igniteConfiguration.setCacheConfiguration(
>>>> expensiveCache
>>>> );
>>>> ```
>>>> What I observe is following - the cache keeps writing data into database, but it does not remove old entries fast enough to prevent crash.
>>>> JVM parameters I use are fairly basic:
>>>> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>>>> The store mechanism is jdbc blob store. Exceptions I get happen to occur in Ignite itself, processing (application code writing cache) or communication thread used to feed cache. I collected one case here:
>>>> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
>>>> The error message in linked gist advice to enable persistence (which I did via cache store!), increase memory limit (which I don't want to do), or enable eviction/expiry policy (which somehow miss behave).
>>>> To me it looks like self defense mechanisms Ignite has are being tricked leading whole application to crash.
>>>> Can you please advise me which settings to tune and how in order to get Ignite more stable under such load?
>>>> Kind regards,
>>>> Łukasz
Re: Safe settings for ignite cache with external store
Posted by Łukasz Dywicki <lu...@code-house.org>.
Dear Jeremy and Stephen,
Thank you for answers. I this issues I face is not bound exclusively to
assigned memory but behavior of eviction policies. After all, without
proper eviction or expiry policy, each amount of memory will eventually
be exceeded. Small size of memory makes it simply easier to test. To be
honest I do not see a point in testing eviction with 50GB of memory, if
it cant be configured for 512MB. Expiry policy proved to work, but it
will not preventing node from crash, as there always might be high write
ratio causing memory to fill.
Both LRU and FIFO eviction policies in default configuration do not take
into account overall memory allocation, they work only with on heap
caches. Unless memory size is explicitly specified they will rely on
over-provisioned memory, or assume that memory assignment is right. This
is possible only if cache entry have a predictable.
Another point is that above eviction policies are dedicated to on-heap
caches and are told to have no affect to off-heap region, why is so? By
looking at sources I see CacheOffheapEvictionManager and
GridCacheEvictionManager, but I can't grasp what enables eviction of
entries from off heap memory managed by Ignite.
Given that my use case is continuous operation with variety of entries
having various sizes I can't assume on-heap cache with LRU nor FIFO,
especially that they do not impact off heap memory. I also can't risk
filling in memory managed by Ignite due to lack of clear eviction policy
during peak load.
As far I can understand from DataRegionConfiguration, its eviction
mechanism works for whole pages, but I haven't seen this happening.
Could it be due to fact that single cache entry in my test exceeds
maximum page size (16MB)?
Best regards,
Łukasz
On 30.01.2023 10:32, Stephen Darlington wrote:
> Ignite is designed to work as a cluster with a lot of memory. A single node with 512Mb of memory just isn’t what most people are testing with. I’ve seen similar issues when people use tiny nodes. I don’t see anything immediately wrong with your configuration, but if it works with more memory, that’s your problem.
>
>> On 28 Jan 2023, at 00:35, Łukasz Dywicki <lu...@code-house.org> wrote:
>>
>> Hello again,
>> I've spent another day on this issue looking at configuration and with cluster larger than configured backup count ignite did the work.
>> After that I reverted to testing of single node with minimal configuration and come to the point that the only way to keep Ignite survive load was setting ExpiryPolicy to very low value (10s).
>>
>> To me it seems that Ignite behavior is to preserve all entries in memory at any cost, even if there is a risk of running into OOM. Is that true?
>> I would like to change this behavior and make sure that entries do not expire because of time as long as there is memory available to store them. They should be evicted by appropriate EvictionPolicy only if memory fills up.
>> To me it looks like DataStoreSettings do not make any impact in this regard. At least setting page eviction to LRU do not change it.
>>
>> Please let me know if I am doing something wrong as I can not prove Ignite to be working stable. Even with such basic objectives I outlined in earlier mail.
>>
>> Kind regards,
>> Łukasz
>>
>> On 27.01.2023 00:58, Łukasz Dywicki wrote:
>>> Dear all,
>>> I come across use of Apache Ignite to cache results of expensive computation operation.
>>> Objectives are basic:
>>> - Keep most of "hot" data in memory
>>> - Offload cold part to cache store
>>> - Keep memory utilization under control (evict entries as needed)
>>> While it sounds basic, it doesn't seem to fit Ignite defaults.
>>> What I am testing now is behavior with large objects which can grow up to 10 mb (serialized) or 25 mb (json representation). Usually objects will stay far below that threshold, but we can't make assumption on that.
>>> I began testing various configurations of Ignite in order to facilitate offloading of memory contents to database. So far I am stuck for two days at Ignite/application itself running out of memory after processing several of such large objects. While I know that storing 10 mb blob in database is not the best idea, I have to test that behavior too.
>>> By observing database contents I see that number of entries there grows, but cache do not seem to be evicted. When I try to switch eviction, it does require onheap to be switched on, and it still fails with LRU eviction policy.
>>> So far I ended up with a named cache and default region configured as below:
>>> ```
>>> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>>> igniteConfiguration.setDataStorageConfiguration(new DataStorageConfiguration()
>>> .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
>>> .setPersistenceEnabled(false)
>>> .setInitialSize(256L * 1024 * 1024)
>>> .setMaxSize(512L * 1024 * 1024)
>>> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
>>> .setSwapPath(null)
>>> .setEvictionThreshold(0.75)
>>> )
>>> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
>>> );
>>> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new CacheConfiguration<>()
>>> .setName(CACHE_NAME)
>>> .setBackups(2)
>>> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
>>> .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
>>> .setWriteThrough(true)
>>> .setOnheapCacheEnabled(true)
>>> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
>>> .setReadThrough(true);
>>> igniteConfiguration.setCacheConfiguration(
>>> expensiveCache
>>> );
>>> ```
>>> What I observe is following - the cache keeps writing data into database, but it does not remove old entries fast enough to prevent crash.
>>> JVM parameters I use are fairly basic:
>>> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>>> The store mechanism is jdbc blob store. Exceptions I get happen to occur in Ignite itself, processing (application code writing cache) or communication thread used to feed cache. I collected one case here:
>>> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
>>> The error message in linked gist advice to enable persistence (which I did via cache store!), increase memory limit (which I don't want to do), or enable eviction/expiry policy (which somehow miss behave).
>>> To me it looks like self defense mechanisms Ignite has are being tricked leading whole application to crash.
>>> Can you please advise me which settings to tune and how in order to get Ignite more stable under such load?
>>> Kind regards,
>>> Łukasz
>
Re: Safe settings for ignite cache with external store
Posted by Stephen Darlington <st...@gridgain.com>.
Ignite is designed to work as a cluster with a lot of memory. A single node with 512Mb of memory just isn’t what most people are testing with. I’ve seen similar issues when people use tiny nodes. I don’t see anything immediately wrong with your configuration, but if it works with more memory, that’s your problem.
> On 28 Jan 2023, at 00:35, Łukasz Dywicki <lu...@code-house.org> wrote:
>
> Hello again,
> I've spent another day on this issue looking at configuration and with cluster larger than configured backup count ignite did the work.
> After that I reverted to testing of single node with minimal configuration and come to the point that the only way to keep Ignite survive load was setting ExpiryPolicy to very low value (10s).
>
> To me it seems that Ignite behavior is to preserve all entries in memory at any cost, even if there is a risk of running into OOM. Is that true?
> I would like to change this behavior and make sure that entries do not expire because of time as long as there is memory available to store them. They should be evicted by appropriate EvictionPolicy only if memory fills up.
> To me it looks like DataStoreSettings do not make any impact in this regard. At least setting page eviction to LRU do not change it.
>
> Please let me know if I am doing something wrong as I can not prove Ignite to be working stable. Even with such basic objectives I outlined in earlier mail.
>
> Kind regards,
> Łukasz
>
> On 27.01.2023 00:58, Łukasz Dywicki wrote:
>> Dear all,
>> I come across use of Apache Ignite to cache results of expensive computation operation.
>> Objectives are basic:
>> - Keep most of "hot" data in memory
>> - Offload cold part to cache store
>> - Keep memory utilization under control (evict entries as needed)
>> While it sounds basic, it doesn't seem to fit Ignite defaults.
>> What I am testing now is behavior with large objects which can grow up to 10 mb (serialized) or 25 mb (json representation). Usually objects will stay far below that threshold, but we can't make assumption on that.
>> I began testing various configurations of Ignite in order to facilitate offloading of memory contents to database. So far I am stuck for two days at Ignite/application itself running out of memory after processing several of such large objects. While I know that storing 10 mb blob in database is not the best idea, I have to test that behavior too.
>> By observing database contents I see that number of entries there grows, but cache do not seem to be evicted. When I try to switch eviction, it does require onheap to be switched on, and it still fails with LRU eviction policy.
>> So far I ended up with a named cache and default region configured as below:
>> ```
>> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>> igniteConfiguration.setDataStorageConfiguration(new DataStorageConfiguration()
>> .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
>> .setPersistenceEnabled(false)
>> .setInitialSize(256L * 1024 * 1024)
>> .setMaxSize(512L * 1024 * 1024)
>> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
>> .setSwapPath(null)
>> .setEvictionThreshold(0.75)
>> )
>> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
>> );
>> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new CacheConfiguration<>()
>> .setName(CACHE_NAME)
>> .setBackups(2)
>> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
>> .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
>> .setWriteThrough(true)
>> .setOnheapCacheEnabled(true)
>> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
>> .setReadThrough(true);
>> igniteConfiguration.setCacheConfiguration(
>> expensiveCache
>> );
>> ```
>> What I observe is following - the cache keeps writing data into database, but it does not remove old entries fast enough to prevent crash.
>> JVM parameters I use are fairly basic:
>> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>> The store mechanism is jdbc blob store. Exceptions I get happen to occur in Ignite itself, processing (application code writing cache) or communication thread used to feed cache. I collected one case here:
>> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
>> The error message in linked gist advice to enable persistence (which I did via cache store!), increase memory limit (which I don't want to do), or enable eviction/expiry policy (which somehow miss behave).
>> To me it looks like self defense mechanisms Ignite has are being tricked leading whole application to crash.
>> Can you please advise me which settings to tune and how in order to get Ignite more stable under such load?
>> Kind regards,
>> Łukasz
Re: Safe settings for ignite cache with external store
Posted by Jeremy McMillan <je...@gridgain.com>.
If your problem is simple and popular, then solutions will chase you.
Consider maybe you might be trying to do something really challenging that
doesn't have any off the shelf solutions.
You might need to approach this as a computer scientist and search for the
best fit, not just best available eviction strategy. I hope you find some
clues, and learn your way to success.
This may be of interest to you.
https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/eviction/CustomEvictionPolicyExample.java
On Fri, Jan 27, 2023, 18:36 Łukasz Dywicki <lu...@code-house.org> wrote:
> Hello again,
> I've spent another day on this issue looking at configuration and with
> cluster larger than configured backup count ignite did the work.
> After that I reverted to testing of single node with minimal
> configuration and come to the point that the only way to keep Ignite
> survive load was setting ExpiryPolicy to very low value (10s).
>
> To me it seems that Ignite behavior is to preserve all entries in memory
> at any cost, even if there is a risk of running into OOM. Is that true?
> I would like to change this behavior and make sure that entries do not
> expire because of time as long as there is memory available to store
> them. They should be evicted by appropriate EvictionPolicy only if
> memory fills up.
> To me it looks like DataStoreSettings do not make any impact in this
> regard. At least setting page eviction to LRU do not change it.
>
> Please let me know if I am doing something wrong as I can not prove
> Ignite to be working stable. Even with such basic objectives I outlined
> in earlier mail.
>
> Kind regards,
> Łukasz
>
> On 27.01.2023 00:58, Łukasz Dywicki wrote:
> > Dear all,
> > I come across use of Apache Ignite to cache results of expensive
> > computation operation.
> > Objectives are basic:
> > - Keep most of "hot" data in memory
> > - Offload cold part to cache store
> > - Keep memory utilization under control (evict entries as needed)
> > While it sounds basic, it doesn't seem to fit Ignite defaults.
> >
> > What I am testing now is behavior with large objects which can grow up
> > to 10 mb (serialized) or 25 mb (json representation). Usually objects
> > will stay far below that threshold, but we can't make assumption on that.
> > I began testing various configurations of Ignite in order to facilitate
> > offloading of memory contents to database. So far I am stuck for two
> > days at Ignite/application itself running out of memory after processing
> > several of such large objects. While I know that storing 10 mb blob in
> > database is not the best idea, I have to test that behavior too.
> >
> > By observing database contents I see that number of entries there grows,
> > but cache do not seem to be evicted. When I try to switch eviction, it
> > does require onheap to be switched on, and it still fails with LRU
> > eviction policy.
> >
> > So far I ended up with a named cache and default region configured as
> > below:
> > ```
> > IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> > igniteConfiguration.setDataStorageConfiguration(new
> > DataStorageConfiguration()
> > .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
> > .setPersistenceEnabled(false)
> > .setInitialSize(256L * 1024 * 1024)
> > .setMaxSize(512L * 1024 * 1024)
> > .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
> > .setSwapPath(null)
> > .setEvictionThreshold(0.75)
> > )
> > .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
> > );
> > CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new
> > CacheConfiguration<>()
> > .setName(CACHE_NAME)
> > .setBackups(2)
> > .setAtomicityMode(CacheAtomicityMode.ATOMIC)
> > .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
> > .setWriteThrough(true)
> > .setOnheapCacheEnabled(true)
> > .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
> > .setReadThrough(true);
> > igniteConfiguration.setCacheConfiguration(
> > expensiveCache
> > );
> > ```
> >
> > What I observe is following - the cache keeps writing data into
> > database, but it does not remove old entries fast enough to prevent
> crash.
> > JVM parameters I use are fairly basic:
> > -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
> > -XX:+DisableExplicitGC
> >
> > The store mechanism is jdbc blob store. Exceptions I get happen to occur
> > in Ignite itself, processing (application code writing cache) or
> > communication thread used to feed cache. I collected one case here:
> > https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
> >
> > The error message in linked gist advice to enable persistence (which I
> > did via cache store!), increase memory limit (which I don't want to do),
> > or enable eviction/expiry policy (which somehow miss behave).
> > To me it looks like self defense mechanisms Ignite has are being tricked
> > leading whole application to crash.
> >
> > Can you please advise me which settings to tune and how in order to get
> > Ignite more stable under such load?
> >
> > Kind regards,
> > Łukasz
> >
> >
> >
>
Re: Safe settings for ignite cache with external store
Posted by Łukasz Dywicki <lu...@code-house.org>.
Hello again,
I've spent another day on this issue looking at configuration and with
cluster larger than configured backup count ignite did the work.
After that I reverted to testing of single node with minimal
configuration and come to the point that the only way to keep Ignite
survive load was setting ExpiryPolicy to very low value (10s).
To me it seems that Ignite behavior is to preserve all entries in memory
at any cost, even if there is a risk of running into OOM. Is that true?
I would like to change this behavior and make sure that entries do not
expire because of time as long as there is memory available to store
them. They should be evicted by appropriate EvictionPolicy only if
memory fills up.
To me it looks like DataStoreSettings do not make any impact in this
regard. At least setting page eviction to LRU do not change it.
Please let me know if I am doing something wrong as I can not prove
Ignite to be working stable. Even with such basic objectives I outlined
in earlier mail.
Kind regards,
Łukasz
On 27.01.2023 00:58, Łukasz Dywicki wrote:
> Dear all,
> I come across use of Apache Ignite to cache results of expensive
> computation operation.
> Objectives are basic:
> - Keep most of "hot" data in memory
> - Offload cold part to cache store
> - Keep memory utilization under control (evict entries as needed)
> While it sounds basic, it doesn't seem to fit Ignite defaults.
>
> What I am testing now is behavior with large objects which can grow up
> to 10 mb (serialized) or 25 mb (json representation). Usually objects
> will stay far below that threshold, but we can't make assumption on that.
> I began testing various configurations of Ignite in order to facilitate
> offloading of memory contents to database. So far I am stuck for two
> days at Ignite/application itself running out of memory after processing
> several of such large objects. While I know that storing 10 mb blob in
> database is not the best idea, I have to test that behavior too.
>
> By observing database contents I see that number of entries there grows,
> but cache do not seem to be evicted. When I try to switch eviction, it
> does require onheap to be switched on, and it still fails with LRU
> eviction policy.
>
> So far I ended up with a named cache and default region configured as
> below:
> ```
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> igniteConfiguration.setDataStorageConfiguration(new
> DataStorageConfiguration()
> .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
> .setPersistenceEnabled(false)
> .setInitialSize(256L * 1024 * 1024)
> .setMaxSize(512L * 1024 * 1024)
> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
> .setSwapPath(null)
> .setEvictionThreshold(0.75)
> )
> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
> );
> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new
> CacheConfiguration<>()
> .setName(CACHE_NAME)
> .setBackups(2)
> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
> .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
> .setWriteThrough(true)
> .setOnheapCacheEnabled(true)
> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
> .setReadThrough(true);
> igniteConfiguration.setCacheConfiguration(
> expensiveCache
> );
> ```
>
> What I observe is following - the cache keeps writing data into
> database, but it does not remove old entries fast enough to prevent crash.
> JVM parameters I use are fairly basic:
> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
> -XX:+DisableExplicitGC
>
> The store mechanism is jdbc blob store. Exceptions I get happen to occur
> in Ignite itself, processing (application code writing cache) or
> communication thread used to feed cache. I collected one case here:
> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
>
> The error message in linked gist advice to enable persistence (which I
> did via cache store!), increase memory limit (which I don't want to do),
> or enable eviction/expiry policy (which somehow miss behave).
> To me it looks like self defense mechanisms Ignite has are being tricked
> leading whole application to crash.
>
> Can you please advise me which settings to tune and how in order to get
> Ignite more stable under such load?
>
> Kind regards,
> Łukasz
>
>
>