You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by userx <ga...@gmail.com> on 2017/08/29 16:00:26 UTC
Limiting the size of Persistent Store and clearing data on restart
Hi All,
1) Like the conventional configuration of
https://apacheignite.readme.io/docs/memory-configuration#section-memory-policies
where in we could limit the size of the memory, how do we define a
configuration which can limit the size of Persistent Store so that say out
of a 500GB HDisk, I don't want this Ignite related data to go beyond 300GB ?
2) Also, is there a configuration such that when Ignite data grid servers
are restarted (say some code change either on client or server side),
whatever was persisted before gets wiped out completely on all the
participating nodes and only fresh persisted data is there ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Limiting-the-size-of-Persistent-Store-and-clearing-data-on-restart-tp16507.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Limiting the size of Persistent Store and clearing data on restart
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Sat, Sep 2, 2017 at 5:11 AM, userx <ga...@gmail.com> wrote:
> Hi all,
>
> Regarding question 2, let me put it this way.
>
> Like ignite has eviction policy for RAM described at
> https://apacheignite.readme.io/docs/evictions
> Is there an eviction policy for a persistent store ? Say for a use case
> where in I cannot allocate a dedicated 1 or 2 tb space (like we do for
> relational dbms), I would want to define an upper limit of say 300 GB and
> if
> $IGNITE_HOME/PersistentStore goes beyond the same same, evict the lru
> cache.
>
>
Evictions for RAM make sense because the data is still on disk and is not
lost. Evictions from disk do not make sense for me, nor am I aware of other
databases that have this feature. I believe it should be handled by users
based on the application requirements.
I think the Ignite community may agree to consider this feature as a
possible enhancement. Can you provide an example of a similar feature you
may have seen in another persistent store or a database?
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Limiting the size of Persistent Store and clearing data on
restart
Posted by userx <ga...@gmail.com>.
Hi all,
Regarding question 2, let me put it this way.
Like ignite has eviction policy for RAM described at
https://apacheignite.readme.io/docs/evictions
Is there an eviction policy for a persistent store ? Say for a use case
where in I cannot allocate a dedicated 1 or 2 tb space (like we do for
relational dbms), I would want to define an upper limit of say 300 GB and if
$IGNITE_HOME/PersistentStore goes beyond the same same, evict the lru cache.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Fwd: Limiting the size of Persistent Store and clearing data on restart
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Cross sending to dev@.
Does anyone know why we still have swapEnabled property in master?
D.
---------- Forwarded message ----------
From: Denis Mekhanikov <dm...@gmail.com>
Date: Fri, Sep 1, 2017 at 9:51 AM
Subject: Re: Limiting the size of Persistent Store and clearing data on
restart
To: user@ignite.apache.org
Sorry, I didn't know that. I found a bunch of matches by "swapEnabled"
keyword on master branch in the project, mostly in benchmarks
configurations. These properties should probably be removed not to confuse
readers.
пт, 1 сент. 2017 г. в 19:23, Dmitriy Setrakyan <ds...@apache.org>:
> On Fri, Sep 1, 2017 at 1:41 PM, Denis Mekhanikov <dm...@gmail.com>
> wrote:
>
>> Regarding your second question: looks like you don't actually need
>> persistence. Its purpose is the opposite: to save cache data between
>> restarts.
>> If you use persistence to store more data than RAM available, then you
>> can enable swap space: https://apacheignite.readme.io/v1.9/docs/off-heap-
>> memory#section-swap-space
>>
>
> Denis, starting with 2.1, Ignite does not have the swap space anymore,
> since it never worked well and kept hurting the performance. We recommend
> now that users do use persistence whenever the data size is bigger than the
> available memory size.
>
Re: Limiting the size of Persistent Store and clearing data on restart
Posted by Denis Mekhanikov <dm...@gmail.com>.
Sorry, I didn't know that. I found a bunch of matches by "swapEnabled"
keyword on master branch in the project, mostly in benchmarks
configurations. These properties should probably be removed not to confuse
readers.
пт, 1 сент. 2017 г. в 19:23, Dmitriy Setrakyan <ds...@apache.org>:
> On Fri, Sep 1, 2017 at 1:41 PM, Denis Mekhanikov <dm...@gmail.com>
> wrote:
>
>> Regarding your second question: looks like you don't actually need
>> persistence. Its purpose is the opposite: to save cache data between
>> restarts.
>> If you use persistence to store more data than RAM available, then you
>> can enable swap space:
>> https://apacheignite.readme.io/v1.9/docs/off-heap-memory#section-swap-space
>>
>
> Denis, starting with 2.1, Ignite does not have the swap space anymore,
> since it never worked well and kept hurting the performance. We recommend
> now that users do use persistence whenever the data size is bigger than the
> available memory size.
>
Re: Limiting the size of Persistent Store and clearing data on restart
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, Sep 1, 2017 at 1:41 PM, Denis Mekhanikov <dm...@gmail.com>
wrote:
> Regarding your second question: looks like you don't actually need
> persistence. Its purpose is the opposite: to save cache data between
> restarts.
> If you use persistence to store more data than RAM available, then you can
> enable swap space: https://apacheignite.readme.io/v1.9/docs/off-heap-mem
> ory#section-swap-space
>
Denis, starting with 2.1, Ignite does not have the swap space anymore,
since it never worked well and kept hurting the performance. We recommend
now that users do use persistence whenever the data size is bigger than the
available memory size.
Re: Limiting the size of Persistent Store and clearing data on restart
Posted by Denis Mekhanikov <dm...@gmail.com>.
Regarding your second question: looks like you don't actually need
persistence. Its purpose is the opposite: to save cache data between
restarts.
If you use persistence to store more data than RAM available, then you can
enable swap space:
https://apacheignite.readme.io/v1.9/docs/off-heap-memory#section-swap-space
вт, 29 авг. 2017 г. в 20:06, ezhuravlev <e....@gmail.com>:
> Hi,
>
> 1) There is no possibility to limit Persistent Store size right now, but
> you
> always can use separate disk volume with size 300GB for your case to use it
> for Persistent Store.
>
> 2) Well it's not usual to use case, but after stopping node you can clear
> data from work folder or just invoke removeAll on cache
>
> Evgenii
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Limiting-the-size-of-Persistent-Store-and-clearing-data-on-restart-tp16507p16515.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Limiting the size of Persistent Store and clearing data on
restart
Posted by ezhuravlev <e....@gmail.com>.
Hi,
1) There is no possibility to limit Persistent Store size right now, but you
always can use separate disk volume with size 300GB for your case to use it
for Persistent Store.
2) Well it's not usual to use case, but after stopping node you can clear
data from work folder or just invoke removeAll on cache
Evgenii
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Limiting-the-size-of-Persistent-Store-and-clearing-data-on-restart-tp16507p16515.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Limiting the size of Persistent Store and clearing data on restart
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Tue, Aug 29, 2017 at 7:00 PM, userx <ga...@gmail.com> wrote:
> Hi All,
>
> 1) Like the conventional configuration of
> https://apacheignite.readme.io/docs/memory-configuration#
> section-memory-policies
> where in we could limit the size of the memory, how do we define a
> configuration which can limit the size of Persistent Store so that say out
> of a 500GB HDisk, I don't want this Ignite related data to go beyond 300GB
> ?
>
You would have to control your data on disk by yourself. Evicting from disk
is very sensitive and use case specific, so it would be almost impossible
to automate it.
>
> 2) Also, is there a configuration such that when Ignite data grid servers
> are restarted (say some code change either on client or server side),
> whatever was persisted before gets wiped out completely on all the
> participating nodes and only fresh persisted data is there ?
>
How about calling destroyCache(...) on startup?