You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by John Wilson <sa...@gmail.com> on 2017/09/01 23:12:16 UTC
Quick questions on Evictions
Hi,
I have been reading through Ignite doc and I still have these questions. I
appreciate your answer.
Assume my Ignite native persistence is *not *enabled:
1. if on-heap cache is also not enabled, then there are no entry-based
evictions, right?
2. if on-heap cache is now enabled, does a write to on-heap cache also
results in a write-through or write-behind behavior to the off-heap entry?
3. If on-heap cache is not enabled but data page eviction mode is
enabled, then where do evicted pages from off-heap go/written to?
and, need confirmation on how data page eviction is implemented:
4. when a data page eviction is initiated, Ignite works by iterating
through each entry in the page and evicting entries one by one. It may
happen that certain entries may be involved in active transactions and
hence certain entries may not be evicted at all.
Thanks,
Re: Quick questions on Evictions
Posted by Dmitry Karachentsev <dk...@gridgain.com>.
Hi,
Assume you have disabled onheapCahe and disabled persistence. In that
case you may configure only datapage eviction mode, then outdated pages
will be thrown away, when no free memory will be available for Ignite.
Also you cannot configure per-entry eviction.
OK, if you enable onheapCache, then Ignite will store on heap every
entry that was read from off-heap (or disk). Next reads of it will not
require off-heap readings, and every update will write to off-heap. To
limit size of onheapCache you may set
CacheConfiguration.setEvictionPolicy(), but it will not evict off-heap
entries.
So, off-heap eviction may be controlled with DataPageEvictionMode only,
and as you suggested, it clears entries one-by-one from page, checking
for current locks (transaction locks as well). If entry is locked, it
won't be evicted.
Thanks!
-Dmitry.
02.09.2017 02:12, John Wilson пишет:
> Hi,
>
> I have been reading through Ignite doc and I still have these
> questions. I appreciate your answer.
>
> Assume my Ignite native persistence is *not *enabled:
>
> 1. if on-heap cache is also not enabled, then there are no
> entry-based evictions, right?
> 2. if on-heap cache is now enabled, does a write to on-heap cache
> also results in a write-through or write-behind behavior to the
> off-heap entry?
> 3. If on-heap cache is not enabled but data page eviction mode is
> enabled, then where do evicted pages from off-heap go/written to?
>
> and, need confirmation on how data page eviction is implemented:
>
> 4. when a data page eviction is initiated, Ignite works by iterating
> through each entry in the page and evicting entries one by one. It may
> happen that certain entries may be involved in active transactions and
> hence certain entries may not be evicted at all.
>
>
> Thanks,
Re: Quick questions on Evictions
Posted by Andrey Mashenkov <an...@gmail.com>.
Yes, you are right.
On Wed, Jun 20, 2018 at 4:13 PM the_palakkaran <ji...@suntecsbs.com> wrote:
> So to conclude, if I have enabled on heap storage for cache(using
> cache.setOnHeapEnabled(true),
> then :
> 1. Still data will be stored off heap, but will be loaded to heap. To
> escape
> out of memory error, I have to set eviction policies.
> 2. Off heap entries will be written to disk based on data page eviction
> mode
> enabled on data region. This size is limited based on initial and maximum
> size of data region.
>
> I hope this is how it works. Big thanks !!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Quick questions on Evictions
Posted by the_palakkaran <ji...@suntecsbs.com>.
So to conclude, if I have enabled on heap storage for cache(using
cache.setOnHeapEnabled(true),
then :
1. Still data will be stored off heap, but will be loaded to heap. To escape
out of memory error, I have to set eviction policies.
2. Off heap entries will be written to disk based on data page eviction mode
enabled on data region. This size is limited based on initial and maximum
size of data region.
I hope this is how it works. Big thanks !!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Quick questions on Evictions
Posted by Andrey Mashenkov <an...@gmail.com>.
DataRegionConfiguration.setMaxSize() should be used to limit offheap memory
usage.
On Tue, Jun 19, 2018 at 2:35 PM the_palakkaran <ji...@suntecsbs.com> wrote:
> So how do I limit cache size if ignite native persistence is enabled using
> dataRegionCfg.setPersistenceEnabled(true)? I don't want it to keep a lot of
> data in memory and others may be kept on disk. That is the requirement.
>
> Also, I do have on heap cache enabled. But I read in many threads that
> Ignite stores everything off heap and only references may be kept in java
> heap. So how do I limit off heap cache usage? I mean I don't want to it to
> use more than a particular amount of off heap memory?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Quick questions on Evictions
Posted by the_palakkaran <ji...@suntecsbs.com>.
So how do I limit cache size if ignite native persistence is enabled using
dataRegionCfg.setPersistenceEnabled(true)? I don't want it to keep a lot of
data in memory and others may be kept on disk. That is the requirement.
Also, I do have on heap cache enabled. But I read in many threads that
Ignite stores everything off heap and only references may be kept in java
heap. So how do I limit off heap cache usage? I mean I don't want to it to
use more than a particular amount of off heap memory?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Quick questions on Evictions
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
DataPageEvictionMode is about algorithm of choosing page to be replaced.
EvictionPolicy is what you are looking for. E.g. FifoEvictionPolicy or
LruEvictionPolicy.
It looks like EvictionPolicies can't be used with persistence as all of
them uses non-persistent structures to track cache entries.
On Tue, Jun 19, 2018 at 8:30 AM the_palakkaran <ji...@suntecsbs.com> wrote:
> Hi,
>
> DataPageEvictionMode is deprecated now, right? What should I do to evict my
> off heap entries? Also, can I limit off heap memory usage?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Quick questions on Evictions
Posted by the_palakkaran <ji...@suntecsbs.com>.
Hi,
DataPageEvictionMode is deprecated now, right? What should I do to evict my
off heap entries? Also, can I limit off heap memory usage?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Quick questions on Evictions
Posted by Dmitry Karachentsev <dk...@gridgain.com>.
Hi John,
1. Actually it is. By default data page eviction is disabled
(DataPageEvictionMode.DISABLED) and when no memory left, it will throw
IgniteOutOfMemoryException.
2. If you have enabled persistence - no data will be lost, dirty pages
will be written do disk. In other words Ignite starts swapping pages
between memory and drive using RANDOM_LRU eviction mode.
3. Ignite 2.0+ works only with pages, so you cannot set eviction per
entry. If page was modified - it will be marked as dirty and fully
written to disk.
Thanks!
-Dmitry.
04.09.2017 21:33, John Wilson пишет:
> I appreciate the nice explanation. I got a few more questions:
>
> 1. For the case where on-heap caching and persistent are both
> disabled, why does Ignite throw out out-dated pages from off-heap?
> Why not throw OOM error since the out-dated pages are not backed
> by persistent store and throwing away results in data loss?
> 2. For off-heap eviction with persistent store enabled, will entries
> evicted from data pages be written to disk (in case they are
> dirty) or will they be thrown away (which would imply that entries
> eligible for eviction must be clean and have already been written
> to disk by checkpointing)?
> 3. Checkpointing works by locating dirty pages and writing them out.
> If a single entry in a data page is dirty (has been updated since
> the last check pointing), will checkpointing write the entire data
> page (all entries) to the partition files or just the dirty entry?
>
> Thanks!
>
> On Mon, Sep 4, 2017 at 8:17 AM, dkarachentsev
> <dkarachentsev@gridgain.com <ma...@gridgain.com>> wrote:
>
> Hi,
>
> Assume you have disabled onheapCahe and disabled persistence. In
> that case
> you may configure only datapage eviction mode, then outdated pages
> will be
> thrown away, when no free memory will be available for Ignite.
> Also you
> cannot configure per-entry eviction.
>
> OK, if you enable onheapCache, then Ignite will store on heap
> every entry
> that was read from off-heap (or disk). Next reads of it will not
> require
> off-heap readings, and every update will write to off-heap. To
> limit size of
> onheapCache you may set CacheConfiguration.setEvictionPolicy(),
> but it will
> not evict off-heap entries.
>
> So, off-heap eviction may be controlled with DataPageEvictionMode
> only, and
> as you suggested, it clears entries one-by-one from page, checking for
> current locks (transaction locks as well). If entry is locked, it
> won't be
> evicted.
>
> Thanks!
> -Dmitry.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> <http://apache-ignite-users.70518.x6.nabble.com/>
>
>
Re: Quick questions on Evictions
Posted by John Wilson <sa...@gmail.com>.
I appreciate the nice explanation. I got a few more questions:
1. For the case where on-heap caching and persistent are both disabled,
why does Ignite throw out out-dated pages from off-heap? Why not throw OOM
error since the out-dated pages are not backed by persistent store and
throwing away results in data loss?
2. For off-heap eviction with persistent store enabled, will entries
evicted from data pages be written to disk (in case they are dirty) or will
they be thrown away (which would imply that entries eligible for eviction
must be clean and have already been written to disk by checkpointing)?
3. Checkpointing works by locating dirty pages and writing them out. If
a single entry in a data page is dirty (has been updated since the last
check pointing), will checkpointing write the entire data page (all
entries) to the partition files or just the dirty entry?
Thanks!
On Mon, Sep 4, 2017 at 8:17 AM, dkarachentsev <dk...@gridgain.com>
wrote:
> Hi,
>
> Assume you have disabled onheapCahe and disabled persistence. In that case
> you may configure only datapage eviction mode, then outdated pages will be
> thrown away, when no free memory will be available for Ignite. Also you
> cannot configure per-entry eviction.
>
> OK, if you enable onheapCache, then Ignite will store on heap every entry
> that was read from off-heap (or disk). Next reads of it will not require
> off-heap readings, and every update will write to off-heap. To limit size
> of
> onheapCache you may set CacheConfiguration.setEvictionPolicy(), but it
> will
> not evict off-heap entries.
>
> So, off-heap eviction may be controlled with DataPageEvictionMode only, and
> as you suggested, it clears entries one-by-one from page, checking for
> current locks (transaction locks as well). If entry is locked, it won't be
> evicted.
>
> Thanks!
> -Dmitry.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Quick questions on Evictions
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
Assume you have disabled onheapCahe and disabled persistence. In that case
you may configure only datapage eviction mode, then outdated pages will be
thrown away, when no free memory will be available for Ignite. Also you
cannot configure per-entry eviction.
OK, if you enable onheapCache, then Ignite will store on heap every entry
that was read from off-heap (or disk). Next reads of it will not require
off-heap readings, and every update will write to off-heap. To limit size of
onheapCache you may set CacheConfiguration.setEvictionPolicy(), but it will
not evict off-heap entries.
So, off-heap eviction may be controlled with DataPageEvictionMode only, and
as you suggested, it clears entries one-by-one from page, checking for
current locks (transaction locks as well). If entry is locked, it won't be
evicted.
Thanks!
-Dmitry.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/