You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Sergey Chugunov <se...@gmail.com> on 2017/02/28 09:28:02 UTC
IGNITE-4758 - introducing cache memory policies
Hello Ignite devs,
I created new subtask <https://issues.apache.org/jira/browse/IGNITE-4758>
under IGNITE-3477 <https://issues.apache.org/jira/browse/IGNITE-3477> with
my vision of how *MemoryPolicy* may be implemented.
In a nutshell my idea is as follows: instead of having only one instance of
*PageMemory* and managing a mapping [cacheId->pageMemory region] I suggest
introducing separate *PageMemory* instance (and corresponding *FreeList*
and *ReuseList* structures) for each *MemoryPolicy* configuration.
This instance will be stored in individual cache contexts instead of shared
cache context as it is right now.
I think this design is a very natural extension of *PageMemory* concept and
should not introduce any performance degradation as no changes are
introduced into *PageMemory* implementation itself.
Please share any thoughts or concerns about suggested design in this thread
or in comments under the subtask
<https://issues.apache.org/jira/browse/IGNITE-4758>.
Thanks,
Sergey.
Re: IGNITE-4758 - introducing cache memory policies
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Sergey, my comments are below...
On Thu, Mar 2, 2017 at 5:22 AM, Sergey Chugunov <se...@gmail.com>
wrote:
> Hello igniters,
>
> Working on the JIRA I found several situations not totally clear to me how
> to handle properly.
>
> First of all I introduced several validation rules for MemoryPolicy
> configurations:
>
> - If user doesn't specify any MemoryPolicy, a default one is created
> implicitly.
> - If user specifies MemoryPolicy[ies] explicitly, one and only one
> default must be specified.
> - Two MemoryPolicies cannot have the same name.
>
All makes sense.
>
> But there are still several questions I would like to think over with
> community:
>
> - What validation rule should be applied to MemoryPolicy size?
> Currently configuration requires to specify size in bytes, but obviously
> values like 1 or 2 don't make sense.
>
> I suggest to apply a lower limit of 1 MB.
>
> - If user doesn't specify any configuration what default MemoryPolicy
> size should be applied?
>
> Previously it was 1 GB, I suggest to use this value. Any arguments to
> change this?
>
This sounds reasonable. We must explicitly print out a message in the log
that 1GB was selected for the default. Also, once the evictions start, i.e.
once we run out of space, we also must print out an explicit message in the
log.
> - Separate instance of PageMemory is needed for system caches which is
> created implicitly.
> What size should it have by default?
> Currently it is used only to store services and datastructures. If user
> doesn't need these it makes sense to use lower size by default.
>
> I think 100 MB would be enough. Surely this setting will be configurable
> via existing MemoryConfiguration interface.
>
100MB by default sounds reasonable to me. However, I would only initiate
this space if user creates his first data structure or his first service.
As usual, we must have a print out in the log that 100MB was allocated and
which config property should be used to change this value.
>
>
> Thanks,
> Sergey.
>
> On Thu, Mar 2, 2017 at 11:12 AM, Sergey Chugunov <
> sergey.chugunov@gmail.com>
> wrote:
>
> > Yes, they use the same PageMemory region.
> >
> > On Wed, Mar 1, 2017 at 10:34 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> >> When 2 caches share the same memory policy, does it mean that they are
> >> using the same Page Memory offheap space?
> >>
> >> On Tue, Feb 28, 2017 at 11:49 PM, Sergey Chugunov <
> >> sergey.chugunov@gmail.com
> >> > wrote:
> >>
> >> > Dmitriy,
> >> >
> >> > Correct, cache configuration may contain a name of memory policy to
> use
> >> for
> >> > this cache.
> >> > If memory policy is not specified, a default one is used.
> >> >
> >> > Thanks,
> >> > Sergey.
> >> >
> >> > On Wed, Mar 1, 2017 at 1:02 AM, Dmitriy Setrakyan <
> >> dsetrakyan@apache.org>
> >> > wrote:
> >> >
> >> > > Thanks Sergey!
> >> > >
> >> > > Just to confirm, the same memory policy could be shared between
> >> different
> >> > > caches, right?
> >> > >
> >> > > D.
> >> > >
> >> > > On Tue, Feb 28, 2017 at 1:28 AM, Sergey Chugunov <
> >> > > sergey.chugunov@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hello Ignite devs,
> >> > > >
> >> > > > I created new subtask <https://issues.apache.org/
> >> > jira/browse/IGNITE-4758
> >> > > >
> >> > > > under IGNITE-3477 <https://issues.apache.org/jir
> >> a/browse/IGNITE-3477>
> >> > > with
> >> > > > my vision of how *MemoryPolicy* may be implemented.
> >> > > >
> >> > > > In a nutshell my idea is as follows: instead of having only one
> >> > instance
> >> > > of
> >> > > > *PageMemory* and managing a mapping [cacheId->pageMemory region] I
> >> > > suggest
> >> > > > introducing separate *PageMemory* instance (and corresponding
> >> > *FreeList*
> >> > > > and *ReuseList* structures) for each *MemoryPolicy* configuration.
> >> > > > This instance will be stored in individual cache contexts instead
> of
> >> > > shared
> >> > > > cache context as it is right now.
> >> > > >
> >> > > > I think this design is a very natural extension of *PageMemory*
> >> concept
> >> > > and
> >> > > > should not introduce any performance degradation as no changes are
> >> > > > introduced into *PageMemory* implementation itself.
> >> > > >
> >> > > > Please share any thoughts or concerns about suggested design in
> this
> >> > > thread
> >> > > > or in comments under the subtask
> >> > > > <https://issues.apache.org/jira/browse/IGNITE-4758>.
> >> > > >
> >> > > > Thanks,
> >> > > > Sergey.
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: IGNITE-4758 - introducing cache memory policies
Posted by Sergey Chugunov <se...@gmail.com>.
Hello igniters,
Working on the JIRA I found several situations not totally clear to me how
to handle properly.
First of all I introduced several validation rules for MemoryPolicy
configurations:
- If user doesn't specify any MemoryPolicy, a default one is created
implicitly.
- If user specifies MemoryPolicy[ies] explicitly, one and only one
default must be specified.
- Two MemoryPolicies cannot have the same name.
But there are still several questions I would like to think over with
community:
- What validation rule should be applied to MemoryPolicy size?
Currently configuration requires to specify size in bytes, but obviously
values like 1 or 2 don't make sense.
I suggest to apply a lower limit of 1 MB.
- If user doesn't specify any configuration what default MemoryPolicy
size should be applied?
Previously it was 1 GB, I suggest to use this value. Any arguments to
change this?
- Separate instance of PageMemory is needed for system caches which is
created implicitly.
What size should it have by default?
Currently it is used only to store services and datastructures. If user
doesn't need these it makes sense to use lower size by default.
I think 100 MB would be enough. Surely this setting will be configurable
via existing MemoryConfiguration interface.
Thanks,
Sergey.
On Thu, Mar 2, 2017 at 11:12 AM, Sergey Chugunov <se...@gmail.com>
wrote:
> Yes, they use the same PageMemory region.
>
> On Wed, Mar 1, 2017 at 10:34 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>> When 2 caches share the same memory policy, does it mean that they are
>> using the same Page Memory offheap space?
>>
>> On Tue, Feb 28, 2017 at 11:49 PM, Sergey Chugunov <
>> sergey.chugunov@gmail.com
>> > wrote:
>>
>> > Dmitriy,
>> >
>> > Correct, cache configuration may contain a name of memory policy to use
>> for
>> > this cache.
>> > If memory policy is not specified, a default one is used.
>> >
>> > Thanks,
>> > Sergey.
>> >
>> > On Wed, Mar 1, 2017 at 1:02 AM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org>
>> > wrote:
>> >
>> > > Thanks Sergey!
>> > >
>> > > Just to confirm, the same memory policy could be shared between
>> different
>> > > caches, right?
>> > >
>> > > D.
>> > >
>> > > On Tue, Feb 28, 2017 at 1:28 AM, Sergey Chugunov <
>> > > sergey.chugunov@gmail.com>
>> > > wrote:
>> > >
>> > > > Hello Ignite devs,
>> > > >
>> > > > I created new subtask <https://issues.apache.org/
>> > jira/browse/IGNITE-4758
>> > > >
>> > > > under IGNITE-3477 <https://issues.apache.org/jir
>> a/browse/IGNITE-3477>
>> > > with
>> > > > my vision of how *MemoryPolicy* may be implemented.
>> > > >
>> > > > In a nutshell my idea is as follows: instead of having only one
>> > instance
>> > > of
>> > > > *PageMemory* and managing a mapping [cacheId->pageMemory region] I
>> > > suggest
>> > > > introducing separate *PageMemory* instance (and corresponding
>> > *FreeList*
>> > > > and *ReuseList* structures) for each *MemoryPolicy* configuration.
>> > > > This instance will be stored in individual cache contexts instead of
>> > > shared
>> > > > cache context as it is right now.
>> > > >
>> > > > I think this design is a very natural extension of *PageMemory*
>> concept
>> > > and
>> > > > should not introduce any performance degradation as no changes are
>> > > > introduced into *PageMemory* implementation itself.
>> > > >
>> > > > Please share any thoughts or concerns about suggested design in this
>> > > thread
>> > > > or in comments under the subtask
>> > > > <https://issues.apache.org/jira/browse/IGNITE-4758>.
>> > > >
>> > > > Thanks,
>> > > > Sergey.
>> > > >
>> > >
>> >
>>
>
>
Re: IGNITE-4758 - introducing cache memory policies
Posted by Sergey Chugunov <se...@gmail.com>.
Yes, they use the same PageMemory region.
On Wed, Mar 1, 2017 at 10:34 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> When 2 caches share the same memory policy, does it mean that they are
> using the same Page Memory offheap space?
>
> On Tue, Feb 28, 2017 at 11:49 PM, Sergey Chugunov <
> sergey.chugunov@gmail.com
> > wrote:
>
> > Dmitriy,
> >
> > Correct, cache configuration may contain a name of memory policy to use
> for
> > this cache.
> > If memory policy is not specified, a default one is used.
> >
> > Thanks,
> > Sergey.
> >
> > On Wed, Mar 1, 2017 at 1:02 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> > wrote:
> >
> > > Thanks Sergey!
> > >
> > > Just to confirm, the same memory policy could be shared between
> different
> > > caches, right?
> > >
> > > D.
> > >
> > > On Tue, Feb 28, 2017 at 1:28 AM, Sergey Chugunov <
> > > sergey.chugunov@gmail.com>
> > > wrote:
> > >
> > > > Hello Ignite devs,
> > > >
> > > > I created new subtask <https://issues.apache.org/
> > jira/browse/IGNITE-4758
> > > >
> > > > under IGNITE-3477 <https://issues.apache.org/jira/browse/IGNITE-3477
> >
> > > with
> > > > my vision of how *MemoryPolicy* may be implemented.
> > > >
> > > > In a nutshell my idea is as follows: instead of having only one
> > instance
> > > of
> > > > *PageMemory* and managing a mapping [cacheId->pageMemory region] I
> > > suggest
> > > > introducing separate *PageMemory* instance (and corresponding
> > *FreeList*
> > > > and *ReuseList* structures) for each *MemoryPolicy* configuration.
> > > > This instance will be stored in individual cache contexts instead of
> > > shared
> > > > cache context as it is right now.
> > > >
> > > > I think this design is a very natural extension of *PageMemory*
> concept
> > > and
> > > > should not introduce any performance degradation as no changes are
> > > > introduced into *PageMemory* implementation itself.
> > > >
> > > > Please share any thoughts or concerns about suggested design in this
> > > thread
> > > > or in comments under the subtask
> > > > <https://issues.apache.org/jira/browse/IGNITE-4758>.
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > >
> >
>
Re: IGNITE-4758 - introducing cache memory policies
Posted by Dmitriy Setrakyan <ds...@apache.org>.
When 2 caches share the same memory policy, does it mean that they are
using the same Page Memory offheap space?
On Tue, Feb 28, 2017 at 11:49 PM, Sergey Chugunov <sergey.chugunov@gmail.com
> wrote:
> Dmitriy,
>
> Correct, cache configuration may contain a name of memory policy to use for
> this cache.
> If memory policy is not specified, a default one is used.
>
> Thanks,
> Sergey.
>
> On Wed, Mar 1, 2017 at 1:02 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Thanks Sergey!
> >
> > Just to confirm, the same memory policy could be shared between different
> > caches, right?
> >
> > D.
> >
> > On Tue, Feb 28, 2017 at 1:28 AM, Sergey Chugunov <
> > sergey.chugunov@gmail.com>
> > wrote:
> >
> > > Hello Ignite devs,
> > >
> > > I created new subtask <https://issues.apache.org/
> jira/browse/IGNITE-4758
> > >
> > > under IGNITE-3477 <https://issues.apache.org/jira/browse/IGNITE-3477>
> > with
> > > my vision of how *MemoryPolicy* may be implemented.
> > >
> > > In a nutshell my idea is as follows: instead of having only one
> instance
> > of
> > > *PageMemory* and managing a mapping [cacheId->pageMemory region] I
> > suggest
> > > introducing separate *PageMemory* instance (and corresponding
> *FreeList*
> > > and *ReuseList* structures) for each *MemoryPolicy* configuration.
> > > This instance will be stored in individual cache contexts instead of
> > shared
> > > cache context as it is right now.
> > >
> > > I think this design is a very natural extension of *PageMemory* concept
> > and
> > > should not introduce any performance degradation as no changes are
> > > introduced into *PageMemory* implementation itself.
> > >
> > > Please share any thoughts or concerns about suggested design in this
> > thread
> > > or in comments under the subtask
> > > <https://issues.apache.org/jira/browse/IGNITE-4758>.
> > >
> > > Thanks,
> > > Sergey.
> > >
> >
>
Re: IGNITE-4758 - introducing cache memory policies
Posted by Sergey Chugunov <se...@gmail.com>.
Dmitriy,
Correct, cache configuration may contain a name of memory policy to use for
this cache.
If memory policy is not specified, a default one is used.
Thanks,
Sergey.
On Wed, Mar 1, 2017 at 1:02 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Thanks Sergey!
>
> Just to confirm, the same memory policy could be shared between different
> caches, right?
>
> D.
>
> On Tue, Feb 28, 2017 at 1:28 AM, Sergey Chugunov <
> sergey.chugunov@gmail.com>
> wrote:
>
> > Hello Ignite devs,
> >
> > I created new subtask <https://issues.apache.org/jira/browse/IGNITE-4758
> >
> > under IGNITE-3477 <https://issues.apache.org/jira/browse/IGNITE-3477>
> with
> > my vision of how *MemoryPolicy* may be implemented.
> >
> > In a nutshell my idea is as follows: instead of having only one instance
> of
> > *PageMemory* and managing a mapping [cacheId->pageMemory region] I
> suggest
> > introducing separate *PageMemory* instance (and corresponding *FreeList*
> > and *ReuseList* structures) for each *MemoryPolicy* configuration.
> > This instance will be stored in individual cache contexts instead of
> shared
> > cache context as it is right now.
> >
> > I think this design is a very natural extension of *PageMemory* concept
> and
> > should not introduce any performance degradation as no changes are
> > introduced into *PageMemory* implementation itself.
> >
> > Please share any thoughts or concerns about suggested design in this
> thread
> > or in comments under the subtask
> > <https://issues.apache.org/jira/browse/IGNITE-4758>.
> >
> > Thanks,
> > Sergey.
> >
>
Re: IGNITE-4758 - introducing cache memory policies
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Thanks Sergey!
Just to confirm, the same memory policy could be shared between different
caches, right?
D.
On Tue, Feb 28, 2017 at 1:28 AM, Sergey Chugunov <se...@gmail.com>
wrote:
> Hello Ignite devs,
>
> I created new subtask <https://issues.apache.org/jira/browse/IGNITE-4758>
> under IGNITE-3477 <https://issues.apache.org/jira/browse/IGNITE-3477> with
> my vision of how *MemoryPolicy* may be implemented.
>
> In a nutshell my idea is as follows: instead of having only one instance of
> *PageMemory* and managing a mapping [cacheId->pageMemory region] I suggest
> introducing separate *PageMemory* instance (and corresponding *FreeList*
> and *ReuseList* structures) for each *MemoryPolicy* configuration.
> This instance will be stored in individual cache contexts instead of shared
> cache context as it is right now.
>
> I think this design is a very natural extension of *PageMemory* concept and
> should not introduce any performance degradation as no changes are
> introduced into *PageMemory* implementation itself.
>
> Please share any thoughts or concerns about suggested design in this thread
> or in comments under the subtask
> <https://issues.apache.org/jira/browse/IGNITE-4758>.
>
> Thanks,
> Sergey.
>