You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Magda <dm...@apache.org> on 2017/04/16 21:01:21 UTC
Page Memory behavior with default memory policy
Guys,
I’ve been working on the documentation for page memory [1]. The doc is still in progress and it will be more profound with code snippets and pictures int the end, so don’t pay to the fact that it mostly consists of dry text only.
So, the question is different. Presently, javadoc part is useless (and will be written by me as well) and it’s not clear what’s the default behavior of the page memory that has only one memory policy (default) that instantiates a single memory region. Looking at the code I see that the default police creates 1 GB region that should be expandable because the policy doesn’t define an eviction algorithm. It means that if my app goes beyond 1 GB then the page memory will take more memory from an OS. Is my understanding correct?
[1] https://apacheignite.readme.io/docs/page-memory <https://apacheignite.readme.io/docs/page-memory>
—
Denis
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Thanks, Alexey! Sounds good.
On Fri, Apr 21, 2017 at 8:34 AM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> Dmitriy,
>
> I have implemented expandable page memory and now I need to merge with the
> changes that Sergey made to make max be 80% of physical memory. Once TC
> passes, I will merge the changes, should not take too long, so hopefully,
> the change will make it to the release.
>
> 2017-04-20 20:40 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> > How long will it take to make the memory expandable? Can it be done for
> > 2.0?
> >
> > On Thu, Apr 20, 2017 at 5:25 AM, Sergey Chugunov <
> > sergey.chugunov@gmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > Replied in the ticket.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > > On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >
> > > wrote:
> > >
> > > > Sergey,
> > > >
> > > > I have responded in the ticket. Can you please provide the current
> and
> > > the
> > > > proposed configuration examples?
> > > >
> > > > D.
> > > >
> > > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > > sergey.chugunov@gmail.com>
> > > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > I created a ticket to implement these improvements, please take a
> > look:
> > > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > > >
> > > > > Besides employing the idea of allocation 80% of physical memory I'm
> > > also
> > > > > suggesting to introduce one more configuration property to specify
> > > > default
> > > > > MemoryPolicy's size in bytes without having to use verbose syntax
> of
> > > > > *memoryPolicyConfiguration
> > > > > *section.
> > > > >
> > > > > Any thoughts on this?
> > > > >
> > > > > Thanks,
> > > > > Sergey.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > > dsetrakyan@apache.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > > alexey.goncharuk@gmail.com> wrote:
> > > > > >
> > > > > > > I don't see why not, this is the way our tests are currently
> > > running.
> > > > > > > Anyways, we can think about efficient dynamic memory expansion
> in
> > > > 2.1,
> > > > > > this
> > > > > > > may be feasible if we free up some space in PageId to encode
> > > segment
> > > > > > > number. There is a ticket for this:
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > > >
> > > > > >
> > > > > > Alexey, if the operating system is already handling this for us,
> I
> > > > don't
> > > > > > see a reason to do it manually.
> > > > > >
> > > > > > I also like what Denis and Semyon are proposing. However, I would
> > not
> > > > > grab
> > > > > > the full free memory. How about 80% of the free memory?
> > > > > >
> > > > > > D.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Alexey Goncharuk <al...@gmail.com>.
Dmitriy,
I have implemented expandable page memory and now I need to merge with the
changes that Sergey made to make max be 80% of physical memory. Once TC
passes, I will merge the changes, should not take too long, so hopefully,
the change will make it to the release.
2017-04-20 20:40 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> How long will it take to make the memory expandable? Can it be done for
> 2.0?
>
> On Thu, Apr 20, 2017 at 5:25 AM, Sergey Chugunov <
> sergey.chugunov@gmail.com>
> wrote:
>
> > Dmitriy,
> >
> > Replied in the ticket.
> >
> > Thanks,
> > Sergey.
> >
> > On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >
> > wrote:
> >
> > > Sergey,
> > >
> > > I have responded in the ticket. Can you please provide the current and
> > the
> > > proposed configuration examples?
> > >
> > > D.
> > >
> > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > sergey.chugunov@gmail.com>
> > > wrote:
> > >
> > > > Guys,
> > > >
> > > > I created a ticket to implement these improvements, please take a
> look:
> > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > >
> > > > Besides employing the idea of allocation 80% of physical memory I'm
> > also
> > > > suggesting to introduce one more configuration property to specify
> > > default
> > > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > > *memoryPolicyConfiguration
> > > > *section.
> > > >
> > > > Any thoughts on this?
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > > >
> > > >
> > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > alexey.goncharuk@gmail.com> wrote:
> > > > >
> > > > > > I don't see why not, this is the way our tests are currently
> > running.
> > > > > > Anyways, we can think about efficient dynamic memory expansion in
> > > 2.1,
> > > > > this
> > > > > > may be feasible if we free up some space in PageId to encode
> > segment
> > > > > > number. There is a ticket for this:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > >
> > > > >
> > > > > Alexey, if the operating system is already handling this for us, I
> > > don't
> > > > > see a reason to do it manually.
> > > > >
> > > > > I also like what Denis and Semyon are proposing. However, I would
> not
> > > > grab
> > > > > the full free memory. How about 80% of the free memory?
> > > > >
> > > > > D.
> > > > >
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
How long will it take to make the memory expandable? Can it be done for 2.0?
On Thu, Apr 20, 2017 at 5:25 AM, Sergey Chugunov <se...@gmail.com>
wrote:
> Dmitriy,
>
> Replied in the ticket.
>
> Thanks,
> Sergey.
>
> On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > Sergey,
> >
> > I have responded in the ticket. Can you please provide the current and
> the
> > proposed configuration examples?
> >
> > D.
> >
> > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > sergey.chugunov@gmail.com>
> > wrote:
> >
> > > Guys,
> > >
> > > I created a ticket to implement these improvements, please take a look:
> > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > >
> > > Besides employing the idea of allocation 80% of physical memory I'm
> also
> > > suggesting to introduce one more configuration property to specify
> > default
> > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > *memoryPolicyConfiguration
> > > *section.
> > >
> > > Any thoughts on this?
> > >
> > > Thanks,
> > > Sergey.
> > >
> > >
> > >
> > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >
> > > wrote:
> > >
> > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com> wrote:
> > > >
> > > > > I don't see why not, this is the way our tests are currently
> running.
> > > > > Anyways, we can think about efficient dynamic memory expansion in
> > 2.1,
> > > > this
> > > > > may be feasible if we free up some space in PageId to encode
> segment
> > > > > number. There is a ticket for this:
> > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > >
> > > >
> > > > Alexey, if the operating system is already handling this for us, I
> > don't
> > > > see a reason to do it manually.
> > > >
> > > > I also like what Denis and Semyon are proposing. However, I would not
> > > grab
> > > > the full free memory. How about 80% of the free memory?
> > > >
> > > > D.
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Sergey Chugunov <se...@gmail.com>.
Dmitriy,
Replied in the ticket.
Thanks,
Sergey.
On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Sergey,
>
> I have responded in the ticket. Can you please provide the current and the
> proposed configuration examples?
>
> D.
>
> On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> sergey.chugunov@gmail.com>
> wrote:
>
> > Guys,
> >
> > I created a ticket to implement these improvements, please take a look:
> > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> >
> > Besides employing the idea of allocation 80% of physical memory I'm also
> > suggesting to introduce one more configuration property to specify
> default
> > MemoryPolicy's size in bytes without having to use verbose syntax of
> > *memoryPolicyConfiguration
> > *section.
> >
> > Any thoughts on this?
> >
> > Thanks,
> > Sergey.
> >
> >
> >
> > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >
> > wrote:
> >
> > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com> wrote:
> > >
> > > > I don't see why not, this is the way our tests are currently running.
> > > > Anyways, we can think about efficient dynamic memory expansion in
> 2.1,
> > > this
> > > > may be feasible if we free up some space in PageId to encode segment
> > > > number. There is a ticket for this:
> > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > >
> > >
> > > Alexey, if the operating system is already handling this for us, I
> don't
> > > see a reason to do it manually.
> > >
> > > I also like what Denis and Semyon are proposing. However, I would not
> > grab
> > > the full free memory. How about 80% of the free memory?
> > >
> > > D.
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Sergey Chugunov <se...@gmail.com>.
yes, we do: IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
On Thu, Apr 20, 2017 at 1:23 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Guys,
>
> Do we have a ticket for this change?
>
> On Thu, Apr 20, 2017 at 1:13 PM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
> > This actually will affect only nodes started in a single jvm, but I agree
> > that expandable memory provides better user experience. Let's try to
> > squeeze this change to 2.0.
> >
> > 2017-04-20 12:32 GMT+03:00 Sergi Vladykin <se...@gmail.com>:
> >
> > > Guys,
> > >
> > > If we have a default of 80% of available memory then just starting few
> > > nodes on my laptop will make it hang. This idea does not work until we
> > have
> > > a dynamically expandable memory pools.
> > >
> > > Sergi
> > >
> > > 2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> > >
> > > > Sergey,
> > > >
> > > > I have responded in the ticket. Can you please provide the current
> and
> > > the
> > > > proposed configuration examples?
> > > >
> > > > D.
> > > >
> > > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > > sergey.chugunov@gmail.com>
> > > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > I created a ticket to implement these improvements, please take a
> > look:
> > > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > > >
> > > > > Besides employing the idea of allocation 80% of physical memory I'm
> > > also
> > > > > suggesting to introduce one more configuration property to specify
> > > > default
> > > > > MemoryPolicy's size in bytes without having to use verbose syntax
> of
> > > > > *memoryPolicyConfiguration
> > > > > *section.
> > > > >
> > > > > Any thoughts on this?
> > > > >
> > > > > Thanks,
> > > > > Sergey.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > > dsetrakyan@apache.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > > alexey.goncharuk@gmail.com> wrote:
> > > > > >
> > > > > > > I don't see why not, this is the way our tests are currently
> > > running.
> > > > > > > Anyways, we can think about efficient dynamic memory expansion
> in
> > > > 2.1,
> > > > > > this
> > > > > > > may be feasible if we free up some space in PageId to encode
> > > segment
> > > > > > > number. There is a ticket for this:
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > > >
> > > > > >
> > > > > > Alexey, if the operating system is already handling this for us,
> I
> > > > don't
> > > > > > see a reason to do it manually.
> > > > > >
> > > > > > I also like what Denis and Semyon are proposing. However, I would
> > not
> > > > > grab
> > > > > > the full free memory. How about 80% of the free memory?
> > > > > >
> > > > > > D.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Guys,
Do we have a ticket for this change?
On Thu, Apr 20, 2017 at 1:13 PM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> This actually will affect only nodes started in a single jvm, but I agree
> that expandable memory provides better user experience. Let's try to
> squeeze this change to 2.0.
>
> 2017-04-20 12:32 GMT+03:00 Sergi Vladykin <se...@gmail.com>:
>
> > Guys,
> >
> > If we have a default of 80% of available memory then just starting few
> > nodes on my laptop will make it hang. This idea does not work until we
> have
> > a dynamically expandable memory pools.
> >
> > Sergi
> >
> > 2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> >
> > > Sergey,
> > >
> > > I have responded in the ticket. Can you please provide the current and
> > the
> > > proposed configuration examples?
> > >
> > > D.
> > >
> > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > sergey.chugunov@gmail.com>
> > > wrote:
> > >
> > > > Guys,
> > > >
> > > > I created a ticket to implement these improvements, please take a
> look:
> > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > >
> > > > Besides employing the idea of allocation 80% of physical memory I'm
> > also
> > > > suggesting to introduce one more configuration property to specify
> > > default
> > > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > > *memoryPolicyConfiguration
> > > > *section.
> > > >
> > > > Any thoughts on this?
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > > >
> > > >
> > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > alexey.goncharuk@gmail.com> wrote:
> > > > >
> > > > > > I don't see why not, this is the way our tests are currently
> > running.
> > > > > > Anyways, we can think about efficient dynamic memory expansion in
> > > 2.1,
> > > > > this
> > > > > > may be feasible if we free up some space in PageId to encode
> > segment
> > > > > > number. There is a ticket for this:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > >
> > > > >
> > > > > Alexey, if the operating system is already handling this for us, I
> > > don't
> > > > > see a reason to do it manually.
> > > > >
> > > > > I also like what Denis and Semyon are proposing. However, I would
> not
> > > > grab
> > > > > the full free memory. How about 80% of the free memory?
> > > > >
> > > > > D.
> > > > >
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Alexey Goncharuk <al...@gmail.com>.
This actually will affect only nodes started in a single jvm, but I agree
that expandable memory provides better user experience. Let's try to
squeeze this change to 2.0.
2017-04-20 12:32 GMT+03:00 Sergi Vladykin <se...@gmail.com>:
> Guys,
>
> If we have a default of 80% of available memory then just starting few
> nodes on my laptop will make it hang. This idea does not work until we have
> a dynamically expandable memory pools.
>
> Sergi
>
> 2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> > Sergey,
> >
> > I have responded in the ticket. Can you please provide the current and
> the
> > proposed configuration examples?
> >
> > D.
> >
> > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > sergey.chugunov@gmail.com>
> > wrote:
> >
> > > Guys,
> > >
> > > I created a ticket to implement these improvements, please take a look:
> > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > >
> > > Besides employing the idea of allocation 80% of physical memory I'm
> also
> > > suggesting to introduce one more configuration property to specify
> > default
> > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > *memoryPolicyConfiguration
> > > *section.
> > >
> > > Any thoughts on this?
> > >
> > > Thanks,
> > > Sergey.
> > >
> > >
> > >
> > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >
> > > wrote:
> > >
> > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com> wrote:
> > > >
> > > > > I don't see why not, this is the way our tests are currently
> running.
> > > > > Anyways, we can think about efficient dynamic memory expansion in
> > 2.1,
> > > > this
> > > > > may be feasible if we free up some space in PageId to encode
> segment
> > > > > number. There is a ticket for this:
> > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > >
> > > >
> > > > Alexey, if the operating system is already handling this for us, I
> > don't
> > > > see a reason to do it manually.
> > > >
> > > > I also like what Denis and Semyon are proposing. However, I would not
> > > grab
> > > > the full free memory. How about 80% of the free memory?
> > > >
> > > > D.
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Sergi Vladykin <se...@gmail.com>.
Guys,
If we have a default of 80% of available memory then just starting few
nodes on my laptop will make it hang. This idea does not work until we have
a dynamically expandable memory pools.
Sergi
2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> Sergey,
>
> I have responded in the ticket. Can you please provide the current and the
> proposed configuration examples?
>
> D.
>
> On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> sergey.chugunov@gmail.com>
> wrote:
>
> > Guys,
> >
> > I created a ticket to implement these improvements, please take a look:
> > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> >
> > Besides employing the idea of allocation 80% of physical memory I'm also
> > suggesting to introduce one more configuration property to specify
> default
> > MemoryPolicy's size in bytes without having to use verbose syntax of
> > *memoryPolicyConfiguration
> > *section.
> >
> > Any thoughts on this?
> >
> > Thanks,
> > Sergey.
> >
> >
> >
> > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >
> > wrote:
> >
> > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com> wrote:
> > >
> > > > I don't see why not, this is the way our tests are currently running.
> > > > Anyways, we can think about efficient dynamic memory expansion in
> 2.1,
> > > this
> > > > may be feasible if we free up some space in PageId to encode segment
> > > > number. There is a ticket for this:
> > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > >
> > >
> > > Alexey, if the operating system is already handling this for us, I
> don't
> > > see a reason to do it manually.
> > >
> > > I also like what Denis and Semyon are proposing. However, I would not
> > grab
> > > the full free memory. How about 80% of the free memory?
> > >
> > > D.
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Sergey,
I have responded in the ticket. Can you please provide the current and the
proposed configuration examples?
D.
On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <se...@gmail.com>
wrote:
> Guys,
>
> I created a ticket to implement these improvements, please take a look:
> IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
>
> Besides employing the idea of allocation 80% of physical memory I'm also
> suggesting to introduce one more configuration property to specify default
> MemoryPolicy's size in bytes without having to use verbose syntax of
> *memoryPolicyConfiguration
> *section.
>
> Any thoughts on this?
>
> Thanks,
> Sergey.
>
>
>
> On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > alexey.goncharuk@gmail.com> wrote:
> >
> > > I don't see why not, this is the way our tests are currently running.
> > > Anyways, we can think about efficient dynamic memory expansion in 2.1,
> > this
> > > may be feasible if we free up some space in PageId to encode segment
> > > number. There is a ticket for this:
> > > https://issues.apache.org/jira/browse/IGNITE-4921
> >
> >
> > Alexey, if the operating system is already handling this for us, I don't
> > see a reason to do it manually.
> >
> > I also like what Denis and Semyon are proposing. However, I would not
> grab
> > the full free memory. How about 80% of the free memory?
> >
> > D.
> >
>
Re: Page Memory behavior with default memory policy
Posted by Sergey Chugunov <se...@gmail.com>.
Guys,
I created a ticket to implement these improvements, please take a look:
IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
Besides employing the idea of allocation 80% of physical memory I'm also
suggesting to introduce one more configuration property to specify default
MemoryPolicy's size in bytes without having to use verbose syntax of
*memoryPolicyConfiguration
*section.
Any thoughts on this?
Thanks,
Sergey.
On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
> > I don't see why not, this is the way our tests are currently running.
> > Anyways, we can think about efficient dynamic memory expansion in 2.1,
> this
> > may be feasible if we free up some space in PageId to encode segment
> > number. There is a ticket for this:
> > https://issues.apache.org/jira/browse/IGNITE-4921
>
>
> Alexey, if the operating system is already handling this for us, I don't
> see a reason to do it manually.
>
> I also like what Denis and Semyon are proposing. However, I would not grab
> the full free memory. How about 80% of the free memory?
>
> D.
>
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> I don't see why not, this is the way our tests are currently running.
> Anyways, we can think about efficient dynamic memory expansion in 2.1, this
> may be feasible if we free up some space in PageId to encode segment
> number. There is a ticket for this:
> https://issues.apache.org/jira/browse/IGNITE-4921
Alexey, if the operating system is already handling this for us, I don't
see a reason to do it manually.
I also like what Denis and Semyon are proposing. However, I would not grab
the full free memory. How about 80% of the free memory?
D.
Re: Page Memory behavior with default memory policy
Posted by Alexey Goncharuk <al...@gmail.com>.
I don't see why not, this is the way our tests are currently running.
Anyways, we can think about efficient dynamic memory expansion in 2.1, this
may be feasible if we free up some space in PageId to encode segment
number. There is a ticket for this:
https://issues.apache.org/jira/browse/IGNITE-4921
2017-04-18 12:04 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> Thanks, Alexey, got it. What happens if a user starts multiple nodes on the
> same box? Will it work the way Denis suggested?
>
> On Tue, Apr 18, 2017 at 1:47 AM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
> > The reason we cannot add another memory region to page memory is
> > performance. Currently, the translation of a page ID to a virtual address
> > is basically a no-op. If we add dynamic memory expansion, we need to
> > maintain some sort of page ID mapping, which adds significant (in my
> view,
> > about 10%) overhead.
> >
> > Agree with Semyon, we can set default size based on available memory on a
> > machine.
> >
> > 2017-04-18 11:38 GMT+03:00 Semyon Boikov <sb...@gridgain.com>:
> >
> > > I think Ignite can behave like JVM: we can have -Xms -Xmx settings with
> > > defaults depending on available memory.
> > >
> > > Thanks
> > >
> > > On Tue, Apr 18, 2017 at 4:56 AM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > > Denis,
> > > >
> > > > If what you are suggesting is true, then we can always allocate about
> > 80%
> > > > of available memory by default. By the way, it must also work on
> > Windows,
> > > > so we should definitely test it.
> > > >
> > > > Alexey G, can you comment?
> > > >
> > > > D.
> > > >
> > > > On Mon, Apr 17, 2017 at 6:17 PM, Denis Magda <dm...@apache.org>
> > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > > Denis, it sounds like with this approach, in case of the
> > > > over-allocation,
> > > > > > the system will just get slower and slower and users will end up
> > > > blaming
> > > > > > Ignite for it. Am I understanding your suggestion correctly?
> > > > >
> > > > >
> > > > > This will not happen (at least in Unix) unless all the nodes really
> > > used
> > > > > all the allocated memory by putting data there or touching all the
> > > memory
> > > > > range somehow else.
> > > > >
> > > > > > How was this handled in Ignite 1.9?
> > > > >
> > > > >
> > > > > If you are talking about the legacy off-heap impl then we requested
> > > small
> > > > > chunks of data from an operating system rather than a continuous
> > memory
> > > > > region as in the page memory. But I would think of the page memory
> as
> > > of
> > > > > Java heap which also can request 8 GB continuous memory region on
> a 8
> > > GB
> > > > > machine following heap settings of an app but an operating system
> > will
> > > > not
> > > > > return the whole range immediately unless Java app occupies the
> whole
> > > > heap
> > > > > or use special parameters.
> > > > >
> > > > > At all, I think it’s safe to use approach suggested by me unless I
> > miss
> > > > > something.
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda <dm...@apache.org>
> > > > wrote:
> > > > > >
> > > > > >> Dmitriy,
> > > > > >>
> > > > > >> All the nodes will request its own continuous memory region that
> > > takes
> > > > > >> 70-80% of all RAM from an underlying operation system. However,
> > the
> > > > > >> operating system will not outfit the nodes with physical pages
> > > mapped
> > > > to
> > > > > >> RAM immediately allowing every node's process to start
> > successfully.
> > > > The
> > > > > >> nodes will communicate to RAM via a virtual memory which in its
> > turn
> > > > > will
> > > > > >> give an access to physical pages whenever is needed applying low
> > > level
> > > > > >> eviction and swapping techniques.
> > > > > >>
> > > > > >
> > > > > > Denis, it sounds like with this approach, in case of the
> > > > over-allocation,
> > > > > > the system will just get slower and slower and users will end up
> > > > blaming
> > > > > > Ignite for it. Am I understanding your suggestion correctly?
> > > > > >
> > > > > > How was this handled in Ignite 1.9?
> > > > > >
> > > > > > D.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Thanks, Alexey, got it. What happens if a user starts multiple nodes on the
same box? Will it work the way Denis suggested?
On Tue, Apr 18, 2017 at 1:47 AM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> The reason we cannot add another memory region to page memory is
> performance. Currently, the translation of a page ID to a virtual address
> is basically a no-op. If we add dynamic memory expansion, we need to
> maintain some sort of page ID mapping, which adds significant (in my view,
> about 10%) overhead.
>
> Agree with Semyon, we can set default size based on available memory on a
> machine.
>
> 2017-04-18 11:38 GMT+03:00 Semyon Boikov <sb...@gridgain.com>:
>
> > I think Ignite can behave like JVM: we can have -Xms -Xmx settings with
> > defaults depending on available memory.
> >
> > Thanks
> >
> > On Tue, Apr 18, 2017 at 4:56 AM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Denis,
> > >
> > > If what you are suggesting is true, then we can always allocate about
> 80%
> > > of available memory by default. By the way, it must also work on
> Windows,
> > > so we should definitely test it.
> > >
> > > Alexey G, can you comment?
> > >
> > > D.
> > >
> > > On Mon, Apr 17, 2017 at 6:17 PM, Denis Magda <dm...@apache.org>
> wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > > Denis, it sounds like with this approach, in case of the
> > > over-allocation,
> > > > > the system will just get slower and slower and users will end up
> > > blaming
> > > > > Ignite for it. Am I understanding your suggestion correctly?
> > > >
> > > >
> > > > This will not happen (at least in Unix) unless all the nodes really
> > used
> > > > all the allocated memory by putting data there or touching all the
> > memory
> > > > range somehow else.
> > > >
> > > > > How was this handled in Ignite 1.9?
> > > >
> > > >
> > > > If you are talking about the legacy off-heap impl then we requested
> > small
> > > > chunks of data from an operating system rather than a continuous
> memory
> > > > region as in the page memory. But I would think of the page memory as
> > of
> > > > Java heap which also can request 8 GB continuous memory region on a 8
> > GB
> > > > machine following heap settings of an app but an operating system
> will
> > > not
> > > > return the whole range immediately unless Java app occupies the whole
> > > heap
> > > > or use special parameters.
> > > >
> > > > At all, I think it’s safe to use approach suggested by me unless I
> miss
> > > > something.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > > wrote:
> > > > >
> > > > > On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda <dm...@apache.org>
> > > wrote:
> > > > >
> > > > >> Dmitriy,
> > > > >>
> > > > >> All the nodes will request its own continuous memory region that
> > takes
> > > > >> 70-80% of all RAM from an underlying operation system. However,
> the
> > > > >> operating system will not outfit the nodes with physical pages
> > mapped
> > > to
> > > > >> RAM immediately allowing every node's process to start
> successfully.
> > > The
> > > > >> nodes will communicate to RAM via a virtual memory which in its
> turn
> > > > will
> > > > >> give an access to physical pages whenever is needed applying low
> > level
> > > > >> eviction and swapping techniques.
> > > > >>
> > > > >
> > > > > Denis, it sounds like with this approach, in case of the
> > > over-allocation,
> > > > > the system will just get slower and slower and users will end up
> > > blaming
> > > > > Ignite for it. Am I understanding your suggestion correctly?
> > > > >
> > > > > How was this handled in Ignite 1.9?
> > > > >
> > > > > D.
> > > >
> > > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Alexey Goncharuk <al...@gmail.com>.
The reason we cannot add another memory region to page memory is
performance. Currently, the translation of a page ID to a virtual address
is basically a no-op. If we add dynamic memory expansion, we need to
maintain some sort of page ID mapping, which adds significant (in my view,
about 10%) overhead.
Agree with Semyon, we can set default size based on available memory on a
machine.
2017-04-18 11:38 GMT+03:00 Semyon Boikov <sb...@gridgain.com>:
> I think Ignite can behave like JVM: we can have -Xms -Xmx settings with
> defaults depending on available memory.
>
> Thanks
>
> On Tue, Apr 18, 2017 at 4:56 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Denis,
> >
> > If what you are suggesting is true, then we can always allocate about 80%
> > of available memory by default. By the way, it must also work on Windows,
> > so we should definitely test it.
> >
> > Alexey G, can you comment?
> >
> > D.
> >
> > On Mon, Apr 17, 2017 at 6:17 PM, Denis Magda <dm...@apache.org> wrote:
> >
> > > Dmitriy,
> > >
> > > > Denis, it sounds like with this approach, in case of the
> > over-allocation,
> > > > the system will just get slower and slower and users will end up
> > blaming
> > > > Ignite for it. Am I understanding your suggestion correctly?
> > >
> > >
> > > This will not happen (at least in Unix) unless all the nodes really
> used
> > > all the allocated memory by putting data there or touching all the
> memory
> > > range somehow else.
> > >
> > > > How was this handled in Ignite 1.9?
> > >
> > >
> > > If you are talking about the legacy off-heap impl then we requested
> small
> > > chunks of data from an operating system rather than a continuous memory
> > > region as in the page memory. But I would think of the page memory as
> of
> > > Java heap which also can request 8 GB continuous memory region on a 8
> GB
> > > machine following heap settings of an app but an operating system will
> > not
> > > return the whole range immediately unless Java app occupies the whole
> > heap
> > > or use special parameters.
> > >
> > > At all, I think it’s safe to use approach suggested by me unless I miss
> > > something.
> > >
> > > —
> > > Denis
> > >
> > > > On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > > wrote:
> > > >
> > > > On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda <dm...@apache.org>
> > wrote:
> > > >
> > > >> Dmitriy,
> > > >>
> > > >> All the nodes will request its own continuous memory region that
> takes
> > > >> 70-80% of all RAM from an underlying operation system. However, the
> > > >> operating system will not outfit the nodes with physical pages
> mapped
> > to
> > > >> RAM immediately allowing every node's process to start successfully.
> > The
> > > >> nodes will communicate to RAM via a virtual memory which in its turn
> > > will
> > > >> give an access to physical pages whenever is needed applying low
> level
> > > >> eviction and swapping techniques.
> > > >>
> > > >
> > > > Denis, it sounds like with this approach, in case of the
> > over-allocation,
> > > > the system will just get slower and slower and users will end up
> > blaming
> > > > Ignite for it. Am I understanding your suggestion correctly?
> > > >
> > > > How was this handled in Ignite 1.9?
> > > >
> > > > D.
> > >
> > >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Semyon Boikov <sb...@gridgain.com>.
I think Ignite can behave like JVM: we can have -Xms -Xmx settings with
defaults depending on available memory.
Thanks
On Tue, Apr 18, 2017 at 4:56 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Denis,
>
> If what you are suggesting is true, then we can always allocate about 80%
> of available memory by default. By the way, it must also work on Windows,
> so we should definitely test it.
>
> Alexey G, can you comment?
>
> D.
>
> On Mon, Apr 17, 2017 at 6:17 PM, Denis Magda <dm...@apache.org> wrote:
>
> > Dmitriy,
> >
> > > Denis, it sounds like with this approach, in case of the
> over-allocation,
> > > the system will just get slower and slower and users will end up
> blaming
> > > Ignite for it. Am I understanding your suggestion correctly?
> >
> >
> > This will not happen (at least in Unix) unless all the nodes really used
> > all the allocated memory by putting data there or touching all the memory
> > range somehow else.
> >
> > > How was this handled in Ignite 1.9?
> >
> >
> > If you are talking about the legacy off-heap impl then we requested small
> > chunks of data from an operating system rather than a continuous memory
> > region as in the page memory. But I would think of the page memory as of
> > Java heap which also can request 8 GB continuous memory region on a 8 GB
> > machine following heap settings of an app but an operating system will
> not
> > return the whole range immediately unless Java app occupies the whole
> heap
> > or use special parameters.
> >
> > At all, I think it’s safe to use approach suggested by me unless I miss
> > something.
> >
> > —
> > Denis
> >
> > > On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan <ds...@apache.org>
> > wrote:
> > >
> > > On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda <dm...@apache.org>
> wrote:
> > >
> > >> Dmitriy,
> > >>
> > >> All the nodes will request its own continuous memory region that takes
> > >> 70-80% of all RAM from an underlying operation system. However, the
> > >> operating system will not outfit the nodes with physical pages mapped
> to
> > >> RAM immediately allowing every node's process to start successfully.
> The
> > >> nodes will communicate to RAM via a virtual memory which in its turn
> > will
> > >> give an access to physical pages whenever is needed applying low level
> > >> eviction and swapping techniques.
> > >>
> > >
> > > Denis, it sounds like with this approach, in case of the
> over-allocation,
> > > the system will just get slower and slower and users will end up
> blaming
> > > Ignite for it. Am I understanding your suggestion correctly?
> > >
> > > How was this handled in Ignite 1.9?
> > >
> > > D.
> >
> >
>
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Denis,
If what you are suggesting is true, then we can always allocate about 80%
of available memory by default. By the way, it must also work on Windows,
so we should definitely test it.
Alexey G, can you comment?
D.
On Mon, Apr 17, 2017 at 6:17 PM, Denis Magda <dm...@apache.org> wrote:
> Dmitriy,
>
> > Denis, it sounds like with this approach, in case of the over-allocation,
> > the system will just get slower and slower and users will end up blaming
> > Ignite for it. Am I understanding your suggestion correctly?
>
>
> This will not happen (at least in Unix) unless all the nodes really used
> all the allocated memory by putting data there or touching all the memory
> range somehow else.
>
> > How was this handled in Ignite 1.9?
>
>
> If you are talking about the legacy off-heap impl then we requested small
> chunks of data from an operating system rather than a continuous memory
> region as in the page memory. But I would think of the page memory as of
> Java heap which also can request 8 GB continuous memory region on a 8 GB
> machine following heap settings of an app but an operating system will not
> return the whole range immediately unless Java app occupies the whole heap
> or use special parameters.
>
> At all, I think it’s safe to use approach suggested by me unless I miss
> something.
>
> —
> Denis
>
> > On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda <dm...@apache.org> wrote:
> >
> >> Dmitriy,
> >>
> >> All the nodes will request its own continuous memory region that takes
> >> 70-80% of all RAM from an underlying operation system. However, the
> >> operating system will not outfit the nodes with physical pages mapped to
> >> RAM immediately allowing every node's process to start successfully. The
> >> nodes will communicate to RAM via a virtual memory which in its turn
> will
> >> give an access to physical pages whenever is needed applying low level
> >> eviction and swapping techniques.
> >>
> >
> > Denis, it sounds like with this approach, in case of the over-allocation,
> > the system will just get slower and slower and users will end up blaming
> > Ignite for it. Am I understanding your suggestion correctly?
> >
> > How was this handled in Ignite 1.9?
> >
> > D.
>
>
Re: Page Memory behavior with default memory policy
Posted by Denis Magda <dm...@apache.org>.
Dmitriy,
> Denis, it sounds like with this approach, in case of the over-allocation,
> the system will just get slower and slower and users will end up blaming
> Ignite for it. Am I understanding your suggestion correctly?
This will not happen (at least in Unix) unless all the nodes really used all the allocated memory by putting data there or touching all the memory range somehow else.
> How was this handled in Ignite 1.9?
If you are talking about the legacy off-heap impl then we requested small chunks of data from an operating system rather than a continuous memory region as in the page memory. But I would think of the page memory as of Java heap which also can request 8 GB continuous memory region on a 8 GB machine following heap settings of an app but an operating system will not return the whole range immediately unless Java app occupies the whole heap or use special parameters.
At all, I think it’s safe to use approach suggested by me unless I miss something.
—
Denis
> On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Dmitriy,
>>
>> All the nodes will request its own continuous memory region that takes
>> 70-80% of all RAM from an underlying operation system. However, the
>> operating system will not outfit the nodes with physical pages mapped to
>> RAM immediately allowing every node's process to start successfully. The
>> nodes will communicate to RAM via a virtual memory which in its turn will
>> give an access to physical pages whenever is needed applying low level
>> eviction and swapping techniques.
>>
>
> Denis, it sounds like with this approach, in case of the over-allocation,
> the system will just get slower and slower and users will end up blaming
> Ignite for it. Am I understanding your suggestion correctly?
>
> How was this handled in Ignite 1.9?
>
> D.
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda <dm...@apache.org> wrote:
> Dmitriy,
>
> All the nodes will request its own continuous memory region that takes
> 70-80% of all RAM from an underlying operation system. However, the
> operating system will not outfit the nodes with physical pages mapped to
> RAM immediately allowing every node's process to start successfully. The
> nodes will communicate to RAM via a virtual memory which in its turn will
> give an access to physical pages whenever is needed applying low level
> eviction and swapping techniques.
>
Denis, it sounds like with this approach, in case of the over-allocation,
the system will just get slower and slower and users will end up blaming
Ignite for it. Am I understanding your suggestion correctly?
How was this handled in Ignite 1.9?
D.
Re: Page Memory behavior with default memory policy
Posted by Denis Magda <dm...@apache.org>.
Sorry, misprint
Alex G., please confirm that we’re *applying* -> *relying* on this low level guarantees in our page memory impl.
> On Apr 17, 2017, at 6:00 PM, Denis Magda <dm...@apache.org> wrote:
>
> Dmitriy,
>
> All the nodes will request its own continuous memory region that takes 70-80% of all RAM from an underlying operation system. However, the operating system will not outfit the nodes with physical pages mapped to RAM immediately allowing every node's process to start successfully. The nodes will communicate to RAM via a virtual memory which in its turn will give an access to physical pages whenever is needed applying low level eviction and swapping techniques.
>
> Alex G., please confirm that we’re applying on this low level guarantees in our page memory impl.
>
> —
> Denis
>
>> On Apr 17, 2017, at 5:50 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>>
>> On Mon, Apr 17, 2017 at 5:46 PM, Denis Magda <dm...@apache.org> wrote:
>>
>>> Guys,
>>>
>>> If a memory region is not expandable can we calculate the size of the
>>> default one automatically rather than setting it to predefined value like 1
>>> GB, 512 MB, etc. ?
>>>
>>> For instance, when a node is being started it finds out how much RAM is
>>> available and requests 70-80% of it for the default memory region usage.
>>> This should help us avoid this usability issue caused by the fact that we
>>> hard code the size.
>>>
>>
>> Denis, what happens if user plans to start multiple nodes on the same box?
>
Re: Page Memory behavior with default memory policy
Posted by Denis Magda <dm...@apache.org>.
Dmitriy,
All the nodes will request its own continuous memory region that takes 70-80% of all RAM from an underlying operation system. However, the operating system will not outfit the nodes with physical pages mapped to RAM immediately allowing every node's process to start successfully. The nodes will communicate to RAM via a virtual memory which in its turn will give an access to physical pages whenever is needed applying low level eviction and swapping techniques.
Alex G., please confirm that we’re applying on this low level guarantees in our page memory impl.
—
Denis
> On Apr 17, 2017, at 5:50 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> On Mon, Apr 17, 2017 at 5:46 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Guys,
>>
>> If a memory region is not expandable can we calculate the size of the
>> default one automatically rather than setting it to predefined value like 1
>> GB, 512 MB, etc. ?
>>
>> For instance, when a node is being started it finds out how much RAM is
>> available and requests 70-80% of it for the default memory region usage.
>> This should help us avoid this usability issue caused by the fact that we
>> hard code the size.
>>
>
> Denis, what happens if user plans to start multiple nodes on the same box?
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Mon, Apr 17, 2017 at 5:46 PM, Denis Magda <dm...@apache.org> wrote:
> Guys,
>
> If a memory region is not expandable can we calculate the size of the
> default one automatically rather than setting it to predefined value like 1
> GB, 512 MB, etc. ?
>
> For instance, when a node is being started it finds out how much RAM is
> available and requests 70-80% of it for the default memory region usage.
> This should help us avoid this usability issue caused by the fact that we
> hard code the size.
>
Denis, what happens if user plans to start multiple nodes on the same box?
Re: Page Memory behavior with default memory policy
Posted by Denis Magda <dm...@apache.org>.
Guys,
If a memory region is not expandable can we calculate the size of the default one automatically rather than setting it to predefined value like 1 GB, 512 MB, etc. ?
For instance, when a node is being started it finds out how much RAM is available and requests 70-80% of it for the default memory region usage. This should help us avoid this usability issue caused by the fact that we hard code the size.
—
Denis
> On Apr 17, 2017, at 2:24 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> Sergey,
>
> What prevents us from making MemoryPolicy expandable? For example, why not
> allocate another 1GB, when user runs out of memory? In my view, this would
> make our memory policies a lot more usable.
>
> D.
>
> On Mon, Apr 17, 2017 at 2:42 AM, Sergey Chugunov <se...@gmail.com>
> wrote:
>
>> Hello Denis,
>>
>> There is a small piece of documentation in *MemoryConfiguration *class,
>> although I think it should be moved to *MemoryPolicyConfiguration *and
>> detailed a lot.
>>
>> I'm going to write some documentation soon about validation rules that are
>> applied to memory policies configuration (there are some restrictions about
>> reserved name, default memory policy name and so on). I'll let you know
>> when it is done.
>>
>> About your question: MemoryPolicy offheap regions were never supposed to be
>> expandable. If user uses default MemoryPolicy (which in fact doesn't have
>> any eviction algorithm configured) he/she will get OutOfMemory error when
>> default size is exhausted.
>>
>> This OOM behavior was left intentionally to be consistent with the previous
>> behavior as configuring some default eviction policy on default
>> MemoryPolicy may lead to data loss and it is better to communicate wrong
>> memory configuration to user even with OOM than silently drop some cached
>> values.
>>
>> Thanks,
>> Sergey.
>>
>> On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda <dm...@apache.org> wrote:
>>
>>> Guys,
>>>
>>> I’ve been working on the documentation for page memory [1]. The doc is
>>> still in progress and it will be more profound with code snippets and
>>> pictures int the end, so don’t pay to the fact that it mostly consists of
>>> dry text only.
>>>
>>> So, the question is different. Presently, javadoc part is useless (and
>>> will be written by me as well) and it’s not clear what’s the default
>>> behavior of the page memory that has only one memory policy (default)
>> that
>>> instantiates a single memory region. Looking at the code I see that the
>>> default police creates 1 GB region that should be expandable because the
>>> policy doesn’t define an eviction algorithm. It means that if my app goes
>>> beyond 1 GB then the page memory will take more memory from an OS. Is my
>>> understanding correct?
>>>
>>> [1] https://apacheignite.readme.io/docs/page-memory <
>>> https://apacheignite.readme.io/docs/page-memory>
>>>
>>> —
>>> Denis
>>
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Sergey,
What prevents us from making MemoryPolicy expandable? For example, why not
allocate another 1GB, when user runs out of memory? In my view, this would
make our memory policies a lot more usable.
D.
On Mon, Apr 17, 2017 at 2:42 AM, Sergey Chugunov <se...@gmail.com>
wrote:
> Hello Denis,
>
> There is a small piece of documentation in *MemoryConfiguration *class,
> although I think it should be moved to *MemoryPolicyConfiguration *and
> detailed a lot.
>
> I'm going to write some documentation soon about validation rules that are
> applied to memory policies configuration (there are some restrictions about
> reserved name, default memory policy name and so on). I'll let you know
> when it is done.
>
> About your question: MemoryPolicy offheap regions were never supposed to be
> expandable. If user uses default MemoryPolicy (which in fact doesn't have
> any eviction algorithm configured) he/she will get OutOfMemory error when
> default size is exhausted.
>
> This OOM behavior was left intentionally to be consistent with the previous
> behavior as configuring some default eviction policy on default
> MemoryPolicy may lead to data loss and it is better to communicate wrong
> memory configuration to user even with OOM than silently drop some cached
> values.
>
> Thanks,
> Sergey.
>
> On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda <dm...@apache.org> wrote:
>
> > Guys,
> >
> > I’ve been working on the documentation for page memory [1]. The doc is
> > still in progress and it will be more profound with code snippets and
> > pictures int the end, so don’t pay to the fact that it mostly consists of
> > dry text only.
> >
> > So, the question is different. Presently, javadoc part is useless (and
> > will be written by me as well) and it’s not clear what’s the default
> > behavior of the page memory that has only one memory policy (default)
> that
> > instantiates a single memory region. Looking at the code I see that the
> > default police creates 1 GB region that should be expandable because the
> > policy doesn’t define an eviction algorithm. It means that if my app goes
> > beyond 1 GB then the page memory will take more memory from an OS. Is my
> > understanding correct?
> >
> > [1] https://apacheignite.readme.io/docs/page-memory <
> > https://apacheignite.readme.io/docs/page-memory>
> >
> > —
> > Denis
>
Re: Page Memory behavior with default memory policy
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Mon, Apr 17, 2017 at 4:40 AM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> Agree, I will rename it if there are no objections.
>
+1
Re: Page Memory behavior with default memory policy
Posted by Alexey Goncharuk <al...@gmail.com>.
Agree, I will rename it if there are no objections.
2017-04-17 14:31 GMT+03:00 Vladimir Ozerov <vo...@gridgain.com>:
> Guys,
>
> I see that we throw our own "OutOfMemoryException" in this case. I think we
> should rename it to "IgniteOufOfMemoryException" to avoid confusion with
> JDK's "OutOfMemoryError". E.g. I looked at some thread dumps recently and
> was pretty sure that there was real OOME, while in reality it was Ignite's
> exception.
>
> On Mon, Apr 17, 2017 at 12:42 PM, Sergey Chugunov <
> sergey.chugunov@gmail.com
> > wrote:
>
> > Hello Denis,
> >
> > There is a small piece of documentation in *MemoryConfiguration *class,
> > although I think it should be moved to *MemoryPolicyConfiguration *and
> > detailed a lot.
> >
> > I'm going to write some documentation soon about validation rules that
> are
> > applied to memory policies configuration (there are some restrictions
> about
> > reserved name, default memory policy name and so on). I'll let you know
> > when it is done.
> >
> > About your question: MemoryPolicy offheap regions were never supposed to
> be
> > expandable. If user uses default MemoryPolicy (which in fact doesn't have
> > any eviction algorithm configured) he/she will get OutOfMemory error when
> > default size is exhausted.
> >
> > This OOM behavior was left intentionally to be consistent with the
> previous
> > behavior as configuring some default eviction policy on default
> > MemoryPolicy may lead to data loss and it is better to communicate wrong
> > memory configuration to user even with OOM than silently drop some cached
> > values.
> >
> > Thanks,
> > Sergey.
> >
> > On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda <dm...@apache.org> wrote:
> >
> > > Guys,
> > >
> > > I’ve been working on the documentation for page memory [1]. The doc is
> > > still in progress and it will be more profound with code snippets and
> > > pictures int the end, so don’t pay to the fact that it mostly consists
> of
> > > dry text only.
> > >
> > > So, the question is different. Presently, javadoc part is useless (and
> > > will be written by me as well) and it’s not clear what’s the default
> > > behavior of the page memory that has only one memory policy (default)
> > that
> > > instantiates a single memory region. Looking at the code I see that the
> > > default police creates 1 GB region that should be expandable because
> the
> > > policy doesn’t define an eviction algorithm. It means that if my app
> goes
> > > beyond 1 GB then the page memory will take more memory from an OS. Is
> my
> > > understanding correct?
> > >
> > > [1] https://apacheignite.readme.io/docs/page-memory <
> > > https://apacheignite.readme.io/docs/page-memory>
> > >
> > > —
> > > Denis
> >
>
Re: Page Memory behavior with default memory policy
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Guys,
I see that we throw our own "OutOfMemoryException" in this case. I think we
should rename it to "IgniteOufOfMemoryException" to avoid confusion with
JDK's "OutOfMemoryError". E.g. I looked at some thread dumps recently and
was pretty sure that there was real OOME, while in reality it was Ignite's
exception.
On Mon, Apr 17, 2017 at 12:42 PM, Sergey Chugunov <sergey.chugunov@gmail.com
> wrote:
> Hello Denis,
>
> There is a small piece of documentation in *MemoryConfiguration *class,
> although I think it should be moved to *MemoryPolicyConfiguration *and
> detailed a lot.
>
> I'm going to write some documentation soon about validation rules that are
> applied to memory policies configuration (there are some restrictions about
> reserved name, default memory policy name and so on). I'll let you know
> when it is done.
>
> About your question: MemoryPolicy offheap regions were never supposed to be
> expandable. If user uses default MemoryPolicy (which in fact doesn't have
> any eviction algorithm configured) he/she will get OutOfMemory error when
> default size is exhausted.
>
> This OOM behavior was left intentionally to be consistent with the previous
> behavior as configuring some default eviction policy on default
> MemoryPolicy may lead to data loss and it is better to communicate wrong
> memory configuration to user even with OOM than silently drop some cached
> values.
>
> Thanks,
> Sergey.
>
> On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda <dm...@apache.org> wrote:
>
> > Guys,
> >
> > I’ve been working on the documentation for page memory [1]. The doc is
> > still in progress and it will be more profound with code snippets and
> > pictures int the end, so don’t pay to the fact that it mostly consists of
> > dry text only.
> >
> > So, the question is different. Presently, javadoc part is useless (and
> > will be written by me as well) and it’s not clear what’s the default
> > behavior of the page memory that has only one memory policy (default)
> that
> > instantiates a single memory region. Looking at the code I see that the
> > default police creates 1 GB region that should be expandable because the
> > policy doesn’t define an eviction algorithm. It means that if my app goes
> > beyond 1 GB then the page memory will take more memory from an OS. Is my
> > understanding correct?
> >
> > [1] https://apacheignite.readme.io/docs/page-memory <
> > https://apacheignite.readme.io/docs/page-memory>
> >
> > —
> > Denis
>
Re: Page Memory behavior with default memory policy
Posted by Sergey Chugunov <se...@gmail.com>.
Hello Denis,
There is a small piece of documentation in *MemoryConfiguration *class,
although I think it should be moved to *MemoryPolicyConfiguration *and
detailed a lot.
I'm going to write some documentation soon about validation rules that are
applied to memory policies configuration (there are some restrictions about
reserved name, default memory policy name and so on). I'll let you know
when it is done.
About your question: MemoryPolicy offheap regions were never supposed to be
expandable. If user uses default MemoryPolicy (which in fact doesn't have
any eviction algorithm configured) he/she will get OutOfMemory error when
default size is exhausted.
This OOM behavior was left intentionally to be consistent with the previous
behavior as configuring some default eviction policy on default
MemoryPolicy may lead to data loss and it is better to communicate wrong
memory configuration to user even with OOM than silently drop some cached
values.
Thanks,
Sergey.
On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda <dm...@apache.org> wrote:
> Guys,
>
> I’ve been working on the documentation for page memory [1]. The doc is
> still in progress and it will be more profound with code snippets and
> pictures int the end, so don’t pay to the fact that it mostly consists of
> dry text only.
>
> So, the question is different. Presently, javadoc part is useless (and
> will be written by me as well) and it’s not clear what’s the default
> behavior of the page memory that has only one memory policy (default) that
> instantiates a single memory region. Looking at the code I see that the
> default police creates 1 GB region that should be expandable because the
> policy doesn’t define an eviction algorithm. It means that if my app goes
> beyond 1 GB then the page memory will take more memory from an OS. Is my
> understanding correct?
>
> [1] https://apacheignite.readme.io/docs/page-memory <
> https://apacheignite.readme.io/docs/page-memory>
>
> —
> Denis