You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Stanislav Lukyanov <st...@gmail.com> on 2018/12/11 16:39:01 UTC

RE: Pre-touch for Ignite off-heap memory

Igniters,

What is being suggested here is an Ignite off-heap’s version of Java’s -XX:+AlwaysPreTouch.
The latter is known to be used to guarantee that the committed memory is backed by physical RAM.
This ensures that
a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
b) JVM fails fast if the Xmx is greater than available RAM
c) New processes will not be able to claim the memory JVM took for itself

Currently one can’t get the same benefits for Ignite because we use off-heap as well as heap.
So, we can implement a similar feature for Ignite – and make sure the users can get all the memory pre-touch benefits if they want.
Of course, it impacts startup time so we should enable it by a configuration property (I’d suggest a system property for now).

Are there any objections to implementing this?

Thanks,
Stan

From: Павлухин Иван
Sent: 31 октября 2018 г. 12:50
To: dev@ignite.apache.org
Subject: Re: Pre-touch for Ignite off-heap memory

Hi,

I did an experiment described above. I created a patch with pre-touch [1]
and started a JVM with an option -XX:+AlwaysPreTouch and configured
Ignite with equal values for initial and max sizes for each data region.
I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
to observe JVM OOM-killed.

[1] https://github.com/apache/ignite/pull/5220
[2]
https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
[3]
https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log

вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:

> Hi guys,
>
> I am not aware that it is possible to run JVM in "allocation-free" fashion.
> If you know that it is possible please share it. As I know JVM allocates
> memory out of garbage collectable area for internal purposes like JIT,
> GC itself. Also native built-it code can request memory allocation from OS,
> e.g. zip facilities. If we imagine that system is running under memory
> allocation
> which is close to a limit then even small allocation request can fail and
> lead
> to OOM killing.
>
> But I think that a simple and useful thing that could be done first is
> making
> an experiment. As Andrey mentioned
> > AFAIK, Ignite always pre-touch first region. So, you can try to set
> region
> > MAX size equal to MIN and get region allocated on node start.
> If it is so then it should not be hard to launch Ignite and observe it
> running
> very close to OS memory limit with Java heap and Ignite off-heap
> pre-touched.
> After that one could check whether it is possible to observe Ignite OOM
> killed.
> Let's say my bet is that it is relatively easy to catch OOM here.
>
> What do you think?
>
> пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
>
>> Andrey,
>>
>> Probability of a OOM kill will be much lower if offheap is pretouched.
>> What
>> do you mean by JVM internal needs? In my understanding if user enables
>> option to pretouch heap and fixes the heap to prevent jvm releasing memory
>> back to OS, then OOM killing is very unlikely.
>>
>> I would agree that pretouch for offheap may be helpful in many cases.
>>
>> --Yakov
>>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>


-- 
Best regards,
Ivan Pavlukhin

Re: Pre-touch for Ignite off-heap memory

Posted by Stanislav Lukyanov <st...@gmail.com>.
Well, I'm not saying "we need all three features all the time", I'm just
summarizing why someone would use pre-touch.
And having c) is better than having none.

Also, we do get a) and b) as well, the timing is just a bit different.
But I find it better to fail at the time of dynamic cache creation (which
should be approached with caution anyway) than
during the normal execution.

Stan

On Tue, Dec 18, 2018 at 4:01 PM Павлухин Иван <vo...@gmail.com> wrote:

> Stan,
>
> In one of your emails you wrote why you need pre-touch:
> > a) JVM doesn’t have to do it during the actual work (avoiding overhead
> for physical page allocation, possible contention with page cache, etc)
> > b) JVM fails fast if the Xmx is greater than available RAM
> > c) New processes will not be able to claim the memory JVM took for itself
>
> As far as I see only c) can be satisfied if lazy region initialization
> is enabled. Am I missing something?
> пн, 17 дек. 2018 г. в 18:11, Stanislav Lukyanov <st...@gmail.com>:
> >
> > I don’t think these two issues require to be solved together, although I
> agree there is some connection.
> >
> >
> >
> > I guess we agree that pre-touch should a be off by default.
> >
> > Similarly, lazy data region initialization from IGNITE-9113 can be on by
> default.
> >
> > We can have two boolean properties controlling each feature, e.g.
> >
> >     IGNITE_EAGER_DATA_REGION_INITIALIZATION
> >
> >     IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
> >
> > Setting both to true will make sure that all memory is committed as
> early as possible.
> >
> > Different combinations will allow for all 4 variants you’ve mentioned. I
> can see a use case for each one:
> >
> > - lazy region init + no pre-touch (default) – easier configuration (at
> least for simple use cases without a lot of regions and node filters) and
> faster startup
> >
> > - lazy region init + pre-touch – allows to reuse server config on a
> client (the goal of the IGNITE-9113) but still allows to fail-fast on cache
> creation
> >
> > - eager region init + no pre-touch – for cases when you want to eagerly
> allocate memory but don’t care about committing physical pages; e.g. if you
> have overcommit disabled then you don’t need to pre-touch everything to
> make sure the memory is there
> >
> > - eager region init + pre-touch – to fail as early as possible if there
> is not enough RAM
> >
> >
> >
> > Stan
> >
> >
> >
> > From: Nikolay Izhikov
> > Sent: 14 декабря 2018 г. 14:42
> > To: dev; vololo100@gmail.com; stanlukyanov@gmail.com
> > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> >
> >
> > Bump.
> >
> >
> >
> > Stanislav, Ivan, can you answer my questions?
> >
> > Let's discuss these improvements.
> >
> >
> >
> > ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:
> >
> >
> >
> > > Hi Stan,
> >
> > >
> >
> > > As I participated in discussion in current thread I would like to
> >
> > > leave a comment.
> >
> > >
> >
> > > Your concerns looks clear for me and if you believe that pre-touch
> >
> > > will help product users then I have nothing against it.
> >
> > > ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
> >
> > > >
> >
> > > > Hello, Stanislav.
> >
> > > >
> >
> > > > As far as I can see, we have controversal ticket based on previous
> >
> > > dev-list discussion:
> >
> > > >
> >
> > > > IGNITE-9113 - Allocate memory for a data region when first cache
> >
> > > assigned to this region is created [1]
> >
> > > >
> >
> > > > I planned to implement it soon.
> >
> > > > Looks like we should have several options of memory(data regions)
> >
> > > allocation:
> >
> > > >
> >
> > > >         - allocate all on startup (AFAIK this is how current
> >
> > > implementation behave)
> >
> > > >         - allocate all on startup AND pre touch.
> >
> > > >         - allocate specific data region for first assignment.
> >
> > > >         - allocate specific data region for first assignment AND pre
> >
> > > touch.
> >
> > > >
> >
> > > > What do you think?
> >
> > > >
> >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > > >
> >
> > > > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> >
> > > > > Igniters,
> >
> > > > >
> >
> > > > > What is being suggested here is an Ignite off-heap’s version of
> Java’s
> >
> > > -XX:+AlwaysPreTouch.
> >
> > > > > The latter is known to be used to guarantee that the committed
> memory
> >
> > > is backed by physical RAM.
> >
> > > > > This ensures that
> >
> > > > > a) JVM doesn’t have to do it during the actual work (avoiding
> overhead
> >
> > > for physical page allocation, possible contention with page cache, etc)
> >
> > > > > b) JVM fails fast if the Xmx is greater than available RAM
> >
> > > > > c) New processes will not be able to claim the memory JVM took for
> >
> > > itself
> >
> > > > >
> >
> > > > > Currently one can’t get the same benefits for Ignite because we use
> >
> > > off-heap as well as heap.
> >
> > > > > So, we can implement a similar feature for Ignite – and make sure
> the
> >
> > > users can get all the memory pre-touch benefits if they want.
> >
> > > > > Of course, it impacts startup time so we should enable it by a
> >
> > > configuration property (I’d suggest a system property for now).
> >
> > > > >
> >
> > > > > Are there any objections to implementing this?
> >
> > > > >
> >
> > > > > Thanks,
> >
> > > > > Stan
> >
> > > > >
> >
> > > > > From: Павлухин Иван
> >
> > > > > Sent: 31 октября 2018 г. 12:50
> >
> > > > > To: dev@ignite.apache.org
> >
> > > > > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> > > > >
> >
> > > > > Hi,
> >
> > > > >
> >
> > > > > I did an experiment described above. I created a patch with
> pre-touch
> >
> > > [1]
> >
> > > > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> >
> > > > > Ignite with equal values for initial and max sizes for each data
> >
> > > region.
> >
> > > > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
> >
> > > easy
> >
> > > > > to observe JVM OOM-killed.
> >
> > > > >
> >
> > > > > [1] https://github.com/apache/ignite/pull/5220
> >
> > > > > [2]
> >
> > > > >
> >
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> >
> > > > > [3]
> >
> > > > >
> >
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> >
> > > > >
> >
> > > > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> >
> > > > >
> >
> > > > > > Hi guys,
> >
> > > > > >
> >
> > > > > > I am not aware that it is possible to run JVM in
> "allocation-free"
> >
> > > fashion.
> >
> > > > > > If you know that it is possible please share it. As I know JVM
> >
> > > allocates
> >
> > > > > > memory out of garbage collectable area for internal purposes like
> >
> > > JIT,
> >
> > > > > > GC itself. Also native built-it code can request memory
> allocation
> >
> > > from OS,
> >
> > > > > > e.g. zip facilities. If we imagine that system is running under
> >
> > > memory
> >
> > > > > > allocation
> >
> > > > > > which is close to a limit then even small allocation request can
> >
> > > fail and
> >
> > > > > > lead
> >
> > > > > > to OOM killing.
> >
> > > > > >
> >
> > > > > > But I think that a simple and useful thing that could be done
> first
> >
> > > is
> >
> > > > > > making
> >
> > > > > > an experiment. As Andrey mentioned
> >
> > > > > > > AFAIK, Ignite always pre-touch first region. So, you can try
> to set
> >
> > > > > >
> >
> > > > > > region
> >
> > > > > > > MAX size equal to MIN and get region allocated on node start.
> >
> > > > > >
> >
> > > > > > If it is so then it should not be hard to launch Ignite and
> observe
> >
> > > it
> >
> > > > > > running
> >
> > > > > > very close to OS memory limit with Java heap and Ignite off-heap
> >
> > > > > > pre-touched.
> >
> > > > > > After that one could check whether it is possible to observe
> Ignite
> >
> > > OOM
> >
> > > > > > killed.
> >
> > > > > > Let's say my bet is that it is relatively easy to catch OOM here.
> >
> > > > > >
> >
> > > > > > What do you think?
> >
> > > > > >
> >
> > > > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yzhdanov@apache.org
> >:
> >
> > > > > >
> >
> > > > > > > Andrey,
> >
> > > > > > >
> >
> > > > > > > Probability of a OOM kill will be much lower if offheap is
> >
> > > pretouched.
> >
> > > > > > > What
> >
> > > > > > > do you mean by JVM internal needs? In my understanding if user
> >
> > > enables
> >
> > > > > > > option to pretouch heap and fixes the heap to prevent jvm
> >
> > > releasing memory
> >
> > > > > > > back to OS, then OOM killing is very unlikely.
> >
> > > > > > >
> >
> > > > > > > I would agree that pretouch for offheap may be helpful in many
> >
> > > cases.
> >
> > > > > > >
> >
> > > > > > > --Yakov
> >
> > > > > > >
> >
> > > > > >
> >
> > > > > >
> >
> > > > > > --
> >
> > > > > > Best regards,
> >
> > > > > > Ivan Pavlukhin
> >
> > > > > >
> >
> > > > >
> >
> > > > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > --
> >
> > > Best regards,
> >
> > > Ivan Pavlukhin
> >
> > >
> >
> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Pre-touch for Ignite off-heap memory

Posted by Павлухин Иван <vo...@gmail.com>.
Stan,

In one of your emails you wrote why you need pre-touch:
> a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
> b) JVM fails fast if the Xmx is greater than available RAM
> c) New processes will not be able to claim the memory JVM took for itself

As far as I see only c) can be satisfied if lazy region initialization
is enabled. Am I missing something?
пн, 17 дек. 2018 г. в 18:11, Stanislav Lukyanov <st...@gmail.com>:
>
> I don’t think these two issues require to be solved together, although I agree there is some connection.
>
>
>
> I guess we agree that pre-touch should a be off by default.
>
> Similarly, lazy data region initialization from IGNITE-9113 can be on by default.
>
> We can have two boolean properties controlling each feature, e.g.
>
>     IGNITE_EAGER_DATA_REGION_INITIALIZATION
>
>     IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
>
> Setting both to true will make sure that all memory is committed as early as possible.
>
> Different combinations will allow for all 4 variants you’ve mentioned. I can see a use case for each one:
>
> - lazy region init + no pre-touch (default) – easier configuration (at least for simple use cases without a lot of regions and node filters) and faster startup
>
> - lazy region init + pre-touch – allows to reuse server config on a client (the goal of the IGNITE-9113) but still allows to fail-fast on cache creation
>
> - eager region init + no pre-touch – for cases when you want to eagerly allocate memory but don’t care about committing physical pages; e.g. if you have overcommit disabled then you don’t need to pre-touch everything to make sure the memory is there
>
> - eager region init + pre-touch – to fail as early as possible if there is not enough RAM
>
>
>
> Stan
>
>
>
> From: Nikolay Izhikov
> Sent: 14 декабря 2018 г. 14:42
> To: dev; vololo100@gmail.com; stanlukyanov@gmail.com
> Subject: Re: Pre-touch for Ignite off-heap memory
>
>
>
> Bump.
>
>
>
> Stanislav, Ivan, can you answer my questions?
>
> Let's discuss these improvements.
>
>
>
> ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:
>
>
>
> > Hi Stan,
>
> >
>
> > As I participated in discussion in current thread I would like to
>
> > leave a comment.
>
> >
>
> > Your concerns looks clear for me and if you believe that pre-touch
>
> > will help product users then I have nothing against it.
>
> > ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
>
> > >
>
> > > Hello, Stanislav.
>
> > >
>
> > > As far as I can see, we have controversal ticket based on previous
>
> > dev-list discussion:
>
> > >
>
> > > IGNITE-9113 - Allocate memory for a data region when first cache
>
> > assigned to this region is created [1]
>
> > >
>
> > > I planned to implement it soon.
>
> > > Looks like we should have several options of memory(data regions)
>
> > allocation:
>
> > >
>
> > >         - allocate all on startup (AFAIK this is how current
>
> > implementation behave)
>
> > >         - allocate all on startup AND pre touch.
>
> > >         - allocate specific data region for first assignment.
>
> > >         - allocate specific data region for first assignment AND pre
>
> > touch.
>
> > >
>
> > > What do you think?
>
> > >
>
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> > >
>
> > > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
>
> > > > Igniters,
>
> > > >
>
> > > > What is being suggested here is an Ignite off-heap’s version of Java’s
>
> > -XX:+AlwaysPreTouch.
>
> > > > The latter is known to be used to guarantee that the committed memory
>
> > is backed by physical RAM.
>
> > > > This ensures that
>
> > > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
>
> > for physical page allocation, possible contention with page cache, etc)
>
> > > > b) JVM fails fast if the Xmx is greater than available RAM
>
> > > > c) New processes will not be able to claim the memory JVM took for
>
> > itself
>
> > > >
>
> > > > Currently one can’t get the same benefits for Ignite because we use
>
> > off-heap as well as heap.
>
> > > > So, we can implement a similar feature for Ignite – and make sure the
>
> > users can get all the memory pre-touch benefits if they want.
>
> > > > Of course, it impacts startup time so we should enable it by a
>
> > configuration property (I’d suggest a system property for now).
>
> > > >
>
> > > > Are there any objections to implementing this?
>
> > > >
>
> > > > Thanks,
>
> > > > Stan
>
> > > >
>
> > > > From: Павлухин Иван
>
> > > > Sent: 31 октября 2018 г. 12:50
>
> > > > To: dev@ignite.apache.org
>
> > > > Subject: Re: Pre-touch for Ignite off-heap memory
>
> > > >
>
> > > > Hi,
>
> > > >
>
> > > > I did an experiment described above. I created a patch with pre-touch
>
> > [1]
>
> > > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
>
> > > > Ignite with equal values for initial and max sizes for each data
>
> > region.
>
> > > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
>
> > easy
>
> > > > to observe JVM OOM-killed.
>
> > > >
>
> > > > [1] https://github.com/apache/ignite/pull/5220
>
> > > > [2]
>
> > > >
>
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
>
> > > > [3]
>
> > > >
>
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
>
> > > >
>
> > > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
>
> > > >
>
> > > > > Hi guys,
>
> > > > >
>
> > > > > I am not aware that it is possible to run JVM in "allocation-free"
>
> > fashion.
>
> > > > > If you know that it is possible please share it. As I know JVM
>
> > allocates
>
> > > > > memory out of garbage collectable area for internal purposes like
>
> > JIT,
>
> > > > > GC itself. Also native built-it code can request memory allocation
>
> > from OS,
>
> > > > > e.g. zip facilities. If we imagine that system is running under
>
> > memory
>
> > > > > allocation
>
> > > > > which is close to a limit then even small allocation request can
>
> > fail and
>
> > > > > lead
>
> > > > > to OOM killing.
>
> > > > >
>
> > > > > But I think that a simple and useful thing that could be done first
>
> > is
>
> > > > > making
>
> > > > > an experiment. As Andrey mentioned
>
> > > > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
>
> > > > >
>
> > > > > region
>
> > > > > > MAX size equal to MIN and get region allocated on node start.
>
> > > > >
>
> > > > > If it is so then it should not be hard to launch Ignite and observe
>
> > it
>
> > > > > running
>
> > > > > very close to OS memory limit with Java heap and Ignite off-heap
>
> > > > > pre-touched.
>
> > > > > After that one could check whether it is possible to observe Ignite
>
> > OOM
>
> > > > > killed.
>
> > > > > Let's say my bet is that it is relatively easy to catch OOM here.
>
> > > > >
>
> > > > > What do you think?
>
> > > > >
>
> > > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
>
> > > > >
>
> > > > > > Andrey,
>
> > > > > >
>
> > > > > > Probability of a OOM kill will be much lower if offheap is
>
> > pretouched.
>
> > > > > > What
>
> > > > > > do you mean by JVM internal needs? In my understanding if user
>
> > enables
>
> > > > > > option to pretouch heap and fixes the heap to prevent jvm
>
> > releasing memory
>
> > > > > > back to OS, then OOM killing is very unlikely.
>
> > > > > >
>
> > > > > > I would agree that pretouch for offheap may be helpful in many
>
> > cases.
>
> > > > > >
>
> > > > > > --Yakov
>
> > > > > >
>
> > > > >
>
> > > > >
>
> > > > > --
>
> > > > > Best regards,
>
> > > > > Ivan Pavlukhin
>
> > > > >
>
> > > >
>
> > > >
>
> >
>
> >
>
> >
>
> > --
>
> > Best regards,
>
> > Ivan Pavlukhin
>
> >
>
>



-- 
Best regards,
Ivan Pavlukhin

RE: Pre-touch for Ignite off-heap memory

Posted by Stanislav Lukyanov <st...@gmail.com>.
I don’t think these two issues require to be solved together, although I agree there is some connection.

I guess we agree that pre-touch should a be off by default.
Similarly, lazy data region initialization from IGNITE-9113 can be on by default.
We can have two boolean properties controlling each feature, e.g. 
    IGNITE_EAGER_DATA_REGION_INITIALIZATION
    IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
Setting both to true will make sure that all memory is committed as early as possible.
Different combinations will allow for all 4 variants you’ve mentioned. I can see a use case for each one:
- lazy region init + no pre-touch (default) – easier configuration (at least for simple use cases without a lot of regions and node filters) and faster startup
- lazy region init + pre-touch – allows to reuse server config on a client (the goal of the IGNITE-9113) but still allows to fail-fast on cache creation
- eager region init + no pre-touch – for cases when you want to eagerly allocate memory but don’t care about committing physical pages; e.g. if you have overcommit disabled then you don’t need to pre-touch everything to make sure the memory is there
- eager region init + pre-touch – to fail as early as possible if there is not enough RAM 

Stan

From: Nikolay Izhikov
Sent: 14 декабря 2018 г. 14:42
To: dev; vololo100@gmail.com; stanlukyanov@gmail.com
Subject: Re: Pre-touch for Ignite off-heap memory

Bump.

Stanislav, Ivan, can you answer my questions?
Let's discuss these improvements.

ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:

> Hi Stan,
>
> As I participated in discussion in current thread I would like to
> leave a comment.
>
> Your concerns looks clear for me and if you believe that pre-touch
> will help product users then I have nothing against it.
> ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
> >
> > Hello, Stanislav.
> >
> > As far as I can see, we have controversal ticket based on previous
> dev-list discussion:
> >
> > IGNITE-9113 - Allocate memory for a data region when first cache
> assigned to this region is created [1]
> >
> > I planned to implement it soon.
> > Looks like we should have several options of memory(data regions)
> allocation:
> >
> >         - allocate all on startup (AFAIK this is how current
> implementation behave)
> >         - allocate all on startup AND pre touch.
> >         - allocate specific data region for first assignment.
> >         - allocate specific data region for first assignment AND pre
> touch.
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > > Igniters,
> > >
> > > What is being suggested here is an Ignite off-heap’s version of Java’s
> -XX:+AlwaysPreTouch.
> > > The latter is known to be used to guarantee that the committed memory
> is backed by physical RAM.
> > > This ensures that
> > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
> for physical page allocation, possible contention with page cache, etc)
> > > b) JVM fails fast if the Xmx is greater than available RAM
> > > c) New processes will not be able to claim the memory JVM took for
> itself
> > >
> > > Currently one can’t get the same benefits for Ignite because we use
> off-heap as well as heap.
> > > So, we can implement a similar feature for Ignite – and make sure the
> users can get all the memory pre-touch benefits if they want.
> > > Of course, it impacts startup time so we should enable it by a
> configuration property (I’d suggest a system property for now).
> > >
> > > Are there any objections to implementing this?
> > >
> > > Thanks,
> > > Stan
> > >
> > > From: Павлухин Иван
> > > Sent: 31 октября 2018 г. 12:50
> > > To: dev@ignite.apache.org
> > > Subject: Re: Pre-touch for Ignite off-heap memory
> > >
> > > Hi,
> > >
> > > I did an experiment described above. I created a patch with pre-touch
> [1]
> > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > > Ignite with equal values for initial and max sizes for each data
> region.
> > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
> easy
> > > to observe JVM OOM-killed.
> > >
> > > [1] https://github.com/apache/ignite/pull/5220
> > > [2]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> > > [3]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> > >
> > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> > >
> > > > Hi guys,
> > > >
> > > > I am not aware that it is possible to run JVM in "allocation-free"
> fashion.
> > > > If you know that it is possible please share it. As I know JVM
> allocates
> > > > memory out of garbage collectable area for internal purposes like
> JIT,
> > > > GC itself. Also native built-it code can request memory allocation
> from OS,
> > > > e.g. zip facilities. If we imagine that system is running under
> memory
> > > > allocation
> > > > which is close to a limit then even small allocation request can
> fail and
> > > > lead
> > > > to OOM killing.
> > > >
> > > > But I think that a simple and useful thing that could be done first
> is
> > > > making
> > > > an experiment. As Andrey mentioned
> > > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > > >
> > > > region
> > > > > MAX size equal to MIN and get region allocated on node start.
> > > >
> > > > If it is so then it should not be hard to launch Ignite and observe
> it
> > > > running
> > > > very close to OS memory limit with Java heap and Ignite off-heap
> > > > pre-touched.
> > > > After that one could check whether it is possible to observe Ignite
> OOM
> > > > killed.
> > > > Let's say my bet is that it is relatively easy to catch OOM here.
> > > >
> > > > What do you think?
> > > >
> > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> > > >
> > > > > Andrey,
> > > > >
> > > > > Probability of a OOM kill will be much lower if offheap is
> pretouched.
> > > > > What
> > > > > do you mean by JVM internal needs? In my understanding if user
> enables
> > > > > option to pretouch heap and fixes the heap to prevent jvm
> releasing memory
> > > > > back to OS, then OOM killing is very unlikely.
> > > > >
> > > > > I would agree that pretouch for offheap may be helpful in many
> cases.
> > > > >
> > > > > --Yakov
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Pre-touch for Ignite off-heap memory

Posted by Nikolay Izhikov <ni...@apache.org>.
Bump.

Stanislav, Ivan, can you answer my questions?
Let's discuss these improvements.

ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:

> Hi Stan,
>
> As I participated in discussion in current thread I would like to
> leave a comment.
>
> Your concerns looks clear for me and if you believe that pre-touch
> will help product users then I have nothing against it.
> ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
> >
> > Hello, Stanislav.
> >
> > As far as I can see, we have controversal ticket based on previous
> dev-list discussion:
> >
> > IGNITE-9113 - Allocate memory for a data region when first cache
> assigned to this region is created [1]
> >
> > I planned to implement it soon.
> > Looks like we should have several options of memory(data regions)
> allocation:
> >
> >         - allocate all on startup (AFAIK this is how current
> implementation behave)
> >         - allocate all on startup AND pre touch.
> >         - allocate specific data region for first assignment.
> >         - allocate specific data region for first assignment AND pre
> touch.
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > > Igniters,
> > >
> > > What is being suggested here is an Ignite off-heap’s version of Java’s
> -XX:+AlwaysPreTouch.
> > > The latter is known to be used to guarantee that the committed memory
> is backed by physical RAM.
> > > This ensures that
> > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
> for physical page allocation, possible contention with page cache, etc)
> > > b) JVM fails fast if the Xmx is greater than available RAM
> > > c) New processes will not be able to claim the memory JVM took for
> itself
> > >
> > > Currently one can’t get the same benefits for Ignite because we use
> off-heap as well as heap.
> > > So, we can implement a similar feature for Ignite – and make sure the
> users can get all the memory pre-touch benefits if they want.
> > > Of course, it impacts startup time so we should enable it by a
> configuration property (I’d suggest a system property for now).
> > >
> > > Are there any objections to implementing this?
> > >
> > > Thanks,
> > > Stan
> > >
> > > From: Павлухин Иван
> > > Sent: 31 октября 2018 г. 12:50
> > > To: dev@ignite.apache.org
> > > Subject: Re: Pre-touch for Ignite off-heap memory
> > >
> > > Hi,
> > >
> > > I did an experiment described above. I created a patch with pre-touch
> [1]
> > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > > Ignite with equal values for initial and max sizes for each data
> region.
> > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
> easy
> > > to observe JVM OOM-killed.
> > >
> > > [1] https://github.com/apache/ignite/pull/5220
> > > [2]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> > > [3]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> > >
> > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> > >
> > > > Hi guys,
> > > >
> > > > I am not aware that it is possible to run JVM in "allocation-free"
> fashion.
> > > > If you know that it is possible please share it. As I know JVM
> allocates
> > > > memory out of garbage collectable area for internal purposes like
> JIT,
> > > > GC itself. Also native built-it code can request memory allocation
> from OS,
> > > > e.g. zip facilities. If we imagine that system is running under
> memory
> > > > allocation
> > > > which is close to a limit then even small allocation request can
> fail and
> > > > lead
> > > > to OOM killing.
> > > >
> > > > But I think that a simple and useful thing that could be done first
> is
> > > > making
> > > > an experiment. As Andrey mentioned
> > > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > > >
> > > > region
> > > > > MAX size equal to MIN and get region allocated on node start.
> > > >
> > > > If it is so then it should not be hard to launch Ignite and observe
> it
> > > > running
> > > > very close to OS memory limit with Java heap and Ignite off-heap
> > > > pre-touched.
> > > > After that one could check whether it is possible to observe Ignite
> OOM
> > > > killed.
> > > > Let's say my bet is that it is relatively easy to catch OOM here.
> > > >
> > > > What do you think?
> > > >
> > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> > > >
> > > > > Andrey,
> > > > >
> > > > > Probability of a OOM kill will be much lower if offheap is
> pretouched.
> > > > > What
> > > > > do you mean by JVM internal needs? In my understanding if user
> enables
> > > > > option to pretouch heap and fixes the heap to prevent jvm
> releasing memory
> > > > > back to OS, then OOM killing is very unlikely.
> > > > >
> > > > > I would agree that pretouch for offheap may be helpful in many
> cases.
> > > > >
> > > > > --Yakov
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Pre-touch for Ignite off-heap memory

Posted by Павлухин Иван <vo...@gmail.com>.
Hi Stan,

As I participated in discussion in current thread I would like to
leave a comment.

Your concerns looks clear for me and if you believe that pre-touch
will help product users then I have nothing against it.
ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
>
> Hello, Stanislav.
>
> As far as I can see, we have controversal ticket based on previous dev-list discussion:
>
> IGNITE-9113 - Allocate memory for a data region when first cache assigned to this region is created [1]
>
> I planned to implement it soon.
> Looks like we should have several options of memory(data regions) allocation:
>
>         - allocate all on startup (AFAIK this is how current implementation behave)
>         - allocate all on startup AND pre touch.
>         - allocate specific data region for first assignment.
>         - allocate specific data region for first assignment AND pre touch.
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > Igniters,
> >
> > What is being suggested here is an Ignite off-heap’s version of Java’s -XX:+AlwaysPreTouch.
> > The latter is known to be used to guarantee that the committed memory is backed by physical RAM.
> > This ensures that
> > a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
> > b) JVM fails fast if the Xmx is greater than available RAM
> > c) New processes will not be able to claim the memory JVM took for itself
> >
> > Currently one can’t get the same benefits for Ignite because we use off-heap as well as heap.
> > So, we can implement a similar feature for Ignite – and make sure the users can get all the memory pre-touch benefits if they want.
> > Of course, it impacts startup time so we should enable it by a configuration property (I’d suggest a system property for now).
> >
> > Are there any objections to implementing this?
> >
> > Thanks,
> > Stan
> >
> > From: Павлухин Иван
> > Sent: 31 октября 2018 г. 12:50
> > To: dev@ignite.apache.org
> > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> > Hi,
> >
> > I did an experiment described above. I created a patch with pre-touch [1]
> > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > Ignite with equal values for initial and max sizes for each data region.
> > I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
> > to observe JVM OOM-killed.
> >
> > [1] https://github.com/apache/ignite/pull/5220
> > [2]
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> > [3]
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> >
> > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> >
> > > Hi guys,
> > >
> > > I am not aware that it is possible to run JVM in "allocation-free" fashion.
> > > If you know that it is possible please share it. As I know JVM allocates
> > > memory out of garbage collectable area for internal purposes like JIT,
> > > GC itself. Also native built-it code can request memory allocation from OS,
> > > e.g. zip facilities. If we imagine that system is running under memory
> > > allocation
> > > which is close to a limit then even small allocation request can fail and
> > > lead
> > > to OOM killing.
> > >
> > > But I think that a simple and useful thing that could be done first is
> > > making
> > > an experiment. As Andrey mentioned
> > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > >
> > > region
> > > > MAX size equal to MIN and get region allocated on node start.
> > >
> > > If it is so then it should not be hard to launch Ignite and observe it
> > > running
> > > very close to OS memory limit with Java heap and Ignite off-heap
> > > pre-touched.
> > > After that one could check whether it is possible to observe Ignite OOM
> > > killed.
> > > Let's say my bet is that it is relatively easy to catch OOM here.
> > >
> > > What do you think?
> > >
> > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> > >
> > > > Andrey,
> > > >
> > > > Probability of a OOM kill will be much lower if offheap is pretouched.
> > > > What
> > > > do you mean by JVM internal needs? In my understanding if user enables
> > > > option to pretouch heap and fixes the heap to prevent jvm releasing memory
> > > > back to OS, then OOM killing is very unlikely.
> > > >
> > > > I would agree that pretouch for offheap may be helpful in many cases.
> > > >
> > > > --Yakov
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
> >



-- 
Best regards,
Ivan Pavlukhin

Re: Pre-touch for Ignite off-heap memory

Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Stanislav.

As far as I can see, we have controversal ticket based on previous dev-list discussion:

IGNITE-9113 - Allocate memory for a data region when first cache assigned to this region is created [1]

I planned to implement it soon.
Looks like we should have several options of memory(data regions) allocation:

	- allocate all on startup (AFAIK this is how current implementation behave)
	- allocate all on startup AND pre touch.
	- allocate specific data region for first assignment.
	- allocate specific data region for first assignment AND pre touch.

What do you think?

[1] https://issues.apache.org/jira/browse/IGNITE-9113

В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> Igniters,
> 
> What is being suggested here is an Ignite off-heap’s version of Java’s -XX:+AlwaysPreTouch.
> The latter is known to be used to guarantee that the committed memory is backed by physical RAM.
> This ensures that
> a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
> b) JVM fails fast if the Xmx is greater than available RAM
> c) New processes will not be able to claim the memory JVM took for itself
> 
> Currently one can’t get the same benefits for Ignite because we use off-heap as well as heap.
> So, we can implement a similar feature for Ignite – and make sure the users can get all the memory pre-touch benefits if they want.
> Of course, it impacts startup time so we should enable it by a configuration property (I’d suggest a system property for now).
> 
> Are there any objections to implementing this?
> 
> Thanks,
> Stan
> 
> From: Павлухин Иван
> Sent: 31 октября 2018 г. 12:50
> To: dev@ignite.apache.org
> Subject: Re: Pre-touch for Ignite off-heap memory
> 
> Hi,
> 
> I did an experiment described above. I created a patch with pre-touch [1]
> and started a JVM with an option -XX:+AlwaysPreTouch and configured
> Ignite with equal values for initial and max sizes for each data region.
> I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
> to observe JVM OOM-killed.
> 
> [1] https://github.com/apache/ignite/pull/5220
> [2]
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> [3]
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> 
> вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> 
> > Hi guys,
> > 
> > I am not aware that it is possible to run JVM in "allocation-free" fashion.
> > If you know that it is possible please share it. As I know JVM allocates
> > memory out of garbage collectable area for internal purposes like JIT,
> > GC itself. Also native built-it code can request memory allocation from OS,
> > e.g. zip facilities. If we imagine that system is running under memory
> > allocation
> > which is close to a limit then even small allocation request can fail and
> > lead
> > to OOM killing.
> > 
> > But I think that a simple and useful thing that could be done first is
> > making
> > an experiment. As Andrey mentioned
> > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > 
> > region
> > > MAX size equal to MIN and get region allocated on node start.
> > 
> > If it is so then it should not be hard to launch Ignite and observe it
> > running
> > very close to OS memory limit with Java heap and Ignite off-heap
> > pre-touched.
> > After that one could check whether it is possible to observe Ignite OOM
> > killed.
> > Let's say my bet is that it is relatively easy to catch OOM here.
> > 
> > What do you think?
> > 
> > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> > 
> > > Andrey,
> > > 
> > > Probability of a OOM kill will be much lower if offheap is pretouched.
> > > What
> > > do you mean by JVM internal needs? In my understanding if user enables
> > > option to pretouch heap and fixes the heap to prevent jvm releasing memory
> > > back to OS, then OOM killing is very unlikely.
> > > 
> > > I would agree that pretouch for offheap may be helpful in many cases.
> > > 
> > > --Yakov
> > > 
> > 
> > 
> > --
> > Best regards,
> > Ivan Pavlukhin
> > 
> 
>