You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Gerus <ag...@gridgain.com> on 2018/10/23 19:08:48 UTC
Pre-touch for Ignite off-heap memory
Hi *Igniters*,
Some time ago I've raised a suggestion for product improvement
https://issues.apache.org/jira/browse/IGNITE-9112
<https://issues.apache.org/jira/browse/IGNITE-9112> . It's all about
off-heap memory allocation. Current implementation can have some
improvements for failure critical systems. Ignite can have OOM in runtime,
because RAM can be used by OS, if it will not be pre-booked by operation
system and this proposal is to address that. Common case is offheap and
thats why memory segment cannot be managed by JVM that has +AlwaysPreTouch
option
Obviously this implementation will make startup longer and thats why it is
proposed to use configuration flag to manage this feature
I think, it will be useful to have this option. Are you supporting this?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Pre-touch for Ignite off-heap memory
Posted by Denis Magda <dm...@apache.org>.
In my understanding, that's an easy operation to support - we just need to
allocate requested space and pre-touch it (iterate and fill out with 0).
Yes, it will increase the launch time but it's a valid use case. Some
companies do memory pre-touching on purpose.
--
Denis
On Thu, Oct 25, 2018 at 7:41 AM Gerus <ag...@gridgain.com> wrote:
> Hi, Ivan
> Let me try to explain. I think that there are 2 goals that are mostly
> applicable for prod systems:
> 1. I can agree with Dave that swap is not a good case for performance and
> disk resource. It is possible that other applications can consume memory
> that is free after Ignite was started. In this case if Ignite will be
> creating new pages in runtime, it can face with OOM.
>
> 2. Ignite user can create wrong configuration. For example, data region can
> exceed available memory by mistake or Ignite can be started on another
> server with less RAM. It can lead to catch OOM in runtime
>
> To summarize: Pre-allocation can detect listed issues on Ignite start by
> allocating OS RAM
>
> Im sure that this option have to be disabled by default, but user should
> have a choice for startup
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Pre-touch for Ignite off-heap memory
Posted by Stanislav Lukyanov <st...@gmail.com>.
Well, I'm not saying "we need all three features all the time", I'm just
summarizing why someone would use pre-touch.
And having c) is better than having none.
Also, we do get a) and b) as well, the timing is just a bit different.
But I find it better to fail at the time of dynamic cache creation (which
should be approached with caution anyway) than
during the normal execution.
Stan
On Tue, Dec 18, 2018 at 4:01 PM Павлухин Иван <vo...@gmail.com> wrote:
> Stan,
>
> In one of your emails you wrote why you need pre-touch:
> > a) JVM doesn’t have to do it during the actual work (avoiding overhead
> for physical page allocation, possible contention with page cache, etc)
> > b) JVM fails fast if the Xmx is greater than available RAM
> > c) New processes will not be able to claim the memory JVM took for itself
>
> As far as I see only c) can be satisfied if lazy region initialization
> is enabled. Am I missing something?
> пн, 17 дек. 2018 г. в 18:11, Stanislav Lukyanov <st...@gmail.com>:
> >
> > I don’t think these two issues require to be solved together, although I
> agree there is some connection.
> >
> >
> >
> > I guess we agree that pre-touch should a be off by default.
> >
> > Similarly, lazy data region initialization from IGNITE-9113 can be on by
> default.
> >
> > We can have two boolean properties controlling each feature, e.g.
> >
> > IGNITE_EAGER_DATA_REGION_INITIALIZATION
> >
> > IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
> >
> > Setting both to true will make sure that all memory is committed as
> early as possible.
> >
> > Different combinations will allow for all 4 variants you’ve mentioned. I
> can see a use case for each one:
> >
> > - lazy region init + no pre-touch (default) – easier configuration (at
> least for simple use cases without a lot of regions and node filters) and
> faster startup
> >
> > - lazy region init + pre-touch – allows to reuse server config on a
> client (the goal of the IGNITE-9113) but still allows to fail-fast on cache
> creation
> >
> > - eager region init + no pre-touch – for cases when you want to eagerly
> allocate memory but don’t care about committing physical pages; e.g. if you
> have overcommit disabled then you don’t need to pre-touch everything to
> make sure the memory is there
> >
> > - eager region init + pre-touch – to fail as early as possible if there
> is not enough RAM
> >
> >
> >
> > Stan
> >
> >
> >
> > From: Nikolay Izhikov
> > Sent: 14 декабря 2018 г. 14:42
> > To: dev; vololo100@gmail.com; stanlukyanov@gmail.com
> > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> >
> >
> > Bump.
> >
> >
> >
> > Stanislav, Ivan, can you answer my questions?
> >
> > Let's discuss these improvements.
> >
> >
> >
> > ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:
> >
> >
> >
> > > Hi Stan,
> >
> > >
> >
> > > As I participated in discussion in current thread I would like to
> >
> > > leave a comment.
> >
> > >
> >
> > > Your concerns looks clear for me and if you believe that pre-touch
> >
> > > will help product users then I have nothing against it.
> >
> > > ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
> >
> > > >
> >
> > > > Hello, Stanislav.
> >
> > > >
> >
> > > > As far as I can see, we have controversal ticket based on previous
> >
> > > dev-list discussion:
> >
> > > >
> >
> > > > IGNITE-9113 - Allocate memory for a data region when first cache
> >
> > > assigned to this region is created [1]
> >
> > > >
> >
> > > > I planned to implement it soon.
> >
> > > > Looks like we should have several options of memory(data regions)
> >
> > > allocation:
> >
> > > >
> >
> > > > - allocate all on startup (AFAIK this is how current
> >
> > > implementation behave)
> >
> > > > - allocate all on startup AND pre touch.
> >
> > > > - allocate specific data region for first assignment.
> >
> > > > - allocate specific data region for first assignment AND pre
> >
> > > touch.
> >
> > > >
> >
> > > > What do you think?
> >
> > > >
> >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > > >
> >
> > > > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> >
> > > > > Igniters,
> >
> > > > >
> >
> > > > > What is being suggested here is an Ignite off-heap’s version of
> Java’s
> >
> > > -XX:+AlwaysPreTouch.
> >
> > > > > The latter is known to be used to guarantee that the committed
> memory
> >
> > > is backed by physical RAM.
> >
> > > > > This ensures that
> >
> > > > > a) JVM doesn’t have to do it during the actual work (avoiding
> overhead
> >
> > > for physical page allocation, possible contention with page cache, etc)
> >
> > > > > b) JVM fails fast if the Xmx is greater than available RAM
> >
> > > > > c) New processes will not be able to claim the memory JVM took for
> >
> > > itself
> >
> > > > >
> >
> > > > > Currently one can’t get the same benefits for Ignite because we use
> >
> > > off-heap as well as heap.
> >
> > > > > So, we can implement a similar feature for Ignite – and make sure
> the
> >
> > > users can get all the memory pre-touch benefits if they want.
> >
> > > > > Of course, it impacts startup time so we should enable it by a
> >
> > > configuration property (I’d suggest a system property for now).
> >
> > > > >
> >
> > > > > Are there any objections to implementing this?
> >
> > > > >
> >
> > > > > Thanks,
> >
> > > > > Stan
> >
> > > > >
> >
> > > > > From: Павлухин Иван
> >
> > > > > Sent: 31 октября 2018 г. 12:50
> >
> > > > > To: dev@ignite.apache.org
> >
> > > > > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> > > > >
> >
> > > > > Hi,
> >
> > > > >
> >
> > > > > I did an experiment described above. I created a patch with
> pre-touch
> >
> > > [1]
> >
> > > > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> >
> > > > > Ignite with equal values for initial and max sizes for each data
> >
> > > region.
> >
> > > > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
> >
> > > easy
> >
> > > > > to observe JVM OOM-killed.
> >
> > > > >
> >
> > > > > [1] https://github.com/apache/ignite/pull/5220
> >
> > > > > [2]
> >
> > > > >
> >
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> >
> > > > > [3]
> >
> > > > >
> >
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> >
> > > > >
> >
> > > > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> >
> > > > >
> >
> > > > > > Hi guys,
> >
> > > > > >
> >
> > > > > > I am not aware that it is possible to run JVM in
> "allocation-free"
> >
> > > fashion.
> >
> > > > > > If you know that it is possible please share it. As I know JVM
> >
> > > allocates
> >
> > > > > > memory out of garbage collectable area for internal purposes like
> >
> > > JIT,
> >
> > > > > > GC itself. Also native built-it code can request memory
> allocation
> >
> > > from OS,
> >
> > > > > > e.g. zip facilities. If we imagine that system is running under
> >
> > > memory
> >
> > > > > > allocation
> >
> > > > > > which is close to a limit then even small allocation request can
> >
> > > fail and
> >
> > > > > > lead
> >
> > > > > > to OOM killing.
> >
> > > > > >
> >
> > > > > > But I think that a simple and useful thing that could be done
> first
> >
> > > is
> >
> > > > > > making
> >
> > > > > > an experiment. As Andrey mentioned
> >
> > > > > > > AFAIK, Ignite always pre-touch first region. So, you can try
> to set
> >
> > > > > >
> >
> > > > > > region
> >
> > > > > > > MAX size equal to MIN and get region allocated on node start.
> >
> > > > > >
> >
> > > > > > If it is so then it should not be hard to launch Ignite and
> observe
> >
> > > it
> >
> > > > > > running
> >
> > > > > > very close to OS memory limit with Java heap and Ignite off-heap
> >
> > > > > > pre-touched.
> >
> > > > > > After that one could check whether it is possible to observe
> Ignite
> >
> > > OOM
> >
> > > > > > killed.
> >
> > > > > > Let's say my bet is that it is relatively easy to catch OOM here.
> >
> > > > > >
> >
> > > > > > What do you think?
> >
> > > > > >
> >
> > > > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yzhdanov@apache.org
> >:
> >
> > > > > >
> >
> > > > > > > Andrey,
> >
> > > > > > >
> >
> > > > > > > Probability of a OOM kill will be much lower if offheap is
> >
> > > pretouched.
> >
> > > > > > > What
> >
> > > > > > > do you mean by JVM internal needs? In my understanding if user
> >
> > > enables
> >
> > > > > > > option to pretouch heap and fixes the heap to prevent jvm
> >
> > > releasing memory
> >
> > > > > > > back to OS, then OOM killing is very unlikely.
> >
> > > > > > >
> >
> > > > > > > I would agree that pretouch for offheap may be helpful in many
> >
> > > cases.
> >
> > > > > > >
> >
> > > > > > > --Yakov
> >
> > > > > > >
> >
> > > > > >
> >
> > > > > >
> >
> > > > > > --
> >
> > > > > > Best regards,
> >
> > > > > > Ivan Pavlukhin
> >
> > > > > >
> >
> > > > >
> >
> > > > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > --
> >
> > > Best regards,
> >
> > > Ivan Pavlukhin
> >
> > >
> >
> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: Pre-touch for Ignite off-heap memory
Posted by Павлухин Иван <vo...@gmail.com>.
Stan,
In one of your emails you wrote why you need pre-touch:
> a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
> b) JVM fails fast if the Xmx is greater than available RAM
> c) New processes will not be able to claim the memory JVM took for itself
As far as I see only c) can be satisfied if lazy region initialization
is enabled. Am I missing something?
пн, 17 дек. 2018 г. в 18:11, Stanislav Lukyanov <st...@gmail.com>:
>
> I don’t think these two issues require to be solved together, although I agree there is some connection.
>
>
>
> I guess we agree that pre-touch should a be off by default.
>
> Similarly, lazy data region initialization from IGNITE-9113 can be on by default.
>
> We can have two boolean properties controlling each feature, e.g.
>
> IGNITE_EAGER_DATA_REGION_INITIALIZATION
>
> IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
>
> Setting both to true will make sure that all memory is committed as early as possible.
>
> Different combinations will allow for all 4 variants you’ve mentioned. I can see a use case for each one:
>
> - lazy region init + no pre-touch (default) – easier configuration (at least for simple use cases without a lot of regions and node filters) and faster startup
>
> - lazy region init + pre-touch – allows to reuse server config on a client (the goal of the IGNITE-9113) but still allows to fail-fast on cache creation
>
> - eager region init + no pre-touch – for cases when you want to eagerly allocate memory but don’t care about committing physical pages; e.g. if you have overcommit disabled then you don’t need to pre-touch everything to make sure the memory is there
>
> - eager region init + pre-touch – to fail as early as possible if there is not enough RAM
>
>
>
> Stan
>
>
>
> From: Nikolay Izhikov
> Sent: 14 декабря 2018 г. 14:42
> To: dev; vololo100@gmail.com; stanlukyanov@gmail.com
> Subject: Re: Pre-touch for Ignite off-heap memory
>
>
>
> Bump.
>
>
>
> Stanislav, Ivan, can you answer my questions?
>
> Let's discuss these improvements.
>
>
>
> ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:
>
>
>
> > Hi Stan,
>
> >
>
> > As I participated in discussion in current thread I would like to
>
> > leave a comment.
>
> >
>
> > Your concerns looks clear for me and if you believe that pre-touch
>
> > will help product users then I have nothing against it.
>
> > ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
>
> > >
>
> > > Hello, Stanislav.
>
> > >
>
> > > As far as I can see, we have controversal ticket based on previous
>
> > dev-list discussion:
>
> > >
>
> > > IGNITE-9113 - Allocate memory for a data region when first cache
>
> > assigned to this region is created [1]
>
> > >
>
> > > I planned to implement it soon.
>
> > > Looks like we should have several options of memory(data regions)
>
> > allocation:
>
> > >
>
> > > - allocate all on startup (AFAIK this is how current
>
> > implementation behave)
>
> > > - allocate all on startup AND pre touch.
>
> > > - allocate specific data region for first assignment.
>
> > > - allocate specific data region for first assignment AND pre
>
> > touch.
>
> > >
>
> > > What do you think?
>
> > >
>
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> > >
>
> > > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
>
> > > > Igniters,
>
> > > >
>
> > > > What is being suggested here is an Ignite off-heap’s version of Java’s
>
> > -XX:+AlwaysPreTouch.
>
> > > > The latter is known to be used to guarantee that the committed memory
>
> > is backed by physical RAM.
>
> > > > This ensures that
>
> > > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
>
> > for physical page allocation, possible contention with page cache, etc)
>
> > > > b) JVM fails fast if the Xmx is greater than available RAM
>
> > > > c) New processes will not be able to claim the memory JVM took for
>
> > itself
>
> > > >
>
> > > > Currently one can’t get the same benefits for Ignite because we use
>
> > off-heap as well as heap.
>
> > > > So, we can implement a similar feature for Ignite – and make sure the
>
> > users can get all the memory pre-touch benefits if they want.
>
> > > > Of course, it impacts startup time so we should enable it by a
>
> > configuration property (I’d suggest a system property for now).
>
> > > >
>
> > > > Are there any objections to implementing this?
>
> > > >
>
> > > > Thanks,
>
> > > > Stan
>
> > > >
>
> > > > From: Павлухин Иван
>
> > > > Sent: 31 октября 2018 г. 12:50
>
> > > > To: dev@ignite.apache.org
>
> > > > Subject: Re: Pre-touch for Ignite off-heap memory
>
> > > >
>
> > > > Hi,
>
> > > >
>
> > > > I did an experiment described above. I created a patch with pre-touch
>
> > [1]
>
> > > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
>
> > > > Ignite with equal values for initial and max sizes for each data
>
> > region.
>
> > > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
>
> > easy
>
> > > > to observe JVM OOM-killed.
>
> > > >
>
> > > > [1] https://github.com/apache/ignite/pull/5220
>
> > > > [2]
>
> > > >
>
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
>
> > > > [3]
>
> > > >
>
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
>
> > > >
>
> > > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
>
> > > >
>
> > > > > Hi guys,
>
> > > > >
>
> > > > > I am not aware that it is possible to run JVM in "allocation-free"
>
> > fashion.
>
> > > > > If you know that it is possible please share it. As I know JVM
>
> > allocates
>
> > > > > memory out of garbage collectable area for internal purposes like
>
> > JIT,
>
> > > > > GC itself. Also native built-it code can request memory allocation
>
> > from OS,
>
> > > > > e.g. zip facilities. If we imagine that system is running under
>
> > memory
>
> > > > > allocation
>
> > > > > which is close to a limit then even small allocation request can
>
> > fail and
>
> > > > > lead
>
> > > > > to OOM killing.
>
> > > > >
>
> > > > > But I think that a simple and useful thing that could be done first
>
> > is
>
> > > > > making
>
> > > > > an experiment. As Andrey mentioned
>
> > > > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
>
> > > > >
>
> > > > > region
>
> > > > > > MAX size equal to MIN and get region allocated on node start.
>
> > > > >
>
> > > > > If it is so then it should not be hard to launch Ignite and observe
>
> > it
>
> > > > > running
>
> > > > > very close to OS memory limit with Java heap and Ignite off-heap
>
> > > > > pre-touched.
>
> > > > > After that one could check whether it is possible to observe Ignite
>
> > OOM
>
> > > > > killed.
>
> > > > > Let's say my bet is that it is relatively easy to catch OOM here.
>
> > > > >
>
> > > > > What do you think?
>
> > > > >
>
> > > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
>
> > > > >
>
> > > > > > Andrey,
>
> > > > > >
>
> > > > > > Probability of a OOM kill will be much lower if offheap is
>
> > pretouched.
>
> > > > > > What
>
> > > > > > do you mean by JVM internal needs? In my understanding if user
>
> > enables
>
> > > > > > option to pretouch heap and fixes the heap to prevent jvm
>
> > releasing memory
>
> > > > > > back to OS, then OOM killing is very unlikely.
>
> > > > > >
>
> > > > > > I would agree that pretouch for offheap may be helpful in many
>
> > cases.
>
> > > > > >
>
> > > > > > --Yakov
>
> > > > > >
>
> > > > >
>
> > > > >
>
> > > > > --
>
> > > > > Best regards,
>
> > > > > Ivan Pavlukhin
>
> > > > >
>
> > > >
>
> > > >
>
> >
>
> >
>
> >
>
> > --
>
> > Best regards,
>
> > Ivan Pavlukhin
>
> >
>
>
--
Best regards,
Ivan Pavlukhin
RE: Pre-touch for Ignite off-heap memory
Posted by Stanislav Lukyanov <st...@gmail.com>.
I don’t think these two issues require to be solved together, although I agree there is some connection.
I guess we agree that pre-touch should a be off by default.
Similarly, lazy data region initialization from IGNITE-9113 can be on by default.
We can have two boolean properties controlling each feature, e.g.
IGNITE_EAGER_DATA_REGION_INITIALIZATION
IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
Setting both to true will make sure that all memory is committed as early as possible.
Different combinations will allow for all 4 variants you’ve mentioned. I can see a use case for each one:
- lazy region init + no pre-touch (default) – easier configuration (at least for simple use cases without a lot of regions and node filters) and faster startup
- lazy region init + pre-touch – allows to reuse server config on a client (the goal of the IGNITE-9113) but still allows to fail-fast on cache creation
- eager region init + no pre-touch – for cases when you want to eagerly allocate memory but don’t care about committing physical pages; e.g. if you have overcommit disabled then you don’t need to pre-touch everything to make sure the memory is there
- eager region init + pre-touch – to fail as early as possible if there is not enough RAM
Stan
From: Nikolay Izhikov
Sent: 14 декабря 2018 г. 14:42
To: dev; vololo100@gmail.com; stanlukyanov@gmail.com
Subject: Re: Pre-touch for Ignite off-heap memory
Bump.
Stanislav, Ivan, can you answer my questions?
Let's discuss these improvements.
ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:
> Hi Stan,
>
> As I participated in discussion in current thread I would like to
> leave a comment.
>
> Your concerns looks clear for me and if you believe that pre-touch
> will help product users then I have nothing against it.
> ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
> >
> > Hello, Stanislav.
> >
> > As far as I can see, we have controversal ticket based on previous
> dev-list discussion:
> >
> > IGNITE-9113 - Allocate memory for a data region when first cache
> assigned to this region is created [1]
> >
> > I planned to implement it soon.
> > Looks like we should have several options of memory(data regions)
> allocation:
> >
> > - allocate all on startup (AFAIK this is how current
> implementation behave)
> > - allocate all on startup AND pre touch.
> > - allocate specific data region for first assignment.
> > - allocate specific data region for first assignment AND pre
> touch.
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > > Igniters,
> > >
> > > What is being suggested here is an Ignite off-heap’s version of Java’s
> -XX:+AlwaysPreTouch.
> > > The latter is known to be used to guarantee that the committed memory
> is backed by physical RAM.
> > > This ensures that
> > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
> for physical page allocation, possible contention with page cache, etc)
> > > b) JVM fails fast if the Xmx is greater than available RAM
> > > c) New processes will not be able to claim the memory JVM took for
> itself
> > >
> > > Currently one can’t get the same benefits for Ignite because we use
> off-heap as well as heap.
> > > So, we can implement a similar feature for Ignite – and make sure the
> users can get all the memory pre-touch benefits if they want.
> > > Of course, it impacts startup time so we should enable it by a
> configuration property (I’d suggest a system property for now).
> > >
> > > Are there any objections to implementing this?
> > >
> > > Thanks,
> > > Stan
> > >
> > > From: Павлухин Иван
> > > Sent: 31 октября 2018 г. 12:50
> > > To: dev@ignite.apache.org
> > > Subject: Re: Pre-touch for Ignite off-heap memory
> > >
> > > Hi,
> > >
> > > I did an experiment described above. I created a patch with pre-touch
> [1]
> > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > > Ignite with equal values for initial and max sizes for each data
> region.
> > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
> easy
> > > to observe JVM OOM-killed.
> > >
> > > [1] https://github.com/apache/ignite/pull/5220
> > > [2]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> > > [3]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> > >
> > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> > >
> > > > Hi guys,
> > > >
> > > > I am not aware that it is possible to run JVM in "allocation-free"
> fashion.
> > > > If you know that it is possible please share it. As I know JVM
> allocates
> > > > memory out of garbage collectable area for internal purposes like
> JIT,
> > > > GC itself. Also native built-it code can request memory allocation
> from OS,
> > > > e.g. zip facilities. If we imagine that system is running under
> memory
> > > > allocation
> > > > which is close to a limit then even small allocation request can
> fail and
> > > > lead
> > > > to OOM killing.
> > > >
> > > > But I think that a simple and useful thing that could be done first
> is
> > > > making
> > > > an experiment. As Andrey mentioned
> > > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > > >
> > > > region
> > > > > MAX size equal to MIN and get region allocated on node start.
> > > >
> > > > If it is so then it should not be hard to launch Ignite and observe
> it
> > > > running
> > > > very close to OS memory limit with Java heap and Ignite off-heap
> > > > pre-touched.
> > > > After that one could check whether it is possible to observe Ignite
> OOM
> > > > killed.
> > > > Let's say my bet is that it is relatively easy to catch OOM here.
> > > >
> > > > What do you think?
> > > >
> > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> > > >
> > > > > Andrey,
> > > > >
> > > > > Probability of a OOM kill will be much lower if offheap is
> pretouched.
> > > > > What
> > > > > do you mean by JVM internal needs? In my understanding if user
> enables
> > > > > option to pretouch heap and fixes the heap to prevent jvm
> releasing memory
> > > > > back to OS, then OOM killing is very unlikely.
> > > > >
> > > > > I would agree that pretouch for offheap may be helpful in many
> cases.
> > > > >
> > > > > --Yakov
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: Pre-touch for Ignite off-heap memory
Posted by Nikolay Izhikov <ni...@apache.org>.
Bump.
Stanislav, Ivan, can you answer my questions?
Let's discuss these improvements.
ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vo...@gmail.com>:
> Hi Stan,
>
> As I participated in discussion in current thread I would like to
> leave a comment.
>
> Your concerns looks clear for me and if you believe that pre-touch
> will help product users then I have nothing against it.
> ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
> >
> > Hello, Stanislav.
> >
> > As far as I can see, we have controversal ticket based on previous
> dev-list discussion:
> >
> > IGNITE-9113 - Allocate memory for a data region when first cache
> assigned to this region is created [1]
> >
> > I planned to implement it soon.
> > Looks like we should have several options of memory(data regions)
> allocation:
> >
> > - allocate all on startup (AFAIK this is how current
> implementation behave)
> > - allocate all on startup AND pre touch.
> > - allocate specific data region for first assignment.
> > - allocate specific data region for first assignment AND pre
> touch.
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > > Igniters,
> > >
> > > What is being suggested here is an Ignite off-heap’s version of Java’s
> -XX:+AlwaysPreTouch.
> > > The latter is known to be used to guarantee that the committed memory
> is backed by physical RAM.
> > > This ensures that
> > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
> for physical page allocation, possible contention with page cache, etc)
> > > b) JVM fails fast if the Xmx is greater than available RAM
> > > c) New processes will not be able to claim the memory JVM took for
> itself
> > >
> > > Currently one can’t get the same benefits for Ignite because we use
> off-heap as well as heap.
> > > So, we can implement a similar feature for Ignite – and make sure the
> users can get all the memory pre-touch benefits if they want.
> > > Of course, it impacts startup time so we should enable it by a
> configuration property (I’d suggest a system property for now).
> > >
> > > Are there any objections to implementing this?
> > >
> > > Thanks,
> > > Stan
> > >
> > > From: Павлухин Иван
> > > Sent: 31 октября 2018 г. 12:50
> > > To: dev@ignite.apache.org
> > > Subject: Re: Pre-touch for Ignite off-heap memory
> > >
> > > Hi,
> > >
> > > I did an experiment described above. I created a patch with pre-touch
> [1]
> > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > > Ignite with equal values for initial and max sizes for each data
> region.
> > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
> easy
> > > to observe JVM OOM-killed.
> > >
> > > [1] https://github.com/apache/ignite/pull/5220
> > > [2]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> > > [3]
> > >
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> > >
> > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> > >
> > > > Hi guys,
> > > >
> > > > I am not aware that it is possible to run JVM in "allocation-free"
> fashion.
> > > > If you know that it is possible please share it. As I know JVM
> allocates
> > > > memory out of garbage collectable area for internal purposes like
> JIT,
> > > > GC itself. Also native built-it code can request memory allocation
> from OS,
> > > > e.g. zip facilities. If we imagine that system is running under
> memory
> > > > allocation
> > > > which is close to a limit then even small allocation request can
> fail and
> > > > lead
> > > > to OOM killing.
> > > >
> > > > But I think that a simple and useful thing that could be done first
> is
> > > > making
> > > > an experiment. As Andrey mentioned
> > > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > > >
> > > > region
> > > > > MAX size equal to MIN and get region allocated on node start.
> > > >
> > > > If it is so then it should not be hard to launch Ignite and observe
> it
> > > > running
> > > > very close to OS memory limit with Java heap and Ignite off-heap
> > > > pre-touched.
> > > > After that one could check whether it is possible to observe Ignite
> OOM
> > > > killed.
> > > > Let's say my bet is that it is relatively easy to catch OOM here.
> > > >
> > > > What do you think?
> > > >
> > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> > > >
> > > > > Andrey,
> > > > >
> > > > > Probability of a OOM kill will be much lower if offheap is
> pretouched.
> > > > > What
> > > > > do you mean by JVM internal needs? In my understanding if user
> enables
> > > > > option to pretouch heap and fixes the heap to prevent jvm
> releasing memory
> > > > > back to OS, then OOM killing is very unlikely.
> > > > >
> > > > > I would agree that pretouch for offheap may be helpful in many
> cases.
> > > > >
> > > > > --Yakov
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: Pre-touch for Ignite off-heap memory
Posted by Павлухин Иван <vo...@gmail.com>.
Hi Stan,
As I participated in discussion in current thread I would like to
leave a comment.
Your concerns looks clear for me and if you believe that pre-touch
will help product users then I have nothing against it.
ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <ni...@apache.org>:
>
> Hello, Stanislav.
>
> As far as I can see, we have controversal ticket based on previous dev-list discussion:
>
> IGNITE-9113 - Allocate memory for a data region when first cache assigned to this region is created [1]
>
> I planned to implement it soon.
> Looks like we should have several options of memory(data regions) allocation:
>
> - allocate all on startup (AFAIK this is how current implementation behave)
> - allocate all on startup AND pre touch.
> - allocate specific data region for first assignment.
> - allocate specific data region for first assignment AND pre touch.
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > Igniters,
> >
> > What is being suggested here is an Ignite off-heap’s version of Java’s -XX:+AlwaysPreTouch.
> > The latter is known to be used to guarantee that the committed memory is backed by physical RAM.
> > This ensures that
> > a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
> > b) JVM fails fast if the Xmx is greater than available RAM
> > c) New processes will not be able to claim the memory JVM took for itself
> >
> > Currently one can’t get the same benefits for Ignite because we use off-heap as well as heap.
> > So, we can implement a similar feature for Ignite – and make sure the users can get all the memory pre-touch benefits if they want.
> > Of course, it impacts startup time so we should enable it by a configuration property (I’d suggest a system property for now).
> >
> > Are there any objections to implementing this?
> >
> > Thanks,
> > Stan
> >
> > From: Павлухин Иван
> > Sent: 31 октября 2018 г. 12:50
> > To: dev@ignite.apache.org
> > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> > Hi,
> >
> > I did an experiment described above. I created a patch with pre-touch [1]
> > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > Ignite with equal values for initial and max sizes for each data region.
> > I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
> > to observe JVM OOM-killed.
> >
> > [1] https://github.com/apache/ignite/pull/5220
> > [2]
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> > [3]
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> >
> > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> >
> > > Hi guys,
> > >
> > > I am not aware that it is possible to run JVM in "allocation-free" fashion.
> > > If you know that it is possible please share it. As I know JVM allocates
> > > memory out of garbage collectable area for internal purposes like JIT,
> > > GC itself. Also native built-it code can request memory allocation from OS,
> > > e.g. zip facilities. If we imagine that system is running under memory
> > > allocation
> > > which is close to a limit then even small allocation request can fail and
> > > lead
> > > to OOM killing.
> > >
> > > But I think that a simple and useful thing that could be done first is
> > > making
> > > an experiment. As Andrey mentioned
> > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > >
> > > region
> > > > MAX size equal to MIN and get region allocated on node start.
> > >
> > > If it is so then it should not be hard to launch Ignite and observe it
> > > running
> > > very close to OS memory limit with Java heap and Ignite off-heap
> > > pre-touched.
> > > After that one could check whether it is possible to observe Ignite OOM
> > > killed.
> > > Let's say my bet is that it is relatively easy to catch OOM here.
> > >
> > > What do you think?
> > >
> > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> > >
> > > > Andrey,
> > > >
> > > > Probability of a OOM kill will be much lower if offheap is pretouched.
> > > > What
> > > > do you mean by JVM internal needs? In my understanding if user enables
> > > > option to pretouch heap and fixes the heap to prevent jvm releasing memory
> > > > back to OS, then OOM killing is very unlikely.
> > > >
> > > > I would agree that pretouch for offheap may be helpful in many cases.
> > > >
> > > > --Yakov
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
> >
--
Best regards,
Ivan Pavlukhin
Re: Pre-touch for Ignite off-heap memory
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Stanislav.
As far as I can see, we have controversal ticket based on previous dev-list discussion:
IGNITE-9113 - Allocate memory for a data region when first cache assigned to this region is created [1]
I planned to implement it soon.
Looks like we should have several options of memory(data regions) allocation:
- allocate all on startup (AFAIK this is how current implementation behave)
- allocate all on startup AND pre touch.
- allocate specific data region for first assignment.
- allocate specific data region for first assignment AND pre touch.
What do you think?
[1] https://issues.apache.org/jira/browse/IGNITE-9113
В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> Igniters,
>
> What is being suggested here is an Ignite off-heap’s version of Java’s -XX:+AlwaysPreTouch.
> The latter is known to be used to guarantee that the committed memory is backed by physical RAM.
> This ensures that
> a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
> b) JVM fails fast if the Xmx is greater than available RAM
> c) New processes will not be able to claim the memory JVM took for itself
>
> Currently one can’t get the same benefits for Ignite because we use off-heap as well as heap.
> So, we can implement a similar feature for Ignite – and make sure the users can get all the memory pre-touch benefits if they want.
> Of course, it impacts startup time so we should enable it by a configuration property (I’d suggest a system property for now).
>
> Are there any objections to implementing this?
>
> Thanks,
> Stan
>
> From: Павлухин Иван
> Sent: 31 октября 2018 г. 12:50
> To: dev@ignite.apache.org
> Subject: Re: Pre-touch for Ignite off-heap memory
>
> Hi,
>
> I did an experiment described above. I created a patch with pre-touch [1]
> and started a JVM with an option -XX:+AlwaysPreTouch and configured
> Ignite with equal values for initial and max sizes for each data region.
> I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
> to observe JVM OOM-killed.
>
> [1] https://github.com/apache/ignite/pull/5220
> [2]
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> [3]
> https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
>
> вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
>
> > Hi guys,
> >
> > I am not aware that it is possible to run JVM in "allocation-free" fashion.
> > If you know that it is possible please share it. As I know JVM allocates
> > memory out of garbage collectable area for internal purposes like JIT,
> > GC itself. Also native built-it code can request memory allocation from OS,
> > e.g. zip facilities. If we imagine that system is running under memory
> > allocation
> > which is close to a limit then even small allocation request can fail and
> > lead
> > to OOM killing.
> >
> > But I think that a simple and useful thing that could be done first is
> > making
> > an experiment. As Andrey mentioned
> > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> >
> > region
> > > MAX size equal to MIN and get region allocated on node start.
> >
> > If it is so then it should not be hard to launch Ignite and observe it
> > running
> > very close to OS memory limit with Java heap and Ignite off-heap
> > pre-touched.
> > After that one could check whether it is possible to observe Ignite OOM
> > killed.
> > Let's say my bet is that it is relatively easy to catch OOM here.
> >
> > What do you think?
> >
> > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> >
> > > Andrey,
> > >
> > > Probability of a OOM kill will be much lower if offheap is pretouched.
> > > What
> > > do you mean by JVM internal needs? In my understanding if user enables
> > > option to pretouch heap and fixes the heap to prevent jvm releasing memory
> > > back to OS, then OOM killing is very unlikely.
> > >
> > > I would agree that pretouch for offheap may be helpful in many cases.
> > >
> > > --Yakov
> > >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>
>
RE: Pre-touch for Ignite off-heap memory
Posted by Stanislav Lukyanov <st...@gmail.com>.
Igniters,
What is being suggested here is an Ignite off-heap’s version of Java’s -XX:+AlwaysPreTouch.
The latter is known to be used to guarantee that the committed memory is backed by physical RAM.
This ensures that
a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical page allocation, possible contention with page cache, etc)
b) JVM fails fast if the Xmx is greater than available RAM
c) New processes will not be able to claim the memory JVM took for itself
Currently one can’t get the same benefits for Ignite because we use off-heap as well as heap.
So, we can implement a similar feature for Ignite – and make sure the users can get all the memory pre-touch benefits if they want.
Of course, it impacts startup time so we should enable it by a configuration property (I’d suggest a system property for now).
Are there any objections to implementing this?
Thanks,
Stan
From: Павлухин Иван
Sent: 31 октября 2018 г. 12:50
To: dev@ignite.apache.org
Subject: Re: Pre-touch for Ignite off-heap memory
Hi,
I did an experiment described above. I created a patch with pre-touch [1]
and started a JVM with an option -XX:+AlwaysPreTouch and configured
Ignite with equal values for initial and max sizes for each data region.
I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
to observe JVM OOM-killed.
[1] https://github.com/apache/ignite/pull/5220
[2]
https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
[3]
https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> Hi guys,
>
> I am not aware that it is possible to run JVM in "allocation-free" fashion.
> If you know that it is possible please share it. As I know JVM allocates
> memory out of garbage collectable area for internal purposes like JIT,
> GC itself. Also native built-it code can request memory allocation from OS,
> e.g. zip facilities. If we imagine that system is running under memory
> allocation
> which is close to a limit then even small allocation request can fail and
> lead
> to OOM killing.
>
> But I think that a simple and useful thing that could be done first is
> making
> an experiment. As Andrey mentioned
> > AFAIK, Ignite always pre-touch first region. So, you can try to set
> region
> > MAX size equal to MIN and get region allocated on node start.
> If it is so then it should not be hard to launch Ignite and observe it
> running
> very close to OS memory limit with Java heap and Ignite off-heap
> pre-touched.
> After that one could check whether it is possible to observe Ignite OOM
> killed.
> Let's say my bet is that it is relatively easy to catch OOM here.
>
> What do you think?
>
> пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
>
>> Andrey,
>>
>> Probability of a OOM kill will be much lower if offheap is pretouched.
>> What
>> do you mean by JVM internal needs? In my understanding if user enables
>> option to pretouch heap and fixes the heap to prevent jvm releasing memory
>> back to OS, then OOM killing is very unlikely.
>>
>> I would agree that pretouch for offheap may be helpful in many cases.
>>
>> --Yakov
>>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
--
Best regards,
Ivan Pavlukhin
Re: Pre-touch for Ignite off-heap memory
Posted by Павлухин Иван <vo...@gmail.com>.
Hi,
I did an experiment described above. I created a patch with pre-touch [1]
and started a JVM with an option -XX:+AlwaysPreTouch and configured
Ignite with equal values for initial and max sizes for each data region.
I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
to observe JVM OOM-killed.
[1] https://github.com/apache/ignite/pull/5220
[2]
https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
[3]
https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vo...@gmail.com>:
> Hi guys,
>
> I am not aware that it is possible to run JVM in "allocation-free" fashion.
> If you know that it is possible please share it. As I know JVM allocates
> memory out of garbage collectable area for internal purposes like JIT,
> GC itself. Also native built-it code can request memory allocation from OS,
> e.g. zip facilities. If we imagine that system is running under memory
> allocation
> which is close to a limit then even small allocation request can fail and
> lead
> to OOM killing.
>
> But I think that a simple and useful thing that could be done first is
> making
> an experiment. As Andrey mentioned
> > AFAIK, Ignite always pre-touch first region. So, you can try to set
> region
> > MAX size equal to MIN and get region allocated on node start.
> If it is so then it should not be hard to launch Ignite and observe it
> running
> very close to OS memory limit with Java heap and Ignite off-heap
> pre-touched.
> After that one could check whether it is possible to observe Ignite OOM
> killed.
> Let's say my bet is that it is relatively easy to catch OOM here.
>
> What do you think?
>
> пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
>
>> Andrey,
>>
>> Probability of a OOM kill will be much lower if offheap is pretouched.
>> What
>> do you mean by JVM internal needs? In my understanding if user enables
>> option to pretouch heap and fixes the heap to prevent jvm releasing memory
>> back to OS, then OOM killing is very unlikely.
>>
>> I would agree that pretouch for offheap may be helpful in many cases.
>>
>> --Yakov
>>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
--
Best regards,
Ivan Pavlukhin
Re: Pre-touch for Ignite off-heap memory
Posted by Павлухин Иван <vo...@gmail.com>.
Hi guys,
I am not aware that it is possible to run JVM in "allocation-free" fashion.
If you know that it is possible please share it. As I know JVM allocates
memory out of garbage collectable area for internal purposes like JIT,
GC itself. Also native built-it code can request memory allocation from OS,
e.g. zip facilities. If we imagine that system is running under memory
allocation
which is close to a limit then even small allocation request can fail and
lead
to OOM killing.
But I think that a simple and useful thing that could be done first is
making
an experiment. As Andrey mentioned
> AFAIK, Ignite always pre-touch first region. So, you can try to set region
> MAX size equal to MIN and get region allocated on node start.
If it is so then it should not be hard to launch Ignite and observe it
running
very close to OS memory limit with Java heap and Ignite off-heap
pre-touched.
After that one could check whether it is possible to observe Ignite OOM
killed.
Let's say my bet is that it is relatively easy to catch OOM here.
What do you think?
пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yz...@apache.org>:
> Andrey,
>
> Probability of a OOM kill will be much lower if offheap is pretouched. What
> do you mean by JVM internal needs? In my understanding if user enables
> option to pretouch heap and fixes the heap to prevent jvm releasing memory
> back to OS, then OOM killing is very unlikely.
>
> I would agree that pretouch for offheap may be helpful in many cases.
>
> --Yakov
>
--
Best regards,
Ivan Pavlukhin
Re: Pre-touch for Ignite off-heap memory
Posted by Yakov Zhdanov <yz...@apache.org>.
Andrey,
Probability of a OOM kill will be much lower if offheap is pretouched. What
do you mean by JVM internal needs? In my understanding if user enables
option to pretouch heap and fixes the heap to prevent jvm releasing memory
back to OS, then OOM killing is very unlikely.
I would agree that pretouch for offheap may be helpful in many cases.
--Yakov
Re: Pre-touch for Ignite off-heap memory
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Alex,
1. Pre-touch will not save from killing node by OS if there will not enough
memory for JVM internal needs, anyway.
Ignite data region setting has MIN and MAX sizes. Ignite split memory
region into segments, first segment is size of MIN.
Next segments are allocated on demand (up to MAX) and have size of 256Mb.
AFAIK, Ignite always pre-touch first region. So, you can try to set region
MAX size equal to MIN and get region allocated on node start.
Do you mean this is not work as expected?
2. We try to add off-heap size checks in configuration validation procedure.
On Thu, Oct 25, 2018 at 5:41 PM Gerus <ag...@gridgain.com> wrote:
> Hi, Ivan
> Let me try to explain. I think that there are 2 goals that are mostly
> applicable for prod systems:
> 1. I can agree with Dave that swap is not a good case for performance and
> disk resource. It is possible that other applications can consume memory
> that is free after Ignite was started. In this case if Ignite will be
> creating new pages in runtime, it can face with OOM.
>
> 2. Ignite user can create wrong configuration. For example, data region can
> exceed available memory by mistake or Ignite can be started on another
> server with less RAM. It can lead to catch OOM in runtime
>
> To summarize: Pre-allocation can detect listed issues on Ignite start by
> allocating OS RAM
>
> Im sure that this option have to be disabled by default, but user should
> have a choice for startup
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Pre-touch for Ignite off-heap memory
Posted by Gerus <ag...@gridgain.com>.
Hi, Ivan
Let me try to explain. I think that there are 2 goals that are mostly
applicable for prod systems:
1. I can agree with Dave that swap is not a good case for performance and
disk resource. It is possible that other applications can consume memory
that is free after Ignite was started. In this case if Ignite will be
creating new pages in runtime, it can face with OOM.
2. Ignite user can create wrong configuration. For example, data region can
exceed available memory by mistake or Ignite can be started on another
server with less RAM. It can lead to catch OOM in runtime
To summarize: Pre-allocation can detect listed issues on Ignite start by
allocating OS RAM
Im sure that this option have to be disabled by default, but user should
have a choice for startup
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Pre-touch for Ignite off-heap memory
Posted by Павлухин Иван <vo...@gmail.com>.
Hi Alex,
I wonder what problem you wish to solve. Pre-touching memory looks like a
solution.
But what is the problem? Am I getting it right that it is desired to avoid
OOM (of some kind)
in unpredictable moment of Ignite process execution? If so there could be
number of ways to tackle it.
I am not aware of means which can reserve physical memory for a process in
such way
that particular process will "own" that memory exclusively. Pre-touching
might not give such guarantees.
Actually, I think that one who deploys failure critical system should
carefully plan resources allocation.
I agree with Andrey here that it is worth adding some foolproof check. In
my mind it could be even preventing
a node startup if a requested data region size is greater than an estimated
amount of available memory.
And if such check (by some unusual reason) is not needed it could be
disabled with some property.
ср, 24 окт. 2018 г. в 14:41, David Harvey <sy...@gmail.com>:
> Denis,
>
> We run must of our production DBs systems without any swapping space,
> because the 10-100x drop in throughput if such systems start paging makes
> them worse than useless. However, we don't get OOM on them until all the
> pages are dirty, since LINUX will page out read-only (code) pages or memory
> mapped files.
>
> Dave Harvey
>
>
>
> On Wed, Oct 24, 2018 at 12:12 AM Denis Magda <dm...@apache.org> wrote:
>
> > Alex,
> >
> > Correct me if I'm wrong, but even if an OS runs out of physical memory (X
> > GB in total) an Ignite node process still can request the X GB from
> virtual
> > memory. Yes, virtual memory can involve swapping and disk to satisfy your
> > request but this shouldn't be a reason of the OOM. Shouldn't OOM happen
> if
> > you're trying to allocate beyond the virtual memory capacity (beyond X
> GB)?
> >
> > Denis
> >
> > On Tue, Oct 23, 2018 at 12:08 PM Gerus <ag...@gridgain.com> wrote:
> >
> > > Hi *Igniters*,
> > > Some time ago I've raised a suggestion for product improvement
> > > https://issues.apache.org/jira/browse/IGNITE-9112
> > > <https://issues.apache.org/jira/browse/IGNITE-9112> . It's all about
> > > off-heap memory allocation. Current implementation can have some
> > > improvements for failure critical systems. Ignite can have OOM in
> > runtime,
> > > because RAM can be used by OS, if it will not be pre-booked by
> operation
> > > system and this proposal is to address that. Common case is offheap and
> > > thats why memory segment cannot be managed by JVM that has
> > +AlwaysPreTouch
> > > option
> > > Obviously this implementation will make startup longer and thats why it
> > is
> > > proposed to use configuration flag to manage this feature
> > > I think, it will be useful to have this option. Are you supporting
> this?
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>
--
Best regards,
Ivan Pavlukhin
Re: Pre-touch for Ignite off-heap memory
Posted by David Harvey <sy...@gmail.com>.
Denis,
We run must of our production DBs systems without any swapping space,
because the 10-100x drop in throughput if such systems start paging makes
them worse than useless. However, we don't get OOM on them until all the
pages are dirty, since LINUX will page out read-only (code) pages or memory
mapped files.
Dave Harvey
On Wed, Oct 24, 2018 at 12:12 AM Denis Magda <dm...@apache.org> wrote:
> Alex,
>
> Correct me if I'm wrong, but even if an OS runs out of physical memory (X
> GB in total) an Ignite node process still can request the X GB from virtual
> memory. Yes, virtual memory can involve swapping and disk to satisfy your
> request but this shouldn't be a reason of the OOM. Shouldn't OOM happen if
> you're trying to allocate beyond the virtual memory capacity (beyond X GB)?
>
> Denis
>
> On Tue, Oct 23, 2018 at 12:08 PM Gerus <ag...@gridgain.com> wrote:
>
> > Hi *Igniters*,
> > Some time ago I've raised a suggestion for product improvement
> > https://issues.apache.org/jira/browse/IGNITE-9112
> > <https://issues.apache.org/jira/browse/IGNITE-9112> . It's all about
> > off-heap memory allocation. Current implementation can have some
> > improvements for failure critical systems. Ignite can have OOM in
> runtime,
> > because RAM can be used by OS, if it will not be pre-booked by operation
> > system and this proposal is to address that. Common case is offheap and
> > thats why memory segment cannot be managed by JVM that has
> +AlwaysPreTouch
> > option
> > Obviously this implementation will make startup longer and thats why it
> is
> > proposed to use configuration flag to manage this feature
> > I think, it will be useful to have this option. Are you supporting this?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
Re: Pre-touch for Ignite off-heap memory
Posted by Andrey Mashenkov <an...@gmail.com>.
Alex,
I think offheap memory preallocation will not be helpful as you expect:
1. We still will not have any guarantee JVM process won't be killed by
OOM-killer due to memory overcommiting e.g. for GC needs.
as JVM allocates memory for internal needs in same process memory space.
2. Startup process will costs as we'll need to touch every offheap page.
We can't try to add some checks on startup to warn a user in case of low
physical memory (based on ram size + swap size + overcommit settings),
but I have no idea how we can estimate memory for JVM internal needs
(threads, GC and others).
On Wed, Oct 24, 2018 at 11:25 AM Andrey Mashenkov <
andrey.mashenkov@gmail.com> wrote:
> Docs [1] says, that OOM can also be thrown when native library can't
> allocate memory chunk if physical memory (ram + swap).
>
> >> Shouldn't OOM happen if you're trying to allocate beyond the virtual
> memory capacity (beyond X GB)?
> With 64-bit addressing you have some exabytes of virtual memory, is it
> possible to have such amount of physical memory?
>
> I'd think JVM process should be killed by OS much earlier due to memory
> overcommit. See [2].
>
> [1]
> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html
> [2]
> http://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/
>
> On Wed, Oct 24, 2018 at 7:12 AM Denis Magda <dm...@apache.org> wrote:
>
>> Alex,
>>
>> Correct me if I'm wrong, but even if an OS runs out of physical memory (X
>> GB in total) an Ignite node process still can request the X GB from
>> virtual
>> memory. Yes, virtual memory can involve swapping and disk to satisfy your
>> request but this shouldn't be a reason of the OOM. Shouldn't OOM happen if
>> you're trying to allocate beyond the virtual memory capacity (beyond X
>> GB)?
>>
>> Denis
>>
>> On Tue, Oct 23, 2018 at 12:08 PM Gerus <ag...@gridgain.com> wrote:
>>
>> > Hi *Igniters*,
>> > Some time ago I've raised a suggestion for product improvement
>> > https://issues.apache.org/jira/browse/IGNITE-9112
>> > <https://issues.apache.org/jira/browse/IGNITE-9112> . It's all about
>> > off-heap memory allocation. Current implementation can have some
>> > improvements for failure critical systems. Ignite can have OOM in
>> runtime,
>> > because RAM can be used by OS, if it will not be pre-booked by operation
>> > system and this proposal is to address that. Common case is offheap and
>> > thats why memory segment cannot be managed by JVM that has
>> +AlwaysPreTouch
>> > option
>> > Obviously this implementation will make startup longer and thats why it
>> is
>> > proposed to use configuration flag to manage this feature
>> > I think, it will be useful to have this option. Are you supporting this?
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>> >
>>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
--
Best regards,
Andrey V. Mashenkov
Re: Pre-touch for Ignite off-heap memory
Posted by Andrey Mashenkov <an...@gmail.com>.
Docs [1] says, that OOM can also be thrown when native library can't
allocate memory chunk if physical memory (ram + swap).
>> Shouldn't OOM happen if you're trying to allocate beyond the virtual
memory capacity (beyond X GB)?
With 64-bit addressing you have some exabytes of virtual memory, is it
possible to have such amount of physical memory?
I'd think JVM process should be killed by OS much earlier due to memory
overcommit. See [2].
[1]
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html
[2]
http://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/
On Wed, Oct 24, 2018 at 7:12 AM Denis Magda <dm...@apache.org> wrote:
> Alex,
>
> Correct me if I'm wrong, but even if an OS runs out of physical memory (X
> GB in total) an Ignite node process still can request the X GB from virtual
> memory. Yes, virtual memory can involve swapping and disk to satisfy your
> request but this shouldn't be a reason of the OOM. Shouldn't OOM happen if
> you're trying to allocate beyond the virtual memory capacity (beyond X GB)?
>
> Denis
>
> On Tue, Oct 23, 2018 at 12:08 PM Gerus <ag...@gridgain.com> wrote:
>
> > Hi *Igniters*,
> > Some time ago I've raised a suggestion for product improvement
> > https://issues.apache.org/jira/browse/IGNITE-9112
> > <https://issues.apache.org/jira/browse/IGNITE-9112> . It's all about
> > off-heap memory allocation. Current implementation can have some
> > improvements for failure critical systems. Ignite can have OOM in
> runtime,
> > because RAM can be used by OS, if it will not be pre-booked by operation
> > system and this proposal is to address that. Common case is offheap and
> > thats why memory segment cannot be managed by JVM that has
> +AlwaysPreTouch
> > option
> > Obviously this implementation will make startup longer and thats why it
> is
> > proposed to use configuration flag to manage this feature
> > I think, it will be useful to have this option. Are you supporting this?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
--
Best regards,
Andrey V. Mashenkov
Re: Pre-touch for Ignite off-heap memory
Posted by Denis Magda <dm...@apache.org>.
Alex,
Correct me if I'm wrong, but even if an OS runs out of physical memory (X
GB in total) an Ignite node process still can request the X GB from virtual
memory. Yes, virtual memory can involve swapping and disk to satisfy your
request but this shouldn't be a reason of the OOM. Shouldn't OOM happen if
you're trying to allocate beyond the virtual memory capacity (beyond X GB)?
Denis
On Tue, Oct 23, 2018 at 12:08 PM Gerus <ag...@gridgain.com> wrote:
> Hi *Igniters*,
> Some time ago I've raised a suggestion for product improvement
> https://issues.apache.org/jira/browse/IGNITE-9112
> <https://issues.apache.org/jira/browse/IGNITE-9112> . It's all about
> off-heap memory allocation. Current implementation can have some
> improvements for failure critical systems. Ignite can have OOM in runtime,
> because RAM can be used by OS, if it will not be pre-booked by operation
> system and this proposal is to address that. Common case is offheap and
> thats why memory segment cannot be managed by JVM that has +AlwaysPreTouch
> option
> Obviously this implementation will make startup longer and thats why it is
> proposed to use configuration flag to manage this feature
> I think, it will be useful to have this option. Are you supporting this?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>