You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Valentin Kulichenko <va...@gmail.com> on 2018/07/20 19:59:29 UTC
Data regions on client nodes
Folks,
Currently do not create any regions or allocate any offheap memory on
client nodes unless it's explicitly configured. This is good behavior,
however there is a usability issue caused by the fact that many users have
the same config file for both server and clients. This can lead to
unexpected excessive memory usage on client side and forces users to
maintain two config files in most cases.
At the same time, the only case when offheap memory can be required on a
client node is using LOCAL caches there, which is a very rare use case.
Having said that, is it possible to allocate memory on client node
dynamically ONLY if a local cache is created there? This would fix the
usability issue without limiting the use of local caches on client side.
-Val
Re: Data regions on client nodes
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Ignite.
Finally, I've implemented this ticket! :)
Jira - https://issues.apache.org/jira/browse/IGNITE-9113
PR - https://github.com/apache/ignite/pull/6388
Tests seems to be OK.
Anyone wants to take a look?
В Чт, 09/08/2018 в 16:41 +0300, Alexey Goncharuk пишет:
> Once the OS gave us the pointer, no exceptions can be raised in the user
> code. If the OS overcommitted the memory, then the process will be halted
> when OOME occurs, not much we can do here.
>
> My point was that dynamic data region allocation can lead to another
> dynamic exception that should be properly handled during cache start. When
> the data region is allocated statically, this exception is handled on node
> start, which is much easier.
>
> ср, 8 авг. 2018 г. в 18:18, Dmitriy Pavlov <dp...@gmail.com>:
>
> > I used to think that OS allocates pages not immediately after call to
> > allocate(), but only during first touch of each page.
> >
> > I'm not sure every OS provides guaranee that 'allocated' memory will never
> > cause OOME. Please correct me if I'm wrong.
> >
> > ср, 8 авг. 2018 г. в 17:38, Dmitriy Setrakyan <ds...@apache.org>:
> >
> > > Alexey,
> > >
> > > I am not sure I understand. If cache creation proceeds, but memory region
> > > was not allocated, how can the cache operate?
> > >
> > > D.
> > >
> > > On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com
> > > > wrote:
> > > > I do not mind making this change, but note that the reason for non-lazy
> > > > region allocation is the need to gracefully handle OOME errors during
> > >
> > > cache
> > > > start. When a region is pre-allocated, no OOME can happen.
> > > >
> > > > If a region is allocated dynamically, then all errors that happened
> > >
> > > during
> > > > the node start before should be properly handled (a client node must be
> > > > stopped, but cache creation should proceed).
> > > >
> > > > пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com>:
> > > >
> > > > > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
> > > > >
> > > > > -Val
> > > > >
> > > > > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <dpavlov.spb@gmail.com
> > > > > wrote:
> > > > >
> > > > > > Maxim, thank you.
> > > > > >
> > > > > > If it seems it is technically possible, we can file ticket for this
> > > > >
> > > > > change.
> > > > > >
> > > > > > I find this proposal reasonable, change makes perfectly sense to
> >
> > me.
> > > > > >
> > > > > > We can wait Alex G. feedback on this change before starting actual
> > > > > > implementation. It can take for a while, because he is travelling
> > >
> > > now.
> > > > > >
> > > > > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <ma...@gmail.com>:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > I can miss some details, but at the first glance we have
> >
> > everething
> > > > we
> > > > > > need
> > > > > > > to defer
> > > > > > > region memory allocation if it has no cache groups assignments.
> >
> > And
> > > > it
> > > > > > > doesn't matter
> > > > > > > where it happens on client or server nodes.
> > > > > > >
> > > > > > > Currently region memory allocation happens at exchange future
> >
> > init
> > > > > > method.
> > > > > > > At the
> > > > > > > node startup method initCachesOnLocalJoin executes. This method
> > > > > >
> > > > > > resposnible
> > > > > > > for
> > > > > > > memory allocation (through initiating cache managers) and it also
> > > > >
> > > > > starts
> > > > > > > caches.
> > > > > > > So, at this point we have all existing caches descriptors and can
> > > >
> > > > find
> > > > > > out
> > > > > > > which
> > > > > > > cache matches which region to defer some regions initialization
> >
> > to
> > > > the
> > > > > > > moment when
> > > > > > > newly cache assings to this region (happend at
> > >
> > > onCacheChangeRequest).
> > > > > > >
> > > > > > > Please, сorrect me if I'm wrong and missing something.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <
> >
> > dpavlov.spb@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Maxim,
> > > > > > > >
> > > > > > > > thank you for stepping in. How do you think, is it possible to
> > > >
> > > > check
> > > > > > > cache
> > > > > > > > assignment to region at stage of memory allocation?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <
> >
> > maxmuzaf@gmail.com
> > > > :
> > > > > > > >
> > > > > > > > > Folks,
> > > > > > > > >
> > > > > > > > > I've checked memory allocation. It looks like we are
> >
> > allocating
> > > > > > memory
> > > > > > > > only
> > > > > > > > > on the first exchange future init on local join occurs on
> >
> > node.
> > > > > Also,
> > > > > > > > seems
> > > > > > > > > like we are allocating only the first chunk of memory (not
> >
> > the
> > > > > whole
> > > > > > > > bunch)
> > > > > > > > > and it calculates as:
> > > > > > > > >
> > > > > > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 *
> > > >
> > > > 1024)
> > > > > > > > >
> > > > > > > > > But, I'm agree with Val. It's better to allocate memory only
> > >
> > > when
> > > > > > when
> > > > > > > > > the first cache assigned to this region.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Also, It seems like we have some problem with user
> >
> > notification
> > > > > about
> > > > > > > > > available
> > > > > > > > > physical resources. For client nodes method requiredOffheap()
> > > > >
> > > > > returns
> > > > > > > > > always
> > > > > > > > > zero [1]. That's why WARN message shown here [2] would be not
> > >
> > > not
> > > > > > quite
> > > > > > > > > right
> > > > > > > > > if we have a lot of client nodes in cluster.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > >
> > > > > https://github.com/apache/ignite/blob/master/modules/
> > > >
> > > > core/src/main/java/org/apache/ignite/internal/managers/discovery/
> > > > GridDiscoveryManager.java#L1501
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > > > >
> > > > >
> > > > > https://github.com/apache/ignite/blob/master/modules/
> > > >
> > > > core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> > > > > > > > >
> > > > > > > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <
> > > > > >
> > > > > > dsetrakyan@apache.org
> > > > > > > > :
> > > > > > > > >
> > > > > > > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > > > > > > > valentin.kulichenko@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Actually, I would go even further: only allocate a data
> > > >
> > > > region
> > > > > > on a
> > > > > > > > > node
> > > > > > > > > > > when the first cache assigned to this region is deployed
> >
> > on
> > > > > that
> > > > > > > > node.
> > > > > > > > > > > Because issue is broader than client nodes and local
> > >
> > > caches.
> > > > > One
> > > > > > > can
> > > > > > > > > have
> > > > > > > > > > > server nodes without any caches as well - running only
> > > > >
> > > > > services,
> > > > > > > for
> > > > > > > > > > > example.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > It would be great if this was possible, but to my
> >
> > knowledge,
> > > > > > regions
> > > > > > > > need
> > > > > > > > > > to be allocated on startup.
> > > > > > > > > >
> > > > > > > > > > Alexey Goncharuk, do you have any suggestions on this?
> > > > > > > > > >
> > > > > > > > > > D.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > --
> > > > > > > > > Maxim Muzafarov
> > > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > --
> > > > > > > Maxim Muzafarov
> > > > > > >
Re: Data regions on client nodes
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Alexey,
I see your point. However, I would still argue that there are more benefits
to the dynamic region allocation than not. For example, if we have this
feature, users would be able to create more regions after the node and
cluster start, which will allow to grow the data size indefinitely in the
cloud.
D.
On Thu, Aug 9, 2018 at 8:41 AM, Alexey Goncharuk <alexey.goncharuk@gmail.com
> wrote:
> Once the OS gave us the pointer, no exceptions can be raised in the user
> code. If the OS overcommitted the memory, then the process will be halted
> when OOME occurs, not much we can do here.
>
> My point was that dynamic data region allocation can lead to another
> dynamic exception that should be properly handled during cache start. When
> the data region is allocated statically, this exception is handled on node
> start, which is much easier.
>
> ср, 8 авг. 2018 г. в 18:18, Dmitriy Pavlov <dp...@gmail.com>:
>
> > I used to think that OS allocates pages not immediately after call to
> > allocate(), but only during first touch of each page.
> >
> > I'm not sure every OS provides guaranee that 'allocated' memory will
> never
> > cause OOME. Please correct me if I'm wrong.
> >
> > ср, 8 авг. 2018 г. в 17:38, Dmitriy Setrakyan <ds...@apache.org>:
> >
> > > Alexey,
> > >
> > > I am not sure I understand. If cache creation proceeds, but memory
> region
> > > was not allocated, how can the cache operate?
> > >
> > > D.
> > >
> > > On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com
> > > > wrote:
> > >
> > > > I do not mind making this change, but note that the reason for
> non-lazy
> > > > region allocation is the need to gracefully handle OOME errors during
> > > cache
> > > > start. When a region is pre-allocated, no OOME can happen.
> > > >
> > > > If a region is allocated dynamically, then all errors that happened
> > > during
> > > > the node start before should be properly handled (a client node must
> be
> > > > stopped, but cache creation should proceed).
> > > >
> > > > пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com>:
> > > >
> > > > > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
> > > > >
> > > > > -Val
> > > > >
> > > > > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <
> dpavlov.spb@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Maxim, thank you.
> > > > > >
> > > > > > If it seems it is technically possible, we can file ticket for
> this
> > > > > change.
> > > > > >
> > > > > > I find this proposal reasonable, change makes perfectly sense to
> > me.
> > > > > >
> > > > > > We can wait Alex G. feedback on this change before starting
> actual
> > > > > > implementation. It can take for a while, because he is travelling
> > > now.
> > > > > >
> > > > > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <maxmuzaf@gmail.com
> >:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > I can miss some details, but at the first glance we have
> > everething
> > > > we
> > > > > > need
> > > > > > > to defer
> > > > > > > region memory allocation if it has no cache groups assignments.
> > And
> > > > it
> > > > > > > doesn't matter
> > > > > > > where it happens on client or server nodes.
> > > > > > >
> > > > > > > Currently region memory allocation happens at exchange future
> > init
> > > > > > method.
> > > > > > > At the
> > > > > > > node startup method initCachesOnLocalJoin executes. This method
> > > > > > resposnible
> > > > > > > for
> > > > > > > memory allocation (through initiating cache managers) and it
> also
> > > > > starts
> > > > > > > caches.
> > > > > > > So, at this point we have all existing caches descriptors and
> can
> > > > find
> > > > > > out
> > > > > > > which
> > > > > > > cache matches which region to defer some regions initialization
> > to
> > > > the
> > > > > > > moment when
> > > > > > > newly cache assings to this region (happend at
> > > onCacheChangeRequest).
> > > > > > >
> > > > > > > Please, сorrect me if I'm wrong and missing something.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <
> > dpavlov.spb@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Maxim,
> > > > > > > >
> > > > > > > > thank you for stepping in. How do you think, is it possible
> to
> > > > check
> > > > > > > cache
> > > > > > > > assignment to region at stage of memory allocation?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <
> > maxmuzaf@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > > Folks,
> > > > > > > > >
> > > > > > > > > I've checked memory allocation. It looks like we are
> > allocating
> > > > > > memory
> > > > > > > > only
> > > > > > > > > on the first exchange future init on local join occurs on
> > node.
> > > > > Also,
> > > > > > > > seems
> > > > > > > > > like we are allocating only the first chunk of memory (not
> > the
> > > > > whole
> > > > > > > > bunch)
> > > > > > > > > and it calculates as:
> > > > > > > > >
> > > > > > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L *
> 1024 *
> > > > 1024)
> > > > > > > > >
> > > > > > > > > But, I'm agree with Val. It's better to allocate memory
> only
> > > when
> > > > > > when
> > > > > > > > > the first cache assigned to this region.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Also, It seems like we have some problem with user
> > notification
> > > > > about
> > > > > > > > > available
> > > > > > > > > physical resources. For client nodes method
> requiredOffheap()
> > > > > returns
> > > > > > > > > always
> > > > > > > > > zero [1]. That's why WARN message shown here [2] would be
> not
> > > not
> > > > > > quite
> > > > > > > > > right
> > > > > > > > > if we have a lot of client nodes in cluster.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > https://github.com/apache/ignite/blob/master/modules/
> > > > core/src/main/java/org/apache/ignite/internal/managers/discovery/
> > > > GridDiscoveryManager.java#L1501
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > https://github.com/apache/ignite/blob/master/modules/
> > > > core/src/main/java/org/apache/ignite/internal/IgniteKernal.
> java#L1489
> > > > > > > > >
> > > > > > > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <
> > > > > > dsetrakyan@apache.org
> > > > > > > >:
> > > > > > > > >
> > > > > > > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > > > > > > > valentin.kulichenko@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Actually, I would go even further: only allocate a data
> > > > region
> > > > > > on a
> > > > > > > > > node
> > > > > > > > > > > when the first cache assigned to this region is
> deployed
> > on
> > > > > that
> > > > > > > > node.
> > > > > > > > > > > Because issue is broader than client nodes and local
> > > caches.
> > > > > One
> > > > > > > can
> > > > > > > > > have
> > > > > > > > > > > server nodes without any caches as well - running only
> > > > > services,
> > > > > > > for
> > > > > > > > > > > example.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > It would be great if this was possible, but to my
> > knowledge,
> > > > > > regions
> > > > > > > > need
> > > > > > > > > > to be allocated on startup.
> > > > > > > > > >
> > > > > > > > > > Alexey Goncharuk, do you have any suggestions on this?
> > > > > > > > > >
> > > > > > > > > > D.
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > > --
> > > > > > > > > Maxim Muzafarov
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > > --
> > > > > > > Maxim Muzafarov
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Data regions on client nodes
Posted by Alexey Goncharuk <al...@gmail.com>.
Once the OS gave us the pointer, no exceptions can be raised in the user
code. If the OS overcommitted the memory, then the process will be halted
when OOME occurs, not much we can do here.
My point was that dynamic data region allocation can lead to another
dynamic exception that should be properly handled during cache start. When
the data region is allocated statically, this exception is handled on node
start, which is much easier.
ср, 8 авг. 2018 г. в 18:18, Dmitriy Pavlov <dp...@gmail.com>:
> I used to think that OS allocates pages not immediately after call to
> allocate(), but only during first touch of each page.
>
> I'm not sure every OS provides guaranee that 'allocated' memory will never
> cause OOME. Please correct me if I'm wrong.
>
> ср, 8 авг. 2018 г. в 17:38, Dmitriy Setrakyan <ds...@apache.org>:
>
> > Alexey,
> >
> > I am not sure I understand. If cache creation proceeds, but memory region
> > was not allocated, how can the cache operate?
> >
> > D.
> >
> > On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk <
> > alexey.goncharuk@gmail.com
> > > wrote:
> >
> > > I do not mind making this change, but note that the reason for non-lazy
> > > region allocation is the need to gracefully handle OOME errors during
> > cache
> > > start. When a region is pre-allocated, no OOME can happen.
> > >
> > > If a region is allocated dynamically, then all errors that happened
> > during
> > > the node start before should be properly handled (a client node must be
> > > stopped, but cache creation should proceed).
> > >
> > > пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com>:
> > >
> > > > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
> > > >
> > > > -Val
> > > >
> > > > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <dpavlov.spb@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Maxim, thank you.
> > > > >
> > > > > If it seems it is technically possible, we can file ticket for this
> > > > change.
> > > > >
> > > > > I find this proposal reasonable, change makes perfectly sense to
> me.
> > > > >
> > > > > We can wait Alex G. feedback on this change before starting actual
> > > > > implementation. It can take for a while, because he is travelling
> > now.
> > > > >
> > > > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <ma...@gmail.com>:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > I can miss some details, but at the first glance we have
> everething
> > > we
> > > > > need
> > > > > > to defer
> > > > > > region memory allocation if it has no cache groups assignments.
> And
> > > it
> > > > > > doesn't matter
> > > > > > where it happens on client or server nodes.
> > > > > >
> > > > > > Currently region memory allocation happens at exchange future
> init
> > > > > method.
> > > > > > At the
> > > > > > node startup method initCachesOnLocalJoin executes. This method
> > > > > resposnible
> > > > > > for
> > > > > > memory allocation (through initiating cache managers) and it also
> > > > starts
> > > > > > caches.
> > > > > > So, at this point we have all existing caches descriptors and can
> > > find
> > > > > out
> > > > > > which
> > > > > > cache matches which region to defer some regions initialization
> to
> > > the
> > > > > > moment when
> > > > > > newly cache assings to this region (happend at
> > onCacheChangeRequest).
> > > > > >
> > > > > > Please, сorrect me if I'm wrong and missing something.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <
> dpavlov.spb@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Maxim,
> > > > > > >
> > > > > > > thank you for stepping in. How do you think, is it possible to
> > > check
> > > > > > cache
> > > > > > > assignment to region at stage of memory allocation?
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <
> maxmuzaf@gmail.com
> > >:
> > > > > > >
> > > > > > > > Folks,
> > > > > > > >
> > > > > > > > I've checked memory allocation. It looks like we are
> allocating
> > > > > memory
> > > > > > > only
> > > > > > > > on the first exchange future init on local join occurs on
> node.
> > > > Also,
> > > > > > > seems
> > > > > > > > like we are allocating only the first chunk of memory (not
> the
> > > > whole
> > > > > > > bunch)
> > > > > > > > and it calculates as:
> > > > > > > >
> > > > > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 *
> > > 1024)
> > > > > > > >
> > > > > > > > But, I'm agree with Val. It's better to allocate memory only
> > when
> > > > > when
> > > > > > > > the first cache assigned to this region.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Also, It seems like we have some problem with user
> notification
> > > > about
> > > > > > > > available
> > > > > > > > physical resources. For client nodes method requiredOffheap()
> > > > returns
> > > > > > > > always
> > > > > > > > zero [1]. That's why WARN message shown here [2] would be not
> > not
> > > > > quite
> > > > > > > > right
> > > > > > > > if we have a lot of client nodes in cluster.
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > https://github.com/apache/ignite/blob/master/modules/
> > > core/src/main/java/org/apache/ignite/internal/managers/discovery/
> > > GridDiscoveryManager.java#L1501
> > > > > > > > [2]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > https://github.com/apache/ignite/blob/master/modules/
> > > core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> > > > > > > >
> > > > > > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <
> > > > > dsetrakyan@apache.org
> > > > > > >:
> > > > > > > >
> > > > > > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > > > > > > valentin.kulichenko@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Actually, I would go even further: only allocate a data
> > > region
> > > > > on a
> > > > > > > > node
> > > > > > > > > > when the first cache assigned to this region is deployed
> on
> > > > that
> > > > > > > node.
> > > > > > > > > > Because issue is broader than client nodes and local
> > caches.
> > > > One
> > > > > > can
> > > > > > > > have
> > > > > > > > > > server nodes without any caches as well - running only
> > > > services,
> > > > > > for
> > > > > > > > > > example.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > It would be great if this was possible, but to my
> knowledge,
> > > > > regions
> > > > > > > need
> > > > > > > > > to be allocated on startup.
> > > > > > > > >
> > > > > > > > > Alexey Goncharuk, do you have any suggestions on this?
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > --
> > > > > > > > --
> > > > > > > > Maxim Muzafarov
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > > --
> > > > > > Maxim Muzafarov
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Data regions on client nodes
Posted by Dmitriy Pavlov <dp...@gmail.com>.
I used to think that OS allocates pages not immediately after call to
allocate(), but only during first touch of each page.
I'm not sure every OS provides guaranee that 'allocated' memory will never
cause OOME. Please correct me if I'm wrong.
ср, 8 авг. 2018 г. в 17:38, Dmitriy Setrakyan <ds...@apache.org>:
> Alexey,
>
> I am not sure I understand. If cache creation proceeds, but memory region
> was not allocated, how can the cache operate?
>
> D.
>
> On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com
> > wrote:
>
> > I do not mind making this change, but note that the reason for non-lazy
> > region allocation is the need to gracefully handle OOME errors during
> cache
> > start. When a region is pre-allocated, no OOME can happen.
> >
> > If a region is allocated dynamically, then all errors that happened
> during
> > the node start before should be properly handled (a client node must be
> > stopped, but cache creation should proceed).
> >
> > пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com>:
> >
> > > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
> > >
> > > -Val
> > >
> > > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <dp...@gmail.com>
> > > wrote:
> > >
> > > > Maxim, thank you.
> > > >
> > > > If it seems it is technically possible, we can file ticket for this
> > > change.
> > > >
> > > > I find this proposal reasonable, change makes perfectly sense to me.
> > > >
> > > > We can wait Alex G. feedback on this change before starting actual
> > > > implementation. It can take for a while, because he is travelling
> now.
> > > >
> > > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <ma...@gmail.com>:
> > > >
> > > > > Guys,
> > > > >
> > > > > I can miss some details, but at the first glance we have everething
> > we
> > > > need
> > > > > to defer
> > > > > region memory allocation if it has no cache groups assignments. And
> > it
> > > > > doesn't matter
> > > > > where it happens on client or server nodes.
> > > > >
> > > > > Currently region memory allocation happens at exchange future init
> > > > method.
> > > > > At the
> > > > > node startup method initCachesOnLocalJoin executes. This method
> > > > resposnible
> > > > > for
> > > > > memory allocation (through initiating cache managers) and it also
> > > starts
> > > > > caches.
> > > > > So, at this point we have all existing caches descriptors and can
> > find
> > > > out
> > > > > which
> > > > > cache matches which region to defer some regions initialization to
> > the
> > > > > moment when
> > > > > newly cache assings to this region (happend at
> onCacheChangeRequest).
> > > > >
> > > > > Please, сorrect me if I'm wrong and missing something.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <dp...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi Maxim,
> > > > > >
> > > > > > thank you for stepping in. How do you think, is it possible to
> > check
> > > > > cache
> > > > > > assignment to region at stage of memory allocation?
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <maxmuzaf@gmail.com
> >:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > I've checked memory allocation. It looks like we are allocating
> > > > memory
> > > > > > only
> > > > > > > on the first exchange future init on local join occurs on node.
> > > Also,
> > > > > > seems
> > > > > > > like we are allocating only the first chunk of memory (not the
> > > whole
> > > > > > bunch)
> > > > > > > and it calculates as:
> > > > > > >
> > > > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 *
> > 1024)
> > > > > > >
> > > > > > > But, I'm agree with Val. It's better to allocate memory only
> when
> > > > when
> > > > > > > the first cache assigned to this region.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Also, It seems like we have some problem with user notification
> > > about
> > > > > > > available
> > > > > > > physical resources. For client nodes method requiredOffheap()
> > > returns
> > > > > > > always
> > > > > > > zero [1]. That's why WARN message shown here [2] would be not
> not
> > > > quite
> > > > > > > right
> > > > > > > if we have a lot of client nodes in cluster.
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://github.com/apache/ignite/blob/master/modules/
> > core/src/main/java/org/apache/ignite/internal/managers/discovery/
> > GridDiscoveryManager.java#L1501
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://github.com/apache/ignite/blob/master/modules/
> > core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> > > > > > >
> > > > > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <
> > > > dsetrakyan@apache.org
> > > > > >:
> > > > > > >
> > > > > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > > > > > valentin.kulichenko@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Actually, I would go even further: only allocate a data
> > region
> > > > on a
> > > > > > > node
> > > > > > > > > when the first cache assigned to this region is deployed on
> > > that
> > > > > > node.
> > > > > > > > > Because issue is broader than client nodes and local
> caches.
> > > One
> > > > > can
> > > > > > > have
> > > > > > > > > server nodes without any caches as well - running only
> > > services,
> > > > > for
> > > > > > > > > example.
> > > > > > > > >
> > > > > > > >
> > > > > > > > It would be great if this was possible, but to my knowledge,
> > > > regions
> > > > > > need
> > > > > > > > to be allocated on startup.
> > > > > > > >
> > > > > > > > Alexey Goncharuk, do you have any suggestions on this?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > --
> > > > > > > --
> > > > > > > Maxim Muzafarov
> > > > > > >
> > > > > >
> > > > > --
> > > > > --
> > > > > Maxim Muzafarov
> > > > >
> > > >
> > >
> >
>
Re: Data regions on client nodes
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Alexey,
I am not sure I understand. If cache creation proceeds, but memory region
was not allocated, how can the cache operate?
D.
On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk <alexey.goncharuk@gmail.com
> wrote:
> I do not mind making this change, but note that the reason for non-lazy
> region allocation is the need to gracefully handle OOME errors during cache
> start. When a region is pre-allocated, no OOME can happen.
>
> If a region is allocated dynamically, then all errors that happened during
> the node start before should be properly handled (a client node must be
> stopped, but cache creation should proceed).
>
> пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
> valentin.kulichenko@gmail.com>:
>
> > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
> >
> > -Val
> >
> > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <dp...@gmail.com>
> > wrote:
> >
> > > Maxim, thank you.
> > >
> > > If it seems it is technically possible, we can file ticket for this
> > change.
> > >
> > > I find this proposal reasonable, change makes perfectly sense to me.
> > >
> > > We can wait Alex G. feedback on this change before starting actual
> > > implementation. It can take for a while, because he is travelling now.
> > >
> > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <ma...@gmail.com>:
> > >
> > > > Guys,
> > > >
> > > > I can miss some details, but at the first glance we have everething
> we
> > > need
> > > > to defer
> > > > region memory allocation if it has no cache groups assignments. And
> it
> > > > doesn't matter
> > > > where it happens on client or server nodes.
> > > >
> > > > Currently region memory allocation happens at exchange future init
> > > method.
> > > > At the
> > > > node startup method initCachesOnLocalJoin executes. This method
> > > resposnible
> > > > for
> > > > memory allocation (through initiating cache managers) and it also
> > starts
> > > > caches.
> > > > So, at this point we have all existing caches descriptors and can
> find
> > > out
> > > > which
> > > > cache matches which region to defer some regions initialization to
> the
> > > > moment when
> > > > newly cache assings to this region (happend at onCacheChangeRequest).
> > > >
> > > > Please, сorrect me if I'm wrong and missing something.
> > > >
> > > >
> > > >
> > > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <dp...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Maxim,
> > > > >
> > > > > thank you for stepping in. How do you think, is it possible to
> check
> > > > cache
> > > > > assignment to region at stage of memory allocation?
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <ma...@gmail.com>:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > I've checked memory allocation. It looks like we are allocating
> > > memory
> > > > > only
> > > > > > on the first exchange future init on local join occurs on node.
> > Also,
> > > > > seems
> > > > > > like we are allocating only the first chunk of memory (not the
> > whole
> > > > > bunch)
> > > > > > and it calculates as:
> > > > > >
> > > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 *
> 1024)
> > > > > >
> > > > > > But, I'm agree with Val. It's better to allocate memory only when
> > > when
> > > > > > the first cache assigned to this region.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Also, It seems like we have some problem with user notification
> > about
> > > > > > available
> > > > > > physical resources. For client nodes method requiredOffheap()
> > returns
> > > > > > always
> > > > > > zero [1]. That's why WARN message shown here [2] would be not not
> > > quite
> > > > > > right
> > > > > > if we have a lot of client nodes in cluster.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/ignite/blob/master/modules/
> core/src/main/java/org/apache/ignite/internal/managers/discovery/
> GridDiscoveryManager.java#L1501
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/ignite/blob/master/modules/
> core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> > > > > >
> > > > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org
> > > > >:
> > > > > >
> > > > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > > > > valentin.kulichenko@gmail.com> wrote:
> > > > > > >
> > > > > > > > Actually, I would go even further: only allocate a data
> region
> > > on a
> > > > > > node
> > > > > > > > when the first cache assigned to this region is deployed on
> > that
> > > > > node.
> > > > > > > > Because issue is broader than client nodes and local caches.
> > One
> > > > can
> > > > > > have
> > > > > > > > server nodes without any caches as well - running only
> > services,
> > > > for
> > > > > > > > example.
> > > > > > > >
> > > > > > >
> > > > > > > It would be great if this was possible, but to my knowledge,
> > > regions
> > > > > need
> > > > > > > to be allocated on startup.
> > > > > > >
> > > > > > > Alexey Goncharuk, do you have any suggestions on this?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > --
> > > > > > --
> > > > > > Maxim Muzafarov
> > > > > >
> > > > >
> > > > --
> > > > --
> > > > Maxim Muzafarov
> > > >
> > >
> >
>
Re: Data regions on client nodes
Posted by Alexey Goncharuk <al...@gmail.com>.
I do not mind making this change, but note that the reason for non-lazy
region allocation is the need to gracefully handle OOME errors during cache
start. When a region is pre-allocated, no OOME can happen.
If a region is allocated dynamically, then all errors that happened during
the node start before should be properly handled (a client node must be
stopped, but cache creation should proceed).
пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko <
valentin.kulichenko@gmail.com>:
> Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
>
> -Val
>
> On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > Maxim, thank you.
> >
> > If it seems it is technically possible, we can file ticket for this
> change.
> >
> > I find this proposal reasonable, change makes perfectly sense to me.
> >
> > We can wait Alex G. feedback on this change before starting actual
> > implementation. It can take for a while, because he is travelling now.
> >
> > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <ma...@gmail.com>:
> >
> > > Guys,
> > >
> > > I can miss some details, but at the first glance we have everething we
> > need
> > > to defer
> > > region memory allocation if it has no cache groups assignments. And it
> > > doesn't matter
> > > where it happens on client or server nodes.
> > >
> > > Currently region memory allocation happens at exchange future init
> > method.
> > > At the
> > > node startup method initCachesOnLocalJoin executes. This method
> > resposnible
> > > for
> > > memory allocation (through initiating cache managers) and it also
> starts
> > > caches.
> > > So, at this point we have all existing caches descriptors and can find
> > out
> > > which
> > > cache matches which region to defer some regions initialization to the
> > > moment when
> > > newly cache assings to this region (happend at onCacheChangeRequest).
> > >
> > > Please, сorrect me if I'm wrong and missing something.
> > >
> > >
> > >
> > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <dp...@gmail.com>
> > wrote:
> > >
> > > > Hi Maxim,
> > > >
> > > > thank you for stepping in. How do you think, is it possible to check
> > > cache
> > > > assignment to region at stage of memory allocation?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <ma...@gmail.com>:
> > > >
> > > > > Folks,
> > > > >
> > > > > I've checked memory allocation. It looks like we are allocating
> > memory
> > > > only
> > > > > on the first exchange future init on local join occurs on node.
> Also,
> > > > seems
> > > > > like we are allocating only the first chunk of memory (not the
> whole
> > > > bunch)
> > > > > and it calculates as:
> > > > >
> > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 * 1024)
> > > > >
> > > > > But, I'm agree with Val. It's better to allocate memory only when
> > when
> > > > > the first cache assigned to this region.
> > > > >
> > > > >
> > > > >
> > > > > Also, It seems like we have some problem with user notification
> about
> > > > > available
> > > > > physical resources. For client nodes method requiredOffheap()
> returns
> > > > > always
> > > > > zero [1]. That's why WARN message shown here [2] would be not not
> > quite
> > > > > right
> > > > > if we have a lot of client nodes in cluster.
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java#L1501
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> > > > >
> > > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >:
> > > > >
> > > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > > > valentin.kulichenko@gmail.com> wrote:
> > > > > >
> > > > > > > Actually, I would go even further: only allocate a data region
> > on a
> > > > > node
> > > > > > > when the first cache assigned to this region is deployed on
> that
> > > > node.
> > > > > > > Because issue is broader than client nodes and local caches.
> One
> > > can
> > > > > have
> > > > > > > server nodes without any caches as well - running only
> services,
> > > for
> > > > > > > example.
> > > > > > >
> > > > > >
> > > > > > It would be great if this was possible, but to my knowledge,
> > regions
> > > > need
> > > > > > to be allocated on startup.
> > > > > >
> > > > > > Alexey Goncharuk, do you have any suggestions on this?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > --
> > > > > --
> > > > > Maxim Muzafarov
> > > > >
> > > >
> > > --
> > > --
> > > Maxim Muzafarov
> > >
> >
>
Re: Data regions on client nodes
Posted by Valentin Kulichenko <va...@gmail.com>.
Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113
-Val
On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <dp...@gmail.com> wrote:
> Maxim, thank you.
>
> If it seems it is technically possible, we can file ticket for this change.
>
> I find this proposal reasonable, change makes perfectly sense to me.
>
> We can wait Alex G. feedback on this change before starting actual
> implementation. It can take for a while, because he is travelling now.
>
> пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <ma...@gmail.com>:
>
> > Guys,
> >
> > I can miss some details, but at the first glance we have everething we
> need
> > to defer
> > region memory allocation if it has no cache groups assignments. And it
> > doesn't matter
> > where it happens on client or server nodes.
> >
> > Currently region memory allocation happens at exchange future init
> method.
> > At the
> > node startup method initCachesOnLocalJoin executes. This method
> resposnible
> > for
> > memory allocation (through initiating cache managers) and it also starts
> > caches.
> > So, at this point we have all existing caches descriptors and can find
> out
> > which
> > cache matches which region to defer some regions initialization to the
> > moment when
> > newly cache assings to this region (happend at onCacheChangeRequest).
> >
> > Please, сorrect me if I'm wrong and missing something.
> >
> >
> >
> > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <dp...@gmail.com>
> wrote:
> >
> > > Hi Maxim,
> > >
> > > thank you for stepping in. How do you think, is it possible to check
> > cache
> > > assignment to region at stage of memory allocation?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <ma...@gmail.com>:
> > >
> > > > Folks,
> > > >
> > > > I've checked memory allocation. It looks like we are allocating
> memory
> > > only
> > > > on the first exchange future init on local join occurs on node. Also,
> > > seems
> > > > like we are allocating only the first chunk of memory (not the whole
> > > bunch)
> > > > and it calculates as:
> > > >
> > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 * 1024)
> > > >
> > > > But, I'm agree with Val. It's better to allocate memory only when
> when
> > > > the first cache assigned to this region.
> > > >
> > > >
> > > >
> > > > Also, It seems like we have some problem with user notification about
> > > > available
> > > > physical resources. For client nodes method requiredOffheap() returns
> > > > always
> > > > zero [1]. That's why WARN message shown here [2] would be not not
> quite
> > > > right
> > > > if we have a lot of client nodes in cluster.
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java#L1501
> > > > [2]
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> > > >
> > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >:
> > > >
> > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com> wrote:
> > > > >
> > > > > > Actually, I would go even further: only allocate a data region
> on a
> > > > node
> > > > > > when the first cache assigned to this region is deployed on that
> > > node.
> > > > > > Because issue is broader than client nodes and local caches. One
> > can
> > > > have
> > > > > > server nodes without any caches as well - running only services,
> > for
> > > > > > example.
> > > > > >
> > > > >
> > > > > It would be great if this was possible, but to my knowledge,
> regions
> > > need
> > > > > to be allocated on startup.
> > > > >
> > > > > Alexey Goncharuk, do you have any suggestions on this?
> > > > >
> > > > > D.
> > > > >
> > > > --
> > > > --
> > > > Maxim Muzafarov
> > > >
> > >
> > --
> > --
> > Maxim Muzafarov
> >
>
Re: Data regions on client nodes
Posted by Dmitry Pavlov <dp...@gmail.com>.
Maxim, thank you.
If it seems it is technically possible, we can file ticket for this change.
I find this proposal reasonable, change makes perfectly sense to me.
We can wait Alex G. feedback on this change before starting actual
implementation. It can take for a while, because he is travelling now.
пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <ma...@gmail.com>:
> Guys,
>
> I can miss some details, but at the first glance we have everething we need
> to defer
> region memory allocation if it has no cache groups assignments. And it
> doesn't matter
> where it happens on client or server nodes.
>
> Currently region memory allocation happens at exchange future init method.
> At the
> node startup method initCachesOnLocalJoin executes. This method resposnible
> for
> memory allocation (through initiating cache managers) and it also starts
> caches.
> So, at this point we have all existing caches descriptors and can find out
> which
> cache matches which region to defer some regions initialization to the
> moment when
> newly cache assings to this region (happend at onCacheChangeRequest).
>
> Please, сorrect me if I'm wrong and missing something.
>
>
>
> On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <dp...@gmail.com> wrote:
>
> > Hi Maxim,
> >
> > thank you for stepping in. How do you think, is it possible to check
> cache
> > assignment to region at stage of memory allocation?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <ma...@gmail.com>:
> >
> > > Folks,
> > >
> > > I've checked memory allocation. It looks like we are allocating memory
> > only
> > > on the first exchange future init on local join occurs on node. Also,
> > seems
> > > like we are allocating only the first chunk of memory (not the whole
> > bunch)
> > > and it calculates as:
> > >
> > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 * 1024)
> > >
> > > But, I'm agree with Val. It's better to allocate memory only when when
> > > the first cache assigned to this region.
> > >
> > >
> > >
> > > Also, It seems like we have some problem with user notification about
> > > available
> > > physical resources. For client nodes method requiredOffheap() returns
> > > always
> > > zero [1]. That's why WARN message shown here [2] would be not not quite
> > > right
> > > if we have a lot of client nodes in cluster.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java#L1501
> > > [2]
> > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> > >
> > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <dsetrakyan@apache.org
> >:
> > >
> > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > >
> > > > > Actually, I would go even further: only allocate a data region on a
> > > node
> > > > > when the first cache assigned to this region is deployed on that
> > node.
> > > > > Because issue is broader than client nodes and local caches. One
> can
> > > have
> > > > > server nodes without any caches as well - running only services,
> for
> > > > > example.
> > > > >
> > > >
> > > > It would be great if this was possible, but to my knowledge, regions
> > need
> > > > to be allocated on startup.
> > > >
> > > > Alexey Goncharuk, do you have any suggestions on this?
> > > >
> > > > D.
> > > >
> > > --
> > > --
> > > Maxim Muzafarov
> > >
> >
> --
> --
> Maxim Muzafarov
>
Re: Data regions on client nodes
Posted by Maxim Muzafarov <ma...@gmail.com>.
Guys,
I can miss some details, but at the first glance we have everething we need
to defer
region memory allocation if it has no cache groups assignments. And it
doesn't matter
where it happens on client or server nodes.
Currently region memory allocation happens at exchange future init method.
At the
node startup method initCachesOnLocalJoin executes. This method resposnible
for
memory allocation (through initiating cache managers) and it also starts
caches.
So, at this point we have all existing caches descriptors and can find out
which
cache matches which region to defer some regions initialization to the
moment when
newly cache assings to this region (happend at onCacheChangeRequest).
Please, сorrect me if I'm wrong and missing something.
On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov <dp...@gmail.com> wrote:
> Hi Maxim,
>
> thank you for stepping in. How do you think, is it possible to check cache
> assignment to region at stage of memory allocation?
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <ma...@gmail.com>:
>
> > Folks,
> >
> > I've checked memory allocation. It looks like we are allocating memory
> only
> > on the first exchange future init on local join occurs on node. Also,
> seems
> > like we are allocating only the first chunk of memory (not the whole
> bunch)
> > and it calculates as:
> >
> > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 * 1024)
> >
> > But, I'm agree with Val. It's better to allocate memory only when when
> > the first cache assigned to this region.
> >
> >
> >
> > Also, It seems like we have some problem with user notification about
> > available
> > physical resources. For client nodes method requiredOffheap() returns
> > always
> > zero [1]. That's why WARN message shown here [2] would be not not quite
> > right
> > if we have a lot of client nodes in cluster.
> >
> >
> > [1]
> >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java#L1501
> > [2]
> >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
> >
> > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <ds...@apache.org>:
> >
> > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Actually, I would go even further: only allocate a data region on a
> > node
> > > > when the first cache assigned to this region is deployed on that
> node.
> > > > Because issue is broader than client nodes and local caches. One can
> > have
> > > > server nodes without any caches as well - running only services, for
> > > > example.
> > > >
> > >
> > > It would be great if this was possible, but to my knowledge, regions
> need
> > > to be allocated on startup.
> > >
> > > Alexey Goncharuk, do you have any suggestions on this?
> > >
> > > D.
> > >
> > --
> > --
> > Maxim Muzafarov
> >
>
--
--
Maxim Muzafarov
Re: Data regions on client nodes
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi Maxim,
thank you for stepping in. How do you think, is it possible to check cache
assignment to region at stage of memory allocation?
Sincerely,
Dmitriy Pavlov
ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov <ma...@gmail.com>:
> Folks,
>
> I've checked memory allocation. It looks like we are allocating memory only
> on the first exchange future init on local join occurs on node. Also, seems
> like we are allocating only the first chunk of memory (not the whole bunch)
> and it calculates as:
>
> Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 * 1024)
>
> But, I'm agree with Val. It's better to allocate memory only when when
> the first cache assigned to this region.
>
>
>
> Also, It seems like we have some problem with user notification about
> available
> physical resources. For client nodes method requiredOffheap() returns
> always
> zero [1]. That's why WARN message shown here [2] would be not not quite
> right
> if we have a lot of client nodes in cluster.
>
>
> [1]
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java#L1501
> [2]
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
>
> сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <ds...@apache.org>:
>
> > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Actually, I would go even further: only allocate a data region on a
> node
> > > when the first cache assigned to this region is deployed on that node.
> > > Because issue is broader than client nodes and local caches. One can
> have
> > > server nodes without any caches as well - running only services, for
> > > example.
> > >
> >
> > It would be great if this was possible, but to my knowledge, regions need
> > to be allocated on startup.
> >
> > Alexey Goncharuk, do you have any suggestions on this?
> >
> > D.
> >
> --
> --
> Maxim Muzafarov
>
Re: Data regions on client nodes
Posted by Maxim Muzafarov <ma...@gmail.com>.
Folks,
I've checked memory allocation. It looks like we are allocating memory only
on the first exchange future init on local join occurs on node. Also, seems
like we are allocating only the first chunk of memory (not the whole bunch)
and it calculates as:
Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 * 1024)
But, I'm agree with Val. It's better to allocate memory only when when
the first cache assigned to this region.
Also, It seems like we have some problem with user notification about
available
physical resources. For client nodes method requiredOffheap() returns always
zero [1]. That's why WARN message shown here [2] would be not not quite
right
if we have a lot of client nodes in cluster.
[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java#L1501
[2]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489
сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan <ds...@apache.org>:
> On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Actually, I would go even further: only allocate a data region on a node
> > when the first cache assigned to this region is deployed on that node.
> > Because issue is broader than client nodes and local caches. One can have
> > server nodes without any caches as well - running only services, for
> > example.
> >
>
> It would be great if this was possible, but to my knowledge, regions need
> to be allocated on startup.
>
> Alexey Goncharuk, do you have any suggestions on this?
>
> D.
>
--
--
Maxim Muzafarov
Re: Data regions on client nodes
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Actually, I would go even further: only allocate a data region on a node
> when the first cache assigned to this region is deployed on that node.
> Because issue is broader than client nodes and local caches. One can have
> server nodes without any caches as well - running only services, for
> example.
>
It would be great if this was possible, but to my knowledge, regions need
to be allocated on startup.
Alexey Goncharuk, do you have any suggestions on this?
D.
Re: Data regions on client nodes
Posted by Valentin Kulichenko <va...@gmail.com>.
Actually, I would go even further: only allocate a data region on a node
when the first cache assigned to this region is deployed on that node.
Because issue is broader than client nodes and local caches. One can have
server nodes without any caches as well - running only services, for
example.
-Val
On Fri, Jul 20, 2018 at 6:30 PM Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Val, thanks for pointing this out.
>
> I would actually not allocate any off-heap memory on the client side unless
> we see Local caches in the configuration. This is such a rare case, that we
> can ignore it altogether.
>
> D.
>
> On Fri, Jul 20, 2018 at 3:59 PM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Folks,
> >
> > Currently do not create any regions or allocate any offheap memory on
> > client nodes unless it's explicitly configured. This is good behavior,
> > however there is a usability issue caused by the fact that many users
> have
> > the same config file for both server and clients. This can lead to
> > unexpected excessive memory usage on client side and forces users to
> > maintain two config files in most cases.
> >
> > At the same time, the only case when offheap memory can be required on a
> > client node is using LOCAL caches there, which is a very rare use case.
> >
> > Having said that, is it possible to allocate memory on client node
> > dynamically ONLY if a local cache is created there? This would fix the
> > usability issue without limiting the use of local caches on client side.
> >
> > -Val
> >
>
Re: Data regions on client nodes
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Val, thanks for pointing this out.
I would actually not allocate any off-heap memory on the client side unless
we see Local caches in the configuration. This is such a rare case, that we
can ignore it altogether.
D.
On Fri, Jul 20, 2018 at 3:59 PM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Folks,
>
> Currently do not create any regions or allocate any offheap memory on
> client nodes unless it's explicitly configured. This is good behavior,
> however there is a usability issue caused by the fact that many users have
> the same config file for both server and clients. This can lead to
> unexpected excessive memory usage on client side and forces users to
> maintain two config files in most cases.
>
> At the same time, the only case when offheap memory can be required on a
> client node is using LOCAL caches there, which is a very rare use case.
>
> Having said that, is it possible to allocate memory on client node
> dynamically ONLY if a local cache is created there? This would fix the
> usability issue without limiting the use of local caches on client side.
>
> -Val
>