Geode memory handling issues

[As...@amat.com]

Hi,

I am facing some issues with geode memory/resource management.
I have simple environment with a single locator and a single server, both are launched via gfsh.
The server is started with initial-heap=max-heap=4GBand with critical-heap-percentage=70%.
I also created a single region of type partition using gfsh.

In addition, I created a simple client application (client-server topology, accessing the region via ClientCache PROXY). The client simply iteratively put data elements (of 100MB each) into the region. Upon putting total size of ~70% of 4GB, I get exception on client side which tells the server is working at low memory as expected - so far so good.
While observing the server metrics (using show metrics) I see that the server holds data entries as expected and the heap usage and total heap size are fine too.

Now, I'm removing all elements from the region - the server metrics after this operation shows that the elements count is 0 (i.e. the server is empty) but still heap usage is high (probably because GC didn't collect freed items). That's shouldn't bother me, but the problem is if I'm trying to put additional elements into the region now I still get the exception on client.
Although the server is empty the client can't put items into it and this is very problematic from user point of view.

I tried to play with GC flags, change GC to G1 but with no success. I can't control GC collection - having an idle empty server with no way to add elements to it.

What am I missing here? Is there some other configuration I should follow?

Thanks.

Assaf Waizman
SW Architect | Process Diagnostics and Control | Applied Materials
9 Oppenheimer Street, Rehovot, 76705.  Israel.
Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax +972.8.948.8848

Re: Geode memory handling issues

[Eric Shu <es...@pivotal.io>]

Data in offheap region is on offheap. It is based on offheap memory size
(calculated by data in regions) in GEODE internally. The resource
management porperty setting is critical-off-heap-percentage.
Link for offheap management.
http://geode.apache.org/docs/guide/managing/heap_use/off_heap_management.html

Regards,
Eric



On Tue, Dec 13, 2016 at 1:38 PM, <As...@amat.com> wrote:

> Indeed the eviction/offheap features or persistency can workaround this, I
> considered the offheap but the problem for my scenario is that upon
> reaching the threshold (which is tested by checking heap status) the
> offheap will take place in I will have throughput hit although I have free
> space in RAM (the server elements usage are less than threshold after some
> keys removal, but since offheap is based on querying the heap new puts will
> go offheap).
>
>
>
> My system requirements can’t afford working with disk while having free
> RAM, it will impact it dramatically.
>
>
>
> Please share if you have more ideas to overcome this.
>
>
>
> Thanks a lot.
>
>
>
> *From:* Eric Shu [mailto:eshu@pivotal.io]
> *Sent:* יום ג 13 דצמבר 2016 23:27
>
> *To:* user@geode.apache.org
> *Subject:* Re: Geode memory handling issues
>
>
>
> Others might chime in as well. I think most users set eviction if they
> know there will be a memory issue. Eviction is kicked in before the
> critical heap setting (Low Memory Exception) with correct settings.
>
>
>
> Also, if it is partitioned region, you can add capacity by adding more
> nodes so that system will be able to proceed. Users may also shut down
> nodes and restart them again -- with persistence enabled. The data will be
> recovered from disc.
>
>
>
> Regards,
>
> Eric
>
>
>
> On Tue, Dec 13, 2016 at 1:16 PM, <As...@amat.com> wrote:
>
> Thanks Dan & Jared,
>
> I carefully read the documentation for the resource management and noticed
> that part that tells the interaction between it and the GC.
> I saw that my server is working with ConcMarkSweep and set the
> InitiatingOccupancyFraction to lower percentage but with no help, GC just
> won’t kick in. I also tried the G1 GC with now help. (also, just to make
> sure that is the cause, I attached to the jvm using yourkit and initiate GC
> manually and indeed the server was able to get put requests again)
>
> For my experience with JVM memory management, counting on GC collection
> can be problematic as you can't really control the collection timing, in
> addition invoking it directly (System.gc) is stated as bad practice. I
> would expect such resource manager to track its elements usage directly and
> not by querying the heap status.
>
> This mem usage issue seems to be very basic, I mean how come GemFire users
> (which I know runs in production) are not facing this problem?
>
> I really appreciate your help guys.
>
> -----Original Message-----
> From: Dan Smith [mailto:dsmith@pivotal.io]
> Sent: יום ג 13 דצמבר 2016 21:44
> To: user@geode.apache.org
> Subject: Re: Geode memory handling issues
>
> Hi Assaf,
>
> +1 for that link Jared send out. In order to the resource manager to
> work, you need to be using ConcMarkSweep with an
> InitiatingOccupancyFraction that's less than your critical and eviction
> heap thresholds. That will cause GC to kick in if your heap is above those
> thresholds.
>
> -Dan
>
> On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io>
> wrote:
> > Hi Assaf,
> >
> > There is some information about tuning the JVM’s garbage collection
> > parameters to work with eviction available here:
> > http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> > tml#resource_manager
> >
> > Best,
> > Jared
> >
> > On Dec 13, 2016, at 11:26 AM, <As...@amat.com>
> > <As...@amat.com> wrote:
> >
> > Hi Eric,
> >
> > Thanks for the quick response!
> > Shutting down the server is problematic for me. In a real scenario I
> > won’t remove all items but only some of them, thus the server still
> > contains data but have free space – still I will get the exception.
> > Shutting down the server will cause me to lose other entries – I’m
> > afraid it’s not applicable to me.
> >
> > I also understand that I can’t really control GC collection, so how
> > does the critical-threshold is expected to work? Once you reach it you
> > can’t add more items to server even if you removed some.
> >
> > Thanks.
> >
> > From: Eric Shu [mailto:eshu@pivotal.io]
> > Sent: יום ג 13 דצמבר 2016 21:21
> > To: user@geode.apache.org
> > Subject: Re: Geode memory handling issues
> >
> > I am not sure if GC collection can be controlled. One possible way to
> > work around this is to shut down the server and restart it.
> >
> > Also you may want to try offheap region to cope this issue?
> >
> > Regards,
> > Eric
> >
> > On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:
> > Hi,
> >
> > I am facing some issues with geode memory/resource management.
> > I have simple environment with a single locator and a single server,
> > both are launched via gfsh.
> > The server is started with initial-heap=max-heap=4GBand with
> > critical-heap-percentage=70%.
> > I also created a single region of type partition using gfsh.
> >
> > In addition, I created a simple client application (client-server
> > topology, accessing the region via ClientCache PROXY). The client
> > simply iteratively put data elements (of 100MB each) into the region.
> > Upon putting total size of ~70% of 4GB, I get exception on client side
> > which tells the server is working at low memory as expected – so far so
> good.
> > While observing the server metrics (using show metrics) I see that the
> > server holds data entries as expected and the heap usage and total
> > heap size are fine too.
> >
> > Now, I’m removing all elements from the region – the server metrics
> > after this operation shows that the elements count is 0 (i.e. the
> > server is empty) but still heap usage is high (probably because GC
> > didn’t collect freed items). That’s shouldn’t bother me, but the
> > problem is if I’m trying to put additional elements into the region now
> I still get the exception on client.
> > Although the server is empty the client can’t put items into it and
> > this is very problematic from user point of view.
> >
> > I tried to play with GC flags, change GC to G1 but with no success. I
> > can’t control GC collection – having an idle empty server with no way
> > to add elements to it.
> >
> > What am I missing here? Is there some other configuration I should
> follow?
> >
> > Thanks.
> >
> > Assaf Waizman
> > SW Architect | Process Diagnostics and Control | Applied Materials
> > 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> > Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax
> > +972.8.948.8848
> >
> >
>
>
>

Re: Geode memory handling issues

[Gang Yan <gy...@pivotal.io>]

Hi Assaf

Maybe， you met  tombstone mechanism.

And you can find some words about  tombstone on  the following link:
http://geode.apache.org/docs/guide/developing/distributed_regions/how_region_versioning_works.html

<How Destroy and Clear Operations Are Resolved>

"When consistency checking is enabled for a region, a Geode member does not
immediately remove an entry from the region when an application destroys
the entry. Instead, the member retains the entry with its current version
stamp for a period of time in order to detect possible conflicts with
operations that have occurred. The retained entry is referred to as a
tombstone. "

Thanks and regards

Gang Yan（闫钢）
Sr. Solution Architect
Customer Success, Pivotal Great China
Mobile:+86 13636347460
gyan@pivotal.io

2016-12-14 6:31 GMT+08:00 Udo Kohlmeyer <uk...@pivotal.io>:

> The way overflow works is when the eviction-threshold is reached, the
> system will ask each region with overflow enabled to start overflowing data
> to disk.
>
> Whilst you have overflow enabled please set the previously mentioned ‑XX:
> CMSInitiatingOccupancyFraction=X and -XX:+UseCMSInitiatingOccupancyOnly.
> Make the InitiatingOccupancyFraction to be a few percent lower than the
> eviction-threshold. Then the GC will kick in clean before having to
> potentially overflow to disk. Also, once overflowed to disk, the gc will
> try and clean up any garbage left behind.
>
> Hope this helps...
>
> --Udo
>
> On 12/13/16 14:22, Assaf_Waizman@amat.com wrote:
>
> Thanks Eric,
>
> I mistakenly thought that offheap is overflow...
> I'll test the offheap for my scenario, hope it will perform well on my
> data sizes and variance.
> One more thing I would like to ask- I saw that the overflow feature can be
> configured per region, does it mean that it also checks internally the mem
> usage for the region (not by querying the heap). If so it might be a
> possible workaround for me too.
>
> Once again, I really appreciate your help.
>
> On 14 Dec 2016, at 0:05, Charlie Black <cb...@pivotal.io> wrote:
>
> It feels like Java hasn't hit the threshold when the GC would kick in.
>
> From a Geode perspective if the entry count is zero all of the references
> to the objects stored in Geode should be removed and are waiting to be
> reclaimed by Java.
>
> If you would like to force a GC try the gfsh command "gc".   GC will
> attempt to run the Java Garbage Collector.   Here is an example run:
>
> gfsh>gc
>
> GC Summary
>
>
>         Member ID/Name          | HeapSize (MB) Before GC | HeapSize(MB)
> After GC | Time Taken for GC in ms
>
> ------------------------------- | ----------------------- |
> --------------------- | -----------------------
>
> 192.168.5.1(foo:23089)<v0>:9696 | 189                     | 142
>         | 84
>
> On Tue, Dec 13, 2016 at 2:40 PM <As...@amat.com> wrote:
>
>> Indeed the eviction/offheap features or persistency can workaround this,
>> I considered the offheap but the problem for my scenario is that upon
>> reaching the threshold (which is tested by checking heap status) the
>> offheap will take place in I will have throughput hit although I have free
>> space in RAM (the server elements usage are less than threshold after some
>> keys removal, but since offheap is based on querying the heap new puts will
>> go offheap).
>>
>>
>>
>> My system requirements can’t afford working with disk while having free
>> RAM, it will impact it dramatically.
>>
>>
>>
>> Please share if you have more ideas to overcome this.
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>> *From:* Eric Shu [mailto:eshu@pivotal.io]
>> *Sent:* יום ג 13 דצמבר 2016 23:27
>>
>>
>> *To:* user@geode.apache.org
>> *Subject:* Re: Geode memory handling issues
>>
>>
>>
>> Others might chime in as well. I think most users set eviction if they
>> know there will be a memory issue. Eviction is kicked in before the
>> critical heap setting (Low Memory Exception) with correct settings.
>>
>>
>>
>> Also, if it is partitioned region, you can add capacity by adding more
>> nodes so that system will be able to proceed. Users may also shut down
>> nodes and restart them again -- with persistence enabled. The data will be
>> recovered from disc.
>>
>>
>>
>> Regards,
>>
>> Eric
>>
>>
>>
>> On Tue, Dec 13, 2016 at 1:16 PM, <As...@amat.com> wrote:
>>
>> Thanks Dan & Jared,
>>
>> I carefully read the documentation for the resource management and
>> noticed that part that tells the interaction between it and the GC.
>> I saw that my server is working with ConcMarkSweep and set the
>> InitiatingOccupancyFraction to lower percentage but with no help, GC just
>> won’t kick in. I also tried the G1 GC with now help. (also, just to make
>> sure that is the cause, I attached to the jvm using yourkit and initiate GC
>> manually and indeed the server was able to get put requests again)
>>
>> For my experience with JVM memory management, counting on GC collection
>> can be problematic as you can't really control the collection timing, in
>> addition invoking it directly (System.gc) is stated as bad practice. I
>> would expect such resource manager to track its elements usage directly and
>> not by querying the heap status.
>>
>> This mem usage issue seems to be very basic, I mean how come GemFire
>> users (which I know runs in production) are not facing this problem?
>>
>> I really appreciate your help guys.
>>
>> -----Original Message-----
>> From: Dan Smith [mailto:dsmith@pivotal.io]
>> Sent: יום ג 13 דצמבר 2016 21:44
>> To: user@geode.apache.org
>> Subject: Re: Geode memory handling issues
>>
>> Hi Assaf,
>>
>> +1 for that link Jared send out. In order to the resource manager to
>> work, you need to be using ConcMarkSweep with an
>> InitiatingOccupancyFraction that's less than your critical and eviction
>> heap thresholds. That will cause GC to kick in if your heap is above those
>> thresholds.
>>
>> -Dan
>>
>> On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io>
>> wrote:
>> > Hi Assaf,
>> >
>> > There is some information about tuning the JVM’s garbage collection
>> > parameters to work with eviction available here:
>> > http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
>> > tml#resource_manager
>> >
>> > Best,
>> > Jared
>> >
>> > On Dec 13, 2016, at 11:26 AM, <As...@amat.com>
>> > <As...@amat.com> wrote:
>> >
>> > Hi Eric,
>> >
>> > Thanks for the quick response!
>> > Shutting down the server is problematic for me. In a real scenario I
>> > won’t remove all items but only some of them, thus the server still
>> > contains data but have free space – still I will get the exception.
>> > Shutting down the server will cause me to lose other entries – I’m
>> > afraid it’s not applicable to me.
>> >
>> > I also understand that I can’t really control GC collection, so how
>> > does the critical-threshold is expected to work? Once you reach it you
>> > can’t add more items to server even if you removed some.
>> >
>> > Thanks.
>> >
>> > From: Eric Shu [mailto:eshu@pivotal.io]
>> > Sent: יום ג 13 דצמבר 2016 21:21
>> > To: user@geode.apache.org
>> > Subject: Re: Geode memory handling issues
>> >
>> > I am not sure if GC collection can be controlled. One possible way to
>> > work around this is to shut down the server and restart it.
>> >
>> > Also you may want to try offheap region to cope this issue?
>> >
>> > Regards,
>> > Eric
>> >
>> > On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:
>> > Hi,
>> >
>> > I am facing some issues with geode memory/resource management.
>> > I have simple environment with a single locator and a single server,
>> > both are launched via gfsh.
>> > The server is started with initial-heap=max-heap=4GBand with
>> > critical-heap-percentage=70%.
>> > I also created a single region of type partition using gfsh.
>> >
>> > In addition, I created a simple client application (client-server
>> > topology, accessing the region via ClientCache PROXY). The client
>> > simply iteratively put data elements (of 100MB each) into the region.
>> > Upon putting total size of ~70% of 4GB, I get exception on client side
>> > which tells the server is working at low memory as expected – so far so
>> good.
>> > While observing the server metrics (using show metrics) I see that the
>> > server holds data entries as expected and the heap usage and total
>> > heap size are fine too.
>> >
>> > Now, I’m removing all elements from the region – the server metrics
>> > after this operation shows that the elements count is 0 (i.e. the
>> > server is empty) but still heap usage is high (probably because GC
>> > didn’t collect freed items). That’s shouldn’t bother me, but the
>> > problem is if I’m trying to put additional elements into the region now
>> I still get the exception on client.
>> > Although the server is empty the client can’t put items into it and
>> > this is very problematic from user point of view.
>> >
>> > I tried to play with GC flags, change GC to G1 but with no success. I
>> > can’t control GC collection – having an idle empty server with no way
>> > to add elements to it.
>> >
>> > What am I missing here? Is there some other configuration I should
>> follow?
>> >
>> > Thanks.
>> >
>> > Assaf Waizman
>> > SW Architect | Process Diagnostics and Control | Applied Materials
>> > 9 Oppenheimer Street, Rehovot, 76705.  Israel.
>> > Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax
>> > +972.8.948.8848
>> >
>> >
>>
>>
>>
> --
> ---
> cblack@pivotal.io | +1.858.480.9722 <(858)%20480-9722>
>
>
>

Re: Geode memory handling issues

[Udo Kohlmeyer <uk...@pivotal.io>]

The way overflow works is when the eviction-threshold is reached, the 
system will ask each region with overflow enabled to start overflowing 
data to disk.

Whilst you have overflow enabled please set the previously mentioned 
\u2011XX:CMSInitiatingOccupancyFraction=X and 
-XX:+UseCMSInitiatingOccupancyOnly. Make the InitiatingOccupancyFraction 
to be a few percent lower than the eviction-threshold. Then the GC will 
kick in clean before having to potentially overflow to disk. Also, once 
overflowed to disk, the gc will try and clean up any garbage left behind.

Hope this helps...

--Udo


On 12/13/16 14:22, Assaf_Waizman@amat.com wrote:
> Thanks Eric,
>
> I mistakenly thought that offheap is overflow...
> I'll test the offheap for my scenario, hope it will perform well on my 
> data sizes and variance.
> One more thing I would like to ask- I saw that the overflow feature 
> can be configured per region, does it mean that it also checks 
> internally the mem usage for the region (not by querying the heap). If 
> so it might be a possible workaround for me too.
>
> Once again, I really appreciate your help.
>
> On 14 Dec 2016, at 0:05, Charlie Black <cblack@pivotal.io 
> <ma...@pivotal.io>> wrote:
>
>> It feels like Java hasn't hit the threshold when the GC would kick in.
>>
>> From a Geode perspective if the entry count is zero all of the 
>> references to the objects stored in Geode should be removed and are 
>> waiting to be reclaimed by Java.
>>
>> If you would like to force a GC try the gfsh command "gc".   GC will 
>> attempt to run the Java Garbage Collector.   Here is an example run:
>>
>> gfsh>gc
>>
>> GC Summary
>>
>>
>> Member ID/Name| HeapSize (MB) Before GC | HeapSize(MB) After GC | 
>> Time Taken for GC in ms
>>
>> ------------------------------- | ----------------------- | 
>> --------------------- | -----------------------
>>
>> 192.168.5.1(foo:23089)<v0>:9696 | 189 | 142 | 84
>>
>>
>> On Tue, Dec 13, 2016 at 2:40 PM <Assaf_Waizman@amat.com 
>> <ma...@amat.com>> wrote:
>>
>>     Indeed the eviction/offheap features or persistency can
>>     workaround this, I considered the offheap but the problem for my
>>     scenario is that upon reaching the threshold (which is tested by
>>     checking heap status) the offheap will take place in I will have
>>     throughput hit although I have free space in RAM (the server
>>     elements usage are less than threshold after some keys removal,
>>     but since offheap is based on querying the heap new puts will go
>>     offheap).
>>
>>     My system requirements can\u2019t afford working with disk while
>>     having free RAM, it will impact it dramatically.
>>
>>     Please share if you have more ideas to overcome this.
>>
>>     Thanks a lot.
>>
>>     *From:*Eric Shu [mailto:eshu@pivotal.io <ma...@pivotal.io>]
>>     *Sent:* \u05d9\u05d5\u05dd \u05d2 13 \u05d3\u05e6\u05de\u05d1\u05e8 2016 23:27
>>
>>
>>     *To:* user@geode.apache.org <ma...@geode.apache.org>
>>     *Subject:* Re: Geode memory handling issues
>>
>>     Others might chime in as well. I think most users set eviction if
>>     they know there will be a memory issue. Eviction is kicked in
>>     before the critical heap setting (Low Memory Exception) with
>>     correct settings.
>>
>>     Also, if it is partitioned region, you can add capacity by adding
>>     more nodes so that system will be able to proceed. Users may also
>>     shut down nodes and restart them again -- with persistence
>>     enabled. The data will be recovered from disc.
>>
>>     Regards,
>>
>>     Eric
>>
>>     On Tue, Dec 13, 2016 at 1:16 PM, <Assaf_Waizman@amat.com
>>     <ma...@amat.com>> wrote:
>>
>>     Thanks Dan & Jared,
>>
>>     I carefully read the documentation for the resource management
>>     and noticed that part that tells the interaction between it and
>>     the GC.
>>     I saw that my server is working with ConcMarkSweep and set the
>>     InitiatingOccupancyFraction to lower percentage but with no help,
>>     GC just won\u2019t kick in. I also tried the G1 GC with now help.
>>     (also, just to make sure that is the cause, I attached to the jvm
>>     using yourkit and initiate GC manually and indeed the server was
>>     able to get put requests again)
>>
>>     For my experience with JVM memory management, counting on GC
>>     collection can be problematic as you can't really control the
>>     collection timing, in addition invoking it directly (System.gc)
>>     is stated as bad practice. I would expect such resource manager
>>     to track its elements usage directly and not by querying the heap
>>     status.
>>
>>     This mem usage issue seems to be very basic, I mean how come
>>     GemFire users (which I know runs in production) are not facing
>>     this problem?
>>
>>     I really appreciate your help guys.
>>
>>     -----Original Message-----
>>     From: Dan Smith [mailto:dsmith@pivotal.io <ma...@pivotal.io>]
>>     Sent: \u05d9\u05d5\u05dd\u05d2 13 \u05d3\u05e6\u05de\u05d1\u05e8 2016 21:44
>>     To: user@geode.apache.org <ma...@geode.apache.org>
>>     Subject: Re: Geode memory handling issues
>>
>>     Hi Assaf,
>>
>>     +1 for that link Jared send out. In order to the resource manager to
>>     work, you need to be using ConcMarkSweep with an
>>     InitiatingOccupancyFraction that's less than your critical and
>>     eviction heap thresholds. That will cause GC to kick in if your
>>     heap is above those thresholds.
>>
>>     -Dan
>>
>>     On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart
>>     <jstewart@pivotal.io <ma...@pivotal.io>> wrote:
>>     > Hi Assaf,
>>     >
>>     > There is some information about tuning the JVM\u2019s garbage collection
>>     > parameters to work with eviction available here:
>>     >
>>     http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
>>     > tml#resource_manager
>>     >
>>     > Best,
>>     > Jared
>>     >
>>     > On Dec 13, 2016, at 11:26 AM, <Assaf_Waizman@amat.com
>>     <ma...@amat.com>>
>>     > <Assaf_Waizman@amat.com <ma...@amat.com>> wrote:
>>     >
>>     > Hi Eric,
>>     >
>>     > Thanks for the quick response!
>>     > Shutting down the server is problematic for me. In a real
>>     scenario I
>>     > won\u2019t remove all items but only some of them, thus the server still
>>     > contains data but have free space \u2013 still I will get the exception.
>>     > Shutting down the server will cause me to lose other entries \u2013 I\u2019m
>>     > afraid it\u2019s not applicable to me.
>>     >
>>     > I also understand that I can\u2019t really control GC collection, so how
>>     > does the critical-threshold is expected to work? Once you reach
>>     it you
>>     > can\u2019t add more items to server even if you removed some.
>>     >
>>     > Thanks.
>>     >
>>     > From: Eric Shu [mailto:eshu@pivotal.io <ma...@pivotal.io>]
>>     > Sent: \u05d9\u05d5\u05dd \u05d2 13 \u05d3\u05e6\u05de\u05d1\u05e8 2016 21:21
>>     > To: user@geode.apache.org <ma...@geode.apache.org>
>>     > Subject: Re: Geode memory handling issues
>>     >
>>     > I am not sure if GC collection can be controlled. One possible
>>     way to
>>     > work around this is to shut down the server and restart it.
>>     >
>>     > Also you may want to try offheap region to cope this issue?
>>     >
>>     > Regards,
>>     > Eric
>>     >
>>     > On Tue, Dec 13, 2016 at 11:02 AM, <Assaf_Waizman@amat.com
>>     <ma...@amat.com>> wrote:
>>     > Hi,
>>     >
>>     > I am facing some issues with geode memory/resource management.
>>     > I have simple environment with a single locator and a single
>>     server,
>>     > both are launched via gfsh.
>>     > The server is started with initial-heap=max-heap=4GBand with
>>     > critical-heap-percentage=70%.
>>     > I also created a single region of type partition using gfsh.
>>     >
>>     > In addition, I created a simple client application (client-server
>>     > topology, accessing the region via ClientCache PROXY). The client
>>     > simply iteratively put data elements (of 100MB each) into the
>>     region.
>>     > Upon putting total size of ~70% of 4GB, I get exception on
>>     client side
>>     > which tells the server is working at low memory as expected \u2013
>>     so far so good.
>>     > While observing the server metrics (using show metrics) I see
>>     that the
>>     > server holds data entries as expected and the heap usage and total
>>     > heap size are fine too.
>>     >
>>     > Now, I\u2019m removing all elements from the region \u2013 the server metrics
>>     > after this operation shows that the elements count is 0 (i.e. the
>>     > server is empty) but still heap usage is high (probably because GC
>>     > didn\u2019t collect freed items). That\u2019s shouldn\u2019t bother me, but the
>>     > problem is if I\u2019m trying to put additional elements into the
>>     region now I still get the exception on client.
>>     > Although the server is empty the client can\u2019t put items into it and
>>     > this is very problematic from user point of view.
>>     >
>>     > I tried to play with GC flags, change GC to G1 but with no
>>     success. I
>>     > can\u2019t control GC collection \u2013 having an idle empty server with
>>     no way
>>     > to add elements to it.
>>     >
>>     > What am I missing here? Is there some other configuration I
>>     should follow?
>>     >
>>     > Thanks.
>>     >
>>     > Assaf Waizman
>>     > SW Architect | Process Diagnostics and Control | Applied Materials
>>     > 9 Oppenheimer Street, Rehovot, 76705. Israel.
>>     > Office +972.8.948.8661 <tel:%2B972.8.948.8661> | Mobile
>>     +972.54.80.10.799 <tel:%2B972.54.80.10.799> | Fax
>>     > +972.8.948.8848 <tel:%2B972.8.948.8848>
>>     >
>>     >
>>
>> -- 
>> ---
>> cblack@pivotal.io <ma...@pivotal.io> | +1.858.480.9722

Re: Geode memory handling issues

[Eric Shu <es...@pivotal.io>]

If it is configured a heap region, overflow is checked against the heap
(querying the heap).
If it is configured a offheap region, overflow is checked against the
offheap memory (calculated by total data in offheap regions).

Regards,
Eric

On Tue, Dec 13, 2016 at 2:22 PM, <As...@amat.com> wrote:

> Thanks Eric,
>
> I mistakenly thought that offheap is overflow...
> I'll test the offheap for my scenario, hope it will perform well on my
> data sizes and variance.
> One more thing I would like to ask- I saw that the overflow feature can be
> configured per region, does it mean that it also checks internally the mem
> usage for the region (not by querying the heap). If so it might be a
> possible workaround for me too.
>
> Once again, I really appreciate your help.
>
> On 14 Dec 2016, at 0:05, Charlie Black <cb...@pivotal.io> wrote:
>
> It feels like Java hasn't hit the threshold when the GC would kick in.
>
> From a Geode perspective if the entry count is zero all of the references
> to the objects stored in Geode should be removed and are waiting to be
> reclaimed by Java.
>
> If you would like to force a GC try the gfsh command "gc".   GC will
> attempt to run the Java Garbage Collector.   Here is an example run:
>
> gfsh>gc
>
> GC Summary
>
>
>         Member ID/Name          | HeapSize (MB) Before GC | HeapSize(MB)
> After GC | Time Taken for GC in ms
>
> ------------------------------- | ----------------------- |
> --------------------- | -----------------------
>
> 192.168.5.1(foo:23089)<v0>:9696 | 189                     | 142
>         | 84
>
> On Tue, Dec 13, 2016 at 2:40 PM <As...@amat.com> wrote:
>
>> Indeed the eviction/offheap features or persistency can workaround this,
>> I considered the offheap but the problem for my scenario is that upon
>> reaching the threshold (which is tested by checking heap status) the
>> offheap will take place in I will have throughput hit although I have free
>> space in RAM (the server elements usage are less than threshold after some
>> keys removal, but since offheap is based on querying the heap new puts will
>> go offheap).
>>
>>
>>
>> My system requirements can’t afford working with disk while having free
>> RAM, it will impact it dramatically.
>>
>>
>>
>> Please share if you have more ideas to overcome this.
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>> *From:* Eric Shu [mailto:eshu@pivotal.io]
>> *Sent:* יום ג 13 דצמבר 2016 23:27
>>
>>
>> *To:* user@geode.apache.org
>> *Subject:* Re: Geode memory handling issues
>>
>>
>>
>> Others might chime in as well. I think most users set eviction if they
>> know there will be a memory issue. Eviction is kicked in before the
>> critical heap setting (Low Memory Exception) with correct settings.
>>
>>
>>
>> Also, if it is partitioned region, you can add capacity by adding more
>> nodes so that system will be able to proceed. Users may also shut down
>> nodes and restart them again -- with persistence enabled. The data will be
>> recovered from disc.
>>
>>
>>
>> Regards,
>>
>> Eric
>>
>>
>>
>> On Tue, Dec 13, 2016 at 1:16 PM, <As...@amat.com> wrote:
>>
>> Thanks Dan & Jared,
>>
>> I carefully read the documentation for the resource management and
>> noticed that part that tells the interaction between it and the GC.
>> I saw that my server is working with ConcMarkSweep and set the
>> InitiatingOccupancyFraction to lower percentage but with no help, GC just
>> won’t kick in. I also tried the G1 GC with now help. (also, just to make
>> sure that is the cause, I attached to the jvm using yourkit and initiate GC
>> manually and indeed the server was able to get put requests again)
>>
>> For my experience with JVM memory management, counting on GC collection
>> can be problematic as you can't really control the collection timing, in
>> addition invoking it directly (System.gc) is stated as bad practice. I
>> would expect such resource manager to track its elements usage directly and
>> not by querying the heap status.
>>
>> This mem usage issue seems to be very basic, I mean how come GemFire
>> users (which I know runs in production) are not facing this problem?
>>
>> I really appreciate your help guys.
>>
>> -----Original Message-----
>> From: Dan Smith [mailto:dsmith@pivotal.io]
>> Sent: יום ג 13 דצמבר 2016 21:44
>> To: user@geode.apache.org
>> Subject: Re: Geode memory handling issues
>>
>> Hi Assaf,
>>
>> +1 for that link Jared send out. In order to the resource manager to
>> work, you need to be using ConcMarkSweep with an
>> InitiatingOccupancyFraction that's less than your critical and eviction
>> heap thresholds. That will cause GC to kick in if your heap is above those
>> thresholds.
>>
>> -Dan
>>
>> On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io>
>> wrote:
>> > Hi Assaf,
>> >
>> > There is some information about tuning the JVM’s garbage collection
>> > parameters to work with eviction available here:
>> > http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
>> > tml#resource_manager
>> >
>> > Best,
>> > Jared
>> >
>> > On Dec 13, 2016, at 11:26 AM, <As...@amat.com>
>> > <As...@amat.com> wrote:
>> >
>> > Hi Eric,
>> >
>> > Thanks for the quick response!
>> > Shutting down the server is problematic for me. In a real scenario I
>> > won’t remove all items but only some of them, thus the server still
>> > contains data but have free space – still I will get the exception.
>> > Shutting down the server will cause me to lose other entries – I’m
>> > afraid it’s not applicable to me.
>> >
>> > I also understand that I can’t really control GC collection, so how
>> > does the critical-threshold is expected to work? Once you reach it you
>> > can’t add more items to server even if you removed some.
>> >
>> > Thanks.
>> >
>> > From: Eric Shu [mailto:eshu@pivotal.io]
>> > Sent: יום ג 13 דצמבר 2016 21:21
>> > To: user@geode.apache.org
>> > Subject: Re: Geode memory handling issues
>> >
>> > I am not sure if GC collection can be controlled. One possible way to
>> > work around this is to shut down the server and restart it.
>> >
>> > Also you may want to try offheap region to cope this issue?
>> >
>> > Regards,
>> > Eric
>> >
>> > On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:
>> > Hi,
>> >
>> > I am facing some issues with geode memory/resource management.
>> > I have simple environment with a single locator and a single server,
>> > both are launched via gfsh.
>> > The server is started with initial-heap=max-heap=4GBand with
>> > critical-heap-percentage=70%.
>> > I also created a single region of type partition using gfsh.
>> >
>> > In addition, I created a simple client application (client-server
>> > topology, accessing the region via ClientCache PROXY). The client
>> > simply iteratively put data elements (of 100MB each) into the region.
>> > Upon putting total size of ~70% of 4GB, I get exception on client side
>> > which tells the server is working at low memory as expected – so far so
>> good.
>> > While observing the server metrics (using show metrics) I see that the
>> > server holds data entries as expected and the heap usage and total
>> > heap size are fine too.
>> >
>> > Now, I’m removing all elements from the region – the server metrics
>> > after this operation shows that the elements count is 0 (i.e. the
>> > server is empty) but still heap usage is high (probably because GC
>> > didn’t collect freed items). That’s shouldn’t bother me, but the
>> > problem is if I’m trying to put additional elements into the region now
>> I still get the exception on client.
>> > Although the server is empty the client can’t put items into it and
>> > this is very problematic from user point of view.
>> >
>> > I tried to play with GC flags, change GC to G1 but with no success. I
>> > can’t control GC collection – having an idle empty server with no way
>> > to add elements to it.
>> >
>> > What am I missing here? Is there some other configuration I should
>> follow?
>> >
>> > Thanks.
>> >
>> > Assaf Waizman
>> > SW Architect | Process Diagnostics and Control | Applied Materials
>> > 9 Oppenheimer Street, Rehovot, 76705.  Israel.
>> > Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax
>> > +972.8.948.8848
>> >
>> >
>>
>>
>>
> --
> ---
> cblack@pivotal.io | +1.858.480.9722 <(858)%20480-9722>
>
>

Re: Geode memory handling issues

[As...@amat.com]

Thanks Eric,

I mistakenly thought that offheap is overflow...
I'll test the offheap for my scenario, hope it will perform well on my data sizes and variance.
One more thing I would like to ask- I saw that the overflow feature can be configured per region, does it mean that it also checks internally the mem usage for the region (not by querying the heap). If so it might be a possible workaround for me too.

Once again, I really appreciate your help.

On 14 Dec 2016, at 0:05, Charlie Black <cb...@pivotal.io>> wrote:

It feels like Java hasn't hit the threshold when the GC would kick in.

From a Geode perspective if the entry count is zero all of the references to the objects stored in Geode should be removed and are waiting to be reclaimed by Java.

If you would like to force a GC try the gfsh command "gc".   GC will attempt to run the Java Garbage Collector.   Here is an example run:

gfsh>gc

GC Summary


        Member ID/Name          | HeapSize (MB) Before GC | HeapSize(MB) After GC | Time Taken for GC in ms

------------------------------- | ----------------------- | --------------------- | -----------------------

192.168.5.1(foo:23089)<v0>:9696 | 189                     | 142                   | 84

On Tue, Dec 13, 2016 at 2:40 PM <As...@amat.com>> wrote:

Indeed the eviction/offheap features or persistency can workaround this, I considered the offheap but the problem for my scenario is that upon reaching the threshold (which is tested by checking heap status) the offheap will take place in I will have throughput hit although I have free space in RAM (the server elements usage are less than threshold after some keys removal, but since offheap is based on querying the heap new puts will go offheap).



My system requirements can’t afford working with disk while having free RAM, it will impact it dramatically.



Please share if you have more ideas to overcome this.



Thanks a lot.



From: Eric Shu [mailto:eshu@pivotal.io<ma...@pivotal.io>]
Sent: יום ג 13 דצמבר 2016 23:27

To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode memory handling issues



Others might chime in as well. I think most users set eviction if they know there will be a memory issue. Eviction is kicked in before the critical heap setting (Low Memory Exception) with correct settings.



Also, if it is partitioned region, you can add capacity by adding more nodes so that system will be able to proceed. Users may also shut down nodes and restart them again -- with persistence enabled. The data will be recovered from disc.



Regards,

Eric



On Tue, Dec 13, 2016 at 1:16 PM, <As...@amat.com>> wrote:

Thanks Dan & Jared,

I carefully read the documentation for the resource management and noticed that part that tells the interaction between it and the GC.
I saw that my server is working with ConcMarkSweep and set the InitiatingOccupancyFraction to lower percentage but with no help, GC just won’t kick in. I also tried the G1 GC with now help. (also, just to make sure that is the cause, I attached to the jvm using yourkit and initiate GC manually and indeed the server was able to get put requests again)

For my experience with JVM memory management, counting on GC collection can be problematic as you can't really control the collection timing, in addition invoking it directly (System.gc) is stated as bad practice. I would expect such resource manager to track its elements usage directly and not by querying the heap status.

This mem usage issue seems to be very basic, I mean how come GemFire users (which I know runs in production) are not facing this problem?

I really appreciate your help guys.

-----Original Message-----
From: Dan Smith [mailto:dsmith@pivotal.io<ma...@pivotal.io>]
Sent: יום ג 13 דצמבר 2016 21:44
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode memory handling issues

Hi Assaf,

+1 for that link Jared send out. In order to the resource manager to
work, you need to be using ConcMarkSweep with an InitiatingOccupancyFraction that's less than your critical and eviction heap thresholds. That will cause GC to kick in if your heap is above those thresholds.

-Dan

On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io>> wrote:
> Hi Assaf,
>
> There is some information about tuning the JVM’s garbage collection
> parameters to work with eviction available here:
> http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> tml#resource_manager
>
> Best,
> Jared
>
> On Dec 13, 2016, at 11:26 AM, <As...@amat.com>>
> <As...@amat.com>> wrote:
>
> Hi Eric,
>
> Thanks for the quick response!
> Shutting down the server is problematic for me. In a real scenario I
> won’t remove all items but only some of them, thus the server still
> contains data but have free space – still I will get the exception.
> Shutting down the server will cause me to lose other entries – I’m
> afraid it’s not applicable to me.
>
> I also understand that I can’t really control GC collection, so how
> does the critical-threshold is expected to work? Once you reach it you
> can’t add more items to server even if you removed some.
>
> Thanks.
>
> From: Eric Shu [mailto:eshu@pivotal.io<ma...@pivotal.io>]
> Sent: יום ג 13 דצמבר 2016 21:21
> To: user@geode.apache.org<ma...@geode.apache.org>
> Subject: Re: Geode memory handling issues
>
> I am not sure if GC collection can be controlled. One possible way to
> work around this is to shut down the server and restart it.
>
> Also you may want to try offheap region to cope this issue?
>
> Regards,
> Eric
>
> On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com>> wrote:
> Hi,
>
> I am facing some issues with geode memory/resource management.
> I have simple environment with a single locator and a single server,
> both are launched via gfsh.
> The server is started with initial-heap=max-heap=4GBand with
> critical-heap-percentage=70%.
> I also created a single region of type partition using gfsh.
>
> In addition, I created a simple client application (client-server
> topology, accessing the region via ClientCache PROXY). The client
> simply iteratively put data elements (of 100MB each) into the region.
> Upon putting total size of ~70% of 4GB, I get exception on client side
> which tells the server is working at low memory as expected – so far so good.
> While observing the server metrics (using show metrics) I see that the
> server holds data entries as expected and the heap usage and total
> heap size are fine too.
>
> Now, I’m removing all elements from the region – the server metrics
> after this operation shows that the elements count is 0 (i.e. the
> server is empty) but still heap usage is high (probably because GC
> didn’t collect freed items). That’s shouldn’t bother me, but the
> problem is if I’m trying to put additional elements into the region now I still get the exception on client.
> Although the server is empty the client can’t put items into it and
> this is very problematic from user point of view.
>
> I tried to play with GC flags, change GC to G1 but with no success. I
> can’t control GC collection – having an idle empty server with no way
> to add elements to it.
>
> What am I missing here? Is there some other configuration I should follow?
>
> Thanks.
>
> Assaf Waizman
> SW Architect | Process Diagnostics and Control | Applied Materials
> 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> Office +972.8.948.8661<tel:%2B972.8.948.8661> | Mobile +972.54.80.10.799<tel:%2B972.54.80.10.799> | Fax
> +972.8.948.8848<tel:%2B972.8.948.8848>
>
>



--
---
cblack@pivotal.io<ma...@pivotal.io> | +1.858.480.9722

Re: Geode memory handling issues

[Charlie Black <cb...@pivotal.io>]

It feels like Java hasn't hit the threshold when the GC would kick in.

From a Geode perspective if the entry count is zero all of the references
to the objects stored in Geode should be removed and are waiting to be
reclaimed by Java.

If you would like to force a GC try the gfsh command "gc".   GC will
attempt to run the Java Garbage Collector.   Here is an example run:

gfsh>gc

GC Summary


        Member ID/Name          | HeapSize (MB) Before GC | HeapSize(MB)
After GC | Time Taken for GC in ms

------------------------------- | ----------------------- |
--------------------- | -----------------------

192.168.5.1(foo:23089)<v0>:9696 | 189                     | 142
      | 84

On Tue, Dec 13, 2016 at 2:40 PM <As...@amat.com> wrote:

> Indeed the eviction/offheap features or persistency can workaround this, I
> considered the offheap but the problem for my scenario is that upon
> reaching the threshold (which is tested by checking heap status) the
> offheap will take place in I will have throughput hit although I have free
> space in RAM (the server elements usage are less than threshold after some
> keys removal, but since offheap is based on querying the heap new puts will
> go offheap).
>
>
>
> My system requirements can’t afford working with disk while having free
> RAM, it will impact it dramatically.
>
>
>
> Please share if you have more ideas to overcome this.
>
>
>
> Thanks a lot.
>
>
>
> *From:* Eric Shu [mailto:eshu@pivotal.io]
> *Sent:* יום ג 13 דצמבר 2016 23:27
>
>
> *To:* user@geode.apache.org
> *Subject:* Re: Geode memory handling issues
>
>
>
> Others might chime in as well. I think most users set eviction if they
> know there will be a memory issue. Eviction is kicked in before the
> critical heap setting (Low Memory Exception) with correct settings.
>
>
>
> Also, if it is partitioned region, you can add capacity by adding more
> nodes so that system will be able to proceed. Users may also shut down
> nodes and restart them again -- with persistence enabled. The data will be
> recovered from disc.
>
>
>
> Regards,
>
> Eric
>
>
>
> On Tue, Dec 13, 2016 at 1:16 PM, <As...@amat.com> wrote:
>
> Thanks Dan & Jared,
>
> I carefully read the documentation for the resource management and noticed
> that part that tells the interaction between it and the GC.
> I saw that my server is working with ConcMarkSweep and set the
> InitiatingOccupancyFraction to lower percentage but with no help, GC just
> won’t kick in. I also tried the G1 GC with now help. (also, just to make
> sure that is the cause, I attached to the jvm using yourkit and initiate GC
> manually and indeed the server was able to get put requests again)
>
> For my experience with JVM memory management, counting on GC collection
> can be problematic as you can't really control the collection timing, in
> addition invoking it directly (System.gc) is stated as bad practice. I
> would expect such resource manager to track its elements usage directly and
> not by querying the heap status.
>
> This mem usage issue seems to be very basic, I mean how come GemFire users
> (which I know runs in production) are not facing this problem?
>
> I really appreciate your help guys.
>
> -----Original Message-----
> From: Dan Smith [mailto:dsmith@pivotal.io]
> Sent: יום ג 13 דצמבר 2016 21:44
> To: user@geode.apache.org
> Subject: Re: Geode memory handling issues
>
> Hi Assaf,
>
> +1 for that link Jared send out. In order to the resource manager to
> work, you need to be using ConcMarkSweep with an
> InitiatingOccupancyFraction that's less than your critical and eviction
> heap thresholds. That will cause GC to kick in if your heap is above those
> thresholds.
>
> -Dan
>
> On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io>
> wrote:
> > Hi Assaf,
> >
> > There is some information about tuning the JVM’s garbage collection
> > parameters to work with eviction available here:
> > http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> > tml#resource_manager
> >
> > Best,
> > Jared
> >
> > On Dec 13, 2016, at 11:26 AM, <As...@amat.com>
> > <As...@amat.com> wrote:
> >
> > Hi Eric,
> >
> > Thanks for the quick response!
> > Shutting down the server is problematic for me. In a real scenario I
> > won’t remove all items but only some of them, thus the server still
> > contains data but have free space – still I will get the exception.
> > Shutting down the server will cause me to lose other entries – I’m
> > afraid it’s not applicable to me.
> >
> > I also understand that I can’t really control GC collection, so how
> > does the critical-threshold is expected to work? Once you reach it you
> > can’t add more items to server even if you removed some.
> >
> > Thanks.
> >
> > From: Eric Shu [mailto:eshu@pivotal.io]
> > Sent: יום ג 13 דצמבר 2016 21:21
> > To: user@geode.apache.org
> > Subject: Re: Geode memory handling issues
> >
> > I am not sure if GC collection can be controlled. One possible way to
> > work around this is to shut down the server and restart it.
> >
> > Also you may want to try offheap region to cope this issue?
> >
> > Regards,
> > Eric
> >
> > On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:
> > Hi,
> >
> > I am facing some issues with geode memory/resource management.
> > I have simple environment with a single locator and a single server,
> > both are launched via gfsh.
> > The server is started with initial-heap=max-heap=4GBand with
> > critical-heap-percentage=70%.
> > I also created a single region of type partition using gfsh.
> >
> > In addition, I created a simple client application (client-server
> > topology, accessing the region via ClientCache PROXY). The client
> > simply iteratively put data elements (of 100MB each) into the region.
> > Upon putting total size of ~70% of 4GB, I get exception on client side
> > which tells the server is working at low memory as expected – so far so
> good.
> > While observing the server metrics (using show metrics) I see that the
> > server holds data entries as expected and the heap usage and total
> > heap size are fine too.
> >
> > Now, I’m removing all elements from the region – the server metrics
> > after this operation shows that the elements count is 0 (i.e. the
> > server is empty) but still heap usage is high (probably because GC
> > didn’t collect freed items). That’s shouldn’t bother me, but the
> > problem is if I’m trying to put additional elements into the region now
> I still get the exception on client.
> > Although the server is empty the client can’t put items into it and
> > this is very problematic from user point of view.
> >
> > I tried to play with GC flags, change GC to G1 but with no success. I
> > can’t control GC collection – having an idle empty server with no way
> > to add elements to it.
> >
> > What am I missing here? Is there some other configuration I should
> follow?
> >
> > Thanks.
> >
> > Assaf Waizman
> > SW Architect | Process Diagnostics and Control | Applied Materials
> > 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> > Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax
> > +972.8.948.8848
> >
> >
>
>
>
-- 
---
cblack@pivotal.io | +1.858.480.9722

Re: Geode memory handling issues

[Udo Kohlmeyer <uk...@pivotal.io>]

Hi there Assaf,

I'm not sure what your eviction-threshold is set to... but I usually set 
that to be around 70% of heap... but regardless of that setting, have 
you tried setting \u2011XX:CMSInitiatingOccupancyFraction=60 and 
-XX:+UseCMSInitiatingOccupancyOnly. This should force the GC to kick in 
when a certain amount of garbage is in the heap.

--Udo

On 12/13/16 13:38, Assaf_Waizman@amat.com wrote:
>
> Indeed the eviction/offheap features or persistency can workaround 
> this, I considered the offheap but the problem for my scenario is that 
> upon reaching the threshold (which is tested by checking heap status) 
> the offheap will take place in I will have throughput hit although I 
> have free space in RAM (the server elements usage are less than 
> threshold after some keys removal, but since offheap is based on 
> querying the heap new puts will go offheap).
>
> My system requirements can\u2019t afford working with disk while having 
> free RAM, it will impact it dramatically.
>
> Please share if you have more ideas to overcome this.
>
> Thanks a lot.
>
> *From:*Eric Shu [mailto:eshu@pivotal.io]
> *Sent:* \u05d9\u05d5\u05dd \u05d2 13 \u05d3\u05e6\u05de\u05d1\u05e8 2016 23:27
> *To:* user@geode.apache.org
> *Subject:* Re: Geode memory handling issues
>
> Others might chime in as well. I think most users set eviction if they 
> know there will be a memory issue. Eviction is kicked in before the 
> critical heap setting (Low Memory Exception) with correct settings.
>
> Also, if it is partitioned region, you can add capacity by adding more 
> nodes so that system will be able to proceed. Users may also shut down 
> nodes and restart them again -- with persistence enabled. The data 
> will be recovered from disc.
>
> Regards,
>
> Eric
>
> On Tue, Dec 13, 2016 at 1:16 PM, <Assaf_Waizman@amat.com 
> <ma...@amat.com>> wrote:
>
> Thanks Dan & Jared,
>
> I carefully read the documentation for the resource management and 
> noticed that part that tells the interaction between it and the GC.
> I saw that my server is working with ConcMarkSweep and set the 
> InitiatingOccupancyFraction to lower percentage but with no help, GC 
> just won\u2019t kick in. I also tried the G1 GC with now help. (also, just 
> to make sure that is the cause, I attached to the jvm using yourkit 
> and initiate GC manually and indeed the server was able to get put 
> requests again)
>
> For my experience with JVM memory management, counting on GC 
> collection can be problematic as you can't really control the 
> collection timing, in addition invoking it directly (System.gc) is 
> stated as bad practice. I would expect such resource manager to track 
> its elements usage directly and not by querying the heap status.
>
> This mem usage issue seems to be very basic, I mean how come GemFire 
> users (which I know runs in production) are not facing this problem?
>
> I really appreciate your help guys.
>
> -----Original Message-----
> From: Dan Smith [mailto:dsmith@pivotal.io <ma...@pivotal.io>]
> Sent: \u05d9\u05d5\u05dd\u05d2 13 \u05d3\u05e6\u05de\u05d1\u05e8 2016 21:44
> To: user@geode.apache.org <ma...@geode.apache.org>
> Subject: Re: Geode memory handling issues
>
> Hi Assaf,
>
> +1 for that link Jared send out. In order to the resource manager to
> work, you need to be using ConcMarkSweep with an 
> InitiatingOccupancyFraction that's less than your critical and 
> eviction heap thresholds. That will cause GC to kick in if your heap 
> is above those thresholds.
>
> -Dan
>
> On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <jstewart@pivotal.io 
> <ma...@pivotal.io>> wrote:
> > Hi Assaf,
> >
> > There is some information about tuning the JVM\u2019s garbage collection
> > parameters to work with eviction available here:
> > http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> > tml#resource_manager
> >
> > Best,
> > Jared
> >
> > On Dec 13, 2016, at 11:26 AM, <Assaf_Waizman@amat.com 
> <ma...@amat.com>>
> > <Assaf_Waizman@amat.com <ma...@amat.com>> wrote:
> >
> > Hi Eric,
> >
> > Thanks for the quick response!
> > Shutting down the server is problematic for me. In a real scenario I
> > won\u2019t remove all items but only some of them, thus the server still
> > contains data but have free space \u2013 still I will get the exception.
> > Shutting down the server will cause me to lose other entries \u2013 I\u2019m
> > afraid it\u2019s not applicable to me.
> >
> > I also understand that I can\u2019t really control GC collection, so how
> > does the critical-threshold is expected to work? Once you reach it you
> > can\u2019t add more items to server even if you removed some.
> >
> > Thanks.
> >
> > From: Eric Shu [mailto:eshu@pivotal.io <ma...@pivotal.io>]
> > Sent: \u05d9\u05d5\u05dd \u05d2 13 \u05d3\u05e6\u05de\u05d1\u05e8 2016 21:21
> > To: user@geode.apache.org <ma...@geode.apache.org>
> > Subject: Re: Geode memory handling issues
> >
> > I am not sure if GC collection can be controlled. One possible way to
> > work around this is to shut down the server and restart it.
> >
> > Also you may want to try offheap region to cope this issue?
> >
> > Regards,
> > Eric
> >
> > On Tue, Dec 13, 2016 at 11:02 AM, <Assaf_Waizman@amat.com 
> <ma...@amat.com>> wrote:
> > Hi,
> >
> > I am facing some issues with geode memory/resource management.
> > I have simple environment with a single locator and a single server,
> > both are launched via gfsh.
> > The server is started with initial-heap=max-heap=4GBand with
> > critical-heap-percentage=70%.
> > I also created a single region of type partition using gfsh.
> >
> > In addition, I created a simple client application (client-server
> > topology, accessing the region via ClientCache PROXY). The client
> > simply iteratively put data elements (of 100MB each) into the region.
> > Upon putting total size of ~70% of 4GB, I get exception on client side
> > which tells the server is working at low memory as expected \u2013 so far 
> so good.
> > While observing the server metrics (using show metrics) I see that the
> > server holds data entries as expected and the heap usage and total
> > heap size are fine too.
> >
> > Now, I\u2019m removing all elements from the region \u2013 the server metrics
> > after this operation shows that the elements count is 0 (i.e. the
> > server is empty) but still heap usage is high (probably because GC
> > didn\u2019t collect freed items). That\u2019s shouldn\u2019t bother me, but the
> > problem is if I\u2019m trying to put additional elements into the region 
> now I still get the exception on client.
> > Although the server is empty the client can\u2019t put items into it and
> > this is very problematic from user point of view.
> >
> > I tried to play with GC flags, change GC to G1 but with no success. I
> > can\u2019t control GC collection \u2013 having an idle empty server with no way
> > to add elements to it.
> >
> > What am I missing here? Is there some other configuration I should 
> follow?
> >
> > Thanks.
> >
> > Assaf Waizman
> > SW Architect | Process Diagnostics and Control | Applied Materials
> > 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> > Office +972.8.948.8661 <tel:%2B972.8.948.8661> | Mobile 
> +972.54.80.10.799 <tel:%2B972.54.80.10.799> | Fax
> > +972.8.948.8848 <tel:%2B972.8.948.8848>
> >
> >
>

RE: Geode memory handling issues

[As...@amat.com]

Indeed the eviction/offheap features or persistency can workaround this, I considered the offheap but the problem for my scenario is that upon reaching the threshold (which is tested by checking heap status) the offheap will take place in I will have throughput hit although I have free space in RAM (the server elements usage are less than threshold after some keys removal, but since offheap is based on querying the heap new puts will go offheap).

My system requirements can’t afford working with disk while having free RAM, it will impact it dramatically.

Please share if you have more ideas to overcome this.

Thanks a lot.

From: Eric Shu [mailto:eshu@pivotal.io]
Sent: יום ג 13 דצמבר 2016 23:27
To: user@geode.apache.org
Subject: Re: Geode memory handling issues

Others might chime in as well. I think most users set eviction if they know there will be a memory issue. Eviction is kicked in before the critical heap setting (Low Memory Exception) with correct settings.

Also, if it is partitioned region, you can add capacity by adding more nodes so that system will be able to proceed. Users may also shut down nodes and restart them again -- with persistence enabled. The data will be recovered from disc.

Regards,
Eric

On Tue, Dec 13, 2016 at 1:16 PM, <As...@amat.com>> wrote:
Thanks Dan & Jared,

I carefully read the documentation for the resource management and noticed that part that tells the interaction between it and the GC.
I saw that my server is working with ConcMarkSweep and set the InitiatingOccupancyFraction to lower percentage but with no help, GC just won’t kick in. I also tried the G1 GC with now help. (also, just to make sure that is the cause, I attached to the jvm using yourkit and initiate GC manually and indeed the server was able to get put requests again)

For my experience with JVM memory management, counting on GC collection can be problematic as you can't really control the collection timing, in addition invoking it directly (System.gc) is stated as bad practice. I would expect such resource manager to track its elements usage directly and not by querying the heap status.

This mem usage issue seems to be very basic, I mean how come GemFire users (which I know runs in production) are not facing this problem?

I really appreciate your help guys.

-----Original Message-----
From: Dan Smith [mailto:dsmith@pivotal.io<ma...@pivotal.io>]
Sent: יום ג 13 דצמבר 2016 21:44
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode memory handling issues
Hi Assaf,

+1 for that link Jared send out. In order to the resource manager to
work, you need to be using ConcMarkSweep with an InitiatingOccupancyFraction that's less than your critical and eviction heap thresholds. That will cause GC to kick in if your heap is above those thresholds.

-Dan

On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io>> wrote:
> Hi Assaf,
>
> There is some information about tuning the JVM’s garbage collection
> parameters to work with eviction available here:
> http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> tml#resource_manager
>
> Best,
> Jared
>
> On Dec 13, 2016, at 11:26 AM, <As...@amat.com>>
> <As...@amat.com>> wrote:
>
> Hi Eric,
>
> Thanks for the quick response!
> Shutting down the server is problematic for me. In a real scenario I
> won’t remove all items but only some of them, thus the server still
> contains data but have free space – still I will get the exception.
> Shutting down the server will cause me to lose other entries – I’m
> afraid it’s not applicable to me.
>
> I also understand that I can’t really control GC collection, so how
> does the critical-threshold is expected to work? Once you reach it you
> can’t add more items to server even if you removed some.
>
> Thanks.
>
> From: Eric Shu [mailto:eshu@pivotal.io<ma...@pivotal.io>]
> Sent: יום ג 13 דצמבר 2016 21:21
> To: user@geode.apache.org<ma...@geode.apache.org>
> Subject: Re: Geode memory handling issues
>
> I am not sure if GC collection can be controlled. One possible way to
> work around this is to shut down the server and restart it.
>
> Also you may want to try offheap region to cope this issue?
>
> Regards,
> Eric
>
> On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com>> wrote:
> Hi,
>
> I am facing some issues with geode memory/resource management.
> I have simple environment with a single locator and a single server,
> both are launched via gfsh.
> The server is started with initial-heap=max-heap=4GBand with
> critical-heap-percentage=70%.
> I also created a single region of type partition using gfsh.
>
> In addition, I created a simple client application (client-server
> topology, accessing the region via ClientCache PROXY). The client
> simply iteratively put data elements (of 100MB each) into the region.
> Upon putting total size of ~70% of 4GB, I get exception on client side
> which tells the server is working at low memory as expected – so far so good.
> While observing the server metrics (using show metrics) I see that the
> server holds data entries as expected and the heap usage and total
> heap size are fine too.
>
> Now, I’m removing all elements from the region – the server metrics
> after this operation shows that the elements count is 0 (i.e. the
> server is empty) but still heap usage is high (probably because GC
> didn’t collect freed items). That’s shouldn’t bother me, but the
> problem is if I’m trying to put additional elements into the region now I still get the exception on client.
> Although the server is empty the client can’t put items into it and
> this is very problematic from user point of view.
>
> I tried to play with GC flags, change GC to G1 but with no success. I
> can’t control GC collection – having an idle empty server with no way
> to add elements to it.
>
> What am I missing here? Is there some other configuration I should follow?
>
> Thanks.
>
> Assaf Waizman
> SW Architect | Process Diagnostics and Control | Applied Materials
> 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> Office +972.8.948.8661<tel:%2B972.8.948.8661> | Mobile +972.54.80.10.799<tel:%2B972.54.80.10.799> | Fax
> +972.8.948.8848<tel:%2B972.8.948.8848>
>
>

Re: Geode memory handling issues

[Eric Shu <es...@pivotal.io>]

Others might chime in as well. I think most users set eviction if they know
there will be a memory issue. Eviction is kicked in before the critical
heap setting (Low Memory Exception) with correct settings.

Also, if it is partitioned region, you can add capacity by adding more
nodes so that system will be able to proceed. Users may also shut down
nodes and restart them again -- with persistence enabled. The data will be
recovered from disc.

Regards,
Eric

On Tue, Dec 13, 2016 at 1:16 PM, <As...@amat.com> wrote:

> Thanks Dan & Jared,
>
> I carefully read the documentation for the resource management and noticed
> that part that tells the interaction between it and the GC.
> I saw that my server is working with ConcMarkSweep and set the
> InitiatingOccupancyFraction to lower percentage but with no help, GC just
> won’t kick in. I also tried the G1 GC with now help. (also, just to make
> sure that is the cause, I attached to the jvm using yourkit and initiate GC
> manually and indeed the server was able to get put requests again)
>
> For my experience with JVM memory management, counting on GC collection
> can be problematic as you can't really control the collection timing, in
> addition invoking it directly (System.gc) is stated as bad practice. I
> would expect such resource manager to track its elements usage directly and
> not by querying the heap status.
>
> This mem usage issue seems to be very basic, I mean how come GemFire users
> (which I know runs in production) are not facing this problem?
>
> I really appreciate your help guys.
>
> -----Original Message-----
> From: Dan Smith [mailto:dsmith@pivotal.io]
> Sent: יום ג 13 דצמבר 2016 21:44
> To: user@geode.apache.org
> Subject: Re: Geode memory handling issues
>
> Hi Assaf,
>
> +1 for that link Jared send out. In order to the resource manager to
> work, you need to be using ConcMarkSweep with an
> InitiatingOccupancyFraction that's less than your critical and eviction
> heap thresholds. That will cause GC to kick in if your heap is above those
> thresholds.
>
> -Dan
>
> On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io>
> wrote:
> > Hi Assaf,
> >
> > There is some information about tuning the JVM’s garbage collection
> > parameters to work with eviction available here:
> > http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> > tml#resource_manager
> >
> > Best,
> > Jared
> >
> > On Dec 13, 2016, at 11:26 AM, <As...@amat.com>
> > <As...@amat.com> wrote:
> >
> > Hi Eric,
> >
> > Thanks for the quick response!
> > Shutting down the server is problematic for me. In a real scenario I
> > won’t remove all items but only some of them, thus the server still
> > contains data but have free space – still I will get the exception.
> > Shutting down the server will cause me to lose other entries – I’m
> > afraid it’s not applicable to me.
> >
> > I also understand that I can’t really control GC collection, so how
> > does the critical-threshold is expected to work? Once you reach it you
> > can’t add more items to server even if you removed some.
> >
> > Thanks.
> >
> > From: Eric Shu [mailto:eshu@pivotal.io]
> > Sent: יום ג 13 דצמבר 2016 21:21
> > To: user@geode.apache.org
> > Subject: Re: Geode memory handling issues
> >
> > I am not sure if GC collection can be controlled. One possible way to
> > work around this is to shut down the server and restart it.
> >
> > Also you may want to try offheap region to cope this issue?
> >
> > Regards,
> > Eric
> >
> > On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:
> > Hi,
> >
> > I am facing some issues with geode memory/resource management.
> > I have simple environment with a single locator and a single server,
> > both are launched via gfsh.
> > The server is started with initial-heap=max-heap=4GBand with
> > critical-heap-percentage=70%.
> > I also created a single region of type partition using gfsh.
> >
> > In addition, I created a simple client application (client-server
> > topology, accessing the region via ClientCache PROXY). The client
> > simply iteratively put data elements (of 100MB each) into the region.
> > Upon putting total size of ~70% of 4GB, I get exception on client side
> > which tells the server is working at low memory as expected – so far so
> good.
> > While observing the server metrics (using show metrics) I see that the
> > server holds data entries as expected and the heap usage and total
> > heap size are fine too.
> >
> > Now, I’m removing all elements from the region – the server metrics
> > after this operation shows that the elements count is 0 (i.e. the
> > server is empty) but still heap usage is high (probably because GC
> > didn’t collect freed items). That’s shouldn’t bother me, but the
> > problem is if I’m trying to put additional elements into the region now
> I still get the exception on client.
> > Although the server is empty the client can’t put items into it and
> > this is very problematic from user point of view.
> >
> > I tried to play with GC flags, change GC to G1 but with no success. I
> > can’t control GC collection – having an idle empty server with no way
> > to add elements to it.
> >
> > What am I missing here? Is there some other configuration I should
> follow?
> >
> > Thanks.
> >
> > Assaf Waizman
> > SW Architect | Process Diagnostics and Control | Applied Materials
> > 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> > Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax
> > +972.8.948.8848
> >
> >
>

RE: Geode memory handling issues

[As...@amat.com]

Thanks Dan & Jared,

I carefully read the documentation for the resource management and noticed that part that tells the interaction between it and the GC.
I saw that my server is working with ConcMarkSweep and set the InitiatingOccupancyFraction to lower percentage but with no help, GC just won’t kick in. I also tried the G1 GC with now help. (also, just to make sure that is the cause, I attached to the jvm using yourkit and initiate GC manually and indeed the server was able to get put requests again)

For my experience with JVM memory management, counting on GC collection can be problematic as you can't really control the collection timing, in addition invoking it directly (System.gc) is stated as bad practice. I would expect such resource manager to track its elements usage directly and not by querying the heap status.

This mem usage issue seems to be very basic, I mean how come GemFire users (which I know runs in production) are not facing this problem?

I really appreciate your help guys.

-----Original Message-----
From: Dan Smith [mailto:dsmith@pivotal.io] 
Sent: יום ג 13 דצמבר 2016 21:44
To: user@geode.apache.org
Subject: Re: Geode memory handling issues

Hi Assaf,

+1 for that link Jared send out. In order to the resource manager to
work, you need to be using ConcMarkSweep with an InitiatingOccupancyFraction that's less than your critical and eviction heap thresholds. That will cause GC to kick in if your heap is above those thresholds.

-Dan

On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io> wrote:
> Hi Assaf,
>
> There is some information about tuning the JVM’s garbage collection 
> parameters to work with eviction available here:
> http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> tml#resource_manager
>
> Best,
> Jared
>
> On Dec 13, 2016, at 11:26 AM, <As...@amat.com> 
> <As...@amat.com> wrote:
>
> Hi Eric,
>
> Thanks for the quick response!
> Shutting down the server is problematic for me. In a real scenario I 
> won’t remove all items but only some of them, thus the server still 
> contains data but have free space – still I will get the exception. 
> Shutting down the server will cause me to lose other entries – I’m 
> afraid it’s not applicable to me.
>
> I also understand that I can’t really control GC collection, so how 
> does the critical-threshold is expected to work? Once you reach it you 
> can’t add more items to server even if you removed some.
>
> Thanks.
>
> From: Eric Shu [mailto:eshu@pivotal.io]
> Sent: יום ג 13 דצמבר 2016 21:21
> To: user@geode.apache.org
> Subject: Re: Geode memory handling issues
>
> I am not sure if GC collection can be controlled. One possible way to 
> work around this is to shut down the server and restart it.
>
> Also you may want to try offheap region to cope this issue?
>
> Regards,
> Eric
>
> On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:
> Hi,
>
> I am facing some issues with geode memory/resource management.
> I have simple environment with a single locator and a single server, 
> both are launched via gfsh.
> The server is started with initial-heap=max-heap=4GBand with 
> critical-heap-percentage=70%.
> I also created a single region of type partition using gfsh.
>
> In addition, I created a simple client application (client-server 
> topology, accessing the region via ClientCache PROXY). The client 
> simply iteratively put data elements (of 100MB each) into the region. 
> Upon putting total size of ~70% of 4GB, I get exception on client side 
> which tells the server is working at low memory as expected – so far so good.
> While observing the server metrics (using show metrics) I see that the 
> server holds data entries as expected and the heap usage and total 
> heap size are fine too.
>
> Now, I’m removing all elements from the region – the server metrics 
> after this operation shows that the elements count is 0 (i.e. the 
> server is empty) but still heap usage is high (probably because GC 
> didn’t collect freed items). That’s shouldn’t bother me, but the 
> problem is if I’m trying to put additional elements into the region now I still get the exception on client.
> Although the server is empty the client can’t put items into it and 
> this is very problematic from user point of view.
>
> I tried to play with GC flags, change GC to G1 but with no success. I 
> can’t control GC collection – having an idle empty server with no way 
> to add elements to it.
>
> What am I missing here? Is there some other configuration I should follow?
>
> Thanks.
>
> Assaf Waizman
> SW Architect | Process Diagnostics and Control | Applied Materials
> 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax 
> +972.8.948.8848
>
>

Re: Geode memory handling issues

[Dan Smith <ds...@pivotal.io>]

Hi Assaf,

+1 for that link Jared send out. In order to the resource manager to
work, you need to be using ConcMarkSweep with an
InitiatingOccupancyFraction that's less than your critical and
eviction heap thresholds. That will cause GC to kick in if your heap
is above those thresholds.

-Dan

On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart <js...@pivotal.io> wrote:
> Hi Assaf,
>
> There is some information about tuning the JVM’s garbage collection
> parameters to work with eviction available here:
> http://geode.apache.org/docs/guide/managing/heap_use/heap_management.html#resource_manager
>
> Best,
> Jared
>
> On Dec 13, 2016, at 11:26 AM, <As...@amat.com>
> <As...@amat.com> wrote:
>
> Hi Eric,
>
> Thanks for the quick response!
> Shutting down the server is problematic for me. In a real scenario I won’t
> remove all items but only some of them, thus the server still contains data
> but have free space – still I will get the exception. Shutting down the
> server will cause me to lose other entries – I’m afraid it’s not applicable
> to me.
>
> I also understand that I can’t really control GC collection, so how does the
> critical-threshold is expected to work? Once you reach it you can’t add more
> items to server even if you removed some.
>
> Thanks.
>
> From: Eric Shu [mailto:eshu@pivotal.io]
> Sent: יום ג 13 דצמבר 2016 21:21
> To: user@geode.apache.org
> Subject: Re: Geode memory handling issues
>
> I am not sure if GC collection can be controlled. One possible way to work
> around this is to shut down the server and restart it.
>
> Also you may want to try offheap region to cope this issue?
>
> Regards,
> Eric
>
> On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:
> Hi,
>
> I am facing some issues with geode memory/resource management.
> I have simple environment with a single locator and a single server, both
> are launched via gfsh.
> The server is started with initial-heap=max-heap=4GBand with
> critical-heap-percentage=70%.
> I also created a single region of type partition using gfsh.
>
> In addition, I created a simple client application (client-server topology,
> accessing the region via ClientCache PROXY). The client simply iteratively
> put data elements (of 100MB each) into the region. Upon putting total size
> of ~70% of 4GB, I get exception on client side which tells the server is
> working at low memory as expected – so far so good.
> While observing the server metrics (using show metrics) I see that the
> server holds data entries as expected and the heap usage and total heap size
> are fine too.
>
> Now, I’m removing all elements from the region – the server metrics after
> this operation shows that the elements count is 0 (i.e. the server is empty)
> but still heap usage is high (probably because GC didn’t collect freed
> items). That’s shouldn’t bother me, but the problem is if I’m trying to put
> additional elements into the region now I still get the exception on client.
> Although the server is empty the client can’t put items into it and this is
> very problematic from user point of view.
>
> I tried to play with GC flags, change GC to G1 but with no success. I can’t
> control GC collection – having an idle empty server with no way to add
> elements to it.
>
> What am I missing here? Is there some other configuration I should follow?
>
> Thanks.
>
> Assaf Waizman
> SW Architect | Process Diagnostics and Control | Applied Materials
> 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax +972.8.948.8848
>
>

Re: Geode memory handling issues

[Jared Stewart <js...@pivotal.io>]

Hi Assaf,

There is some information about tuning the JVM’s garbage collection parameters to work with eviction available here: http://geode.apache.org/docs/guide/managing/heap_use/heap_management.html#resource_manager <http://geode.apache.org/docs/guide/managing/heap_use/heap_management.html#resource_manager>

Best,
Jared

> On Dec 13, 2016, at 11:26 AM, <As...@amat.com> <As...@amat.com> wrote:
> 
> Hi Eric,
>  
> Thanks for the quick response!
> Shutting down the server is problematic for me. In a real scenario I won’t remove all items but only some of them, thus the server still contains data but have free space – still I will get the exception. Shutting down the server will cause me to lose other entries – I’m afraid it’s not applicable to me.
>  
> I also understand that I can’t really control GC collection, so how does the critical-threshold is expected to work? Once you reach it you can’t add more items to server even if you removed some.
>  
> Thanks.
>  
> From: Eric Shu [mailto:eshu@pivotal.io] 
> Sent: יום ג 13 דצמבר 2016 21:21
> To: user@geode.apache.org
> Subject: Re: Geode memory handling issues
>  
> I am not sure if GC collection can be controlled. One possible way to work around this is to shut down the server and restart it.
>  
> Also you may want to try offheap region to cope this issue?
>  
> Regards,
> Eric
>  
> On Tue, Dec 13, 2016 at 11:02 AM, <Assaf_Waizman@amat.com <ma...@amat.com>> wrote:
> Hi,
>  
> I am facing some issues with geode memory/resource management.
> I have simple environment with a single locator and a single server, both are launched via gfsh.
> The server is started with initial-heap=max-heap=4GBand with critical-heap-percentage=70%.
> I also created a single region of type partition using gfsh.
>  
> In addition, I created a simple client application (client-server topology, accessing the region via ClientCache PROXY). The client simply iteratively put data elements (of 100MB each) into the region. Upon putting total size of ~70% of 4GB, I get exception on client side which tells the server is working at low memory as expected – so far so good.
> While observing the server metrics (using show metrics) I see that the server holds data entries as expected and the heap usage and total heap size are fine too.
>  
> Now, I’m removing all elements from the region – the server metrics after this operation shows that the elements count is 0 (i.e. the server is empty) but still heap usage is high (probably because GC didn’t collect freed items). That’s shouldn’t bother me, but the problem is if I’m trying to put additional elements into the region now I still get the exception on client.
> Although the server is empty the client can’t put items into it and this is very problematic from user point of view.
>  
> I tried to play with GC flags, change GC to G1 but with no success. I can’t control GC collection – having an idle empty server with no way to add elements to it.
>  
> What am I missing here? Is there some other configuration I should follow?
>  
> Thanks.
>  
> Assaf Waizman
> SW Architect | Process Diagnostics and Control | Applied Materials 
> 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> Office +972.8.948.8661 <tel:%2B972.8.948.8661> | Mobile +972.54.80.10.799 | Fax +972.8.948.8848 <tel:%2B972.8.948.8848>

RE: Geode memory handling issues

[As...@amat.com]

Hi Eric,

Thanks for the quick response!
Shutting down the server is problematic for me. In a real scenario I won’t remove all items but only some of them, thus the server still contains data but have free space – still I will get the exception. Shutting down the server will cause me to lose other entries – I’m afraid it’s not applicable to me.

I also understand that I can’t really control GC collection, so how does the critical-threshold is expected to work? Once you reach it you can’t add more items to server even if you removed some.

Thanks.

From: Eric Shu [mailto:eshu@pivotal.io]
Sent: יום ג 13 דצמבר 2016 21:21
To: user@geode.apache.org
Subject: Re: Geode memory handling issues

I am not sure if GC collection can be controlled. One possible way to work around this is to shut down the server and restart it.

Also you may want to try offheap region to cope this issue?

Regards,
Eric

On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com>> wrote:
Hi,

I am facing some issues with geode memory/resource management.
I have simple environment with a single locator and a single server, both are launched via gfsh.
The server is started with initial-heap=max-heap=4GBand with critical-heap-percentage=70%.
I also created a single region of type partition using gfsh.

In addition, I created a simple client application (client-server topology, accessing the region via ClientCache PROXY). The client simply iteratively put data elements (of 100MB each) into the region. Upon putting total size of ~70% of 4GB, I get exception on client side which tells the server is working at low memory as expected – so far so good.
While observing the server metrics (using show metrics) I see that the server holds data entries as expected and the heap usage and total heap size are fine too.

Now, I’m removing all elements from the region – the server metrics after this operation shows that the elements count is 0 (i.e. the server is empty) but still heap usage is high (probably because GC didn’t collect freed items). That’s shouldn’t bother me, but the problem is if I’m trying to put additional elements into the region now I still get the exception on client.
Although the server is empty the client can’t put items into it and this is very problematic from user point of view.

I tried to play with GC flags, change GC to G1 but with no success. I can’t control GC collection – having an idle empty server with no way to add elements to it.

What am I missing here? Is there some other configuration I should follow?

Thanks.

Assaf Waizman
SW Architect | Process Diagnostics and Control | Applied Materials
9 Oppenheimer Street, Rehovot, 76705.  Israel.
Office +972.8.948.8661<tel:%2B972.8.948.8661> | Mobile +972.54.80.10.799 | Fax +972.8.948.8848<tel:%2B972.8.948.8848>

Re: Geode memory handling issues

[Eric Shu <es...@pivotal.io>]

I am not sure if GC collection can be controlled. One possible way to work
around this is to shut down the server and restart it.

Also you may want to try offheap region to cope this issue?

Regards,
Eric

On Tue, Dec 13, 2016 at 11:02 AM, <As...@amat.com> wrote:

> Hi,
>
>
>
> I am facing some issues with geode memory/resource management.
>
> I have simple environment with a single locator and a single server, both
> are launched via gfsh.
>
> The server is started with initial-heap=max-heap=4GBand with
> critical-heap-percentage=70%.
>
> I also created a single region of type partition using gfsh.
>
>
>
> In addition, I created a simple client application (client-server
> topology, accessing the region via ClientCache PROXY). The client simply
> iteratively put data elements (of 100MB each) into the region. Upon putting
> total size of ~70% of 4GB, I get exception on client side which tells the
> server is working at low memory as expected – so far so good.
>
> While observing the server metrics (using show metrics) I see that the
> server holds data entries as expected and the heap usage and total heap
> size are fine too.
>
>
>
> Now, I’m removing all elements from the region – the server metrics after
> this operation shows that the elements count is 0 (i.e. the server is
> empty) but still heap usage is high (probably because GC didn’t collect
> freed items). That’s shouldn’t bother me, but the problem is if I’m trying
> to put additional elements into the region now I still get the exception on
> client.
>
> Although the server is empty the client can’t put items into it and this
> is very problematic from user point of view.
>
>
>
> I tried to play with GC flags, change GC to G1 but with no success. I
> can’t control GC collection – having an idle empty server with no way to
> add elements to it.
>
>
>
> What am I missing here? Is there some other configuration I should follow?
>
>
>
> Thanks.
>
>
>
> Assaf Waizman
>
> SW Architect | Process Diagnostics and Control | Applied Materials
> 9 Oppenheimer Street, Rehovot, 76705.  Israel.
> Office +972.8.948.8661 | Mobile +972.54.80.10.799 | Fax +972.8.948.8848
>