You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by "Catanzarite, P. (Paul)" <Pa...@voya.com> on 2015/12/02 18:51:55 UTC
Locator high cpu
We have a gemfire client/server cluster that we've noticed our locator is running a sustained 50 - 100% cpu utilization. We stopped all work in this environment (test) and the cpu has remained the same. We are running gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look at to see what may be causing this?
Thanks.
-Paul
---------------------------------------------------------
NOTICE: The information contained in this electronic mail message is confidential and intended only for certain recipients. If you are not an intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication and any attachments is strictly prohibited. If you have received this communication in error, please notify the sender by reply transmission and delete the message without copying or disclosing it.
============================================================================================
Re: Locator high cpu
Posted by Jens Deppe <jd...@pivotal.io>.
Sounds like it might be a GC issue. You can run 'jstat -gc <PID> 1s' to get
a view of the memory. OC is 'Old Gen Configured' (your max heap) and 'OU'
is Old Gen currently in use. If OU is close to OC, over time, and the FGC
column (Full GC count) increasing steadily then the CPU is definitely being
spent on GC.
You'll want to do a heap dump to see what's holding the memory - 'jmap
-histo:live <PID>' to get a nice histogram of the current heap or 'jmap
-dump:live,format=b,file=<dump-file-name>' to dump the actual heap for
actual object inspection.
--Jens
On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <
Paul.Catanzarite@voya.com> wrote:
> We have a gemfire client/server cluster that we’ve noticed our locator is
> running a sustained 50 – 100% cpu utilization. We stopped all work in this
> environment (test) and the cpu has remained the same. We are running
> gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look
> at to see what may be causing this?
>
> Thanks.
>
> -Paul
>
>
> ---------------------------------------------------------
>
> NOTICE: The information contained in this electronic mail message is
> confidential and intended only for certain recipients. If you are not an
> intended recipient, you are hereby notified that any disclosure,
> reproduction, distribution or other use of this communication and any
> attachments is strictly prohibited. If you have received this
> communication in error, please notify the sender by reply transmission and
> delete the message without copying or disclosing it.
>
>
> ============================================================================================
>
RE: Locator high cpu
Posted by "Catanzarite, P. (Paul)" <Pa...@voya.com>.
# free -m
total used free shared buffers cached
Mem: 3825 3424 400 0 268 345
-/+ buffers/cache: 2811 1013
Swap: 8191 12 8179
I will send along thread dump output as soon as I can get it.
Thanks again for everyone’s help.
From: Barry Oglesby [mailto:boglesby@pivotal.io]
Sent: Wednesday, December 02, 2015 02:18 PM
To: user@geode.incubator.apache.org
Subject: Re: Locator high cpu
You're pretty near the limit of this box. Based on the top command output you included, there is 431944k free. I don't think that includes the buffers and cached though so it looks like you have ~1GB free, but you don't want to use all the memory. I actually don't think I would increase the memory yet (or maybe even at all). It might be better to see what is using the memory to verify there isn't a leak.
I like using free -m. The example below tells me there is 44307m free on my box (from the -/+ buffers/cache line).
free -m
total used free shared buffers cached
Mem: 48292 24423 23868 2408 759 19679
-/+ buffers/cache: 3984 44307
Swap: 15358 0 15358
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:24 AM, Catanzarite, P. (Paul) <Pa...@voya.com>> wrote:
Thanks for the feedback.
On a server with 4gb memory, what is a safe max heap size to set for the locator?
From: Barry Oglesby [mailto:boglesby@pivotal.io<ma...@pivotal.io>]
Sent: Wednesday, December 02, 2015 01:20 PM
To: user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>
Subject: Re: Locator high cpu
Sorry, I sent that before I meant to.
It looks like GC issues. You'll probably need to increase your max heap when you start the locator.
| currentHeapSize | 1905
| maximumHeapSize | 2031
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:19 AM, Barry Oglesby <bo...@pivotal.io>> wrote:
As Jens said, it looks like GC issues.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:13 AM, Catanzarite, P. (Paul) <Pa...@voya.com>> wrote:
It will take me some time to get the thread dump but here is some additional info.
Server info:
VM with 2 CPUs and 4gb memory and only this one locator is running on this server
=> uptime
12:57:17 up 11 days, 3:10, 1 user, load average: 0.34, 0.46, 0.46
=> top
top - 12:57:46 up 11 days, 3:10, 1 user, load average: 0.53, 0.50, 0.47
Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.2%us, 0.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3916900k total, 3484956k used, 431944k free, 266008k buffers
Swap: 8388604k total, 26100k used, 8362504k free, 350836k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4952 gemfire 20 0 3680m 2.3g 8192 S 105.7 61.8 3529:13 java
Pulse shows locator heap using 1.9gb (out of 2gb configured)
Member Metrics
Category | Metric | Value
------------- | -------------------------------- | ------
member | upTime | 962112
| cpuUsage | 60
| currentHeapSize | 1905
| maximumHeapSize | 2031
jvm | jvmThreads | 92
| fileDescriptorLimit | 32768
| totalFileDescriptorOpen | 120
From: Barry Oglesby [mailto:boglesby@pivotal.io<ma...@pivotal.io>]
Sent: Wednesday, December 02, 2015 12:56 PM
To: user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>
Subject: Re: Locator high cpu
What is the memory usage of the locator? Do you have any JMX clients or Pulse running? Can you get a few thread dumps of the locator JVM? That should help determine what the threads are doing.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <Pa...@voya.com>> wrote:
We have a gemfire client/server cluster that we’ve noticed our locator is running a sustained 50 – 100% cpu utilization. We stopped all work in this environment (test) and the cpu has remained the same. We are running gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look at to see what may be causing this?
Thanks.
-Paul
---------------------------------------------------------
NOTICE: The information contained in this electronic mail message is confidential and intended only for certain recipients. If you are not an intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication and any attachments is strictly prohibited. If you have received this communication in error, please notify the sender by reply transmission and delete the message without copying or disclosing it.
============================================================================================
Re: Locator high cpu
Posted by Barry Oglesby <bo...@pivotal.io>.
You're pretty near the limit of this box. Based on the top command output
you included, there is 431944k free. I don't think that includes the
buffers and cached though so it looks like you have ~1GB free, but you
don't want to use all the memory. I actually don't think I would increase
the memory yet (or maybe even at all). It might be better to see what is
using the memory to verify there isn't a leak.
I like using free -m. The example below tells me there is 44307m free on my
box (from the -/+ buffers/cache line).
free -m
total used free shared buffers cached
Mem: 48292 24423 23868 2408 759 19679
-/+ buffers/cache: 3984 44307
Swap: 15358 0 15358
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:24 AM, Catanzarite, P. (Paul) <
Paul.Catanzarite@voya.com> wrote:
> Thanks for the feedback.
>
>
>
> On a server with 4gb memory, what is a safe max heap size to set for the
> locator?
>
>
>
> *From:* Barry Oglesby [mailto:boglesby@pivotal.io]
> *Sent:* Wednesday, December 02, 2015 01:20 PM
> *To:* user@geode.incubator.apache.org
> *Subject:* Re: Locator high cpu
>
>
>
> Sorry, I sent that before I meant to.
>
>
>
> It looks like GC issues. You'll probably need to increase your max heap
> when you start the locator.
>
>
>
> | currentHeapSize | 1905
>
> | maximumHeapSize | 2031
>
>
>
>
> Barry Oglesby
>
> GemFire Advanced Customer Engineering (ACE)
>
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
>
>
>
> On Wed, Dec 2, 2015 at 10:19 AM, Barry Oglesby <bo...@pivotal.io>
> wrote:
>
> As Jens said, it looks like GC issues.
>
>
>
>
> Barry Oglesby
>
> GemFire Advanced Customer Engineering (ACE)
>
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
>
>
>
> On Wed, Dec 2, 2015 at 10:13 AM, Catanzarite, P. (Paul) <
> Paul.Catanzarite@voya.com> wrote:
>
> It will take me some time to get the thread dump but here is some
> additional info.
>
>
>
> Server info:
>
> VM with 2 CPUs and 4gb memory and only this one locator is running on this
> server
>
>
>
> => uptime
>
> 12:57:17 up 11 days, 3:10, 1 user, load average: 0.34, 0.46, 0.46
>
>
>
> => top
>
> top - 12:57:46 up 11 days, 3:10, 1 user, load average: 0.53, 0.50, 0.47
>
> Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
>
> Cpu(s): 11.2%us, 0.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
>
> Mem: 3916900k total, 3484956k used, 431944k free, 266008k buffers
>
> Swap: 8388604k total, 26100k used, 8362504k free, 350836k cached
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 4952 gemfire 20 0 3680m 2.3g 8192 S 105.7 61.8 3529:13 java
>
>
>
>
>
> Pulse shows locator heap using 1.9gb (out of 2gb configured)
>
>
>
> Member Metrics
>
>
>
> Category | Metric | Value
>
> ------------- | -------------------------------- | ------
>
> member | upTime | 962112
>
> | cpuUsage | 60
>
> | currentHeapSize | 1905
>
> | maximumHeapSize | 2031
>
> jvm | jvmThreads | 92
>
> | fileDescriptorLimit | 32768
>
> | totalFileDescriptorOpen | 120
>
>
>
>
>
>
>
>
>
> *From:* Barry Oglesby [mailto:boglesby@pivotal.io]
> *Sent:* Wednesday, December 02, 2015 12:56 PM
> *To:* user@geode.incubator.apache.org
> *Subject:* Re: Locator high cpu
>
>
>
> What is the memory usage of the locator? Do you have any JMX clients or
> Pulse running? Can you get a few thread dumps of the locator JVM? That
> should help determine what the threads are doing.
>
>
> Barry Oglesby
>
> GemFire Advanced Customer Engineering (ACE)
>
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
>
>
>
> On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <
> Paul.Catanzarite@voya.com> wrote:
>
> We have a gemfire client/server cluster that we’ve noticed our locator is
> running a sustained 50 – 100% cpu utilization. We stopped all work in this
> environment (test) and the cpu has remained the same. We are running
> gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look
> at to see what may be causing this?
>
> Thanks.
>
> -Paul
>
>
> ---------------------------------------------------------
>
> NOTICE: The information contained in this electronic mail message is
> confidential and intended only for certain recipients. If you are not an
> intended recipient, you are hereby notified that any disclosure,
> reproduction, distribution or other use of this communication and any
> attachments is strictly prohibited. If you have received this
> communication in error, please notify the sender by reply transmission and
> delete the message without copying or disclosing it.
>
>
> ============================================================================================
>
>
>
>
>
>
>
RE: Locator high cpu
Posted by "Catanzarite, P. (Paul)" <Pa...@voya.com>.
Thanks for the feedback.
On a server with 4gb memory, what is a safe max heap size to set for the locator?
From: Barry Oglesby [mailto:boglesby@pivotal.io]
Sent: Wednesday, December 02, 2015 01:20 PM
To: user@geode.incubator.apache.org
Subject: Re: Locator high cpu
Sorry, I sent that before I meant to.
It looks like GC issues. You'll probably need to increase your max heap when you start the locator.
| currentHeapSize | 1905
| maximumHeapSize | 2031
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:19 AM, Barry Oglesby <bo...@pivotal.io>> wrote:
As Jens said, it looks like GC issues.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:13 AM, Catanzarite, P. (Paul) <Pa...@voya.com>> wrote:
It will take me some time to get the thread dump but here is some additional info.
Server info:
VM with 2 CPUs and 4gb memory and only this one locator is running on this server
=> uptime
12:57:17 up 11 days, 3:10, 1 user, load average: 0.34, 0.46, 0.46
=> top
top - 12:57:46 up 11 days, 3:10, 1 user, load average: 0.53, 0.50, 0.47
Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.2%us, 0.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3916900k total, 3484956k used, 431944k free, 266008k buffers
Swap: 8388604k total, 26100k used, 8362504k free, 350836k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4952 gemfire 20 0 3680m 2.3g 8192 S 105.7 61.8 3529:13 java
Pulse shows locator heap using 1.9gb (out of 2gb configured)
Member Metrics
Category | Metric | Value
------------- | -------------------------------- | ------
member | upTime | 962112
| cpuUsage | 60
| currentHeapSize | 1905
| maximumHeapSize | 2031
jvm | jvmThreads | 92
| fileDescriptorLimit | 32768
| totalFileDescriptorOpen | 120
From: Barry Oglesby [mailto:boglesby@pivotal.io<ma...@pivotal.io>]
Sent: Wednesday, December 02, 2015 12:56 PM
To: user@geode.incubator.apache.org<ma...@geode.incubator.apache.org>
Subject: Re: Locator high cpu
What is the memory usage of the locator? Do you have any JMX clients or Pulse running? Can you get a few thread dumps of the locator JVM? That should help determine what the threads are doing.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <Pa...@voya.com>> wrote:
We have a gemfire client/server cluster that we’ve noticed our locator is running a sustained 50 – 100% cpu utilization. We stopped all work in this environment (test) and the cpu has remained the same. We are running gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look at to see what may be causing this?
Thanks.
-Paul
---------------------------------------------------------
NOTICE: The information contained in this electronic mail message is confidential and intended only for certain recipients. If you are not an intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication and any attachments is strictly prohibited. If you have received this communication in error, please notify the sender by reply transmission and delete the message without copying or disclosing it.
============================================================================================
Re: Locator high cpu
Posted by Barry Oglesby <bo...@pivotal.io>.
Sorry, I sent that before I meant to.
It looks like GC issues. You'll probably need to increase your max heap
when you start the locator.
| currentHeapSize | 1905
| maximumHeapSize | 2031
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:19 AM, Barry Oglesby <bo...@pivotal.io> wrote:
> As Jens said, it looks like GC issues.
>
>
> Barry Oglesby
> GemFire Advanced Customer Engineering (ACE)
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
> On Wed, Dec 2, 2015 at 10:13 AM, Catanzarite, P. (Paul) <
> Paul.Catanzarite@voya.com> wrote:
>
>> It will take me some time to get the thread dump but here is some
>> additional info.
>>
>>
>>
>> Server info:
>>
>> VM with 2 CPUs and 4gb memory and only this one locator is running on
>> this server
>>
>>
>>
>> => uptime
>>
>> 12:57:17 up 11 days, 3:10, 1 user, load average: 0.34, 0.46, 0.46
>>
>>
>>
>> => top
>>
>> top - 12:57:46 up 11 days, 3:10, 1 user, load average: 0.53, 0.50, 0.47
>>
>> Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
>>
>> Cpu(s): 11.2%us, 0.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si,
>> 0.0%st
>>
>> Mem: 3916900k total, 3484956k used, 431944k free, 266008k buffers
>>
>> Swap: 8388604k total, 26100k used, 8362504k free, 350836k cached
>>
>>
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>> 4952 gemfire 20 0 3680m 2.3g 8192 S 105.7 61.8 3529:13 java
>>
>>
>>
>>
>>
>> Pulse shows locator heap using 1.9gb (out of 2gb configured)
>>
>>
>>
>> Member Metrics
>>
>>
>>
>> Category | Metric | Value
>>
>> ------------- | -------------------------------- | ------
>>
>> member | upTime | 962112
>>
>> | cpuUsage | 60
>>
>> | currentHeapSize | 1905
>>
>> | maximumHeapSize | 2031
>>
>> jvm | jvmThreads | 92
>>
>> | fileDescriptorLimit | 32768
>>
>> | totalFileDescriptorOpen | 120
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From:* Barry Oglesby [mailto:boglesby@pivotal.io]
>> *Sent:* Wednesday, December 02, 2015 12:56 PM
>> *To:* user@geode.incubator.apache.org
>> *Subject:* Re: Locator high cpu
>>
>>
>>
>> What is the memory usage of the locator? Do you have any JMX clients or
>> Pulse running? Can you get a few thread dumps of the locator JVM? That
>> should help determine what the threads are doing.
>>
>>
>> Barry Oglesby
>>
>> GemFire Advanced Customer Engineering (ACE)
>>
>> For immediate support please contact Pivotal Support at
>> http://support.pivotal.io/
>>
>>
>>
>>
>>
>> On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <
>> Paul.Catanzarite@voya.com> wrote:
>>
>> We have a gemfire client/server cluster that we’ve noticed our locator is
>> running a sustained 50 – 100% cpu utilization. We stopped all work in this
>> environment (test) and the cpu has remained the same. We are running
>> gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look
>> at to see what may be causing this?
>>
>> Thanks.
>>
>> -Paul
>>
>>
>> ---------------------------------------------------------
>>
>> NOTICE: The information contained in this electronic mail message is
>> confidential and intended only for certain recipients. If you are not an
>> intended recipient, you are hereby notified that any disclosure,
>> reproduction, distribution or other use of this communication and any
>> attachments is strictly prohibited. If you have received this
>> communication in error, please notify the sender by reply transmission and
>> delete the message without copying or disclosing it.
>>
>>
>> ============================================================================================
>>
>>
>>
>
>
Re: Locator high cpu
Posted by Barry Oglesby <bo...@pivotal.io>.
As Jens said, it looks like GC issues.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:13 AM, Catanzarite, P. (Paul) <
Paul.Catanzarite@voya.com> wrote:
> It will take me some time to get the thread dump but here is some
> additional info.
>
>
>
> Server info:
>
> VM with 2 CPUs and 4gb memory and only this one locator is running on this
> server
>
>
>
> => uptime
>
> 12:57:17 up 11 days, 3:10, 1 user, load average: 0.34, 0.46, 0.46
>
>
>
> => top
>
> top - 12:57:46 up 11 days, 3:10, 1 user, load average: 0.53, 0.50, 0.47
>
> Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
>
> Cpu(s): 11.2%us, 0.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
>
> Mem: 3916900k total, 3484956k used, 431944k free, 266008k buffers
>
> Swap: 8388604k total, 26100k used, 8362504k free, 350836k cached
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 4952 gemfire 20 0 3680m 2.3g 8192 S 105.7 61.8 3529:13 java
>
>
>
>
>
> Pulse shows locator heap using 1.9gb (out of 2gb configured)
>
>
>
> Member Metrics
>
>
>
> Category | Metric | Value
>
> ------------- | -------------------------------- | ------
>
> member | upTime | 962112
>
> | cpuUsage | 60
>
> | currentHeapSize | 1905
>
> | maximumHeapSize | 2031
>
> jvm | jvmThreads | 92
>
> | fileDescriptorLimit | 32768
>
> | totalFileDescriptorOpen | 120
>
>
>
>
>
>
>
>
>
> *From:* Barry Oglesby [mailto:boglesby@pivotal.io]
> *Sent:* Wednesday, December 02, 2015 12:56 PM
> *To:* user@geode.incubator.apache.org
> *Subject:* Re: Locator high cpu
>
>
>
> What is the memory usage of the locator? Do you have any JMX clients or
> Pulse running? Can you get a few thread dumps of the locator JVM? That
> should help determine what the threads are doing.
>
>
> Barry Oglesby
>
> GemFire Advanced Customer Engineering (ACE)
>
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
>
>
>
> On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <
> Paul.Catanzarite@voya.com> wrote:
>
> We have a gemfire client/server cluster that we’ve noticed our locator is
> running a sustained 50 – 100% cpu utilization. We stopped all work in this
> environment (test) and the cpu has remained the same. We are running
> gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look
> at to see what may be causing this?
>
> Thanks.
>
> -Paul
>
>
> ---------------------------------------------------------
>
> NOTICE: The information contained in this electronic mail message is
> confidential and intended only for certain recipients. If you are not an
> intended recipient, you are hereby notified that any disclosure,
> reproduction, distribution or other use of this communication and any
> attachments is strictly prohibited. If you have received this
> communication in error, please notify the sender by reply transmission and
> delete the message without copying or disclosing it.
>
>
> ============================================================================================
>
>
>
Re: Locator high cpu
Posted by Barry Oglesby <bo...@pivotal.io>.
Paul,
That is pretty high memory usage for a locator. Jens idea to get a
histogram or heap dump is a real good one. That'll help determine if there
is a memory leak.
Thanks,
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/
On Wed, Dec 2, 2015 at 10:13 AM, Catanzarite, P. (Paul) <
Paul.Catanzarite@voya.com> wrote:
> It will take me some time to get the thread dump but here is some
> additional info.
>
>
>
> Server info:
>
> VM with 2 CPUs and 4gb memory and only this one locator is running on this
> server
>
>
>
> => uptime
>
> 12:57:17 up 11 days, 3:10, 1 user, load average: 0.34, 0.46, 0.46
>
>
>
> => top
>
> top - 12:57:46 up 11 days, 3:10, 1 user, load average: 0.53, 0.50, 0.47
>
> Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
>
> Cpu(s): 11.2%us, 0.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
>
> Mem: 3916900k total, 3484956k used, 431944k free, 266008k buffers
>
> Swap: 8388604k total, 26100k used, 8362504k free, 350836k cached
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 4952 gemfire 20 0 3680m 2.3g 8192 S 105.7 61.8 3529:13 java
>
>
>
>
>
> Pulse shows locator heap using 1.9gb (out of 2gb configured)
>
>
>
> Member Metrics
>
>
>
> Category | Metric | Value
>
> ------------- | -------------------------------- | ------
>
> member | upTime | 962112
>
> | cpuUsage | 60
>
> | currentHeapSize | 1905
>
> | maximumHeapSize | 2031
>
> jvm | jvmThreads | 92
>
> | fileDescriptorLimit | 32768
>
> | totalFileDescriptorOpen | 120
>
>
>
>
>
>
>
>
>
> *From:* Barry Oglesby [mailto:boglesby@pivotal.io]
> *Sent:* Wednesday, December 02, 2015 12:56 PM
> *To:* user@geode.incubator.apache.org
> *Subject:* Re: Locator high cpu
>
>
>
> What is the memory usage of the locator? Do you have any JMX clients or
> Pulse running? Can you get a few thread dumps of the locator JVM? That
> should help determine what the threads are doing.
>
>
> Barry Oglesby
>
> GemFire Advanced Customer Engineering (ACE)
>
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
>
>
>
> On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <
> Paul.Catanzarite@voya.com> wrote:
>
> We have a gemfire client/server cluster that we’ve noticed our locator is
> running a sustained 50 – 100% cpu utilization. We stopped all work in this
> environment (test) and the cpu has remained the same. We are running
> gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look
> at to see what may be causing this?
>
> Thanks.
>
> -Paul
>
>
> ---------------------------------------------------------
>
> NOTICE: The information contained in this electronic mail message is
> confidential and intended only for certain recipients. If you are not an
> intended recipient, you are hereby notified that any disclosure,
> reproduction, distribution or other use of this communication and any
> attachments is strictly prohibited. If you have received this
> communication in error, please notify the sender by reply transmission and
> delete the message without copying or disclosing it.
>
>
> ============================================================================================
>
>
>
RE: Locator high cpu
Posted by "Catanzarite, P. (Paul)" <Pa...@voya.com>.
It will take me some time to get the thread dump but here is some additional info.
Server info:
VM with 2 CPUs and 4gb memory and only this one locator is running on this server
=> uptime
12:57:17 up 11 days, 3:10, 1 user, load average: 0.34, 0.46, 0.46
=> top
top - 12:57:46 up 11 days, 3:10, 1 user, load average: 0.53, 0.50, 0.47
Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.2%us, 0.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3916900k total, 3484956k used, 431944k free, 266008k buffers
Swap: 8388604k total, 26100k used, 8362504k free, 350836k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4952 gemfire 20 0 3680m 2.3g 8192 S 105.7 61.8 3529:13 java
Pulse shows locator heap using 1.9gb (out of 2gb configured)
Member Metrics
Category | Metric | Value
------------- | -------------------------------- | ------
member | upTime | 962112
| cpuUsage | 60
| currentHeapSize | 1905
| maximumHeapSize | 2031
jvm | jvmThreads | 92
| fileDescriptorLimit | 32768
| totalFileDescriptorOpen | 120
From: Barry Oglesby [mailto:boglesby@pivotal.io]
Sent: Wednesday, December 02, 2015 12:56 PM
To: user@geode.incubator.apache.org
Subject: Re: Locator high cpu
What is the memory usage of the locator? Do you have any JMX clients or Pulse running? Can you get a few thread dumps of the locator JVM? That should help determine what the threads are doing.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at http://support.pivotal.io/
On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <Pa...@voya.com>> wrote:
We have a gemfire client/server cluster that we’ve noticed our locator is running a sustained 50 – 100% cpu utilization. We stopped all work in this environment (test) and the cpu has remained the same. We are running gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look at to see what may be causing this?
Thanks.
-Paul
---------------------------------------------------------
NOTICE: The information contained in this electronic mail message is confidential and intended only for certain recipients. If you are not an intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication and any attachments is strictly prohibited. If you have received this communication in error, please notify the sender by reply transmission and delete the message without copying or disclosing it.
============================================================================================
Re: Locator high cpu
Posted by Barry Oglesby <bo...@pivotal.io>.
What is the memory usage of the locator? Do you have any JMX clients or
Pulse running? Can you get a few thread dumps of the locator JVM? That
should help determine what the threads are doing.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/
On Wed, Dec 2, 2015 at 9:51 AM, Catanzarite, P. (Paul) <
Paul.Catanzarite@voya.com> wrote:
> We have a gemfire client/server cluster that we’ve noticed our locator is
> running a sustained 50 – 100% cpu utilization. We stopped all work in this
> environment (test) and the cpu has remained the same. We are running
> gemfire v8.1, java 1.7 on linux VM RHEL 6. Any suggestions on what to look
> at to see what may be causing this?
>
> Thanks.
>
> -Paul
>
>
> ---------------------------------------------------------
>
> NOTICE: The information contained in this electronic mail message is
> confidential and intended only for certain recipients. If you are not an
> intended recipient, you are hereby notified that any disclosure,
> reproduction, distribution or other use of this communication and any
> attachments is strictly prohibited. If you have received this
> communication in error, please notify the sender by reply transmission and
> delete the message without copying or disclosing it.
>
>
> ============================================================================================
>