You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 曹志富 <ca...@gmail.com> on 2015/10/20 06:40:21 UTC
unusual GC log
INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 -
ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 -
ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 -
ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 -
ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 -
ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 -
ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 -
ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 -
ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 -
ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480;
CMS Perm Gen: 33751256 -> 33751192
; Par Eden Space: 7192 -> 611360336;
INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 -
ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 -
ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 -
ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 -
ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space:
671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 -
ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space:
671088640 -> 0; Par Survivor Sp
ace: 83886080 -> 83886072
INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 -
ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden Space:
671088640 -> 0; Par Survivor Sp
ace: 83886072 -> 83886080
after seconds node mark down.
My node config is : 8GB heap NEW_HEAP size is 800MB
NODE hardware is :4CORE 32GBRAM
--------------------------------------
Ranger Tsao
Re: unusual GC log
Posted by Brice Dutheil <br...@gmail.com>.
I remember similar troubles with GC, when our write load was to intensive.
We were BATCH LOGGED statements in order to have the same write timestamp
for a related upserts (usually 10 to 20 statements), however we had
outliers in our data that affected our denormalizations / projections.
Our outliers were mostly in the form of very lare lists/maps in a single
row, which somehow affected the duration of the Memtable in the heap longer
that it should. Additionaly with the denormalization with primary key on
elements of these collections it also added the same number of upsert
statement.
So you may want to check your write workload. Also since you are using 2.1
you may want to try off-heap memtables (the default is heap_buffers).
memtable_allocation_type: offheap_objects
We had to do heap dumpd to confirm this, also we were on 2.0 at this time,
and we tweaked our write workload before.
-- Brice
On Wed, Oct 21, 2015 at 8:08 AM, 曹志富 <ca...@gmail.com> wrote:
> only use LCS & STCS
>
> --------------------------------------
> Ranger Tsao
>
> 2015-10-21 13:58 GMT+08:00 Jeff Jirsa <je...@crowdstrike.com>:
>
>> 2.1.5 and 2.1.6 have a known memory leak. Didn’t typically manifest in
>> the behavior you’re seeing, but it’s definitely there (and especially
>> prevalent if you use DTCS).
>>
>>
>>
>> From: 曹志富
>> Reply-To: "user@cassandra.apache.org"
>> Date: Tuesday, October 20, 2015 at 10:47 PM
>> To: "user@cassandra.apache.org"
>> Subject: Re: unusual GC log
>>
>> I had try increase jvm heap to 12GB ,but also has this problem.
>>
>> I will try again with increase jvm heap.
>>
>> Thanks!!!
>>
>> --------------------------------------
>> Ranger Tsao
>>
>> 2015-10-21 13:44 GMT+08:00 Jason Wee <pe...@gmail.com>:
>>
>>> that both survivor spaces with 100% usage and tenure generation 100%
>>> usage is something not good. that column fgc and fgct shown full gc is
>>> happening too often and takes second. Perhaps the immediate solution is to
>>> increase heap to the cassandra jvm. Then add more nodes.
>>>
>>> how much cassandra load per node and cassandra cluster load, do you
>>> know? and file descriptor to run this cassandra instance? i'm using oracle
>>> jdk 7 update 72 64bit server edition, with 14GB to the c* jvm heap with
>>> system memory at 32GB.
>>>
>>> On Wed, Oct 21, 2015 at 12:37 PM, 曹志富 <ca...@gmail.com> wrote:
>>>
>>>> jstack info is :
>>>>
>>>>
>>>>
>>>>
>>>> --------------------------------------
>>>> Ranger Tsao
>>>>
>>>> 2015-10-21 9:18 GMT+08:00 曹志富 <ca...@gmail.com>:
>>>>
>>>>> C* version is 2.1.6.
>>>>> CentOS release 6.5 (Final)
>>>>> Sun JDK 1.7.0_71 64bit.
>>>>>
>>>>> attach is my config setting.
>>>>>
>>>>> Thank you very much!!!
>>>>>
>>>>> --------------------------------------
>>>>> Ranger Tsao
>>>>>
>>>>> 2015-10-20 15:43 GMT+08:00 Graham Sanderson <gr...@vast.com>:
>>>>>
>>>>>> What version of C* are you running? any special settings in
>>>>>> cassandra.yaml; are you running with stock GC settings in cassandra-env.sh?
>>>>>> what JDK/OS?
>>>>>>
>>>>>> On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
>>>>>>
>>>>>> INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 -
>>>>>> ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space:
>>>>>> 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 -
>>>>>> ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space:
>>>>>> 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 -
>>>>>> ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden
>>>>>> Space: 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 -
>>>>>> ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden
>>>>>> Space: 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 -
>>>>>> ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space:
>>>>>> 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 -
>>>>>> ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden
>>>>>> Space: 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 -
>>>>>> ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden
>>>>>> Space: 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 -
>>>>>> ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden
>>>>>> Space: 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 -
>>>>>> ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480;
>>>>>> CMS Perm Gen: 33751256 -> 33751192
>>>>>> ; Par Eden Space: 7192 -> 611360336;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 -
>>>>>> ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space:
>>>>>> 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 -
>>>>>> ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space:
>>>>>> 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 -
>>>>>> ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space:
>>>>>> 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 -
>>>>>> ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space:
>>>>>> 671088640 -> 0;
>>>>>> INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 -
>>>>>> ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space:
>>>>>> 671088640 -> 0; Par Survivor Sp
>>>>>> ace: 83886080 -> 83886072
>>>>>> INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 -
>>>>>> ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden
>>>>>> Space: 671088640 -> 0; Par Survivor Sp
>>>>>> ace: 83886072 -> 83886080
>>>>>>
>>>>>> after seconds node mark down.
>>>>>>
>>>>>> My node config is : 8GB heap NEW_HEAP size is 800MB
>>>>>>
>>>>>> NODE hardware is :4CORE 32GBRAM
>>>>>>
>>>>>> --------------------------------------
>>>>>> Ranger Tsao
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: unusual GC log
Posted by 曹志富 <ca...@gmail.com>.
only use LCS & STCS
--------------------------------------
Ranger Tsao
2015-10-21 13:58 GMT+08:00 Jeff Jirsa <je...@crowdstrike.com>:
> 2.1.5 and 2.1.6 have a known memory leak. Didn’t typically manifest in the
> behavior you’re seeing, but it’s definitely there (and especially prevalent
> if you use DTCS).
>
>
>
> From: 曹志富
> Reply-To: "user@cassandra.apache.org"
> Date: Tuesday, October 20, 2015 at 10:47 PM
> To: "user@cassandra.apache.org"
> Subject: Re: unusual GC log
>
> I had try increase jvm heap to 12GB ,but also has this problem.
>
> I will try again with increase jvm heap.
>
> Thanks!!!
>
> --------------------------------------
> Ranger Tsao
>
> 2015-10-21 13:44 GMT+08:00 Jason Wee <pe...@gmail.com>:
>
>> that both survivor spaces with 100% usage and tenure generation 100%
>> usage is something not good. that column fgc and fgct shown full gc is
>> happening too often and takes second. Perhaps the immediate solution is to
>> increase heap to the cassandra jvm. Then add more nodes.
>>
>> how much cassandra load per node and cassandra cluster load, do you know?
>> and file descriptor to run this cassandra instance? i'm using oracle jdk 7
>> update 72 64bit server edition, with 14GB to the c* jvm heap with system
>> memory at 32GB.
>>
>> On Wed, Oct 21, 2015 at 12:37 PM, 曹志富 <ca...@gmail.com> wrote:
>>
>>> jstack info is :
>>>
>>>
>>>
>>>
>>> --------------------------------------
>>> Ranger Tsao
>>>
>>> 2015-10-21 9:18 GMT+08:00 曹志富 <ca...@gmail.com>:
>>>
>>>> C* version is 2.1.6.
>>>> CentOS release 6.5 (Final)
>>>> Sun JDK 1.7.0_71 64bit.
>>>>
>>>> attach is my config setting.
>>>>
>>>> Thank you very much!!!
>>>>
>>>> --------------------------------------
>>>> Ranger Tsao
>>>>
>>>> 2015-10-20 15:43 GMT+08:00 Graham Sanderson <gr...@vast.com>:
>>>>
>>>>> What version of C* are you running? any special settings in
>>>>> cassandra.yaml; are you running with stock GC settings in cassandra-env.sh?
>>>>> what JDK/OS?
>>>>>
>>>>> On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
>>>>>
>>>>> INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 -
>>>>> ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space:
>>>>> 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 -
>>>>> ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space:
>>>>> 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 -
>>>>> ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden
>>>>> Space: 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 -
>>>>> ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden
>>>>> Space: 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 -
>>>>> ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space:
>>>>> 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 -
>>>>> ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden
>>>>> Space: 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 -
>>>>> ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden
>>>>> Space: 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 -
>>>>> ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden
>>>>> Space: 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 -
>>>>> ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480;
>>>>> CMS Perm Gen: 33751256 -> 33751192
>>>>> ; Par Eden Space: 7192 -> 611360336;
>>>>> INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 -
>>>>> ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space:
>>>>> 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 -
>>>>> ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space:
>>>>> 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 -
>>>>> ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space:
>>>>> 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 -
>>>>> ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space:
>>>>> 671088640 -> 0;
>>>>> INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 -
>>>>> ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space:
>>>>> 671088640 -> 0; Par Survivor Sp
>>>>> ace: 83886080 -> 83886072
>>>>> INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 -
>>>>> ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden
>>>>> Space: 671088640 -> 0; Par Survivor Sp
>>>>> ace: 83886072 -> 83886080
>>>>>
>>>>> after seconds node mark down.
>>>>>
>>>>> My node config is : 8GB heap NEW_HEAP size is 800MB
>>>>>
>>>>> NODE hardware is :4CORE 32GBRAM
>>>>>
>>>>> --------------------------------------
>>>>> Ranger Tsao
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: unusual GC log
Posted by Jeff Jirsa <je...@crowdstrike.com>.
2.1.5 and 2.1.6 have a known memory leak. Didn’t typically manifest in the behavior you’re seeing, but it’s definitely there (and especially prevalent if you use DTCS).
From: 曹志富
Reply-To: "user@cassandra.apache.org"
Date: Tuesday, October 20, 2015 at 10:47 PM
To: "user@cassandra.apache.org"
Subject: Re: unusual GC log
I had try increase jvm heap to 12GB ,but also has this problem.
I will try again with increase jvm heap.
Thanks!!!
--------------------------------------
Ranger Tsao
2015-10-21 13:44 GMT+08:00 Jason Wee <pe...@gmail.com>:
that both survivor spaces with 100% usage and tenure generation 100% usage is something not good. that column fgc and fgct shown full gc is happening too often and takes second. Perhaps the immediate solution is to increase heap to the cassandra jvm. Then add more nodes.
how much cassandra load per node and cassandra cluster load, do you know? and file descriptor to run this cassandra instance? i'm using oracle jdk 7 update 72 64bit server edition, with 14GB to the c* jvm heap with system memory at 32GB.
On Wed, Oct 21, 2015 at 12:37 PM, 曹志富 <ca...@gmail.com> wrote:
jstack info is :
--------------------------------------
Ranger Tsao
2015-10-21 9:18 GMT+08:00 曹志富 <ca...@gmail.com>:
C* version is 2.1.6.
CentOS release 6.5 (Final)
Sun JDK 1.7.0_71 64bit.
attach is my config setting.
Thank you very much!!!
--------------------------------------
Ranger Tsao
2015-10-20 15:43 GMT+08:00 Graham Sanderson <gr...@vast.com>:
What version of C* are you running? any special settings in cassandra.yaml; are you running with stock GC settings in cassandra-env.sh? what JDK/OS?
On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 - ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 - ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 - ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 - ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 - ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 - ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 - ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 - ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 - ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480; CMS Perm Gen: 33751256 -> 33751192
; Par Eden Space: 7192 -> 611360336;
INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 - ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 - ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 - ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 - ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space: 671088640 -> 0;
INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 - ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space: 671088640 -> 0; Par Survivor Sp
ace: 83886080 -> 83886072
INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 - ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden Space: 671088640 -> 0; Par Survivor Sp
ace: 83886072 -> 83886080
after seconds node mark down.
My node config is : 8GB heap NEW_HEAP size is 800MB
NODE hardware is :4CORE 32GBRAM
--------------------------------------
Ranger Tsao
Re: unusual GC log
Posted by 曹志富 <ca...@gmail.com>.
I had try increase jvm heap to 12GB ,but also has this problem.
I will try again with increase jvm heap.
Thanks!!!
--------------------------------------
Ranger Tsao
2015-10-21 13:44 GMT+08:00 Jason Wee <pe...@gmail.com>:
> that both survivor spaces with 100% usage and tenure generation 100% usage
> is something not good. that column fgc and fgct shown full gc is happening
> too often and takes second. Perhaps the immediate solution is to increase
> heap to the cassandra jvm. Then add more nodes.
>
> how much cassandra load per node and cassandra cluster load, do you know?
> and file descriptor to run this cassandra instance? i'm using oracle jdk 7
> update 72 64bit server edition, with 14GB to the c* jvm heap with system
> memory at 32GB.
>
> On Wed, Oct 21, 2015 at 12:37 PM, 曹志富 <ca...@gmail.com> wrote:
>
>> jstack info is :
>>
>>
>>
>>
>> --------------------------------------
>> Ranger Tsao
>>
>> 2015-10-21 9:18 GMT+08:00 曹志富 <ca...@gmail.com>:
>>
>>> C* version is 2.1.6.
>>> CentOS release 6.5 (Final)
>>> Sun JDK 1.7.0_71 64bit.
>>>
>>> attach is my config setting.
>>>
>>> Thank you very much!!!
>>>
>>> --------------------------------------
>>> Ranger Tsao
>>>
>>> 2015-10-20 15:43 GMT+08:00 Graham Sanderson <gr...@vast.com>:
>>>
>>>> What version of C* are you running? any special settings in
>>>> cassandra.yaml; are you running with stock GC settings in cassandra-env.sh?
>>>> what JDK/OS?
>>>>
>>>> On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
>>>>
>>>> INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 -
>>>> ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space:
>>>> 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 -
>>>> ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space:
>>>> 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 -
>>>> ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden
>>>> Space: 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 -
>>>> ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden
>>>> Space: 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 -
>>>> ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space:
>>>> 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 -
>>>> ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden
>>>> Space: 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 -
>>>> ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden
>>>> Space: 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 -
>>>> ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden
>>>> Space: 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 -
>>>> ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480;
>>>> CMS Perm Gen: 33751256 -> 33751192
>>>> ; Par Eden Space: 7192 -> 611360336;
>>>> INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 -
>>>> ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space:
>>>> 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 -
>>>> ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space:
>>>> 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 -
>>>> ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space:
>>>> 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 -
>>>> ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space:
>>>> 671088640 -> 0;
>>>> INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 -
>>>> ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space:
>>>> 671088640 -> 0; Par Survivor Sp
>>>> ace: 83886080 -> 83886072
>>>> INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 -
>>>> ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden
>>>> Space: 671088640 -> 0; Par Survivor Sp
>>>> ace: 83886072 -> 83886080
>>>>
>>>> after seconds node mark down.
>>>>
>>>> My node config is : 8GB heap NEW_HEAP size is 800MB
>>>>
>>>> NODE hardware is :4CORE 32GBRAM
>>>>
>>>> --------------------------------------
>>>> Ranger Tsao
>>>>
>>>>
>>>>
>>>
>>
>
Re: unusual GC log
Posted by Jason Wee <pe...@gmail.com>.
that both survivor spaces with 100% usage and tenure generation 100% usage
is something not good. that column fgc and fgct shown full gc is happening
too often and takes second. Perhaps the immediate solution is to increase
heap to the cassandra jvm. Then add more nodes.
how much cassandra load per node and cassandra cluster load, do you know?
and file descriptor to run this cassandra instance? i'm using oracle jdk 7
update 72 64bit server edition, with 14GB to the c* jvm heap with system
memory at 32GB.
On Wed, Oct 21, 2015 at 12:37 PM, 曹志富 <ca...@gmail.com> wrote:
> jstack info is :
>
>
>
>
> --------------------------------------
> Ranger Tsao
>
> 2015-10-21 9:18 GMT+08:00 曹志富 <ca...@gmail.com>:
>
>> C* version is 2.1.6.
>> CentOS release 6.5 (Final)
>> Sun JDK 1.7.0_71 64bit.
>>
>> attach is my config setting.
>>
>> Thank you very much!!!
>>
>> --------------------------------------
>> Ranger Tsao
>>
>> 2015-10-20 15:43 GMT+08:00 Graham Sanderson <gr...@vast.com>:
>>
>>> What version of C* are you running? any special settings in
>>> cassandra.yaml; are you running with stock GC settings in cassandra-env.sh?
>>> what JDK/OS?
>>>
>>> On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
>>>
>>> INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 -
>>> ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space:
>>> 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 -
>>> ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space:
>>> 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 -
>>> ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden
>>> Space: 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 -
>>> ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden
>>> Space: 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 -
>>> ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space:
>>> 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 -
>>> ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden
>>> Space: 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 -
>>> ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden
>>> Space: 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 -
>>> ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden
>>> Space: 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 -
>>> ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480;
>>> CMS Perm Gen: 33751256 -> 33751192
>>> ; Par Eden Space: 7192 -> 611360336;
>>> INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 -
>>> ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space:
>>> 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 -
>>> ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space:
>>> 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 -
>>> ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space:
>>> 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 -
>>> ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space:
>>> 671088640 -> 0;
>>> INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 -
>>> ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space:
>>> 671088640 -> 0; Par Survivor Sp
>>> ace: 83886080 -> 83886072
>>> INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 -
>>> ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden
>>> Space: 671088640 -> 0; Par Survivor Sp
>>> ace: 83886072 -> 83886080
>>>
>>> after seconds node mark down.
>>>
>>> My node config is : 8GB heap NEW_HEAP size is 800MB
>>>
>>> NODE hardware is :4CORE 32GBRAM
>>>
>>> --------------------------------------
>>> Ranger Tsao
>>>
>>>
>>>
>>
>
Re: unusual GC log
Posted by 曹志富 <ca...@gmail.com>.
jstack info is :
--------------------------------------
Ranger Tsao
2015-10-21 9:18 GMT+08:00 曹志富 <ca...@gmail.com>:
> C* version is 2.1.6.
> CentOS release 6.5 (Final)
> Sun JDK 1.7.0_71 64bit.
>
> attach is my config setting.
>
> Thank you very much!!!
>
> --------------------------------------
> Ranger Tsao
>
> 2015-10-20 15:43 GMT+08:00 Graham Sanderson <gr...@vast.com>:
>
>> What version of C* are you running? any special settings in
>> cassandra.yaml; are you running with stock GC settings in cassandra-env.sh?
>> what JDK/OS?
>>
>> On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
>>
>> INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 -
>> ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space:
>> 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 -
>> ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space:
>> 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 -
>> ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden
>> Space: 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 -
>> ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden
>> Space: 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 -
>> ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space:
>> 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 -
>> ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden
>> Space: 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 -
>> ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden
>> Space: 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 -
>> ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden
>> Space: 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 -
>> ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480;
>> CMS Perm Gen: 33751256 -> 33751192
>> ; Par Eden Space: 7192 -> 611360336;
>> INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 -
>> ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space:
>> 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 -
>> ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space:
>> 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 -
>> ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space:
>> 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 -
>> ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space:
>> 671088640 -> 0;
>> INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 -
>> ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space:
>> 671088640 -> 0; Par Survivor Sp
>> ace: 83886080 -> 83886072
>> INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 -
>> ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden
>> Space: 671088640 -> 0; Par Survivor Sp
>> ace: 83886072 -> 83886080
>>
>> after seconds node mark down.
>>
>> My node config is : 8GB heap NEW_HEAP size is 800MB
>>
>> NODE hardware is :4CORE 32GBRAM
>>
>> --------------------------------------
>> Ranger Tsao
>>
>>
>>
>
Re: unusual GC log
Posted by 曹志富 <ca...@gmail.com>.
C* version is 2.1.6.
CentOS release 6.5 (Final)
Sun JDK 1.7.0_71 64bit.
attach is my config setting.
Thank you very much!!!
--------------------------------------
Ranger Tsao
2015-10-20 15:43 GMT+08:00 Graham Sanderson <gr...@vast.com>:
> What version of C* are you running? any special settings in
> cassandra.yaml; are you running with stock GC settings in cassandra-env.sh?
> what JDK/OS?
>
> On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
>
> INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 -
> ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space:
> 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 -
> ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space:
> 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 -
> ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden
> Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 -
> ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden
> Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 -
> ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space:
> 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 -
> ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden
> Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 -
> ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden
> Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 -
> ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden
> Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 -
> ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480;
> CMS Perm Gen: 33751256 -> 33751192
> ; Par Eden Space: 7192 -> 611360336;
> INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 -
> ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space:
> 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 -
> ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space:
> 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 -
> ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space:
> 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 -
> ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space:
> 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 -
> ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space:
> 671088640 -> 0; Par Survivor Sp
> ace: 83886080 -> 83886072
> INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 -
> ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden
> Space: 671088640 -> 0; Par Survivor Sp
> ace: 83886072 -> 83886080
>
> after seconds node mark down.
>
> My node config is : 8GB heap NEW_HEAP size is 800MB
>
> NODE hardware is :4CORE 32GBRAM
>
> --------------------------------------
> Ranger Tsao
>
>
>
Re: unusual GC log
Posted by Graham Sanderson <gr...@vast.com>.
What version of C* are you running? any special settings in cassandra.yaml; are you running with stock GC settings in cassandra-env.sh? what JDK/OS?
> On Oct 19, 2015, at 11:40 PM, 曹志富 <ca...@gmail.com> wrote:
>
> INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 - ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 - ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:42:53,847 GCInspector.java:252 - ParNew GC in 406ms. CMS Old Gen: 5114688368 -> 5513119264; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:42:57,118 GCInspector.java:252 - ParNew GC in 421ms. CMS Old Gen: 5513119264 -> 5926324736; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:00,041 GCInspector.java:252 - ParNew GC in 437ms. CMS Old Gen: 5926324736 -> 6324793584; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:03,029 GCInspector.java:252 - ParNew GC in 429ms. CMS Old Gen: 6324793584 -> 6693672608; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:05,566 GCInspector.java:252 - ParNew GC in 339ms. CMS Old Gen: 6693672608 -> 6989128592; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:08,431 GCInspector.java:252 - ParNew GC in 421ms. CMS Old Gen: 6266493464 -> 6662041272; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:11,131 GCInspector.java:252 - ConcurrentMarkSweep GC in 215ms. CMS Old Gen: 5926324736 -> 4574418480; CMS Perm Gen: 33751256 -> 33751192
> ; Par Eden Space: 7192 -> 611360336;
> INFO [Service Thread] 2015-10-20 10:43:11,848 GCInspector.java:252 - ParNew GC in 511ms. CMS Old Gen: 4574418480 -> 4996166672; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:14,915 GCInspector.java:252 - ParNew GC in 395ms. CMS Old Gen: 4996167912 -> 5380926744; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:18,335 GCInspector.java:252 - ParNew GC in 432ms. CMS Old Gen: 5380926744 -> 5811659120; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:21,492 GCInspector.java:252 - ParNew GC in 439ms. CMS Old Gen: 5811659120 -> 6270861936; Par Eden Space: 671088640 -> 0;
> INFO [Service Thread] 2015-10-20 10:43:24,698 GCInspector.java:252 - ParNew GC in 490ms. CMS Old Gen: 6270861936 -> 6668734208; Par Eden Space: 671088640 -> 0; Par Survivor Sp
> ace: 83886080 -> 83886072
> INFO [Service Thread] 2015-10-20 10:43:27,963 GCInspector.java:252 - ParNew GC in 457ms. CMS Old Gen: 6668734208 -> 7072885208; Par Eden Space: 671088640 -> 0; Par Survivor Sp
> ace: 83886072 -> 83886080
>
> after seconds node mark down.
>
> My node config is : 8GB heap NEW_HEAP size is 800MB
>
> NODE hardware is :4CORE 32GBRAM
>
> --------------------------------------
> Ranger Tsao