You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by prakash kadel <pr...@gmail.com> on 2013/10/07 17:32:15 UTC
You Are Dead Exception due to promotion failure
Hello,
I am getting this YADE all the time
HBASE_HEAPSIZE=8000
Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
-XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly
-XX:NewSize=256m -XX:MaxNewSize=256m
it seems there is promotion failure and the CMS take too long
2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100 secs]
[Times: user=0.27 sys=0.01, real=0.33 secs]
2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700 secs]
[Times: user=0.26 sys=0.02, real=0.19 secs]
2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570 secs]
[Times: user=0.25 sys=0.02, real=0.24 secs]
2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130 secs]
[Times: user=0.20 sys=0.03, real=0.25 secs]
2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280 secs]
[Times: user=0.23 sys=0.04, real=0.30 secs]
2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark: 7660656K(7929856K)]
7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06, real=3.77
secs]
2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100 secs]
[Times: user=0.21 sys=0.03, real=0.46 secs]
2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230 secs]
[Times: user=0.24 sys=0.06, real=0.35 secs]
2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530 secs]
[Times: user=0.23 sys=0.03, real=0.63 secs]
2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840 secs]
[Times: user=0.24 sys=0.02, real=0.32 secs]
2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480 secs]
[Times: user=0.31 sys=0.03, real=0.49 secs]
2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380 secs]
[Times: user=0.21 sys=0.06, real=0.34 secs]
2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
235745K->235745K(235968K), 0.3339710 secs][CMS2013-10-07T01:25:29.750+0900:
[CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
real=88.77 secs]
(concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810 secs]
8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)] icms_dc=9
, 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark: 2882954K(7929856K)]
2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00, real=0.09
secs]
Heap
par new generation total 235968K, used 197697K [0x0000000606e00000,
0x0000000616e00000, 0x0000000616e00000)
eden space 209792K, 94% used [0x0000000606e00000, 0x0000000612f10718,
0x0000000613ae0000)
from space 26176K, 0% used [0x0000000615470000, 0x0000000615470000,
0x0000000616e00000)
to space 26176K, 0% used [0x0000000613ae0000, 0x0000000613ae0000,
0x0000000615470000)
concurrent mark-sweep generation total 7929856K, used 2882954K
[0x0000000616e00000, 0x00000007fae00000, 0x00000007fae00000)
concurrent-mark-sweep perm gen total 53340K, used 31960K
[0x00000007fae00000, 0x00000007fe217000, 0x0000000800000000)
What is wrong here? please give me some suggestions.
Sincerely,
Prakash
答复: You Are Dead Exception due to promotion failure
Posted by 谢良 <xi...@xiaomi.com>.
The OS will pick up a process to kill according to an internal oom score,weight...(very likely is HBase if it ran long time and consumed lots of memory)
________________________________________
发件人: prakash kadel [prakash.kadel@gmail.com]
发送时间: 2013年10月8日 6:13
收件人: user@hbase.apache.org
主题: Re: You Are Dead Exception due to promotion failure
BTW,
if i disable the swap at all. What will happen in the above situation?
currently it starts swapping at 90%
Sincerely
On Tue, Oct 8, 2013 at 7:09 AM, prakash kadel <pr...@gmail.com>wrote:
> thanks,
>
> yup, it seems so. I have 48 gb memory. i see it swaps at that point.
>
> btw, why is the CMS not kicking in early? do you have any idea?
>
> sincerely
>
>
>
> On Tue, Oct 8, 2013 at 3:00 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> This line:
>>
>> [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
>> real=88.77
>> secs]
>>
>> Is suspicious. Are you swapping?
>>
>> J-D
>>
>>
>> On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel <prakash.kadel@gmail.com
>> >wrote:
>>
>> > Also,
>> > why is the CMS not kicking in early, i have set XX:+
>> > UseCMSInitiatingOccupancyOnly???
>> >
>> > Sincerely,
>> > Prakash
>> >
>> >
>> > On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <prakash.kadel@gmail.com
>> > >wrote:
>> >
>> > > Hello,
>> > > I am getting this YADE all the time
>> > >
>> > > HBASE_HEAPSIZE=8000
>> > >
>> > > Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
>> > > -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode
>> -XX:+UseParNewGC
>> > > -XX:CMSInitiatingOccupancyFraction=50
>> -XX:+UseCMSInitiatingOccupancyOnly
>> > > -XX:NewSize=256m -XX:MaxNewSize=256m
>> > >
>> > > it seems there is promotion failure and the CMS take too long
>> > >
>> > > 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
>> > > 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100
>> secs]
>> > > [Times: user=0.27 sys=0.01, real=0.33 secs]
>> > > 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
>> > > 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700
>> secs]
>> > > [Times: user=0.26 sys=0.02, real=0.19 secs]
>> > > 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
>> > > 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570
>> secs]
>> > > [Times: user=0.25 sys=0.02, real=0.24 secs]
>> > > 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
>> > > 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130
>> secs]
>> > > [Times: user=0.20 sys=0.03, real=0.25 secs]
>> > > 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
>> > > 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280
>> secs]
>> > > [Times: user=0.23 sys=0.04, real=0.30 secs]
>> > > 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark:
>> > 7660656K(7929856K)]
>> > > 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06,
>> real=3.77
>> > > secs]
>> > > 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
>> > > 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100
>> secs]
>> > > [Times: user=0.21 sys=0.03, real=0.46 secs]
>> > > 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
>> > > 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230
>> secs]
>> > > [Times: user=0.24 sys=0.06, real=0.35 secs]
>> > > 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
>> > > 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530
>> secs]
>> > > [Times: user=0.23 sys=0.03, real=0.63 secs]
>> > > 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
>> > > 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840
>> secs]
>> > > [Times: user=0.24 sys=0.02, real=0.32 secs]
>> > > 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
>> > > 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480
>> secs]
>> > > [Times: user=0.31 sys=0.03, real=0.49 secs]
>> > > 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
>> > > 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380
>> secs]
>> > > [Times: user=0.21 sys=0.06, real=0.34 secs]
>> > > 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
>> > > 235745K->235745K(235968K), 0.3339710
>> > secs][CMS2013-10-07T01:25:29.750+0900:
>> > > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
>> > > real=88.77 secs]
>> > > (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810
>> > secs]
>> > > 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)]
>> > icms_dc=9
>> > > , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
>> > > 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark:
>> > 2882954K(7929856K)]
>> > > 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00,
>> real=0.09
>> > > secs]
>> > > Heap
>> > > par new generation total 235968K, used 197697K [0x0000000606e00000,
>> > > 0x0000000616e00000, 0x0000000616e00000)
>> > > eden space 209792K, 94% used [0x0000000606e00000,
>> 0x0000000612f10718,
>> > > 0x0000000613ae0000)
>> > > from space 26176K, 0% used [0x0000000615470000,
>> 0x0000000615470000,
>> > > 0x0000000616e00000)
>> > > to space 26176K, 0% used [0x0000000613ae0000,
>> 0x0000000613ae0000,
>> > > 0x0000000615470000)
>> > > concurrent mark-sweep generation total 7929856K, used 2882954K
>> > > [0x0000000616e00000, 0x00000007fae00000, 0x00000007fae00000)
>> > > concurrent-mark-sweep perm gen total 53340K, used 31960K
>> > > [0x00000007fae00000, 0x00000007fe217000, 0x0000000800000000)
>> > >
>> > > What is wrong here? please give me some suggestions.
>> > >
>> > >
>> > > Sincerely,
>> > > Prakash
>> > >
>> >
>>
>
>
Re: You Are Dead Exception due to promotion failure
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Swapping and Java simply don't go well together. You need to ensure that
the committed memory is smaller than the available memory. Also see
http://hbase.apache.org/book.html#perf.os.swap
I haven't looked closely at your GC output but even if CMS was kicking as
early as it's supposed to, the fact that you are swapping might just screw
up everything.
J-D
On Mon, Oct 7, 2013 at 3:13 PM, prakash kadel <pr...@gmail.com>wrote:
> BTW,
> if i disable the swap at all. What will happen in the above situation?
> currently it starts swapping at 90%
>
> Sincerely
>
>
> On Tue, Oct 8, 2013 at 7:09 AM, prakash kadel <prakash.kadel@gmail.com
> >wrote:
>
> > thanks,
> >
> > yup, it seems so. I have 48 gb memory. i see it swaps at that point.
> >
> > btw, why is the CMS not kicking in early? do you have any idea?
> >
> > sincerely
> >
> >
> >
> > On Tue, Oct 8, 2013 at 3:00 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> This line:
> >>
> >> [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> >> real=88.77
> >> secs]
> >>
> >> Is suspicious. Are you swapping?
> >>
> >> J-D
> >>
> >>
> >> On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel <prakash.kadel@gmail.com
> >> >wrote:
> >>
> >> > Also,
> >> > why is the CMS not kicking in early, i have set XX:+
> >> > UseCMSInitiatingOccupancyOnly???
> >> >
> >> > Sincerely,
> >> > Prakash
> >> >
> >> >
> >> > On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <
> prakash.kadel@gmail.com
> >> > >wrote:
> >> >
> >> > > Hello,
> >> > > I am getting this YADE all the time
> >> > >
> >> > > HBASE_HEAPSIZE=8000
> >> > >
> >> > > Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
> >> > > -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode
> >> -XX:+UseParNewGC
> >> > > -XX:CMSInitiatingOccupancyFraction=50
> >> -XX:+UseCMSInitiatingOccupancyOnly
> >> > > -XX:NewSize=256m -XX:MaxNewSize=256m
> >> > >
> >> > > it seems there is promotion failure and the CMS take too long
> >> > >
> >> > > 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
> >> > > 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100
> >> secs]
> >> > > [Times: user=0.27 sys=0.01, real=0.33 secs]
> >> > > 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
> >> > > 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700
> >> secs]
> >> > > [Times: user=0.26 sys=0.02, real=0.19 secs]
> >> > > 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
> >> > > 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570
> >> secs]
> >> > > [Times: user=0.25 sys=0.02, real=0.24 secs]
> >> > > 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
> >> > > 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130
> >> secs]
> >> > > [Times: user=0.20 sys=0.03, real=0.25 secs]
> >> > > 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
> >> > > 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280
> >> secs]
> >> > > [Times: user=0.23 sys=0.04, real=0.30 secs]
> >> > > 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark:
> >> > 7660656K(7929856K)]
> >> > > 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06,
> >> real=3.77
> >> > > secs]
> >> > > 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
> >> > > 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100
> >> secs]
> >> > > [Times: user=0.21 sys=0.03, real=0.46 secs]
> >> > > 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
> >> > > 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230
> >> secs]
> >> > > [Times: user=0.24 sys=0.06, real=0.35 secs]
> >> > > 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
> >> > > 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530
> >> secs]
> >> > > [Times: user=0.23 sys=0.03, real=0.63 secs]
> >> > > 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
> >> > > 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840
> >> secs]
> >> > > [Times: user=0.24 sys=0.02, real=0.32 secs]
> >> > > 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
> >> > > 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480
> >> secs]
> >> > > [Times: user=0.31 sys=0.03, real=0.49 secs]
> >> > > 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
> >> > > 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380
> >> secs]
> >> > > [Times: user=0.21 sys=0.06, real=0.34 secs]
> >> > > 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
> >> > > 235745K->235745K(235968K), 0.3339710
> >> > secs][CMS2013-10-07T01:25:29.750+0900:
> >> > > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30
> sys=3.74,
> >> > > real=88.77 secs]
> >> > > (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810
> >> > secs]
> >> > > 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)]
> >> > icms_dc=9
> >> > > , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
> >> > > 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark:
> >> > 2882954K(7929856K)]
> >> > > 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00,
> >> real=0.09
> >> > > secs]
> >> > > Heap
> >> > > par new generation total 235968K, used 197697K
> [0x0000000606e00000,
> >> > > 0x0000000616e00000, 0x0000000616e00000)
> >> > > eden space 209792K, 94% used [0x0000000606e00000,
> >> 0x0000000612f10718,
> >> > > 0x0000000613ae0000)
> >> > > from space 26176K, 0% used [0x0000000615470000,
> >> 0x0000000615470000,
> >> > > 0x0000000616e00000)
> >> > > to space 26176K, 0% used [0x0000000613ae0000,
> >> 0x0000000613ae0000,
> >> > > 0x0000000615470000)
> >> > > concurrent mark-sweep generation total 7929856K, used 2882954K
> >> > > [0x0000000616e00000, 0x00000007fae00000, 0x00000007fae00000)
> >> > > concurrent-mark-sweep perm gen total 53340K, used 31960K
> >> > > [0x00000007fae00000, 0x00000007fe217000, 0x0000000800000000)
> >> > >
> >> > > What is wrong here? please give me some suggestions.
> >> > >
> >> > >
> >> > > Sincerely,
> >> > > Prakash
> >> > >
> >> >
> >>
> >
> >
>
Re: You Are Dead Exception due to promotion failure
Posted by prakash kadel <pr...@gmail.com>.
BTW,
if i disable the swap at all. What will happen in the above situation?
currently it starts swapping at 90%
Sincerely
On Tue, Oct 8, 2013 at 7:09 AM, prakash kadel <pr...@gmail.com>wrote:
> thanks,
>
> yup, it seems so. I have 48 gb memory. i see it swaps at that point.
>
> btw, why is the CMS not kicking in early? do you have any idea?
>
> sincerely
>
>
>
> On Tue, Oct 8, 2013 at 3:00 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> This line:
>>
>> [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
>> real=88.77
>> secs]
>>
>> Is suspicious. Are you swapping?
>>
>> J-D
>>
>>
>> On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel <prakash.kadel@gmail.com
>> >wrote:
>>
>> > Also,
>> > why is the CMS not kicking in early, i have set XX:+
>> > UseCMSInitiatingOccupancyOnly???
>> >
>> > Sincerely,
>> > Prakash
>> >
>> >
>> > On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <prakash.kadel@gmail.com
>> > >wrote:
>> >
>> > > Hello,
>> > > I am getting this YADE all the time
>> > >
>> > > HBASE_HEAPSIZE=8000
>> > >
>> > > Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
>> > > -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode
>> -XX:+UseParNewGC
>> > > -XX:CMSInitiatingOccupancyFraction=50
>> -XX:+UseCMSInitiatingOccupancyOnly
>> > > -XX:NewSize=256m -XX:MaxNewSize=256m
>> > >
>> > > it seems there is promotion failure and the CMS take too long
>> > >
>> > > 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
>> > > 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100
>> secs]
>> > > [Times: user=0.27 sys=0.01, real=0.33 secs]
>> > > 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
>> > > 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700
>> secs]
>> > > [Times: user=0.26 sys=0.02, real=0.19 secs]
>> > > 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
>> > > 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570
>> secs]
>> > > [Times: user=0.25 sys=0.02, real=0.24 secs]
>> > > 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
>> > > 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130
>> secs]
>> > > [Times: user=0.20 sys=0.03, real=0.25 secs]
>> > > 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
>> > > 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280
>> secs]
>> > > [Times: user=0.23 sys=0.04, real=0.30 secs]
>> > > 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark:
>> > 7660656K(7929856K)]
>> > > 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06,
>> real=3.77
>> > > secs]
>> > > 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
>> > > 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100
>> secs]
>> > > [Times: user=0.21 sys=0.03, real=0.46 secs]
>> > > 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
>> > > 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230
>> secs]
>> > > [Times: user=0.24 sys=0.06, real=0.35 secs]
>> > > 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
>> > > 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530
>> secs]
>> > > [Times: user=0.23 sys=0.03, real=0.63 secs]
>> > > 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
>> > > 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840
>> secs]
>> > > [Times: user=0.24 sys=0.02, real=0.32 secs]
>> > > 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
>> > > 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480
>> secs]
>> > > [Times: user=0.31 sys=0.03, real=0.49 secs]
>> > > 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
>> > > 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380
>> secs]
>> > > [Times: user=0.21 sys=0.06, real=0.34 secs]
>> > > 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
>> > > 235745K->235745K(235968K), 0.3339710
>> > secs][CMS2013-10-07T01:25:29.750+0900:
>> > > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
>> > > real=88.77 secs]
>> > > (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810
>> > secs]
>> > > 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)]
>> > icms_dc=9
>> > > , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
>> > > 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark:
>> > 2882954K(7929856K)]
>> > > 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00,
>> real=0.09
>> > > secs]
>> > > Heap
>> > > par new generation total 235968K, used 197697K [0x0000000606e00000,
>> > > 0x0000000616e00000, 0x0000000616e00000)
>> > > eden space 209792K, 94% used [0x0000000606e00000,
>> 0x0000000612f10718,
>> > > 0x0000000613ae0000)
>> > > from space 26176K, 0% used [0x0000000615470000,
>> 0x0000000615470000,
>> > > 0x0000000616e00000)
>> > > to space 26176K, 0% used [0x0000000613ae0000,
>> 0x0000000613ae0000,
>> > > 0x0000000615470000)
>> > > concurrent mark-sweep generation total 7929856K, used 2882954K
>> > > [0x0000000616e00000, 0x00000007fae00000, 0x00000007fae00000)
>> > > concurrent-mark-sweep perm gen total 53340K, used 31960K
>> > > [0x00000007fae00000, 0x00000007fe217000, 0x0000000800000000)
>> > >
>> > > What is wrong here? please give me some suggestions.
>> > >
>> > >
>> > > Sincerely,
>> > > Prakash
>> > >
>> >
>>
>
>
Re: You Are Dead Exception due to promotion failure
Posted by prakash kadel <pr...@gmail.com>.
thanks,
yup, it seems so. I have 48 gb memory. i see it swaps at that point.
btw, why is the CMS not kicking in early? do you have any idea?
sincerely
On Tue, Oct 8, 2013 at 3:00 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> This line:
>
> [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> real=88.77
> secs]
>
> Is suspicious. Are you swapping?
>
> J-D
>
>
> On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel <prakash.kadel@gmail.com
> >wrote:
>
> > Also,
> > why is the CMS not kicking in early, i have set XX:+
> > UseCMSInitiatingOccupancyOnly???
> >
> > Sincerely,
> > Prakash
> >
> >
> > On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <prakash.kadel@gmail.com
> > >wrote:
> >
> > > Hello,
> > > I am getting this YADE all the time
> > >
> > > HBASE_HEAPSIZE=8000
> > >
> > > Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
> > > -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode
> -XX:+UseParNewGC
> > > -XX:CMSInitiatingOccupancyFraction=50
> -XX:+UseCMSInitiatingOccupancyOnly
> > > -XX:NewSize=256m -XX:MaxNewSize=256m
> > >
> > > it seems there is promotion failure and the CMS take too long
> > >
> > > 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
> > > 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100
> secs]
> > > [Times: user=0.27 sys=0.01, real=0.33 secs]
> > > 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
> > > 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700
> secs]
> > > [Times: user=0.26 sys=0.02, real=0.19 secs]
> > > 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
> > > 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570
> secs]
> > > [Times: user=0.25 sys=0.02, real=0.24 secs]
> > > 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
> > > 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130
> secs]
> > > [Times: user=0.20 sys=0.03, real=0.25 secs]
> > > 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
> > > 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280
> secs]
> > > [Times: user=0.23 sys=0.04, real=0.30 secs]
> > > 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark:
> > 7660656K(7929856K)]
> > > 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06,
> real=3.77
> > > secs]
> > > 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
> > > 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100
> secs]
> > > [Times: user=0.21 sys=0.03, real=0.46 secs]
> > > 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
> > > 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230
> secs]
> > > [Times: user=0.24 sys=0.06, real=0.35 secs]
> > > 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
> > > 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530
> secs]
> > > [Times: user=0.23 sys=0.03, real=0.63 secs]
> > > 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
> > > 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840
> secs]
> > > [Times: user=0.24 sys=0.02, real=0.32 secs]
> > > 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
> > > 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480
> secs]
> > > [Times: user=0.31 sys=0.03, real=0.49 secs]
> > > 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
> > > 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380
> secs]
> > > [Times: user=0.21 sys=0.06, real=0.34 secs]
> > > 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
> > > 235745K->235745K(235968K), 0.3339710
> > secs][CMS2013-10-07T01:25:29.750+0900:
> > > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> > > real=88.77 secs]
> > > (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810
> > secs]
> > > 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)]
> > icms_dc=9
> > > , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
> > > 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark:
> > 2882954K(7929856K)]
> > > 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00,
> real=0.09
> > > secs]
> > > Heap
> > > par new generation total 235968K, used 197697K [0x0000000606e00000,
> > > 0x0000000616e00000, 0x0000000616e00000)
> > > eden space 209792K, 94% used [0x0000000606e00000,
> 0x0000000612f10718,
> > > 0x0000000613ae0000)
> > > from space 26176K, 0% used [0x0000000615470000, 0x0000000615470000,
> > > 0x0000000616e00000)
> > > to space 26176K, 0% used [0x0000000613ae0000, 0x0000000613ae0000,
> > > 0x0000000615470000)
> > > concurrent mark-sweep generation total 7929856K, used 2882954K
> > > [0x0000000616e00000, 0x00000007fae00000, 0x00000007fae00000)
> > > concurrent-mark-sweep perm gen total 53340K, used 31960K
> > > [0x00000007fae00000, 0x00000007fe217000, 0x0000000800000000)
> > >
> > > What is wrong here? please give me some suggestions.
> > >
> > >
> > > Sincerely,
> > > Prakash
> > >
> >
>
Re: You Are Dead Exception due to promotion failure
Posted by Jean-Daniel Cryans <jd...@apache.org>.
It reads that it spent 89 seconds doing a CMS concurrent mark, but really
just spent 14 seconds of user CPU and 4 seconds of system CPU doing it.
Where are the other 70 seconds? It's often just swapping, and less likely
it can also be CPU starvation.
J-D
On Fri, Nov 1, 2013 at 1:40 AM, Asaf Mesika <as...@gmail.com> wrote:
> Can you please explain why is this suspicious?
>
> On Monday, October 7, 2013, Jean-Daniel Cryans wrote:
>
> > This line:
> >
> > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> > real=88.77
> > secs]
> >
> > Is suspicious. Are you swapping?
> >
> > J-D
> >
> >
> > On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel <prakash.kadel@gmail.com
> <javascript:;>
> > >wrote:
> >
> > > Also,
> > > why is the CMS not kicking in early, i have set XX:+
> > > UseCMSInitiatingOccupancyOnly???
> > >
> > > Sincerely,
> > > Prakash
> > >
> > >
> > > On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <
> prakash.kadel@gmail.com
> > > >wrote:
> > >
> > > > Hello,
> > > > I am getting this YADE all the time
> > > >
> > > > HBASE_HEAPSIZE=8000
> > > >
> > > > Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
> > > > -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode
> > -XX:+UseParNewGC
> > > > -XX:CMSInitiatingOccupancyFraction=50
> > -XX:+UseCMSInitiatingOccupancyOnly
> > > > -XX:NewSize=256m -XX:MaxNewSize=256m
> > > >
> > > > it seems there is promotion failure and the CMS take too long
> > > >
> > > > 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
> > > > 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100
> > secs]
> > > > [Times: user=0.27 sys=0.01, real=0.33 secs]
> > > > 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
> > > > 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700
> > secs]
> > > > [Times: user=0.26 sys=0.02, real=0.19 secs]
> > > > 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
> > > > 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570
> > secs]
> > > > [Times: user=0.25 sys=0.02, real=0.24 secs]
> > > > 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
> > > > 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130
> > secs]
> > > > [Times: user=0.20 sys=0.03, real=0.25 secs]
> > > > 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
> > > > 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280
> > secs]
> > > > [Times: user=0.23 sys=0.04, real=0.30 secs]
> > > > 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark:
> > > 7660656K(7929856K)]
> > > > 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06,
> > real=3.77
> > > > secs]
> > > > 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
> > > > 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100
> > secs]
> > > > [Times: user=0.21 sys=0.03, real=0.46 secs]
> > > > 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
> > > > 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230
> > secs]
> > > > [Times: user=0.24 sys=0.06, real=0.35 secs]
> > > > 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
> > > > 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530
> > secs]
> > > > [Times: user=0.23 sys=0.03, real=0.63 secs]
> > > > 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
> > > > 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840
> > secs]
> > > > [Times: user=0.24 sys=0.02, real=0.32 secs]
> > > > 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
> > > > 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480
> > secs]
> > > > [Times: user=0.31 sys=0.03, real=0.49 secs]
> > > > 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
> > > > 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380
> > secs]
> > > > [Times: user=0.21 sys=0.06, real=0.34 secs]
> > > > 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
> > > > 235745K->235745K(235968K), 0.3339710
> > > secs][CMS2013-10-07T01:25:29.750+0900:
> > > > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30
> sys=3.74,
> > > > real=88.77 secs]
> > > > (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810
> > > secs]
> > > > 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)]
> > > icms_dc=9
> > > > , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
> > > > 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark:
> > > 2882954K(7929856K)]
> > > > 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00,
> > real=0.09
> > > > secs]
> > > > Heap
> > > > par new generation total 235968K, used 197697K
> [0x0000000606e00000,
> > > > 0x0000000616e00000, 0x0000000616e00000)
> > > > eden space 209792K, 94% used [0x0000000606e00000,
> > 0x0000000612f10718,
> > > > 0x0000000613ae0000)
> > > > from space 26176K, 0% used [0x0000000615470000,
> 0x0000000615470000,
> > > >
>
Re: You Are Dead Exception due to promotion failure
Posted by Asaf Mesika <as...@gmail.com>.
Can you please explain why is this suspicious?
On Monday, October 7, 2013, Jean-Daniel Cryans wrote:
> This line:
>
> [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> real=88.77
> secs]
>
> Is suspicious. Are you swapping?
>
> J-D
>
>
> On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel <prakash.kadel@gmail.com<javascript:;>
> >wrote:
>
> > Also,
> > why is the CMS not kicking in early, i have set XX:+
> > UseCMSInitiatingOccupancyOnly???
> >
> > Sincerely,
> > Prakash
> >
> >
> > On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <prakash.kadel@gmail.com
> > >wrote:
> >
> > > Hello,
> > > I am getting this YADE all the time
> > >
> > > HBASE_HEAPSIZE=8000
> > >
> > > Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
> > > -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode
> -XX:+UseParNewGC
> > > -XX:CMSInitiatingOccupancyFraction=50
> -XX:+UseCMSInitiatingOccupancyOnly
> > > -XX:NewSize=256m -XX:MaxNewSize=256m
> > >
> > > it seems there is promotion failure and the CMS take too long
> > >
> > > 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
> > > 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100
> secs]
> > > [Times: user=0.27 sys=0.01, real=0.33 secs]
> > > 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
> > > 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700
> secs]
> > > [Times: user=0.26 sys=0.02, real=0.19 secs]
> > > 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
> > > 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570
> secs]
> > > [Times: user=0.25 sys=0.02, real=0.24 secs]
> > > 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
> > > 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130
> secs]
> > > [Times: user=0.20 sys=0.03, real=0.25 secs]
> > > 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
> > > 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280
> secs]
> > > [Times: user=0.23 sys=0.04, real=0.30 secs]
> > > 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark:
> > 7660656K(7929856K)]
> > > 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06,
> real=3.77
> > > secs]
> > > 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
> > > 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100
> secs]
> > > [Times: user=0.21 sys=0.03, real=0.46 secs]
> > > 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
> > > 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230
> secs]
> > > [Times: user=0.24 sys=0.06, real=0.35 secs]
> > > 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
> > > 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530
> secs]
> > > [Times: user=0.23 sys=0.03, real=0.63 secs]
> > > 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
> > > 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840
> secs]
> > > [Times: user=0.24 sys=0.02, real=0.32 secs]
> > > 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
> > > 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480
> secs]
> > > [Times: user=0.31 sys=0.03, real=0.49 secs]
> > > 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
> > > 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380
> secs]
> > > [Times: user=0.21 sys=0.06, real=0.34 secs]
> > > 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
> > > 235745K->235745K(235968K), 0.3339710
> > secs][CMS2013-10-07T01:25:29.750+0900:
> > > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> > > real=88.77 secs]
> > > (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810
> > secs]
> > > 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)]
> > icms_dc=9
> > > , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
> > > 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark:
> > 2882954K(7929856K)]
> > > 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00,
> real=0.09
> > > secs]
> > > Heap
> > > par new generation total 235968K, used 197697K [0x0000000606e00000,
> > > 0x0000000616e00000, 0x0000000616e00000)
> > > eden space 209792K, 94% used [0x0000000606e00000,
> 0x0000000612f10718,
> > > 0x0000000613ae0000)
> > > from space 26176K, 0% used [0x0000000615470000, 0x0000000615470000,
> > >
Re: You Are Dead Exception due to promotion failure
Posted by Jean-Daniel Cryans <jd...@apache.org>.
This line:
[CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
real=88.77
secs]
Is suspicious. Are you swapping?
J-D
On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel <pr...@gmail.com>wrote:
> Also,
> why is the CMS not kicking in early, i have set XX:+
> UseCMSInitiatingOccupancyOnly???
>
> Sincerely,
> Prakash
>
>
> On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <prakash.kadel@gmail.com
> >wrote:
>
> > Hello,
> > I am getting this YADE all the time
> >
> > HBASE_HEAPSIZE=8000
> >
> > Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
> > -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode -XX:+UseParNewGC
> > -XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly
> > -XX:NewSize=256m -XX:MaxNewSize=256m
> >
> > it seems there is promotion failure and the CMS take too long
> >
> > 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
> > 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100 secs]
> > [Times: user=0.27 sys=0.01, real=0.33 secs]
> > 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
> > 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700 secs]
> > [Times: user=0.26 sys=0.02, real=0.19 secs]
> > 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
> > 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570 secs]
> > [Times: user=0.25 sys=0.02, real=0.24 secs]
> > 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
> > 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130 secs]
> > [Times: user=0.20 sys=0.03, real=0.25 secs]
> > 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
> > 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280 secs]
> > [Times: user=0.23 sys=0.04, real=0.30 secs]
> > 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark:
> 7660656K(7929856K)]
> > 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06, real=3.77
> > secs]
> > 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
> > 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100 secs]
> > [Times: user=0.21 sys=0.03, real=0.46 secs]
> > 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
> > 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230 secs]
> > [Times: user=0.24 sys=0.06, real=0.35 secs]
> > 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
> > 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530 secs]
> > [Times: user=0.23 sys=0.03, real=0.63 secs]
> > 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
> > 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840 secs]
> > [Times: user=0.24 sys=0.02, real=0.32 secs]
> > 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
> > 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480 secs]
> > [Times: user=0.31 sys=0.03, real=0.49 secs]
> > 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
> > 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380 secs]
> > [Times: user=0.21 sys=0.06, real=0.34 secs]
> > 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
> > 235745K->235745K(235968K), 0.3339710
> secs][CMS2013-10-07T01:25:29.750+0900:
> > [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> > real=88.77 secs]
> > (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810
> secs]
> > 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)]
> icms_dc=9
> > , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
> > 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark:
> 2882954K(7929856K)]
> > 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00, real=0.09
> > secs]
> > Heap
> > par new generation total 235968K, used 197697K [0x0000000606e00000,
> > 0x0000000616e00000, 0x0000000616e00000)
> > eden space 209792K, 94% used [0x0000000606e00000, 0x0000000612f10718,
> > 0x0000000613ae0000)
> > from space 26176K, 0% used [0x0000000615470000, 0x0000000615470000,
> > 0x0000000616e00000)
> > to space 26176K, 0% used [0x0000000613ae0000, 0x0000000613ae0000,
> > 0x0000000615470000)
> > concurrent mark-sweep generation total 7929856K, used 2882954K
> > [0x0000000616e00000, 0x00000007fae00000, 0x00000007fae00000)
> > concurrent-mark-sweep perm gen total 53340K, used 31960K
> > [0x00000007fae00000, 0x00000007fe217000, 0x0000000800000000)
> >
> > What is wrong here? please give me some suggestions.
> >
> >
> > Sincerely,
> > Prakash
> >
>
Re: You Are Dead Exception due to promotion failure
Posted by prakash kadel <pr...@gmail.com>.
Also,
why is the CMS not kicking in early, i have set XX:+
UseCMSInitiatingOccupancyOnly???
Sincerely,
Prakash
On Tue, Oct 8, 2013 at 12:32 AM, prakash kadel <pr...@gmail.com>wrote:
> Hello,
> I am getting this YADE all the time
>
> HBASE_HEAPSIZE=8000
>
> Settings: -ea -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=200
> -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSIncrementalMode -XX:+UseParNewGC
> -XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:NewSize=256m -XX:MaxNewSize=256m
>
> it seems there is promotion failure and the CMS take too long
>
> 2013-10-07T01:22:55.784+0900: [GC [ParNew: 235968K->26176K(235968K),
> 0.3219980 secs] 7709485K->7538063K(8165824K) icms_dc=0 , 0.3221100 secs]
> [Times: user=0.27 sys=0.01, real=0.33 secs]
> 2013-10-07T01:23:07.361+0900: [GC [ParNew: 235842K->26176K(235968K),
> 0.1899680 secs] 7747729K->7578713K(8165824K) icms_dc=0 , 0.1900700 secs]
> [Times: user=0.26 sys=0.02, real=0.19 secs]
> 2013-10-07T01:23:20.154+0900: [GC [ParNew: 235803K->26176K(235968K),
> 0.2428200 secs] 7788341K->7615284K(8165824K) icms_dc=0 , 0.2429570 secs]
> [Times: user=0.25 sys=0.02, real=0.24 secs]
> 2013-10-07T01:23:34.594+0900: [GC [ParNew: 235889K->26176K(235968K),
> 0.2440980 secs] 7824998K->7651179K(8165824K) icms_dc=0 , 0.2442130 secs]
> [Times: user=0.20 sys=0.03, real=0.25 secs]
> 2013-10-07T01:23:47.666+0900: [GC [ParNew: 235906K->26176K(235968K),
> 0.2998100 secs] 7860909K->7686832K(8165824K) icms_dc=3 , 0.3020280 secs]
> [Times: user=0.23 sys=0.04, real=0.30 secs]
> 2013-10-07T01:23:57.216+0900: [GC [1 CMS-initial-mark: 7660656K(7929856K)]
> 7788778K(8165824K), 3.7665320 secs] [Times: user=0.07 sys=0.06, real=3.77
> secs]
> 2013-10-07T01:24:05.508+0900: [GC [ParNew: 235811K->26176K(235968K),
> 0.4632860 secs] 7896468K->7721167K(8165824K) icms_dc=3 , 0.4634100 secs]
> [Times: user=0.21 sys=0.03, real=0.46 secs]
> 2013-10-07T01:24:19.889+0900: [GC [ParNew: 235812K->26176K(235968K),
> 0.3531980 secs] 7930804K->7755633K(8165824K) icms_dc=3 , 0.3533230 secs]
> [Times: user=0.24 sys=0.06, real=0.35 secs]
> 2013-10-07T01:24:32.832+0900: [GC [ParNew: 235968K->26176K(235968K),
> 0.6298370 secs] 7965425K->7790643K(8165824K) icms_dc=3 , 0.6299530 secs]
> [Times: user=0.23 sys=0.03, real=0.63 secs]
> 2013-10-07T01:24:43.629+0900: [GC [ParNew: 235800K->26176K(235968K),
> 0.3190580 secs] 8000268K->7825555K(8165824K) icms_dc=3 , 0.3191840 secs]
> [Times: user=0.24 sys=0.02, real=0.32 secs]
> 2013-10-07T01:24:56.005+0900: [GC [ParNew: 235848K->26176K(235968K),
> 0.4839400 secs] 8035228K->7860300K(8165824K) icms_dc=3 , 0.4840480 secs]
> [Times: user=0.31 sys=0.03, real=0.49 secs]
> 2013-10-07T01:25:07.282+0900: [GC [ParNew: 235750K->26176K(235968K),
> 0.3423250 secs] 8069875K->7895852K(8165824K) icms_dc=9 , 0.3424380 secs]
> [Times: user=0.21 sys=0.06, real=0.34 secs]
> 2013-10-07T01:25:19.853+0900: [GC [ParNew (promotion failed):
> 235745K->235745K(235968K), 0.3339710 secs][CMS2013-10-07T01:25:29.750+0900:
> [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74,
> real=88.77 secs]
> (concurrent mode failure): 7899125K->2882954K(7929856K), 42.8279810 secs]
> 8105422K->2882954K(8165824K), [CMS Perm : 31956K->31861K(53340K)] icms_dc=9
> , 43.1621090 secs] [Times: user=10.40 sys=1.89, real=43.16 secs]
> 2013-10-07T01:26:08.288+0900: [GC [1 CMS-initial-mark: 2882954K(7929856K)]
> 2978434K(8165824K), 0.0965830 secs] [Times: user=0.04 sys=0.00, real=0.09
> secs]
> Heap
> par new generation total 235968K, used 197697K [0x0000000606e00000,
> 0x0000000616e00000, 0x0000000616e00000)
> eden space 209792K, 94% used [0x0000000606e00000, 0x0000000612f10718,
> 0x0000000613ae0000)
> from space 26176K, 0% used [0x0000000615470000, 0x0000000615470000,
> 0x0000000616e00000)
> to space 26176K, 0% used [0x0000000613ae0000, 0x0000000613ae0000,
> 0x0000000615470000)
> concurrent mark-sweep generation total 7929856K, used 2882954K
> [0x0000000616e00000, 0x00000007fae00000, 0x00000007fae00000)
> concurrent-mark-sweep perm gen total 53340K, used 31960K
> [0x00000007fae00000, 0x00000007fe217000, 0x0000000800000000)
>
> What is wrong here? please give me some suggestions.
>
>
> Sincerely,
> Prakash
>