You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by karthi keyan <ka...@gmail.com> on 2016/04/14 08:00:59 UTC
Region server crashes after GC pause time
Hi ,
i got this issue in HBase while at peak time handling more requests . can
any one pls guide me to resolve the Long GC pauses in hbase .
JDK-7 , JVM heap 512m
HBase 0.98.13
INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
host machine (eg GC): pause of approximately 1466ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms
INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
host machine (eg GC): pause of approximately 2304ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
host machine (eg GC): pause of approximately 2287ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
zookeeper.ClientCnxn: Client session timed out, have not heard from server
in 6819ms for sessionid 0x1540ab48b280004, closing socket connection and
attempting reconnect
INFO [SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
zookeeper.ClientCnxn: Client session timed out, have not heard from server
in 6819ms for sessionid 0x1540ab48b280005, closing socket connection and
attempting reconnect
Once after this HBase Region Server moved to Dead state.
Best,
Karthik
RE: Region server crashes after GC pause time
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Sure, here you go.
Note that a number of them are from "optimal settings" available in a number of presentations.
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC -XX:MaxTenuringThreshold=2 -XX:SurvivorRatio=8 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32768 -XX:NewSize=750m -XX:MaxNewSize=750m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UnlockExperimentalVMOptions -XX:+PrintTenuringDistribution -Xloggc:/home/vchadoop/var/logs/hbase/regionserver-gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=4M -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+LogVMOutput -XX:LogFile=/home/vchadoop/var/logs/hbase/vm.log -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=200
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, April 22, 2016 9:56 AM
To: user@hbase.apache.org
Cc: karthi keyan <ka...@gmail.com>; dingdongchao@baidu.com
Subject: Re: Region server crashes after GC pause time
Jayesh:
Is it possible for you to share the JVM parameters ?
Thanks
On Fri, Apr 22, 2016 at 7:48 AM, Thakrar, Jayesh < jthakrar@conversantmedia.com> wrote:
> Karthik,
>
> Yes, tuning can help - but the biggest help is to give "sufficient"
> memory to the regionserver.
> And "sufficient" is relative - e.g. we have a 75GB heap (our increases
> were like this - 8, 12, 16, 24, 32, 45, 60 and 75 GB) Note that we had
> started off with 45 GB RAM and are now running on 148 GB RAM servers.
>
> Now we are very stable at 75GB and with appropriate JVM tuning our GCs
> are well contained (sub-100 ms).
>
> Also, although I like G1, it did not fare well for us at 75 GB - with
> the best tuning we could get upto 20-25 second pauses (HBase 1.0).
> CMS seems to work best for us.
>
> Jayesh
>
> -----Original Message-----
> From: karthi keyan [mailto:karthi93.sankar@gmail.com]
> Sent: Friday, April 22, 2016 7:04 AM
> To: user@hbase.apache.org; dingdongchao@baidu.com
> Subject: Re: Region server crashes after GC pause time
>
> Hi Ding,
>
> I have increased the Heap to 2G , still getting out of memory exception .
> Actually i had write the data to HBase at 40K writes/sec .
> Is there any parameter to tune up , as my knowledge "-
> XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase.
> Is there any other parameter required to resolve this issue???
>
> Best,
> Karthik
>
> On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao
> <di...@baidu.com>
> wrote:
>
> > Dump the jvm heap，analysis the the heap and find which query(s) cost
> > so many memory？
> > In my ever bad case，the RS crashed for Long GC pauses because of a
> > big query on Batch Get operation.
> >
> >
> > In addition，I think you can increase the men of JVM, 512m is so
> > small for RS.
> >
> >
> >
> > 在 16/4/14 14:00， "karthi keyan" <ka...@gmail.com> 写入:
> >
> > >Hi ,
> > >
> > >i got this issue in HBase while at peak time handling more requests .
> > >can any one pls guide me to resolve the Long GC pauses in hbase .
> > >
> > >JDK-7 , JVM heap 512m
> > >
> > >HBase 0.98.13
> > >
> > >
> > > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in
> > >JVM or host machine (eg GC): pause of approximately 1466ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms INFO
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> > >host machine (eg GC): pause of approximately 2304ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms INFO
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> > >host machine (eg GC): pause of approximately 2287ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> > >
> > > INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from
> > >server in 6819ms for sessionid 0x1540ab48b280004, closing socket
> > >connection and attempting reconnect INFO
> > >[SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from
> > >server in 6819ms for sessionid 0x1540ab48b280005, closing socket
> > >connection and attempting reconnect
> > >
> > >Once after this HBase Region Server moved to Dead state.
> > >
> > >Best,
> > >Karthik
> >
> >
>
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s). Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender. If you have
> received this email in error, please immediately notify the sender via
> return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
Re: Region server crashes after GC pause time
Posted by Ted Yu <yu...@gmail.com>.
Jayesh:
Is it possible for you to share the JVM parameters ?
Thanks
On Fri, Apr 22, 2016 at 7:48 AM, Thakrar, Jayesh <
jthakrar@conversantmedia.com> wrote:
> Karthik,
>
> Yes, tuning can help - but the biggest help is to give "sufficient" memory
> to the regionserver.
> And "sufficient" is relative - e.g. we have a 75GB heap (our increases
> were like this - 8, 12, 16, 24, 32, 45, 60 and 75 GB)
> Note that we had started off with 45 GB RAM and are now running on 148 GB
> RAM servers.
>
> Now we are very stable at 75GB and with appropriate JVM tuning our GCs are
> well contained (sub-100 ms).
>
> Also, although I like G1, it did not fare well for us at 75 GB - with the
> best tuning we could get upto 20-25 second pauses (HBase 1.0).
> CMS seems to work best for us.
>
> Jayesh
>
> -----Original Message-----
> From: karthi keyan [mailto:karthi93.sankar@gmail.com]
> Sent: Friday, April 22, 2016 7:04 AM
> To: user@hbase.apache.org; dingdongchao@baidu.com
> Subject: Re: Region server crashes after GC pause time
>
> Hi Ding,
>
> I have increased the Heap to 2G , still getting out of memory exception .
> Actually i had write the data to HBase at 40K writes/sec .
> Is there any parameter to tune up , as my knowledge "-
> XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase.
> Is there any other parameter required to resolve this issue???
>
> Best,
> Karthik
>
> On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao <di...@baidu.com>
> wrote:
>
> > Dump the jvm heap，analysis the the heap and find which query(s) cost
> > so many memory？
> > In my ever bad case，the RS crashed for Long GC pauses because of a big
> > query on Batch Get operation.
> >
> >
> > In addition，I think you can increase the men of JVM, 512m is so small
> > for RS.
> >
> >
> >
> > 在 16/4/14 14:00， "karthi keyan" <ka...@gmail.com> 写入:
> >
> > >Hi ,
> > >
> > >i got this issue in HBase while at peak time handling more requests .
> > >can any one pls guide me to resolve the Long GC pauses in hbase .
> > >
> > >JDK-7 , JVM heap 512m
> > >
> > >HBase 0.98.13
> > >
> > >
> > > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM
> > >or host machine (eg GC): pause of approximately 1466ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms INFO
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host
> > >machine (eg GC): pause of approximately 2304ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms INFO
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host
> > >machine (eg GC): pause of approximately 2287ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> > >
> > > INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from
> > >server in 6819ms for sessionid 0x1540ab48b280004, closing socket
> > >connection and attempting reconnect INFO
> > >[SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from
> > >server in 6819ms for sessionid 0x1540ab48b280005, closing socket
> > >connection and attempting reconnect
> > >
> > >Once after this HBase Region Server moved to Dead state.
> > >
> > >Best,
> > >Karthik
> >
> >
>
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s). Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender. If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>
RE: Region server crashes after GC pause time
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Karthik,
Yes, tuning can help - but the biggest help is to give "sufficient" memory to the regionserver.
And "sufficient" is relative - e.g. we have a 75GB heap (our increases were like this - 8, 12, 16, 24, 32, 45, 60 and 75 GB)
Note that we had started off with 45 GB RAM and are now running on 148 GB RAM servers.
Now we are very stable at 75GB and with appropriate JVM tuning our GCs are well contained (sub-100 ms).
Also, although I like G1, it did not fare well for us at 75 GB - with the best tuning we could get upto 20-25 second pauses (HBase 1.0).
CMS seems to work best for us.
Jayesh
-----Original Message-----
From: karthi keyan [mailto:karthi93.sankar@gmail.com]
Sent: Friday, April 22, 2016 7:04 AM
To: user@hbase.apache.org; dingdongchao@baidu.com
Subject: Re: Region server crashes after GC pause time
Hi Ding,
I have increased the Heap to 2G , still getting out of memory exception .
Actually i had write the data to HBase at 40K writes/sec .
Is there any parameter to tune up , as my knowledge "- XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase.
Is there any other parameter required to resolve this issue???
Best,
Karthik
On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao <di...@baidu.com>
wrote:
> Dump the jvm heap，analysis the the heap and find which query(s) cost
> so many memory？
> In my ever bad case，the RS crashed for Long GC pauses because of a big
> query on Batch Get operation.
>
>
> In addition，I think you can increase the men of JVM, 512m is so small
> for RS.
>
>
>
> 在 16/4/14 14:00， "karthi keyan" <ka...@gmail.com> 写入:
>
> >Hi ,
> >
> >i got this issue in HBase while at peak time handling more requests .
> >can any one pls guide me to resolve the Long GC pauses in hbase .
> >
> >JDK-7 , JVM heap 512m
> >
> >HBase 0.98.13
> >
> >
> > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM
> >or host machine (eg GC): pause of approximately 1466ms GC pool
> >'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms INFO
> >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host
> >machine (eg GC): pause of approximately 2304ms GC pool
> >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms INFO
> >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host
> >machine (eg GC): pause of approximately 2287ms GC pool
> >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> >
> > INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
> >zookeeper.ClientCnxn: Client session timed out, have not heard from
> >server in 6819ms for sessionid 0x1540ab48b280004, closing socket
> >connection and attempting reconnect INFO
> >[SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
> >zookeeper.ClientCnxn: Client session timed out, have not heard from
> >server in 6819ms for sessionid 0x1540ab48b280005, closing socket
> >connection and attempting reconnect
> >
> >Once after this HBase Region Server moved to Dead state.
> >
> >Best,
> >Karthik
>
>
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
Re: Region server crashes after GC pause time
Posted by Sumit Nigam <su...@yahoo.com.INVALID>.
Hi Karthik,
As Ding said, eventually you need to watch your heap objects to understand what produces so many objects in the first place.
CMS occupancy % defaults to 70%. Default values of block cache and in-memory memstore sizes add up to around 75-80%. Which means you could get CMS to start walking your heap with just these 2. And we are not even talking hbase scan objects, etc. If even after analysis of your heap you are unable to figure a way out, then you may want to consider G1 collector (esp if you are having > 15-20G heaps).
It may also give you temporary respite by increasing zookeeper max session timeout value as that makes region server be 'alive' on zookeeper longer. The downside is that it may now take a little longer to mark DEAD a really dead server and allows zookeeper clients to now negotiate a longer time out. If that is possible or not depends on your use case really.
You could also put block cache off-heap - that is almost 40% saving of heap. But then, be very sure that you are unable to understand your heap data in region servers. You may also benefit by lowering your heap size and launching more region servers instead.
Thanks,Sumit
From: karthi keyan <ka...@gmail.com>
To: user@hbase.apache.org; dingdongchao@baidu.com
Sent: Friday, April 22, 2016 5:34 PM
Subject: Re: Region server crashes after GC pause time
Hi Ding,
I have increased the Heap to 2G , still getting out of memory exception .
Actually i had write the data to HBase at 40K writes/sec .
Is there any parameter to tune up , as my knowledge "-
XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase.
Is there any other parameter required to resolve this issue???
Best,
Karthik
On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao <di...@baidu.com>
wrote:
> Dump the jvm heap，analysis the the heap and find which query(s) cost so
> many memory？
> In my ever bad case，the RS crashed for Long GC pauses because of a big
> query on Batch Get operation.
>
>
> In addition，I think you can increase the men of JVM, 512m is so small for
> RS.
>
>
>
> 在 16/4/14 14:00， "karthi keyan" <ka...@gmail.com> 写入:
>
> >Hi ,
> >
> >i got this issue in HBase while at peak time handling more requests . can
> >any one pls guide me to resolve the Long GC pauses in hbase .
> >
> >JDK-7 , JVM heap 512m
> >
> >HBase 0.98.13
> >
> >
> > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> >host machine (eg GC): pause of approximately 1466ms
> >GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms
> > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> >host machine (eg GC): pause of approximately 2304ms
> >GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> >host machine (eg GC): pause of approximately 2287ms
> >GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> >
> > INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
> >zookeeper.ClientCnxn: Client session timed out, have not heard from server
> >in 6819ms for sessionid 0x1540ab48b280004, closing socket connection and
> >attempting reconnect
> > INFO [SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
> >zookeeper.ClientCnxn: Client session timed out, have not heard from server
> >in 6819ms for sessionid 0x1540ab48b280005, closing socket connection and
> >attempting reconnect
> >
> >Once after this HBase Region Server moved to Dead state.
> >
> >Best,
> >Karthik
>
>
Re: Region server crashes after GC pause time
Posted by karthi keyan <ka...@gmail.com>.
Hi Ding,
I have increased the Heap to 2G , still getting out of memory exception .
Actually i had write the data to HBase at 40K writes/sec .
Is there any parameter to tune up , as my knowledge "-
XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase.
Is there any other parameter required to resolve this issue???
Best,
Karthik
On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao <di...@baidu.com>
wrote:
> Dump the jvm heap，analysis the the heap and find which query(s) cost so
> many memory？
> In my ever bad case，the RS crashed for Long GC pauses because of a big
> query on Batch Get operation.
>
>
> In addition，I think you can increase the men of JVM, 512m is so small for
> RS.
>
>
>
> 在 16/4/14 14:00， "karthi keyan" <ka...@gmail.com> 写入:
>
> >Hi ,
> >
> >i got this issue in HBase while at peak time handling more requests . can
> >any one pls guide me to resolve the Long GC pauses in hbase .
> >
> >JDK-7 , JVM heap 512m
> >
> >HBase 0.98.13
> >
> >
> > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> >host machine (eg GC): pause of approximately 1466ms
> >GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms
> > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> >host machine (eg GC): pause of approximately 2304ms
> >GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
> >host machine (eg GC): pause of approximately 2287ms
> >GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> >
> > INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
> >zookeeper.ClientCnxn: Client session timed out, have not heard from server
> >in 6819ms for sessionid 0x1540ab48b280004, closing socket connection and
> >attempting reconnect
> > INFO [SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
> >zookeeper.ClientCnxn: Client session timed out, have not heard from server
> >in 6819ms for sessionid 0x1540ab48b280005, closing socket connection and
> >attempting reconnect
> >
> >Once after this HBase Region Server moved to Dead state.
> >
> >Best,
> >Karthik
>
>
Re: Region server crashes after GC pause time
Posted by "Ding,Dongchao" <di...@baidu.com>.
Dump the jvm heap，analysis the the heap and find which query(s) cost so
many memory？
In my ever bad case，the RS crashed for Long GC pauses because of a big
query on Batch Get operation.
In addition，I think you can increase the men of JVM, 512m is so small for
RS.
在 16/4/14 14:00， "karthi keyan" <ka...@gmail.com> 写入:
>Hi ,
>
>i got this issue in HBase while at peak time handling more requests . can
>any one pls guide me to resolve the Long GC pauses in hbase .
>
>JDK-7 , JVM heap 512m
>
>HBase 0.98.13
>
>
> INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
>host machine (eg GC): pause of approximately 1466ms
>GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms
> INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
>host machine (eg GC): pause of approximately 2304ms
>GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or
>host machine (eg GC): pause of approximately 2287ms
>GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
>
> INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
>zookeeper.ClientCnxn: Client session timed out, have not heard from server
>in 6819ms for sessionid 0x1540ab48b280004, closing socket connection and
>attempting reconnect
> INFO [SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
>zookeeper.ClientCnxn: Client session timed out, have not heard from server
>in 6819ms for sessionid 0x1540ab48b280005, closing socket connection and
>attempting reconnect
>
>Once after this HBase Region Server moved to Dead state.
>
>Best,
>Karthik