You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by gehaijiang <ge...@aliyun.com> on 2017/03/10 05:24:02 UTC
hbase CMS gc pause serious program
CMS GC program:
2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800: 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs] 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07 sys=0.35, real=2.09 secs]2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, real=2.05 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, real=2.41 secs]
2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-abortable-preclean-start]
2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, real=2.62 secs]
2017-03-10T10:15:41.012+0800: 4555931.648: [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 sys=0.87, real=4.72 secs]
2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: 2011637K->340736K(3067136K), 2.0773980 secs] 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, real=2.07 secs]
regionserver JVM config:export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m -XX:MaxPermSize=256m -Xms96G -Xmx96G"
export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
-XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -verbose:gc
-XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
attachment： hdfs-site.xml
Re: hbase CMS gc pause serious program
Posted by Anoop John <an...@gmail.com>.
96 GB of heap size? !!
What kind of work load is this?
-Anoop-
On Fri, Mar 10, 2017 at 10:54 AM, gehaijiang <ge...@aliyun.com> wrote:
> CMS GC program:
>
> 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800:
> 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs]
> 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07
> sys=0.35, real=2.09 secs]
> 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800:
> 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs]
> 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94
> sys=0.34, real=2.05 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep:
> 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
>
> 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset:
> 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
> 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800:
> 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs]
> 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63
> sys=0.36, real=2.41 secs]
> 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark:
> 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
> 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427
> secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean:
> 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
> 2017-03-10T10:15:36.291+0800: 4555926.928:
> [CMS-concurrent-abortable-preclean-start]
> 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800:
> 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs]
> 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79
> sys=0.48, real=2.62 secs]
> 2017-03-10T10:15:41.012+0800: 4555931.648:
> [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51
> sys=0.87, real=4.72 secs]
> 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K
> (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652:
> [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew:
> 2011637K->340736K(3067136K), 2.0773980 secs]
> 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09
> sys=0.38, real=2.07 secs]
>
>
> regionserver JVM config:
>
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m
> -XX:MaxPermSize=256m -Xms96G -Xmx96G"
> export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
> -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
> -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -verbose:gc
> -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
>
>
> attachment： hdfs-site.xml
回复：hbase CMS gc pause serious program
Posted by gehaijiang <ge...@aliyun.com>.
No use bucket cache， I Can try testing environment。
hbase-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.master</name>
<value>qihe015005:60000</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://fscluster/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>10.15.5.120:2181,10.15.5.107:2181,10.15.5.55:2181,10.15.5.56:2181,10.15.2.31:2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/hadoop/data/hbase/zookeeper</value>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/hbasecluster</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/home/hadoop/data/hbase/tmp</value>
</property>
<property>
<name>hbase.fs.tmp.dir</name>
<value>/home/hadoop/data/hbase/tmp/hbase-staging</value>
</property>
<property>
<name>hbase.local.dir</name>
<value>/home/hadoop/data/hbase/local</value>
</property>
<property>
<name>hbase.master.logcleaner.ttl</name>
<value>600000</value>
</property>
<property>
<name>hbase.regionserver.logroll.period</name>
<value>3600000</value>
</property>
<property>
<name>hbase.regionserver.global.memstore.size</name>
<value>0.4</value>
</property>
<property>
<name>hbase.regionserver.global.memstore.size.lower.limit</name>
<value>0.35</value>
</property>
<property>
<name>hbase.regionserver.region.split.policy</name>
<value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value>
</property>
<property>
<name>hbase.regionserver.regionSplitLimit</name>
<value>1600</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>120000</value>
</property>
<property>
<name>hbase.normalizer.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.normalizer.period</name>
<value>3600000</value>
</property>
<property>
<name>hbase.server.thread.wakefrequency</name>
<value>10000</value>
</property>
<property>
<name>hbase.server.versionfile.writeattempts</name>
<value>3</value>
</property>
<property>
<name>hbase.hregion.memstore.flush.size</name>
<value>134217728</value>
</property>
<property>
<name>hbase.hregion.memstore.block.multiplier</name>
<value>4</value>
</property>
<property>
<name>hbase.hregion.memstore.mslab.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>6442450944</value>
</property>
<property>
<name>hbase.hregion.majorcompaction</name>
<value>604800000</value>
</property>
<property>
<name>hbase.hstore.compactionThreshold</name>
<value>5</value>
</property>
<property>
<name>hbase.hstore.flusher.count</name>
<value>8</value>
</property>
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>16</value>
</property>
<property>
<name>hbase.hstore.blockingWaitTime</name>
<value>30000</value>
</property>
<property>
<name>hbase.hstore.compaction.min</name>
<value>6</value>
</property>
<property>
<name>hbase.hstore.compaction.max</name>
<value>12</value>
</property>
<property>
<name>hbase.hstore.compaction.min.size</name>
<value>134217728</value>
</property>
<property>
<name>hbase.hstore.compaction.ratio</name>
<value>1.2F</value>
</property>
<property>
<name>hbase.regionserver.thread.compaction.throttle</name>
<value>2684354560</value>
</property>
<property>
<name>hbase.hstore.compaction.kv.max</name>
<value>100</value>
</property>
<property>
<name>hbase.storescanner.parallel.seek.enable</name>
<value>true</value>
</property>
<property>
<name>hbase.storescanner.parallel.seek.threads</name>
<value>10</value>
</property>
<property>
<name>hfile.block.cache.size</name>
<value>0.4</value>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>90000</value>
</property>
<property>
<name>hbase.server.compactchecker.interval.multiplier</name>
<value>1000</value>
</property>
<property>
<name>hbase.security.authentication</name>
<value>simple</value>
</property>
<property>
<name>hbase.regionserver.storefile.refresh.period</name>
<value>15000</value>
</property>
<property>
<name>hbase.region.replica.replication.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.replication</name>
<value>true</value>
</property>
<property>
<name>hbase.ipc.warn.response.time</name>
<value>3000</value>
</property>
<property>
<name>hbase.ipc.warn.response.size</name>
<value>10485760</value>
</property>
<property>
<name>hbase.quota.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.regionserver.handler.count</name>
<value>180</value>
</property>
<property>
<name>hbase.snapshot.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.rest.port</name>
<value>20550</value>
</property>
<property>
<name>hbase.rest.info.port</name>
<value>8085</value>
</property>
<property>
<name>>hbase.rest.readonly</name>
<value>false</value>
</property>
<property>
<name>hbase.rest.threads.min</name>
<value>2</value>
</property>
<property>
<name>hbase.thrift.minWorkerThreads</name>
<value>200</value>
</property>
<property>
<name>hbase.thrift.info.port</name>
<value>9095</value>
</property>
<property>
<name>thrift.accept-backlog</name>
<value>511</value>
</property>
</configuration>
------------------------------------------------------------------发件人：Ted Yu <yu...@gmail.com>发送时间：2017年3月10日(星期五) 21:31收件人：user <us...@hbase.apache.org>抄 送：gehaijiang <ge...@aliyun.com>主 题：Re: hbase CMS gc pause serious program
Attachment didn't go through.
Do you use bucket cache ? It would reduce GC pause.
On Mar 9, 2017, at 9:24 PM, gehaijiang <ge...@aliyun.com> wrote:
CMS GC program:
2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800: 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs] 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07 sys=0.35, real=2.09 secs]2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, real=2.05 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, real=2.41 secs]
2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-abortable-preclean-start]
2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, real=2.62 secs]
2017-03-10T10:15:41.012+0800: 4555931.648: [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 sys=0.87, real=4.72 secs]
2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: 2011637K->340736K(3067136K), 2.0773980 secs] 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, real=2.07 secs]
regionserver JVM config:export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m -XX:MaxPermSize=256m -Xms96G -Xmx96G"
export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
-XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -verbose:gc
-XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
attachment： hdfs-site.xml
Re: hbase CMS gc pause serious program
Posted by Ted Yu <yu...@gmail.com>.
Attachment didn't go through.
Do you use bucket cache ? It would reduce GC pause.
> On Mar 9, 2017, at 9:24 PM, gehaijiang <ge...@aliyun.com> wrote:
>
> CMS GC program:
>
> 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800: 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs] 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07 sys=0.35, real=2.09 secs]
> 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, real=2.05 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
> 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
> 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, real=2.41 secs]
> 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-abortable-preclean-start]
> 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, real=2.62 secs]
> 2017-03-10T10:15:41.012+0800: 4555931.648: [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 sys=0.87, real=4.72 secs]
> 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: 2011637K->340736K(3067136K), 2.0773980 secs] 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, real=2.07 secs]
>
> regionserver JVM config:
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m -XX:MaxPermSize=256m -Xms96G -Xmx96G"
> export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
> -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -verbose:gc
> -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
>
> attachment： hdfs-site.xml
回复：hbase CMS gc pause serious program
Posted by gehaijiang <ge...@aliyun.com>.
hbase 1.1.2 ， JDK8 compatibility issues ？
------------------------------------------------------------------发件人：Sean Busbey <bu...@apache.org>发送时间：2017年3月10日(星期五) 23:48收件人：user <us...@hbase.apache.org>; gehaijiang <ge...@aliyun.com>主 题：Re: hbase CMS gc pause serious program
For heap sizes larger than ~16GiB you should be using Java 8 and the
G1GC collector. You'll need to plan some time to tune G1GC for your
workload.
There are some pointers doing so in this blog post:
https://blogs.apache.org/hbase/entry/tuning_g1gc_for_your_hbase
On Thu, Mar 9, 2017 at 11:24 PM, gehaijiang <ge...@aliyun.com> wrote:
> CMS GC program:
>
> 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800:
> 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs]
> 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07
> sys=0.35, real=2.09 secs]
> 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800:
> 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs]
> 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94
> sys=0.34, real=2.05 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep:
> 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
>
> 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset:
> 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
> 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800:
> 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs]
> 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63
> sys=0.36, real=2.41 secs]
> 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark:
> 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
> 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427
> secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean:
> 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
> 2017-03-10T10:15:36.291+0800: 4555926.928:
> [CMS-concurrent-abortable-preclean-start]
> 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800:
> 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs]
> 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79
> sys=0.48, real=2.62 secs]
> 2017-03-10T10:15:41.012+0800: 4555931.648:
> [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51
> sys=0.87, real=4.72 secs]
> 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K
> (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652:
> [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew:
> 2011637K->340736K(3067136K), 2.0773980 secs]
> 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09
> sys=0.38, real=2.07 secs]
>
>
> regionserver JVM config:
>
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m
> -XX:MaxPermSize=256m -Xms96G -Xmx96G"
> export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
> -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
> -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -verbose:gc
> -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
>
>
> attachment： hdfs-site.xml
Re: hbase CMS gc pause serious program
Posted by Sean Busbey <bu...@apache.org>.
For heap sizes larger than ~16GiB you should be using Java 8 and the
G1GC collector. You'll need to plan some time to tune G1GC for your
workload.
There are some pointers doing so in this blog post:
https://blogs.apache.org/hbase/entry/tuning_g1gc_for_your_hbase
On Thu, Mar 9, 2017 at 11:24 PM, gehaijiang <ge...@aliyun.com> wrote:
> CMS GC program:
>
> 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800:
> 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs]
> 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07
> sys=0.35, real=2.09 secs]
> 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800:
> 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs]
> 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94
> sys=0.34, real=2.05 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep:
> 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
>
> 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset:
> 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
> 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800:
> 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs]
> 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63
> sys=0.36, real=2.41 secs]
> 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark:
> 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
> 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427
> secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean:
> 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
> 2017-03-10T10:15:36.291+0800: 4555926.928:
> [CMS-concurrent-abortable-preclean-start]
> 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800:
> 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs]
> 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79
> sys=0.48, real=2.62 secs]
> 2017-03-10T10:15:41.012+0800: 4555931.648:
> [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51
> sys=0.87, real=4.72 secs]
> 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K
> (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652:
> [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew:
> 2011637K->340736K(3067136K), 2.0773980 secs]
> 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09
> sys=0.38, real=2.07 secs]
>
>
> regionserver JVM config:
>
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m
> -XX:MaxPermSize=256m -Xms96G -Xmx96G"
> export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
> -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
> -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -verbose:gc
> -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
>
>
> attachment： hdfs-site.xml