You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Dima Spivak (JIRA)" <ji...@apache.org> on 2016/08/29 21:24:20 UTC
[jira] [Resolved] (HBASE-16514) RegionServer shutdown by some
unknown reason.
[ https://issues.apache.org/jira/browse/HBASE-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dima Spivak resolved HBASE-16514.
---------------------------------
Resolution: Invalid
Hi [~dixingxing@yeah.net],
Sorry to hear that HBase is misbehaving, but the issue tracker is only used for tracking bugs and new features. For help with your issue, please subscribe to and then email the HBase user list as described on [this page|https://hbase.apache.org/mail-lists.html].
All the best,
Dima
> RegionServer shutdown by some unknown reason.
> ---------------------------------------------
>
> Key: HBASE-16514
> URL: https://issues.apache.org/jira/browse/HBASE-16514
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 1.1.2
> Environment: apache hadoop 2.4.1, hbase 1.1.2
> Reporter: Xingxing Di
>
> The regionserver process was shutdown by some reason.
> I looked into regionserver's log , it looks like detected jvm pause about 9122ms (zk session timout is 40s), and after 10 seconds it start another process to delete regionserver's ephemeral node in zk, and regionserver process was gone, here is the detail log:
> {quote}
> 2016-08-14 22:16:53,139 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 9022ms
> GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=9122ms
> 2016-08-14 22:17:03,121 WARN [main] util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
> 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=olap3.data.lq
> 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_09-icedtea
> 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
> 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre
> 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/data/sysdir/hbase-1.1.2/conf:/usr/lib/jvm/java-1.7.0-openjdk.x86_64/lib/tools.jar:/data/sysdir/hbase-1.1.2:/data/sysdir/hbase-1.1.2/lib/activation-1.1.jar:/data/sysdir/hbase-1.1.2/lib/aopalliance-1.0.jar:/data/sysdir/hbase-1.1.2/lib/apacheds-i18n-2.0.0-M15.jar:/data/sysdir/hbase-1.1.2/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/data/sysdir/hbase-1.1.2/lib/api-asn1-api-1.0.0-M20.jar:/data/sysdir/hbase-1.1.2/lib/api-util-1.0.0-M20.jar:/data/sysdir/hbase-1.1.2/lib/asm-3.1.jar:/data/sysdir/hbase-1.1.2/lib/avro-1.7.4.jar:/data/sysdir/hbase-1.1.2/lib/commons-beanutils-1.7.0.jar:/data/sysdir/hbase-1.1.2/lib/commons-beanutils-core-1.8.0.jar:/data/sysdir/hbase-1.1.2/lib/commons-cli-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-codec-1.9.jar:/data/sysdir/hbase-1.1.2/lib/commons-collections-3.2.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-compress-1.4.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-configuration-1.6.jar:/data/sysdir/hbase-1.1.2/lib/commons-daemon-1.0.13.jar:/data/sysdir/hbase-1.1.2/lib/commons-digester-1.8.jar:/data/sysdir/hbase-1.1.2/lib/commons-el-1.0.jar:/data/sysdir/hbase-1.1.2/lib/commons-httpclient-3.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-io-2.4.jar:/data/sysdir/hbase-1.1.2/lib/commons-lang-2.6.jar:/data/sysdir/hbase-1.1.2/lib/commons-logging-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-math-2.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-math3-3.1.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-net-3.1.jar:/data/sysdir/hbase-1.1.2/lib/disruptor-3.3.0.jar:/data/sysdir/hbase-1.1.2/lib/findbugs-annotations-1.3.9-1.jar:/data/sysdir/hbase-1.1.2/lib/guava-12.0.1.jar:/data/sysdir/hbase-1.1.2/lib/guice-3.0.jar:/data/sysdir/hbase-1.1.2/lib/guice-servlet-3.0.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-annotations-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-auth-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-client-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-hdfs-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-app-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-core-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-api-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-client-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-server-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hbase-annotations-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-annotations-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-client-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-examples-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-hadoop2-compat-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-hadoop-compat-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-it-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-it-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-prefix-tree-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-procedure-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-protocol-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-resource-bundle-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-rest-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-server-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-server-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-shell-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-thrift-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/htrace-core-3.1.0-incubating.jar:/data/sysdir/hbase-1.1.2/lib/httpclient-4.2.5.jar:/data/sysdir/hbase-1.1.2/lib/httpcore-4.1.3.jar:/data/sysdir/hbase-1.1.2/lib/jackson-core-asl-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-jaxrs-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-mapper-asl-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-xc-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jamon-runtime-2.3.1.jar:/data/sysdir/hbase-1.1.2/lib/jasper-compiler-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/javax.inject-1.jar:/data/sysdir/hbase-1.1.2/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hbase-1.1.2/lib/jaxb-api-2.2.2.jar:/data/sysdir/hbase-1.1.2/lib/jaxb-impl-2.2.3-1.jar:/data/sysdir/hbase-1.1.2/lib/jcodings-1.0.8.jar:/data/sysdir/hbase-1.1.2/lib/jersey-client-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-core-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-guice-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-json-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-server-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jets3t-0.9.0.jar:/data/sysdir/hbase-1.1.2/lib/jettison-1.3.3.jar:/data/sysdir/hbase-1.1.2/lib/jetty-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/jetty-sslengine-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/jetty-util-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/joni-2.1.2.jar:/data/sysdir/hbase-1.1.2/lib/jruby-complete-1.6.8.jar:/data/sysdir/hbase-1.1.2/lib/jsch-0.1.42.jar:/data/sysdir/hbase-1.1.2/lib/jsp-2.1-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/jsp-api-2.1-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/jsr305-1.3.9.jar:/data/sysdir/hbase-1.1.2/lib/junit-4.11.jar:/data/sysdir/hbase-1.1.2/lib/leveldbjni-all-1.8.jar:/data/sysdir/hbase-1.1.2/lib/libthrift-0.9.0.jar:/data/sysdir/hbase-1.1.2/lib/log4j-1.2.17.jar:/data/sysdir/hbase-1.1.2/lib/metrics-core-2.2.0.jar:/data/sysdir/hbase-1.1.2/lib/netty-3.2.4.Final.jar:/data/sysdir/hbase-1.1.2/lib/netty-all-4.0.23.Final.jar:/data/sysdir/hbase-1.1.2/lib/paranamer-2.3.jar:/data/sysdir/hbase-1.1.2/lib/protobuf-java-2.5.0.jar:/data/sysdir/hbase-1.1.2/lib/servlet-api-2.5-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/servlet-api-2.5.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-api-1.7.7.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/hbase-1.1.2/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hbase-1.1.2/lib/spymemcached-2.11.6.jar:/data/sysdir/hbase-1.1.2/lib/xmlenc-0.52.jar:/data/sysdir/hbase-1.1.2/lib/xz-1.0.jar:/data/sysdir/hbase-1.1.2/lib/zookeeper-3.4.6.jar:/data/sysdir/hadoop-2.4.1-hbase/etc/hadoop:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-net-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jets3t-0.9.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/httpclient-4.2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-collections-3.2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jettison-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-digester-1.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/activation-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/xmlenc-0.52.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/netty-3.6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/stax-api-1.0-2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/hadoop-auth-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/hadoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/httpcore-4.2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/paranamer-2.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsch-0.1.42.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-configuration-1.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/mockito-all-1.8.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-math3-3.1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/junit-4.8.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-common-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-common-2.4.1-path-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-nfs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/hadoop-hdfs-2.4.1-patch-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/hadoop-hdfs-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/hadoop-hdfs-nfs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/javax.inject-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jline-0.9.94.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jettison-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/activation-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/aopalliance-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/guice-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-client-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/javax.inject-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hadoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/junit-4.10.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/contrib/capacity-scheduler/*.jar
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/data/sysdir/hadoop-2.4.1-hbase/lib/native
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.13.6
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/data/home/hbase
> 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/data/sysdir/hbase-1.1.2/conf
> 2016-08-14 22:17:03,180 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.201.52:2181,192.168.201.53:2181,192.168.201.63:2181,192.168.201.64:2181,192.168.201.65:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@38987d26
> 2016-08-14 22:17:03,206 INFO [main-SendThread(192.168.201.65:2181)] zookeeper.ClientCnxn: Opening socket connection to server 192.168.201.65/192.168.201.65:2181. Will not attempt to authenticate using SASL (unknown error)
> 2016-08-14 22:17:03,228 INFO [main-SendThread(192.168.201.65:2181)] zookeeper.ClientCnxn: Socket connection established to 192.168.201.65/192.168.201.65:2181, initiating session
> 2016-08-14 22:17:03,234 INFO [main-SendThread(192.168.201.65:2181)] zookeeper.ClientCnxn: Session establishment complete on server 192.168.201.65/192.168.201.65:2181, sessionid = 0x556683846220a79, negotiated timeout = 30000
> {quote}
> Then i looked into zk's log, i found these :
> {quote}
> 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x0 zxid 0xf73ca5859 createSession 30000
> 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x1 zxid 0xf73ca585a delete '/olapHbase/rs/olap3.data.lq%2C16020%2C1470799848293
> {quote}
> And i looked master logs as below :
> {quote}
> 2016-08-14 22:16:43,942 INFO [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving table KYLIN_F5O605Z3QZ state from null to ENABLING
> 2016-08-14 22:16:43,952 INFO [ProcedureExecutorThread-0] master.AssignmentManager: Bulk assigning 1 region(s) across 4 server(s), round-robin=true
> 2016-08-14 22:16:43,953 INFO [olap1.data.lq,16000,1469605579716-GeneralBulkAssigner-2] master.AssignmentManager: Assigning 1 region(s) to olap3.data.lq,16020,1470799848293
> 2016-08-14 22:16:43,958 INFO [olap1.data.lq,16000,1469605579716-GeneralBulkAssigner-2] master.RegionStates: Transition {6e207f0307d4213d6c0e195115df1b8d state=OFFLINE, ts=1471184203953, server=null} to {6e207f0307d4213d6c0e195115df1b8d state=PENDING_OPEN, ts=1471184203958, server=olap3.data.lq,16020,1470799848293}
> 2016-08-14 22:16:43,985 INFO [AM.ZK.Worker-pool2-t4001] master.RegionStates: Transition {6e207f0307d4213d6c0e195115df1b8d state=PENDING_OPEN, ts=1471184203958, server=olap3.data.lq,16020,1470799848293} to {6e207f0307d4213d6c0e195115df1b8d state=OPENING, ts=1471184203985, server=olap3.data.lq,16020,1470799848293}
> 2016-08-14 22:17:03,238 INFO [main-EventThread] zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [olap3.data.lq,16020,1470799848293]
> 2016-08-14 22:17:03,240 WARN [MASTER_SERVER_OPERATIONS-olap1:16000-2] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
> 2016-08-14 22:17:04,008 INFO [ProcedureExecutorThread-0] master.AssignmentManager: Bulk assigning done
> 2016-08-14 22:17:04,011 INFO [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving table KYLIN_F5O605Z3QZ state from ENABLING to ENABLED
> 2016-08-14 22:17:04,251 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] handler.ServerShutdownHandler: Splitting logs for olap3.data.lq,16020,1470799848293 before assignment; region count=2033
> 2016-08-14 22:17:04,261 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] master.SplitLogManager: dead splitlog workers [olap3.data.lq,16020,1470799848293]
> 2016-08-14 22:17:04,263 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] master.SplitLogManager: started splitting 2 logs in [hdfs://olapHbaseCluster:8020/hbase/WALs/olap3.data.lq,16020,1470799848293-splitting] for [olap3.data.lq,16020,1470799848293]
> 2016-08-14 22:17:04,268 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C1470799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.1471182516751 acquired by olap1.data.lq,16020,1470799983323
> 2016-08-14 22:17:04,277 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C1470799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.1471180125256 acquired by olap2.data.lq,16020,1470800075996
> 2016-08-14 22:17:04,288 WARN [olap1.data.lq,16000,1469605579716_ChoreService_1] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
> ....
> {quote}
> I'm not falimliar with hbase , may be it's not a bug , but i can not figure out why this happend. I will be grateful if somebody can help me.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)