You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Xingxing Di (JIRA)" <ji...@apache.org> on 2016/08/29 09:36:20 UTC
[jira] [Created] (HBASE-16514) RegionServer shutdown by some
unknown reason.
Xingxing Di created HBASE-16514:
-----------------------------------
Summary: RegionServer shutdown by some unknown reason.
Key: HBASE-16514
URL: https://issues.apache.org/jira/browse/HBASE-16514
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.1.2
Environment: apache hadoop 2.4.1, hbase 1.1.2
Reporter: Xingxing Di
The regionserver process was shutdown by some reason, the regionserver log is :
2016-08-14 22:16:53,139 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 9022ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=9122ms
2016-08-14 22:17:03,121 WARN [main] util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=olap3.data.lq
2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_09-icedtea
2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre
2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/data/sysdir/hbase-1.1.2/conf:/usr/lib/jvm/java-1.7.0-openjdk.x86_64/lib/tools.jar:/data/sysdir/hbase-1.1.2:/data/sysdir/hbase-1.1.2/lib/activation-1.1.jar:/data/sysdir/hbase-1.1.2/lib/aopalliance-1.0.jar:/data/sysdir/hbase-1.1.2/lib/apacheds-i18n-2.0.0-M15.jar:/data/sysdir/hbase-1.1.2/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/data/sysdir/hbase-1.1.2/lib/api-asn1-api-1.0.0-M20.jar:/data/sysdir/hbase-1.1.2/lib/api-util-1.0.0-M20.jar:/data/sysdir/hbase-1.1.2/lib/asm-3.1.jar:/data/sysdir/hbase-1.1.2/lib/avro-1.7.4.jar:/data/sysdir/hbase-1.1.2/lib/commons-beanutils-1.7.0.jar:/data/sysdir/hbase-1.1.2/lib/commons-beanutils-core-1.8.0.jar:/data/sysdir/hbase-1.1.2/lib/commons-cli-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-codec-1.9.jar:/data/sysdir/hbase-1.1.2/lib/commons-collections-3.2.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-compress-1.4.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-configuration-1.6.jar:/data/sysdir/hbase-1.1.2/lib/commons-daemon-1.0.13.jar:/data/sysdir/hbase-1.1.2/lib/commons-digester-1.8.jar:/data/sysdir/hbase-1.1.2/lib/commons-el-1.0.jar:/data/sysdir/hbase-1.1.2/lib/commons-httpclient-3.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-io-2.4.jar:/data/sysdir/hbase-1.1.2/lib/commons-lang-2.6.jar:/data/sysdir/hbase-1.1.2/lib/commons-logging-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-math-2.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-math3-3.1.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-net-3.1.jar:/data/sysdir/hbase-1.1.2/lib/disruptor-3.3.0.jar:/data/sysdir/hbase-1.1.2/lib/findbugs-annotations-1.3.9-1.jar:/data/sysdir/hbase-1.1.2/lib/guava-12.0.1.jar:/data/sysdir/hbase-1.1.2/lib/guice-3.0.jar:/data/sysdir/hbase-1.1.2/lib/guice-servlet-3.0.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-annotations-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-auth-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-client-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-hdfs-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-app-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-core-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-api-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-client-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-server-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hbase-annotations-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-annotations-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-client-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-examples-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-hadoop2-compat-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-hadoop-compat-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-it-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-it-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-prefix-tree-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-procedure-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-protocol-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-resource-bundle-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-rest-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-server-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-server-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-shell-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-thrift-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/htrace-core-3.1.0-incubating.jar:/data/sysdir/hbase-1.1.2/lib/httpclient-4.2.5.jar:/data/sysdir/hbase-1.1.2/lib/httpcore-4.1.3.jar:/data/sysdir/hbase-1.1.2/lib/jackson-core-asl-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-jaxrs-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-mapper-asl-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-xc-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jamon-runtime-2.3.1.jar:/data/sysdir/hbase-1.1.2/lib/jasper-compiler-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/javax.inject-1.jar:/data/sysdir/hbase-1.1.2/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hbase-1.1.2/lib/jaxb-api-2.2.2.jar:/data/sysdir/hbase-1.1.2/lib/jaxb-impl-2.2.3-1.jar:/data/sysdir/hbase-1.1.2/lib/jcodings-1.0.8.jar:/data/sysdir/hbase-1.1.2/lib/jersey-client-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-core-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-guice-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-json-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-server-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jets3t-0.9.0.jar:/data/sysdir/hbase-1.1.2/lib/jettison-1.3.3.jar:/data/sysdir/hbase-1.1.2/lib/jetty-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/jetty-sslengine-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/jetty-util-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/joni-2.1.2.jar:/data/sysdir/hbase-1.1.2/lib/jruby-complete-1.6.8.jar:/data/sysdir/hbase-1.1.2/lib/jsch-0.1.42.jar:/data/sysdir/hbase-1.1.2/lib/jsp-2.1-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/jsp-api-2.1-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/jsr305-1.3.9.jar:/data/sysdir/hbase-1.1.2/lib/junit-4.11.jar:/data/sysdir/hbase-1.1.2/lib/leveldbjni-all-1.8.jar:/data/sysdir/hbase-1.1.2/lib/libthrift-0.9.0.jar:/data/sysdir/hbase-1.1.2/lib/log4j-1.2.17.jar:/data/sysdir/hbase-1.1.2/lib/metrics-core-2.2.0.jar:/data/sysdir/hbase-1.1.2/lib/netty-3.2.4.Final.jar:/data/sysdir/hbase-1.1.2/lib/netty-all-4.0.23.Final.jar:/data/sysdir/hbase-1.1.2/lib/paranamer-2.3.jar:/data/sysdir/hbase-1.1.2/lib/protobuf-java-2.5.0.jar:/data/sysdir/hbase-1.1.2/lib/servlet-api-2.5-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/servlet-api-2.5.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-api-1.7.7.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/hbase-1.1.2/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hbase-1.1.2/lib/spymemcached-2.11.6.jar:/data/sysdir/hbase-1.1.2/lib/xmlenc-0.52.jar:/data/sysdir/hbase-1.1.2/lib/xz-1.0.jar:/data/sysdir/hbase-1.1.2/lib/zookeeper-3.4.6.jar:/data/sysdir/hadoop-2.4.1-hbase/etc/hadoop:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-net-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jets3t-0.9.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/httpclient-4.2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-collections-3.2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jettison-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-digester-1.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/activation-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/xmlenc-0.52.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/netty-3.6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/stax-api-1.0-2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/hadoop-auth-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/hadoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/httpcore-4.2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/paranamer-2.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsch-0.1.42.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-configuration-1.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/mockito-all-1.8.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-math3-3.1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/junit-4.8.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-common-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-common-2.4.1-path-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/hadoop-nfs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/hadoop-hdfs-2.4.1-patch-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/hadoop-hdfs-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/hadoop-hdfs-nfs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/javax.inject-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jline-0.9.94.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jettison-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/activation-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/aopalliance-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/guice-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jersey-client-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/javax.inject-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hadoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/junit-4.10.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/contrib/capacity-scheduler/*.jar
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/data/sysdir/hadoop-2.4.1-hbase/lib/native
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.13.6
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/data/home/hbase
2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/data/sysdir/hbase-1.1.2/conf
2016-08-14 22:17:03,180 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.201.52:2181,192.168.201.53:2181,192.168.201.63:2181,192.168.201.64:2181,192.168.201.65:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@38987d26
2016-08-14 22:17:03,206 INFO [main-SendThread(192.168.201.65:2181)] zookeeper.ClientCnxn: Opening socket connection to server 192.168.201.65/192.168.201.65:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-14 22:17:03,228 INFO [main-SendThread(192.168.201.65:2181)] zookeeper.ClientCnxn: Socket connection established to 192.168.201.65/192.168.201.65:2181, initiating session
2016-08-14 22:17:03,234 INFO [main-SendThread(192.168.201.65:2181)] zookeeper.ClientCnxn: Session establishment complete on server 192.168.201.65/192.168.201.65:2181, sessionid = 0x556683846220a79, negotiated timeout = 30000
Then i looked into zk logs, i found these logs :
8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x0 zxid 0xf73ca5859 createSession 30000
8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x1 zxid 0xf73ca585a delete '/olapHbase/rs/olap3.data.lq%2C16020%2C1470799848293
And i looked master logs as below :
2016-08-14 22:16:43,942 INFO [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving table KYLIN_F5O605Z3QZ state from null to ENABLING
2016-08-14 22:16:43,952 INFO [ProcedureExecutorThread-0] master.AssignmentManager: Bulk assigning 1 region(s) across 4 server(s), round-robin=true
2016-08-14 22:16:43,953 INFO [olap1.data.lq,16000,1469605579716-GeneralBulkAssigner-2] master.AssignmentManager: Assigning 1 region(s) to olap3.data.lq,16020,1470799848293
2016-08-14 22:16:43,958 INFO [olap1.data.lq,16000,1469605579716-GeneralBulkAssigner-2] master.RegionStates: Transition {6e207f0307d4213d6c0e195115df1b8d state=OFFLINE, ts=1471184203953, server=null} to {6e207f0307d4213d6c0e195115df1b8d state=PENDING_OPEN, ts=1471184203958, server=olap3.data.lq,16020,1470799848293}
2016-08-14 22:16:43,985 INFO [AM.ZK.Worker-pool2-t4001] master.RegionStates: Transition {6e207f0307d4213d6c0e195115df1b8d state=PENDING_OPEN, ts=1471184203958, server=olap3.data.lq,16020,1470799848293} to {6e207f0307d4213d6c0e195115df1b8d state=OPENING, ts=1471184203985, server=olap3.data.lq,16020,1470799848293}
2016-08-14 22:17:03,238 INFO [main-EventThread] zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [olap3.data.lq,16020,1470799848293]
2016-08-14 22:17:03,240 WARN [MASTER_SERVER_OPERATIONS-olap1:16000-2] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
2016-08-14 22:17:04,008 INFO [ProcedureExecutorThread-0] master.AssignmentManager: Bulk assigning done
2016-08-14 22:17:04,011 INFO [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving table KYLIN_F5O605Z3QZ state from ENABLING to ENABLED
2016-08-14 22:17:04,251 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] handler.ServerShutdownHandler: Splitting logs for olap3.data.lq,16020,1470799848293 before assignment; region count=2033
2016-08-14 22:17:04,261 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] master.SplitLogManager: dead splitlog workers [olap3.data.lq,16020,1470799848293]
2016-08-14 22:17:04,263 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] master.SplitLogManager: started splitting 2 logs in [hdfs://olapHbaseCluster:8020/hbase/WALs/olap3.data.lq,16020,1470799848293-splitting] for [olap3.data.lq,16020,1470799848293]
2016-08-14 22:17:04,268 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C1470799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.1471182516751 acquired by olap1.data.lq,16020,1470799983323
2016-08-14 22:17:04,277 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C1470799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.1471180125256 acquired by olap2.data.lq,16020,1470800075996
2016-08-14 22:17:04,288 WARN [olap1.data.lq,16000,1469605579716_ChoreService_1] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
....
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)