You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Reid Chan (JIRA)" <ji...@apache.org> on 2018/08/14 10:45:00 UTC
[jira] [Resolved] (HBASE-21049) RUNNABLE:SERVER_CRASH_START;
ServerCrashProcedure
[ https://issues.apache.org/jira/browse/HBASE-21049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reid Chan resolved HBASE-21049.
-------------------------------
Resolution: Not A Bug
> RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> -------------------------------------------------
>
> Key: HBASE-21049
> URL: https://issues.apache.org/jira/browse/HBASE-21049
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.1.0
> Reporter: apcahephoenix
> Priority: Major
>
> Power off and restart(Hadoop and HBase), Master is initializing - Hbase ServerManager: but crash processing already in progress
> command jps, HMaster and HRegionServer is live
>
> LOG:
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 64091
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 64091
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> 2018-08-14 17:25:00,173 INFO [main] master.HMaster: STARTING service HMaster
> 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: HBase 2.1.0
> 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: Source code repository revision=4531d1c947a25b28a9a994b60c791a112c12a2b4
> 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: Compiled by hbase on Wed Aug 1 11:25:59 2018
> 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: From source with checksum fc32566f7e030ff71458fbf6dc77bce9
> 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine: hbase.tmp.dir: /tmp/hbase-root
> 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine: hbase.rootdir: hdfs://192.168.101.114:9000/hbase
> 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine: hbase.cluster.distributed: true
> 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine: hbase.zookeeper.quorum: 192.168.101.114:2181
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:PATH=/opt/apache-phoenix-5.0.0-HBase-2.0-bin/bin:/opt/hbase-2.1.0/bin:/opt/hadoop-2.8.4/bin:/opt/jdk1.8.0_172/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:HADOOP_CONF_DIR=/opt/hadoop-2.8.4/etc/hadoop
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:HISTCONTROL=ignoredups
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:JAVA_LIBRARY_PATH=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native:
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVER_OPTS= -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:HBASE_CONF_DIR=/opt/hbase-2.1.0/conf
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:HDFS_DATANODE_SECURE_USER=root
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:MAIL=/var/spool/mail/root
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:PHOENIX_HOME=/opt/apache-phoenix-5.0.0-HBase-2.0-bin
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=:/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native:
> 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: env:LOGNAME=root
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS=
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:PWD=/opt/hbase-2.1.0/bin
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HADOOP_PREFIX=/opt/hadoop-2.8.4
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HADOOP_INSTALL=/opt/hadoop-2.8.4
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:LESSOPEN=||/usr/bin/lesspipe.sh %s
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:SELINUX_USE_CURRENT_RANGE=
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:YARN_CONF_DIR=/opt/hadoop-2.8.4/etc/hadoop
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HADOOP_YARN_HOME=/opt/hadoop-2.8.4
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS= -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HADOOP_HOME=/opt/hadoop-2.8.4
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_NICENESS=0
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_OPTS= -XX:+UseConcMarkSweepGC -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070 -Dhbase.log.dir=/opt/hbase-2.1.0/logs -Dhbase.log.file=hbase-root-master-hbase-114.log -Dhbase.home.dir=/opt/hbase-2.1.0 -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native: -Dhbase.security.logger=INFO,RFAS
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HDFS_DATANODE_USER=root
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: 1:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:SHLVL=4
> 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-root-master-hbase-114.log
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HISTSIZE=1000
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/opt/jdk1.8.0_172
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:TERM=xterm
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:LANG=zh_CN.UTF-8
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:XDG_SESSION_ID=1
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:SELINUX_LEVEL_REQUESTED=
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HADOOP_LIBEXEC_DIR=/opt/hadoop-2.8.4/libexec
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:SELINUX_ROLE_REQUESTED=
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HADOOP_HDFS_HOME=/opt/hadoop-2.8.4
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HADOOP_MAPRED_HOME=/opt/hadoop-2.8.4
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HADOOP_COMMON_HOME=/opt/hadoop-2.8.4
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HADOOP_OPTS=-Djava.library.path=/opt/hadoop-2.8.4/lib:/opt/hadoop-2.8.4/lib/native
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=root
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/tmp/hbase-root-master.znode
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:SSH_TTY=/dev/pts/0
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:SSH_CLIENT=192.168.98.129 35604 22
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-root-master-hbase-114
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/opt/hbase-2.1.0/logs
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:USER=root
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: dparty/commons-logging-1.2.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/log4j-1.2.17.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/phoenix-5.0.0-HBase-2.0-server.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/phoenix-core-5.0.0-HBase-2.0.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/opt/hadoop-2.8.4/etc/hadoop:/opt/hadoop-2.8.4/share/hadoop/common/lib/*:/opt/hadoop-2.8.4/share/hadoop/common/*:/opt/hadoop-2.8.4/share/hadoop/hdfs:/opt/hadoop-2.8.4/share/hadoop/hdfs/lib/*:/opt/hadoop-2.8.4/share/hadoop/hdfs/*:/opt/hadoop-2.8.4/share/hadoop/yarn/lib/*:/opt/hadoop-2.8.4/share/hadoop/yarn/*:/opt/hadoop-2.8.4/share/hadoop/mapreduce/lib/*:/opt/hadoop-2.8.4/share/hadoop/mapreduce/*:/opt/hadoop-2.8.4/contrib/capacity-scheduler/*.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HDFS_NAMENODE_USER=root
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:SSH_CONNECTION=192.168.98.129 35604 192.168.101.114 22
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HBASE_AUTOSTART_FILE=/tmp/hbase-root-master.autostart
> 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:HOSTNAME=hbase-114
> 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: env:HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-2.8.4/lib/native
> 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: env:XDG_RUNTIME_DIR=/run/user/0
> 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: env:HDFS_SECONDARYNAMENODE_USER=root
> 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS=
> 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/opt/hbase-2.1.0
> 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: env:HOME=/root
> 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4
> 2018-08-14 17:25:00,521 INFO [main] util.ServerCommandLine: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.172-b11
> 2018-08-14 17:25:00,521 INFO [main] util.ServerCommandLine: vmInputArguments=[-Dproc_master, -XX:OnOutOfMemoryError=kill -9 %p, -XX:+UseConcMarkSweepGC, -Xdebug, -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070, -Dhbase.log.dir=/opt/hbase-2.1.0/logs, -Dhbase.log.file=hbase-root-master-hbase-114.log, -Dhbase.home.dir=/opt/hbase-2.1.0, -Dhbase.id.str=root, -Dhbase.root.logger=INFO,RFA, -Djava.library.path=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native:, -Dhbase.security.logger=INFO,RFAS]
> 2018-08-14 17:25:00,886 INFO [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> 2018-08-14 17:25:01,258 INFO [main] regionserver.RSRpcServices: master/hbase-114:16000 server-side Connection retries=3
> 2018-08-14 17:25:01,278 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30
> 2018-08-14 17:25:01,280 INFO [main] ipc.RpcExecutor: Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=300, handlerCount=20
> 2018-08-14 17:25:01,280 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=300, handlerCount=3
> 2018-08-14 17:25:01,418 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService
> 2018-08-14 17:25:01,632 INFO [main] ipc.NettyRpcServer: Bind to /192.168.101.114:16000
> 2018-08-14 17:25:01,688 INFO [main] hfile.CacheConfig: Allocating onheap LruBlockCache size=1.55 GB, blockSize=64 KB
> 2018-08-14 17:25:01,694 INFO [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache\{blockCount=0, currentSize=1.16 MB, freeSize=1.55 GB, maxSize=1.55 GB, heapSize=1.16 MB, minSize=1.47 GB, minFactor=0.95, multiSize=752.80 MB, multiFactor=0.5, singleSize=376.40 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
> 2018-08-14 17:25:01,695 INFO [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache\{blockCount=0, currentSize=1.16 MB, freeSize=1.55 GB, maxSize=1.55 GB, heapSize=1.16 MB, minSize=1.47 GB, minFactor=0.95, multiSize=752.80 MB, multiFactor=0.5, singleSize=376.40 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
> 2018-08-14 17:25:02,160 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2018-08-14 17:25:02,163 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2018-08-14 17:25:02,233 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=master:16000 connecting to ZooKeeper ensemble=192.168.101.114:2181
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=hbase-114
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_172
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/opt/jdk1.8.0_172/jre
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: o-2.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.4-tests.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.4.jar:/opt/hadoop-2.8.4/contrib/capacity-scheduler/*.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native:
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
> 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/hbase-2.1.0/bin
> 2018-08-14 17:25:02,240 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.101.114:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@4ae2e781
> 2018-08-14 17:25:02,256 INFO [main-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Opening socket connection to server 192.168.101.114/192.168.101.114:2181. Will not attempt to authenticate using SASL (unknown error)
> 2018-08-14 17:25:02,264 INFO [main-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Socket connection established to 192.168.101.114/192.168.101.114:2181, initiating session
> 2018-08-14 17:25:02,282 INFO [main-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Session establishment complete on server 192.168.101.114/192.168.101.114:2181, sessionid = 0x10004dac1970000, negotiated timeout = 40000
> 2018-08-14 17:25:02,352 INFO [main] util.log: Logging initialized @2552ms
> 2018-08-14 17:25:02,413 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined
> 2018-08-14 17:25:02,426 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
> 2018-08-14 17:25:02,426 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter)
> 2018-08-14 17:25:02,428 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master
> 2018-08-14 17:25:02,428 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
> 2018-08-14 17:25:02,428 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
> 2018-08-14 17:25:02,455 INFO [main] http.HttpServer: Jetty bound to port 16010
> 2018-08-14 17:25:02,456 INFO [main] server.Server: jetty-9.3.19.v20170502
> 2018-08-14 17:25:02,489 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@49232c6f\{/logs,file:///opt/hbase-2.1.0/logs/,AVAILABLE}
> 2018-08-14 17:25:02,490 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@279126f5\{/static,file:///opt/hbase-2.1.0/hbase-webapps/static/,AVAILABLE}
> 2018-08-14 17:25:02,582 INFO [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@537b3b2e\{/,file:///opt/hbase-2.1.0/hbase-webapps/master/,AVAILABLE}{file:/opt/hbase-2.1.0/hbase-webapps/master}
> 2018-08-14 17:25:02,587 INFO [main] server.AbstractConnector: Started ServerConnector@550c973e\{HTTP/1.1,[http/1.1]}{0.0.0.0:16010}
> 2018-08-14 17:25:02,587 INFO [main] server.Server: Started @2787ms
> 2018-08-14 17:25:02,590 INFO [main] master.HMaster: hbase.rootdir=hdfs://192.168.101.114:9000/hbase, hbase.cluster.distributed=true
> 2018-08-14 17:25:02,606 INFO [Thread-14] master.HMaster: Adding backup master ZNode /hbase/backup-masters/hbase-114,16000,1534238700547
> 2018-08-14 17:25:02,685 INFO [Thread-14] master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/hbase-114,16000,1534238700547 from backup master directory
> 2018-08-14 17:25:02,691 INFO [Thread-14] master.ActiveMasterManager: Registered as active master=hbase-114,16000,1534238700547
> 2018-08-14 17:25:02,697 INFO [Thread-14] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 713, initial count 0
> 2018-08-14 17:25:02,698 INFO [Thread-14] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 792, initial count 0
> 2018-08-14 17:25:02,992 INFO [Thread-14] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2018-08-14 17:25:03,001 INFO [Thread-14] coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 rescan nodes
> 2018-08-14 17:25:03,094 INFO [Thread-14] zookeeper.ReadOnlyZKClient: Connect 0x66461af1 to 192.168.101.114:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
> 2018-08-14 17:25:03,100 INFO [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1] zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.101.114:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$52/1619197561@6e451c19
> 2018-08-14 17:25:03,101 INFO [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Opening socket connection to server 192.168.101.114/192.168.101.114:2181. Will not attempt to authenticate using SASL (unknown error)
> 2018-08-14 17:25:03,101 INFO [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Socket connection established to 192.168.101.114/192.168.101.114:2181, initiating session
> 2018-08-14 17:25:03,104 INFO [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Session establishment complete on server 192.168.101.114/192.168.101.114:2181, sessionid = 0x10004dac1970001, negotiated timeout = 40000
> 2018-08-14 17:25:03,145 INFO [Thread-14] procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker count=160
> 2018-08-14 17:25:03,149 INFO [Thread-14] util.FSHDFSUtils: Recover lease on dfs file hdfs://192.168.101.114:9000/hbase/MasterProcWALs/pv2-00000000000000000004.log
> 2018-08-14 17:25:03,153 INFO [Thread-14] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://192.168.101.114:9000/hbase/MasterProcWALs/pv2-00000000000000000004.log after 4ms
> 2018-08-14 17:25:03,188 WARN [Thread-14] util.CommonFSUtils: Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated.
> 2018-08-14 17:25:03,189 INFO [Thread-14] wal.WALProcedureStore: Rolled new Procedure Store WAL, id=5
> 2018-08-14 17:25:03,190 INFO [Thread-14] procedure2.ProcedureExecutor: Recovered WALProcedureStore lease in 42msec
> 2018-08-14 17:25:03,224 INFO [Thread-14] procedure2.ProcedureExecutor: Loaded WALProcedureStore in 33msec
> 2018-08-14 17:25:03,224 INFO [Thread-14] procedure2.RemoteProcedureDispatcher: Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150
> 2018-08-14 17:25:03,261 WARN [Thread-14] master.ServerManager: Expiration of hbase-116,16020,1534237430655 but server not online
> 2018-08-14 17:25:03,261 INFO [Thread-14] master.ServerManager: Processing expiration of hbase-116,16020,1534237430655 on hbase-114,16000,1534238700547
> 2018-08-14 17:25:03,481 WARN [Thread-14] master.ServerManager: Expiration of hbase-115,16020,1534237425729 but server not online
> 2018-08-14 17:25:03,481 INFO [Thread-14] master.ServerManager: Processing expiration of hbase-115,16020,1534237425729 on hbase-114,16000,1534238700547
> 2018-08-14 17:25:03,622 INFO [Thread-14] balancer.BaseLoadBalancer: slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false
> 2018-08-14 17:25:03,629 INFO [Thread-14] balancer.StochasticLoadBalancer: Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc.
> 2018-08-14 17:25:03,669 INFO [Thread-14] master.HMaster: Active/primary master=hbase-114,16000,1534238700547, sessionid=0x10004dac1970000, setting cluster-up flag (Was=false)
> 2018-08-14 17:25:03,771 INFO [PEWorker-4] procedure.ServerCrashProcedure: Start pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=hbase-115,16020,1534237425729, splitWal=true, meta=false
> 2018-08-14 17:25:03,772 INFO [Thread-14] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1534238733772
> 2018-08-14 17:25:03,774 INFO [PEWorker-3] procedure.ServerCrashProcedure: Start pid=11, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=hbase-116,16020,1534237430655, splitWal=true, meta=false
> 2018-08-14 17:25:03,775 INFO [Thread-14] cleaner.CleanerChore: Cleaner pool size is 1
> 2018-08-14 17:25:03,776 INFO [Thread-14] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=192.168.101.114:2181
> 2018-08-14 17:25:03,776 INFO [Thread-14] zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.101.114:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@46bb7405
> 2018-08-14 17:25:03,777 INFO [Thread-14-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Opening socket connection to server 192.168.101.114/192.168.101.114:2181. Will not attempt to authenticate using SASL (unknown error)
> 2018-08-14 17:25:03,777 INFO [Thread-14-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Socket connection established to 192.168.101.114/192.168.101.114:2181, initiating session
> 2018-08-14 17:25:03,777 INFO [Thread-14] cleaner.LogCleaner: Creating OldWALs cleaners with size=2
> 2018-08-14 17:25:03,780 INFO [Thread-14-SendThread(192.168.101.114:2181)] zookeeper.ClientCnxn: Session establishment complete on server 192.168.101.114/192.168.101.114:2181, sessionid = 0x10004dac1970006, negotiated timeout = 40000
> 2018-08-14 17:25:03,967 INFO [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16000] master.ServerManager: Registering regionserver=hbase-116,16020,1534238701517
> 2018-08-14 17:25:03,967 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] master.ServerManager: Registering regionserver=hbase-115,16020,1534238702258
> 2018-08-14 17:25:04,022 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [hbase-116,16020,1534238701517]
> 2018-08-14 17:25:04,023 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [hbase-115,16020,1534238702258]
> 2018-08-14 17:25:33,877 INFO [WALProcedureStoreSyncThread] wal.ProcedureWALFile: Archiving hdfs://192.168.101.114:9000/hbase/MasterProcWALs/pv2-00000000000000000004.log to hdfs://192.168.101.114:9000/hbase/oldWALs/pv2-00000000000000000004.log
> 2018-08-14 17:26:59,875 WARN [qtp1304765785-87] servlet.ServletHandler: /master-status
> org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
> at org.apache.hadoop.hbase.master.HMaster.isInMaintenanceMode(HMaster.java:2890)
> at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:277)
> at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:395)
> at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:386)
> at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:81)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:48)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1374)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)