can not start backup master on slave node

[Invkrh <in...@gmail.com>]

Hi,

I am running a hbase cluster on Amazon EC2.

*HBase Version = 0.98.4-hadoop1*

The cluster is on fully distributed mode with one master and three slaves.

On the master:

slaves' host names are added to conf/backup-masters
$HBASE_HOME/conf is rsync'ed on cluster

The hbase is started normally, but the backup-master process is not started
on slaves node.

"$ jps" on slaves nodes does not show a HMaster process.

After checking log, I found the following:

2014-10-01 14:00:22,864 FATAL [master:ip-10-239-180-222:*60000*]
master.HMaster: Unhandled exception. Starting shutdown.
java.net.BindException: *Address already in use*
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at
org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
	at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:585)
	at java.lang.Thread.run(Thread.java:745)
2014-10-01 14:00:22,866 INFO  [master:ip-10-239-180-222:60000]
master.HMaster: Aborting
2014-10-01 14:00:22,866 DEBUG [master:ip-10-239-180-222:60000]
master.HMaster: Stopping service threads
2014-10-01 14:00:22,866 INFO  [master:ip-10-239-180-222:60000]
ipc.RpcServer: Stopping server on 60000
2014-10-01 14:00:22,866 INFO  [RpcServer.listener,port=60000] ipc.RpcServer:
RpcServer.listener,port=60000: stopping
2014-10-01 14:00:22,867 INFO  [master:ip-10-239-180-222:60000]
master.HMaster: Stopping infoServer
2014-10-01 14:00:22,868 INFO  [RpcServer.responder] ipc.RpcServer:
RpcServer.responder: stopped
2014-10-01 14:00:22,868 INFO  [RpcServer.responder] ipc.RpcServer:
RpcServer.responder: stopping
2014-10-01 14:00:22,874 INFO  [master:ip-10-239-180-222:60000]
zookeeper.ZooKeeper: Session: 0x348cc02ea6a0001 closed
2014-10-01 14:00:22,874 INFO  [master:ip-10-239-180-222:60000]
master.HMaster: HMaster main thread exiting
2014-10-01 14:00:22,874 INFO  [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2014-10-01 14:00:22,875 ERROR [main] master.HMasterCommandLine: Master
exiting
java.lang.RuntimeException: HMaster Aborted
	at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
	at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2793)

It seems like the port 60000 is used by some processes.

However, "$ netstat -apn | grep 60000" shows nothing, but
# on slave node 1
tcp        0      0 ::ffff:10.239.180.222:39082 ::ffff:10.154.231.185:60000
ESTABLISHED 10315/java 
# on slave node 2
tcp        0      0 ::ffff:10.233.58.90:40845   ::ffff:10.154.231.185:60000
ESTABLISHED 10518/java          
# on slave node 3
tcp        0      0 ::ffff:10.234.163.74:58339  ::ffff:10.154.231.185:60000
ESTABLISHED 10229/java          

where 10.154.231.185 is the master's IP, (10315, 10518, 10229) are
repectively pid of region server on slave nodes which have TCP connection on
master's 60000 port.

So far, I am sure that the port 60000 is not used, if I am not missing
anything.

Any help is highly appreciated. Thank you.

Hao



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/can-not-start-backup-master-on-slave-node-tp4064557.html
Sent from the HBase User mailing list archive at Nabble.com.