You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by kushum sharma <ku...@gmail.com> on 2011/02/18 11:29:07 UTC
HBase setup problem
Hi,
I've deployed hbase on 5 nodes cluster of amazon ec2 successfully and was
working fine.
The next day when I logon I changed the configuration file(regionservers
list,slaves, masters,dfs rootdir etc)
in hadoop and hbase /conf directory for the new IP address which is dynamic
for ec2
then the hbase master immediately stopped after startup and generated the
error message in log files, it tries to connect
to the old region server address and failed. Again I removed
both hadoop and hbase for fresh installation but it again generates the
error in
connection with an old ip address on port 60020.
I would be grateful if someone could getme out of it.
Thanks.
log of hbase-root-master
..........
2011-02-18 05:13:32,760 DEBUG
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED
event for 70236052; deleting unassigned node
2011-02-18 05:13:32,760 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x2e3842768e0003 Deleting existing unassigned node for 70236052
that is in expected state RS_ZK_REGION_OPENED
2011-02-18 05:13:32,831 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x2e3842768e0003 Successfully deleted unassigned node for
region 70236052 in expected state RS_ZK_REGION_OPENED
2011-02-18 05:13:32,831 DEBUG
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region
-ROOT-,,0.70236052 on ip-10-203-13-80.ec2.internal,60020,1298024003961
2011-02-18 05:13:52,794 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020
2011-02-18 05:14:13,811 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020
2011-02-18 05:14:34,817 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020
2011-02-18 05:14:34,817 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.net.SocketTimeoutException: 20000 millis timeout while waiting for
channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending
remote=ip-10-114-29-91.ec2.internal/10.114.29.91:60020]
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:384)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:283)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:478)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:435)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
2011-02-18 05:14:34,841 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2011-02-18 05:14:34,841 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service threads
2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: exiting
..................................
Re: HBase setup problem
Posted by Ted Dunning <td...@maprtech.com>.
You might do well to build a host file so that you can make the host names
stable over time.
On Fri, Feb 18, 2011 at 2:29 AM, kushum sharma <ku...@gmail.com> wrote:
> Hi,
> I've deployed hbase on 5 nodes cluster of amazon ec2 successfully and was
> working fine.
> The next day when I logon I changed the configuration file(regionservers
> list,slaves, masters,dfs rootdir etc)
> in hadoop and hbase /conf directory for the new IP address which is dynamic
> for ec2
> then the hbase master immediately stopped after startup and generated the
> error message in log files, it tries to connect
> to the old region server address and failed. Again I removed
> both hadoop and hbase for fresh installation but it again generates the
> error in
> connection with an old ip address on port 60020.
> I would be grateful if someone could getme out of it.
> Thanks.
>
> log of hbase-root-master
> ..........
>
> 2011-02-18 05:13:32,760 DEBUG
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED
> event for 70236052; deleting unassigned node
> 2011-02-18 05:13:32,760 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x2e3842768e0003 Deleting existing unassigned node for
> 70236052
> that is in expected state RS_ZK_REGION_OPENED
> 2011-02-18 05:13:32,831 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x2e3842768e0003 Successfully deleted unassigned node for
> region 70236052 in expected state RS_ZK_REGION_OPENED
> 2011-02-18 05:13:32,831 DEBUG
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region
> -ROOT-,,0.70236052 on ip-10-203-13-80.ec2.internal,60020,1298024003961
> 2011-02-18 05:13:52,794 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
> connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020
> 2011-02-18 05:14:13,811 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
> connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020
> 2011-02-18 05:14:34,817 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
> connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020
> 2011-02-18 05:14:34,817 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unhandled exception. Starting shutdown.
> java.net.SocketTimeoutException: 20000 millis timeout while waiting for
> channel to be ready for connect. ch :
> java.nio.channels.SocketChannel[connection-pending
> remote=ip-10-114-29-91.ec2.internal/10.114.29.91:60020]
> at
>
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
> at
>
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:384)
> at
>
> org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:283)
> at
>
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:478)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:435)
> at
>
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
> 2011-02-18 05:14:34,841 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
> 2011-02-18 05:14:34,841 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60000
> 2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60000: exiting
> 2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60000: exiting
> ..................................
>