You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hari Sreekumar <hs...@clickable.com> on 2011/05/30 18:19:31 UTC
Problem starting up HBase in pseudo distributed mode
Hi,
I am trying to set up hbase is pseudo distributed mode on one of the
machines and I am getting this error when I try to use hbase shell. If I try
the list command, it just hangs.
hbase(main):002:0> create 't1', 'f1'
ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException:
org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out
(10000ms)
HMaster Log:
2011-05-30 21:32:35,021 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE
for too long, reassigning -ROOT-,,0.70236052 to a random server
2011-05-30 21:32:35,021 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=-ROOT-,,0.70236052 state=OFFLINE, ts=1306771315023
2011-05-30 21:32:35,021 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
for region -ROOT-,,0.70236052; plan=hri=-ROOT-,,0.70236052, src=,
dest=localhost,54093,1306770984643
2011-05-30 21:32:35,021 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
-ROOT-,,0.70236052 to localhost,54093,1306770984643
2011-05-30 21:32:35,021 DEBUG org.apache.hadoop.hbase.master.ServerManager:
New connection to localhost,54093,1306770984643
2011-05-30 21:32:35,023 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
127.0.0.1:54093 could not be reached after 1 tries, giving up.
2011-05-30 21:32:35,023 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
-ROOT-,,0.70236052 to serverName=localhost,54093,1306770984643,
load=(requests=0, regions=0, usedHeap=36, maxHeap=991), trying to assign
elsewhere instead; retry=0
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /
127.0.0.1:54093 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
at
org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:541)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:901)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:730)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:710)
at
org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor.chore(AssignmentManager.java:1605)
at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy9.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
... 8 more
2011-05-30 21:32:35,025 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Unable to find a viable
location to assign region -ROOT-,,0.70236052
2011-05-30 21:33:12,788 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@3248bc64;
hsa=null
I am running it with java 1.6.0_25 on Ubuntu 11.04 desktop. Another thing I
noticed is that it is now looking for a different port once I restarted the
machine:
2011-05-30 21:47:10,858 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
-ROOT-,,0.70236052 to localhost,37283,1306772219858
2011-05-30 21:47:10,858 DEBUG org.apache.hadoop.hbase.master.ServerManager:
New connection to localhost,37283,1306772219858
2011-05-30 21:47:10,874 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
127.0.0.1:37283 could not be reached after 1 tries, giving up.
2011-05-30 21:47:10,875 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
-ROOT-,,0.70236052 to serverName=localhost,37283,1306772219858,
load=(requests=0, regions=0, usedHeap=36, maxHeap=991), trying to assign
elsewhere instead; retry=0
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /
127.0.0.1:37283 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
...
...
Any ideas? ..
Hari
Re: Problem starting up HBase in pseudo distributed mode
Posted by gen <ge...@gmail.com>.
Excellent! thanks so much mate, in Ubuntu 11.04 there's a duplicate loop back
address, removing one fixed it... thanks again!
Re: Problem starting up HBase in pseudo distributed mode
Posted by Hari Sreekumar <hs...@clickable.com>.
This got fixed when I removed the 127.0.1.1 entry in /etc/hosts and moved
the entries corresponding to it to 127.0.0.1. (
http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/18878)
hair
On Tue, May 31, 2011 at 12:43 AM, Sean Bigdatafun <sean.bigdatafun@gmail.com
> wrote:
> Hi Hari,
>
> I am experiencing the same problem as you do, I think. (My system is also
> Ubuntu 11.04).
>
>
> Please take a look at my thread and see if it is the same problem you are
> experiencing.topic "0.90.1 HMaster malfunction in pseudo-distributed mode".
>
> Hopefully this question get answered after the long weekend.
>
>
>
>
>
> On Mon, May 30, 2011 at 9:19 AM, Hari Sreekumar <hsreekumar@clickable.com
> >wrote:
>
> > Hi,
> >
> > I am trying to set up hbase is pseudo distributed mode on one of the
> > machines and I am getting this error when I try to use hbase shell. If I
> > try
> > the list command, it just hangs.
> >
> > hbase(main):002:0> create 't1', 'f1'
> >
> > ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException:
> > org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out
> > (10000ms)
> >
> >
> >
> > HMaster Log:
> >
> > 2011-05-30 21:32:35,021 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE
> > for too long, reassigning -ROOT-,,0.70236052 to a random server
> > 2011-05-30 21:32:35,021 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> > was=-ROOT-,,0.70236052 state=OFFLINE, ts=1306771315023
> > 2011-05-30 21:32:35,021 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
> > for region -ROOT-,,0.70236052; plan=hri=-ROOT-,,0.70236052, src=,
> > dest=localhost,54093,1306770984643
> > 2011-05-30 21:32:35,021 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> > -ROOT-,,0.70236052 to localhost,54093,1306770984643
> > 2011-05-30 21:32:35,021 DEBUG
> org.apache.hadoop.hbase.master.ServerManager:
> > New connection to localhost,54093,1306770984643
> > 2011-05-30 21:32:35,023 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> > 127.0.0.1:54093 could not be reached after 1 tries, giving up.
> > 2011-05-30 21:32:35,023 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
> > -ROOT-,,0.70236052 to serverName=localhost,54093,1306770984643,
> > load=(requests=0, regions=0, usedHeap=36, maxHeap=991), trying to assign
> > elsewhere instead; retry=0
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up
> > proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /
> > 127.0.0.1:54093 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
> > at
> >
> >
> org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606)
> > at
> >
> >
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:541)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:901)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:730)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:710)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor.chore(AssignmentManager.java:1605)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
> > Caused by: java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> > at
> >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> > at $Proxy9.getProtocolVersion(Unknown Source)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> > ... 8 more
> > 2011-05-30 21:32:35,025 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Unable to find a viable
> > location to assign region -ROOT-,,0.70236052
> > 2011-05-30 21:33:12,788 DEBUG
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Lookedup root region location,
> >
> >
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@3248bc64
> > ;
> > hsa=null
> >
> > I am running it with java 1.6.0_25 on Ubuntu 11.04 desktop. Another thing
> I
> > noticed is that it is now looking for a different port once I restarted
> the
> > machine:
> >
> > 2011-05-30 21:47:10,858 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> > -ROOT-,,0.70236052 to localhost,37283,1306772219858
> > 2011-05-30 21:47:10,858 DEBUG
> org.apache.hadoop.hbase.master.ServerManager:
> > New connection to localhost,37283,1306772219858
> > 2011-05-30 21:47:10,874 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> > 127.0.0.1:37283 could not be reached after 1 tries, giving up.
> > 2011-05-30 21:47:10,875 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
> > -ROOT-,,0.70236052 to serverName=localhost,37283,1306772219858,
> > load=(requests=0, regions=0, usedHeap=36, maxHeap=991), trying to assign
> > elsewhere instead; retry=0
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up
> > proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /
> > 127.0.0.1:37283 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
> > ...
> > ...
> >
> > Any ideas? ..
> > Hari
> >
>
>
>
> --
> --Sean
>
Re: Problem starting up HBase in pseudo distributed mode
Posted by Mark Kerzner <ma...@shmsoft.com>.
Guys, keep in mind that EC2 re-adds the line pointing to 127.0.1.1 on every
stop/start of the instance. That is a double gotcha. Good luck to all those who
chance upon this solution.
Re: Problem starting up HBase in pseudo distributed mode
Posted by Mark Kerzner <ma...@shmsoft.com>.
Thank you, thank you, thank you! I would never have HBase on EC2 working without
this advice.
Mark :)
Re: Problem starting up HBase in pseudo distributed mode
Posted by Gargi Nagar <gr...@gmail.com>.
Hi Hari,
For the error..
> > ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException:
> > org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out
> > (10000ms)
A very non-tech stuff, but i was facing the same issue and it got fixed on its
own when i restarted the system.
Regards,
Gargi
Re: Problem starting up HBase in pseudo distributed mode
Posted by Sean Bigdatafun <se...@gmail.com>.
Hi Hari,
I am experiencing the same problem as you do, I think. (My system is also
Ubuntu 11.04).
Please take a look at my thread and see if it is the same problem you are
experiencing.topic "0.90.1 HMaster malfunction in pseudo-distributed mode".
Hopefully this question get answered after the long weekend.
On Mon, May 30, 2011 at 9:19 AM, Hari Sreekumar <hs...@clickable.com>wrote:
> Hi,
>
> I am trying to set up hbase is pseudo distributed mode on one of the
> machines and I am getting this error when I try to use hbase shell. If I
> try
> the list command, it just hangs.
>
> hbase(main):002:0> create 't1', 'f1'
>
> ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException:
> org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out
> (10000ms)
>
>
>
> HMaster Log:
>
> 2011-05-30 21:32:35,021 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE
> for too long, reassigning -ROOT-,,0.70236052 to a random server
> 2011-05-30 21:32:35,021 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=-ROOT-,,0.70236052 state=OFFLINE, ts=1306771315023
> 2011-05-30 21:32:35,021 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
> for region -ROOT-,,0.70236052; plan=hri=-ROOT-,,0.70236052, src=,
> dest=localhost,54093,1306770984643
> 2011-05-30 21:32:35,021 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> -ROOT-,,0.70236052 to localhost,54093,1306770984643
> 2011-05-30 21:32:35,021 DEBUG org.apache.hadoop.hbase.master.ServerManager:
> New connection to localhost,54093,1306770984643
> 2011-05-30 21:32:35,023 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 127.0.0.1:54093 could not be reached after 1 tries, giving up.
> 2011-05-30 21:32:35,023 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
> -ROOT-,,0.70236052 to serverName=localhost,54093,1306770984643,
> load=(requests=0, regions=0, usedHeap=36, maxHeap=991), trying to assign
> elsewhere instead; retry=0
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
> proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /
> 127.0.0.1:54093 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
> at
>
> org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606)
> at
>
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:541)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:901)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:730)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:710)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor.chore(AssignmentManager.java:1605)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
>
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy9.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> ... 8 more
> 2011-05-30 21:32:35,025 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Unable to find a viable
> location to assign region -ROOT-,,0.70236052
> 2011-05-30 21:33:12,788 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Lookedup root region location,
>
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@3248bc64
> ;
> hsa=null
>
> I am running it with java 1.6.0_25 on Ubuntu 11.04 desktop. Another thing I
> noticed is that it is now looking for a different port once I restarted the
> machine:
>
> 2011-05-30 21:47:10,858 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> -ROOT-,,0.70236052 to localhost,37283,1306772219858
> 2011-05-30 21:47:10,858 DEBUG org.apache.hadoop.hbase.master.ServerManager:
> New connection to localhost,37283,1306772219858
> 2011-05-30 21:47:10,874 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 127.0.0.1:37283 could not be reached after 1 tries, giving up.
> 2011-05-30 21:47:10,875 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
> -ROOT-,,0.70236052 to serverName=localhost,37283,1306772219858,
> load=(requests=0, regions=0, usedHeap=36, maxHeap=991), trying to assign
> elsewhere instead; retry=0
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
> proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /
> 127.0.0.1:37283 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
> ...
> ...
>
> Any ideas? ..
> Hari
>
--
--Sean