You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 王仲远 <zh...@163.com> on 2010/01/13 13:32:42 UTC
Problems with Hbase Restart
I have two problems when I use hbase 0.20.2.
I used several machines to make a cluster and deployed hadoop and hbase on them. I followed the "Getting Started" and installed Hbase 0.20.2. I could start hbase sucessfully and created table and inserted data.
However, when I accessed the master page: http://10.37.17.252:60010/master.jsp , I found the first problem: the master node also appear on Region Servers and appear twice. But I didn't set the master node as regionserver in conf/regionservers:
Region Servers
Address Start Code Load
127.0.0.1:60030 1263383321075 requests=0, regions=0, usedHeap=0, maxHeap=0
127.0.0.1:60030 1263383321096 requests=0, regions=0, usedHeap=0, maxHeap=0
WAMDM1.ruc.edu.cn:60030 1263383350174 requests=0, regions=0, usedHeap=24, maxHeap=991
WAMDM2.ruc.edu.cn:60030 1263383320980 requests=0, regions=1, usedHeap=32, maxHeap=991
WAMDM3.ruc.edu.cn:60030 1263383320985 requests=0, regions=1, usedHeap=33, maxHeap=991
WAMDM4.ruc.edu.cn:60030 1263383322246 requests=0, regions=0, usedHeap=33, maxHeap=991
However, the hbase seemed to work fine. But When I try to restart the hbase, I found the second problem: I run bin/stop-hbase.sh and then run bin/ start-hbase.sh. But at this time, the hbase go down. When I tried to access the master page: http://10.37.17.252:60010/master.jsp , I got the following error message:
HTTP ERROR: 500
Trying to contact region server null for region , row '', but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
RequestURI=/master.jsp
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Powered by Jetty://
The error messages in the master log:
2010-01-13 18:34:04,424 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:04,425 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:04,425 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:06,425 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:06,429 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:06,430 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:06,430 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:08,430 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:08,434 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:08,435 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:08,435 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-01-13 18:34:08,451 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:08,451 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:08,451 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:10,451 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:10,456 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:10,465 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:10,465 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:12,465 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:12,469 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:12,470 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:14,474 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:14,475 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:14,475 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:16,475 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:16,480 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:16,480 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:16,480 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:18,480 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:18,485 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:18,486 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:18,486 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-01-13 18:34:18,500 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:18,501 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:18,501 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:20,501 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:20,506 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:20,506 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:20,506 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:22,507 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:22,511 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:22,511 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:24,516 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:24,516 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:24,516 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:26,517 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:26,521 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:26,522 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:26,522 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:28,522 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:28,526 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:28,527 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:28,527 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-01-13 18:34:28,547 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:28,547 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:28,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:30,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:30,552 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:30,552 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:30,552 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2010-01-13 18:34:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
2010-01-13 18:34:32,552 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2010-01-13 18:34:32,557 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
2010-01-13 18:34:32,558 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
2010-01-13 18:34:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
2010-01-13 18:34:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:34:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-01-13 18:35:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
2010-01-13 18:35:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
2010-01-13 18:35:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:35:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-01-13 18:35:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
2010-01-13 18:36:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
2010-01-13 18:36:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
2010-01-13 18:36:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:36:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-01-13 18:36:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
2010-01-13 18:37:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
2010-01-13 18:37:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
2010-01-13 18:37:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:37:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-01-13 18:37:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
2010-01-13 18:38:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
2010-01-13 18:38:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
2010-01-13 18:38:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:38:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-01-13 18:38:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
2010-01-13 18:39:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
2010-01-13 18:39:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
2010-01-13 18:39:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:39:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-01-13 18:39:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
2010-01-13 18:40:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
2010-01-13 18:40:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
2010-01-13 18:40:32,740 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
2010-01-13 18:40:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
.....
Can somebody help me. Thank you.
------------------
Zhongyuan
2010-01-13
Re: Re: Problems with Hbase Restart
Posted by Jean-Daniel Cryans <jd...@apache.org>.
I fixed it in 0.20.3
J-D
2010/1/14 stack <st...@duboce.net>:
> Its an exception thrown when you have logging set to DEBUG. Its just an
> emission to give a sense of where zk setup is during startup. The issue of
> nonode for the master is soon resolved after master succeeds writing its
> address into zk and startup proceeds.
>
> You are not the first to finger this exception as possible issue. Its
> confusing. Let us try and change this one at least before we cut the new
> 0.20.3RC.
>
> St.Ack
>
> 2010/1/14 王仲远 <zh...@163.com>
>
>> Hi, J-D,
>> Thank you for your response and suggestion. I found the problem was
>> caused by /etc/hosts in 10.37.248. In the file, the first line is:
>> 127.0.0.1 WAMDM1 localhost.localdomain localhost
>> and We used the hostname in our regionservers file.
>>
>> Then I had corrected the hosts file and solved the problems.
>>
>> However, when I restarted the hbase, I get some exceptions as follow:
>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /hbase/master
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:332)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrapper.java:318)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddress(ZooKeeperWrapper.java:250)
>> at
>> org.apache.hadoop.hbase.master.ZKMasterAddressWatcher.waitForMasterAddressAvailability(ZKMasterAddressWatcher.java:82)
>> at
>> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:267)
>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:254)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>
>> But it seemed that Hbase still worked well. What's the matter with this
>> exception?
>>
>> Any suggestion? Thank you~~
>>
>>
>> Zhongyuan
>> 2010-01-15
>>
>>
>> -------------------------------------------------------------
>> From: Jean-Daniel Cryans
>> Sent: 2010-01-14 01:40:33
>> To: hbase-user
>> CC:
>> Subject: Re: Problems with Hbase Restart
>>
>> It seems your master's region server reported itself as 127.0.0.1
>> Then there's something wrong with 10.37.17.248, you should check this
>> machine's region server log.
>> J-D
>> 2010/1/13 王仲远 <zh...@163.com>:
>> > I have two problems when I use hbase 0.20.2.
>> >
>> > I used several machines to make a cluster and deployed hadoop and hbase
>> on them. I followed the "Getting Started" and installed Hbase 0.20.2. I
>> could start hbase sucessfully and created table and inserted data.
>> >
>> > However, when I accessed the master page:
>> http://10.37.17.252:60010/master.jsp , I found the first problem: the
>> master node also appear on Region Servers and appear twice. But I didn't set
>> the master node as regionserver in conf/regionservers:
>> > Region Servers
>> > Address Start Code Load
>> > 127.0.0.1:60030 1263383321075 requests=0, regions=0, usedHeap=0,
>> maxHeap=0
>> > 127.0.0.1:60030 1263383321096 requests=0, regions=0, usedHeap=0,
>> maxHeap=0
>> > WAMDM1.ruc.edu.cn:60030 1263383350174 requests=0, regions=0,
>> usedHeap=24, maxHeap=991
>> > WAMDM2.ruc.edu.cn:60030 1263383320980 requests=0, regions=1,
>> usedHeap=32, maxHeap=991
>> > WAMDM3.ruc.edu.cn:60030 1263383320985 requests=0, regions=1,
>> usedHeap=33, maxHeap=991
>> > WAMDM4.ruc.edu.cn:60030 1263383322246 requests=0, regions=0,
>> usedHeap=33, maxHeap=991
>> >
>> > However, the hbase seemed to work fine. But When I try to restart the
>> hbase, I found the second problem: I run bin/stop-hbase.sh and then run bin/
>> start-hbase.sh. But at this time, the hbase go down. When I tried to access
>> the master page: http://10.37.17.252:60010/master.jsp , I got the
>> following error message:
>> >
>> > HTTP ERROR: 500
>> > Trying to contact region server null for region , row '', but failed
>> after 3 attempts.
>> > Exceptions:
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> >
>> > RequestURI=/master.jsp
>> >
>> > Caused by:
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> contact region server null for region , row '', but failed after 3 attempts.
>> > Exceptions:
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> >
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
>> > at
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
>> > at
>> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
>> > at
>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> > at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> > at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> > at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> > at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> > at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> > at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> > at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> > at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>> > at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> > at
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> > at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> > at
>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> > at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> > at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>> >
>> > Powered by Jetty://
>> >
>> >
>> > The error messages in the master log:
>> > 2010-01-13 18:34:04,424 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:04,425 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:04,425 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:06,425 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:06,429 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:06,430 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:06,430 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:08,430 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:08,434 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:08,435 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:08,435 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
>> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
>> > at
>> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
>> > at
>> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
>> > at
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
>> > at
>> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
>> > at
>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> > at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> > at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> > at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> > at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> > at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> > at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> > at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> > at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>> > at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> > at
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> > at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> > at
>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> > at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> > at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>> > 2010-01-13 18:34:08,451 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:08,451 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:08,451 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:10,451 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:10,456 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:10,465 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:10,465 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:12,465 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:12,469 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:12,470 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:14,474 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:14,475 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:14,475 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:16,475 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:16,480 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:16,480 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:16,480 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:18,480 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:18,485 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:18,486 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:18,486 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
>> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
>> > at
>> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
>> > at
>> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
>> > at
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
>> > at
>> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
>> > at
>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> > at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> > at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> > at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> > at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> > at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> > at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> > at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> > at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>> > at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> > at
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> > at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> > at
>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> > at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> > at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>> > 2010-01-13 18:34:18,500 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:18,501 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:18,501 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:20,501 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:20,506 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:20,506 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:20,506 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:22,507 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:22,511 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:22,511 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:24,516 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:24,516 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:24,516 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:26,517 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:26,521 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:26,522 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:26,522 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:28,522 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:28,526 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:28,527 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:28,527 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
>> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
>> > at
>> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
>> > at
>> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
>> > at
>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
>> > at
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
>> > at
>> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
>> > at
>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> > at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> > at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> > at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> > at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> > at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> > at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> > at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> > at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>> > at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> > at
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> > at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> > at
>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> > at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> > at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>> > 2010-01-13 18:34:28,547 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:28,547 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:28,547 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:30,547 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:30,552 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:30,552 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:30,552 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
>> location changed. Sleeping.
>> > 2010-01-13 18:34:31,575 INFO
>> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
>> average load 0.16666666666666666
>> > 2010-01-13 18:34:32,552 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
>> finding root region.
>> > 2010-01-13 18:34:32,557 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
>> /hbase/root-region-server got 10.37.17.248:60020
>> > 2010-01-13 18:34:32,558 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> > 2010-01-13 18:34:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 0 .META. region(s) scanned
>> > 2010-01-13 18:34:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:34:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
>> Scan ROOT region
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
>> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>> > 2010-01-13 18:35:31,575 INFO
>> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
>> average load 0.16666666666666666
>> > 2010-01-13 18:35:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> > 2010-01-13 18:35:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:35:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
>> Scan ROOT region
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
>> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>> > 2010-01-13 18:35:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 0 .META. region(s) scanned
>> > 2010-01-13 18:36:31,575 INFO
>> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
>> average load 0.16666666666666666
>> > 2010-01-13 18:36:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> > 2010-01-13 18:36:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:36:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
>> Scan ROOT region
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
>> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>> > 2010-01-13 18:36:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 0 .META. region(s) scanned
>> > 2010-01-13 18:37:31,575 INFO
>> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
>> average load 0.16666666666666666
>> > 2010-01-13 18:37:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> > 2010-01-13 18:37:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:37:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
>> Scan ROOT region
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
>> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>> > 2010-01-13 18:37:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 0 .META. region(s) scanned
>> > 2010-01-13 18:38:31,575 INFO
>> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
>> average load 0.16666666666666666
>> > 2010-01-13 18:38:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> > 2010-01-13 18:38:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:38:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
>> Scan ROOT region
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
>> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>> > 2010-01-13 18:38:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 0 .META. region(s) scanned
>> > 2010-01-13 18:39:31,575 INFO
>> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
>> average load 0.16666666666666666
>> > 2010-01-13 18:39:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> > 2010-01-13 18:39:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:39:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
>> Scan ROOT region
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
>> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>> > 2010-01-13 18:39:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 0 .META. region(s) scanned
>> > 2010-01-13 18:40:31,575 INFO
>> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
>> average load 0.16666666666666666
>> > 2010-01-13 18:40:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> > 2010-01-13 18:40:32,740 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
>> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
>> > 2010-01-13 18:40:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
>> Scan ROOT region
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
>> up proxy to /10.37.17.248:60020 after attempts=1
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
>> > at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
>> > at
>> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
>> > at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
>> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>> > .....
>> >
>> >
>> > Can somebody help me. Thank you.
>> > ------------------
>> > Zhongyuan
>> > 2010-01-13
>> >
>> >
>> >
>>
>
Re: Re: Problems with Hbase Restart
Posted by stack <st...@duboce.net>.
Its an exception thrown when you have logging set to DEBUG. Its just an
emission to give a sense of where zk setup is during startup. The issue of
nonode for the master is soon resolved after master succeeds writing its
address into zk and startup proceeds.
You are not the first to finger this exception as possible issue. Its
confusing. Let us try and change this one at least before we cut the new
0.20.3RC.
St.Ack
2010/1/14 王仲远 <zh...@163.com>
> Hi, J-D,
> Thank you for your response and suggestion. I found the problem was
> caused by /etc/hosts in 10.37.248. In the file, the first line is:
> 127.0.0.1 WAMDM1 localhost.localdomain localhost
> and We used the hostname in our regionservers file.
>
> Then I had corrected the hosts file and solved the problems.
>
> However, when I restarted the hbase, I get some exceptions as follow:
> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/master
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:332)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrapper.java:318)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddress(ZooKeeperWrapper.java:250)
> at
> org.apache.hadoop.hbase.master.ZKMasterAddressWatcher.waitForMasterAddressAvailability(ZKMasterAddressWatcher.java:82)
> at
> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:267)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:254)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>
> But it seemed that Hbase still worked well. What's the matter with this
> exception?
>
> Any suggestion? Thank you~~
>
>
> Zhongyuan
> 2010-01-15
>
>
> -------------------------------------------------------------
> From: Jean-Daniel Cryans
> Sent: 2010-01-14 01:40:33
> To: hbase-user
> CC:
> Subject: Re: Problems with Hbase Restart
>
> It seems your master's region server reported itself as 127.0.0.1
> Then there's something wrong with 10.37.17.248, you should check this
> machine's region server log.
> J-D
> 2010/1/13 王仲远 <zh...@163.com>:
> > I have two problems when I use hbase 0.20.2.
> >
> > I used several machines to make a cluster and deployed hadoop and hbase
> on them. I followed the "Getting Started" and installed Hbase 0.20.2. I
> could start hbase sucessfully and created table and inserted data.
> >
> > However, when I accessed the master page:
> http://10.37.17.252:60010/master.jsp , I found the first problem: the
> master node also appear on Region Servers and appear twice. But I didn't set
> the master node as regionserver in conf/regionservers:
> > Region Servers
> > Address Start Code Load
> > 127.0.0.1:60030 1263383321075 requests=0, regions=0, usedHeap=0,
> maxHeap=0
> > 127.0.0.1:60030 1263383321096 requests=0, regions=0, usedHeap=0,
> maxHeap=0
> > WAMDM1.ruc.edu.cn:60030 1263383350174 requests=0, regions=0,
> usedHeap=24, maxHeap=991
> > WAMDM2.ruc.edu.cn:60030 1263383320980 requests=0, regions=1,
> usedHeap=32, maxHeap=991
> > WAMDM3.ruc.edu.cn:60030 1263383320985 requests=0, regions=1,
> usedHeap=33, maxHeap=991
> > WAMDM4.ruc.edu.cn:60030 1263383322246 requests=0, regions=0,
> usedHeap=33, maxHeap=991
> >
> > However, the hbase seemed to work fine. But When I try to restart the
> hbase, I found the second problem: I run bin/stop-hbase.sh and then run bin/
> start-hbase.sh. But at this time, the hbase go down. When I tried to access
> the master page: http://10.37.17.252:60010/master.jsp , I got the
> following error message:
> >
> > HTTP ERROR: 500
> > Trying to contact region server null for region , row '', but failed
> after 3 attempts.
> > Exceptions:
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> >
> > RequestURI=/master.jsp
> >
> > Caused by:
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact region server null for region , row '', but failed after 3 attempts.
> > Exceptions:
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> >
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> > at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> > at
> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> > at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> > at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> > at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> > at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:324)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> > at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> > at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> > at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> > at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> > at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> >
> > Powered by Jetty://
> >
> >
> > The error messages in the master log:
> > 2010-01-13 18:34:04,424 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:04,425 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:04,425 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:06,425 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:06,429 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:06,430 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:06,430 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:08,430 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:08,434 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:08,435 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:08,435 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> > at
> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> > at
> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> > at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> > at
> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> > at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> > at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> > at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> > at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:324)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> > at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> > at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> > at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> > at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> > at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> > 2010-01-13 18:34:08,451 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:08,451 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:08,451 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:10,451 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:10,456 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:10,465 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:10,465 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:12,465 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:12,469 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:12,470 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:14,474 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:14,475 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:14,475 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:16,475 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:16,480 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:16,480 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:16,480 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:18,480 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:18,485 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:18,486 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:18,486 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> > at
> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> > at
> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> > at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> > at
> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> > at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> > at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> > at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> > at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:324)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> > at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> > at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> > at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> > at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> > at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> > 2010-01-13 18:34:18,500 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:18,501 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:18,501 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:20,501 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:20,506 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:20,506 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:20,506 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:22,507 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:22,511 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:22,511 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:24,516 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:24,516 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:24,516 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:26,517 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:26,521 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:26,522 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:26,522 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:28,522 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:28,526 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:28,527 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:28,527 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> > at
> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> > at
> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> > at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> > at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> > at
> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> > at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> > at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> > at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> > at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:324)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> > at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> > at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> > at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> > at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> > at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> > 2010-01-13 18:34:28,547 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:28,547 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:28,547 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:30,547 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:30,552 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:30,552 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:30,552 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> > 2010-01-13 18:34:31,575 INFO
> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
> average load 0.16666666666666666
> > 2010-01-13 18:34:32,552 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> > 2010-01-13 18:34:32,557 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.37.17.248:60020
> > 2010-01-13 18:34:32,558 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
> regionname: -ROOT-,,0, startKey: <>}
> > 2010-01-13 18:34:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 0 .META. region(s) scanned
> > 2010-01-13 18:34:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:34:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> > at
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> > at
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> > 2010-01-13 18:35:31,575 INFO
> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
> average load 0.16666666666666666
> > 2010-01-13 18:35:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
> regionname: -ROOT-,,0, startKey: <>}
> > 2010-01-13 18:35:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:35:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> > at
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> > at
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> > 2010-01-13 18:35:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 0 .META. region(s) scanned
> > 2010-01-13 18:36:31,575 INFO
> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
> average load 0.16666666666666666
> > 2010-01-13 18:36:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
> regionname: -ROOT-,,0, startKey: <>}
> > 2010-01-13 18:36:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:36:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> > at
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> > at
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> > 2010-01-13 18:36:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 0 .META. region(s) scanned
> > 2010-01-13 18:37:31,575 INFO
> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
> average load 0.16666666666666666
> > 2010-01-13 18:37:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
> regionname: -ROOT-,,0, startKey: <>}
> > 2010-01-13 18:37:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:37:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> > at
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> > at
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> > 2010-01-13 18:37:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 0 .META. region(s) scanned
> > 2010-01-13 18:38:31,575 INFO
> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
> average load 0.16666666666666666
> > 2010-01-13 18:38:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
> regionname: -ROOT-,,0, startKey: <>}
> > 2010-01-13 18:38:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:38:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> > at
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> > at
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> > 2010-01-13 18:38:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 0 .META. region(s) scanned
> > 2010-01-13 18:39:31,575 INFO
> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
> average load 0.16666666666666666
> > 2010-01-13 18:39:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
> regionname: -ROOT-,,0, startKey: <>}
> > 2010-01-13 18:39:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:39:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> > at
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> > at
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> > 2010-01-13 18:39:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 0 .META. region(s) scanned
> > 2010-01-13 18:40:31,575 INFO
> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead,
> average load 0.16666666666666666
> > 2010-01-13 18:40:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020,
> regionname: -ROOT-,,0, startKey: <>}
> > 2010-01-13 18:40:32,740 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
> 10.37.17.248:60020 could not be reached after 1 tries, giving up.
> > 2010-01-13 18:40:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
> up proxy to /10.37.17.248:60020 after attempts=1
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> > at
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> > at
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> > at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> > .....
> >
> >
> > Can somebody help me. Thank you.
> > ------------------
> > Zhongyuan
> > 2010-01-13
> >
> >
> >
>
Re: Re: Problems with Hbase Restart
Posted by 王仲远 <zh...@163.com>.
Hi, J-D,
Thank you for your response and suggestion. I found the problem was caused by /etc/hosts in 10.37.248. In the file, the first line is:
127.0.0.1 WAMDM1 localhost.localdomain localhost
and We used the hostname in our regionservers file.
Then I had corrected the hosts file and solved the problems.
However, when I restarted the hbase, I get some exceptions as follow:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:332)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrapper.java:318)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddress(ZooKeeperWrapper.java:250)
at org.apache.hadoop.hbase.master.ZKMasterAddressWatcher.waitForMasterAddressAvailability(ZKMasterAddressWatcher.java:82)
at org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:267)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:254)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
But it seemed that Hbase still worked well. What's the matter with this exception?
Any suggestion? Thank you~~
Zhongyuan
2010-01-15
-------------------------------------------------------------
From: Jean-Daniel Cryans
Sent: 2010-01-14 01:40:33
To: hbase-user
CC:
Subject: Re: Problems with Hbase Restart
It seems your master's region server reported itself as 127.0.0.1
Then there's something wrong with 10.37.17.248, you should check this
machine's region server log.
J-D
2010/1/13 王仲远 <zh...@163.com>:
> I have two problems when I use hbase 0.20.2.
>
> I used several machines to make a cluster and deployed hadoop and hbase on them. I followed the "Getting Started" and installed Hbase 0.20.2. I could start hbase sucessfully and created table and inserted data.
>
> However, when I accessed the master page: http://10.37.17.252:60010/master.jsp , I found the first problem: the master node also appear on Region Servers and appear twice. But I didn't set the master node as regionserver in conf/regionservers:
> Region Servers
> Address Start Code Load
> 127.0.0.1:60030 1263383321075 requests=0, regions=0, usedHeap=0, maxHeap=0
> 127.0.0.1:60030 1263383321096 requests=0, regions=0, usedHeap=0, maxHeap=0
> WAMDM1.ruc.edu.cn:60030 1263383350174 requests=0, regions=0, usedHeap=24, maxHeap=991
> WAMDM2.ruc.edu.cn:60030 1263383320980 requests=0, regions=1, usedHeap=32, maxHeap=991
> WAMDM3.ruc.edu.cn:60030 1263383320985 requests=0, regions=1, usedHeap=33, maxHeap=991
> WAMDM4.ruc.edu.cn:60030 1263383322246 requests=0, regions=0, usedHeap=33, maxHeap=991
>
> However, the hbase seemed to work fine. But When I try to restart the hbase, I found the second problem: I run bin/stop-hbase.sh and then run bin/ start-hbase.sh. But at this time, the hbase go down. When I tried to access the master page: http://10.37.17.252:60010/master.jsp , I got the following error message:
>
> HTTP ERROR: 500
> Trying to contact region server null for region , row '', but failed after 3 attempts.
> Exceptions:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
>
> RequestURI=/master.jsp
>
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 3 attempts.
> Exceptions:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
>
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> Powered by Jetty://
>
>
> The error messages in the master log:
> 2010-01-13 18:34:04,424 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:04,425 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:04,425 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:06,425 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:06,429 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:06,430 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:06,430 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:08,430 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:08,434 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:08,435 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:08,435 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> 2010-01-13 18:34:08,451 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:08,451 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:08,451 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:10,451 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:10,456 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:10,465 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:10,465 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:12,465 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:12,469 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:12,470 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:14,474 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:14,475 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:14,475 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:16,475 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:16,480 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:16,480 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:16,480 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:18,480 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:18,485 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:18,486 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:18,486 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> 2010-01-13 18:34:18,500 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:18,501 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:18,501 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:20,501 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:20,506 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:20,506 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:20,506 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:22,507 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:22,511 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:22,511 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:24,516 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:24,516 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:24,516 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:26,517 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:26,521 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:26,522 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:26,522 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:28,522 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:28,526 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:28,527 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:28,527 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> 2010-01-13 18:34:28,547 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:28,547 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:28,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:30,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:30,552 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:30,552 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:30,552 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:34:32,552 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:32,557 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:32,558 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:34:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:34:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:35:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:35:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:35:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:35:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:35:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:36:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:36:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:36:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:36:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:36:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:37:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:37:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:37:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:37:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:37:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:38:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:38:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:38:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:38:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:38:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:39:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:39:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:39:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:39:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:39:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:40:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:40:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:40:32,740 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:40:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> .....
>
>
> Can somebody help me. Thank you.
> ------------------
> Zhongyuan
> 2010-01-13
>
>
>
Re: Problems with Hbase Restart
Posted by Jean-Daniel Cryans <jd...@apache.org>.
It seems your master's region server reported itself as 127.0.0.1
Then there's something wrong with 10.37.17.248, you should check this
machine's region server log.
J-D
2010/1/13 王仲远 <zh...@163.com>:
> I have two problems when I use hbase 0.20.2.
>
> I used several machines to make a cluster and deployed hadoop and hbase on them. I followed the "Getting Started" and installed Hbase 0.20.2. I could start hbase sucessfully and created table and inserted data.
>
> However, when I accessed the master page: http://10.37.17.252:60010/master.jsp , I found the first problem: the master node also appear on Region Servers and appear twice. But I didn't set the master node as regionserver in conf/regionservers:
> Region Servers
> Address Start Code Load
> 127.0.0.1:60030 1263383321075 requests=0, regions=0, usedHeap=0, maxHeap=0
> 127.0.0.1:60030 1263383321096 requests=0, regions=0, usedHeap=0, maxHeap=0
> WAMDM1.ruc.edu.cn:60030 1263383350174 requests=0, regions=0, usedHeap=24, maxHeap=991
> WAMDM2.ruc.edu.cn:60030 1263383320980 requests=0, regions=1, usedHeap=32, maxHeap=991
> WAMDM3.ruc.edu.cn:60030 1263383320985 requests=0, regions=1, usedHeap=33, maxHeap=991
> WAMDM4.ruc.edu.cn:60030 1263383322246 requests=0, regions=0, usedHeap=33, maxHeap=991
>
> However, the hbase seemed to work fine. But When I try to restart the hbase, I found the second problem: I run bin/stop-hbase.sh and then run bin/ start-hbase.sh. But at this time, the hbase go down. When I tried to access the master page: http://10.37.17.252:60010/master.jsp , I got the following error message:
>
> HTTP ERROR: 500
> Trying to contact region server null for region , row '', but failed after 3 attempts.
> Exceptions:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
>
> RequestURI=/master.jsp
>
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 3 attempts.
> Exceptions:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
>
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> Powered by Jetty://
>
>
> The error messages in the master log:
> 2010-01-13 18:34:04,424 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:04,425 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:04,425 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:06,425 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:06,429 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:06,430 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:06,430 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:08,430 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:08,434 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:08,435 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:08,435 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> 2010-01-13 18:34:08,451 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:08,451 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:08,451 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:10,451 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:10,456 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:10,465 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:10,465 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:12,465 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:12,469 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:12,470 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:14,474 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:14,475 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:14,475 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:16,475 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:16,480 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:16,480 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:16,480 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:18,480 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:18,485 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:18,486 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:18,486 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> 2010-01-13 18:34:18,500 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:18,501 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:18,501 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:20,501 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:20,506 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:20,506 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:20,506 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:22,507 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:22,511 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:22,511 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:24,516 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:24,516 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:24,516 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:26,517 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:26,521 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:26,522 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:26,522 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:28,522 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:28,526 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:28,527 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:28,527 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
> at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> 2010-01-13 18:34:28,547 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:28,547 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:28,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:30,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:30,552 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:30,552 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:30,552 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
> 2010-01-13 18:34:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:34:32,552 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
> 2010-01-13 18:34:32,557 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.37.17.248:60020
> 2010-01-13 18:34:32,558 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:34:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:34:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:34:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:35:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:35:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:35:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:35:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:35:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:36:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:36:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:36:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:36:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:36:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:37:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:37:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:37:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:37:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:37:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:38:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:38:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:38:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:38:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:38:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:39:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:39:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:39:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:39:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> 2010-01-13 18:39:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
> 2010-01-13 18:40:31,575 INFO org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, average load 0.16666666666666666
> 2010-01-13 18:40:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-01-13 18:40:32,740 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.37.17.248:60020 could not be reached after 1 tries, giving up.
> 2010-01-13 18:40:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156)
> at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
> at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> .....
>
>
> Can somebody help me. Thank you.
> ------------------
> Zhongyuan
> 2010-01-13
>
>
>