You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dmitriy Lyubimov <dl...@gmail.com> on 2010/09/22 23:45:01 UTC
.META. not found
Hi,
after restart of hbase cluster, which was running for a while (and perhaps
under some stress), my hbase UI fails with the message below. the logs also
have the same message.
My other clients are spitting similar messages
tried to do compactions, flush on .meta. table, which seem to succeed but
otherwise hbase is still defunct.
hbase hdfs directory seems to be looking ok .
how can i troubleshoot it further?
it's 0.20.5.
Thanks in advance.
-Dmitriy
HTTP ERROR: 500
Trying to contact region server null for region , row '', but failed
after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
RequestURI=/master.jsp
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server null for region , row '', but failed after 3
attempts.
Exceptions:
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1037)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:75)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:48)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:152)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
on the clients :
org.apache.hadoop.hbase.TableNotFoundException: .META.
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:698)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)
...
Re: .META. not found
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Ok, thanks, it fixed itself. Patience is the key i guess : )
thanks a bunch.
but i see that a lot of people running into this problem with -ROOT- now and
then. i wonder why that is.
On Wed, Sep 22, 2010 at 3:10 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Thanks.
>
> already tried this.
>
> now i am getting another stack traces
>
> HTTP ERROR: 500
>
> Trying to contact region server 192.168.2.34:60020 for region .META.,,1, row '', but failed after 3 attempts.
>
> Exceptions:
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
>
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
>
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
>
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> RequestURI=/master.jsp
> Caused by:
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.2.34:60020 for region .META.,,1, row '', but failed after 3 attempts.
>
> Exceptions:
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
>
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
>
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
>
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
>
>
>
> On Wed, Sep 22, 2010 at 3:08 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
>> I had this problem yesterday, which I think was caused by the -ROOT-
>> region
>> losing it's only row: the one that points to .META.
>>
>> To fix it i
>> * stopped the HMaster
>> * changed it's config file to point to /hbase2 instead of /hbase
>> * started it so that it would generate a ROOT region in the new directory
>> * stopped it
>> * changed back to /hbase
>> * used 'hadoop dfs cp.... ' to copy ROOT's hdfs directory from /hbase2 to
>> /hbase
>> * started HMaster back up
>>
>> apparently it doesn't care what the content of the ROOT region is on
>> startup
>>
>>
>> On Wed, Sep 22, 2010 at 5:45 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > after restart of hbase cluster, which was running for a while (and
>> perhaps
>> > under some stress), my hbase UI fails with the message below. the logs
>> also
>> > have the same message.
>> >
>> > My other clients are spitting similar messages
>> >
>> > tried to do compactions, flush on .meta. table, which seem to succeed
>> but
>> > otherwise hbase is still defunct.
>> >
>> > hbase hdfs directory seems to be looking ok .
>> >
>> > how can i troubleshoot it further?
>> >
>> > it's 0.20.5.
>> >
>> > Thanks in advance.
>> > -Dmitriy
>> >
>> >
>> > HTTP ERROR: 500
>> >
>> > Trying to contact region server null for region , row '', but failed
>> > after 3 attempts.
>> > Exceptions:
>> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> >
>> > RequestURI=/master.jsp
>> > Caused by:
>> >
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> > contact region server null for region , row '', but failed after 3
>> > attempts.
>> > Exceptions:
>> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1037)
>> > at
>> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:75)
>> > at
>> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:48)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
>> > at
>> >
>> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:152)
>> > at
>> > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> > at
>> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> > at
>> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> > at
>> >
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> > at
>> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> > at
>> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> > at
>> > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> > at
>> >
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> > at
>> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>> > at
>> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> > at
>> >
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> > at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> > at
>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> > at
>> >
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> > at
>> >
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>> >
>> >
>> >
>> > on the clients :
>> >
>> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:698)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)
>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)
>> > ...
>> >
>>
>
>
Re: .META. not found
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Thanks.
already tried this.
now i am getting another stack traces
HTTP ERROR: 500
Trying to contact region server 192.168.2.34:60020 for region
.META.,,1, row '', but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
RequestURI=/master.jsp
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server 192.168.2.34:60020 for region .META.,,1, row '',
but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
On Wed, Sep 22, 2010 at 3:08 PM, Matt Corgan <mc...@hotpads.com> wrote:
> I had this problem yesterday, which I think was caused by the -ROOT- region
> losing it's only row: the one that points to .META.
>
> To fix it i
> * stopped the HMaster
> * changed it's config file to point to /hbase2 instead of /hbase
> * started it so that it would generate a ROOT region in the new directory
> * stopped it
> * changed back to /hbase
> * used 'hadoop dfs cp.... ' to copy ROOT's hdfs directory from /hbase2 to
> /hbase
> * started HMaster back up
>
> apparently it doesn't care what the content of the ROOT region is on
> startup
>
>
> On Wed, Sep 22, 2010 at 5:45 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > Hi,
> >
> > after restart of hbase cluster, which was running for a while (and
> perhaps
> > under some stress), my hbase UI fails with the message below. the logs
> also
> > have the same message.
> >
> > My other clients are spitting similar messages
> >
> > tried to do compactions, flush on .meta. table, which seem to succeed but
> > otherwise hbase is still defunct.
> >
> > hbase hdfs directory seems to be looking ok .
> >
> > how can i troubleshoot it further?
> >
> > it's 0.20.5.
> >
> > Thanks in advance.
> > -Dmitriy
> >
> >
> > HTTP ERROR: 500
> >
> > Trying to contact region server null for region , row '', but failed
> > after 3 attempts.
> > Exceptions:
> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> >
> > RequestURI=/master.jsp
> > Caused by:
> >
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> > contact region server null for region , row '', but failed after 3
> > attempts.
> > Exceptions:
> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> >
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1037)
> > at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:75)
> > at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:48)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454)
> > at
> > org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> > at
> >
> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:152)
> > at
> > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > at
> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> > at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> > at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> > at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > at
> > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> > at
> >
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> > at
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:324)
> > at
> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> > at
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> > at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> > at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> > at
> >
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> > at
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> >
> >
> >
> > on the clients :
> >
> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:698)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)
> > ...
> >
>
Re: .META. not found
Posted by Matt Corgan <mc...@hotpads.com>.
I had this problem yesterday, which I think was caused by the -ROOT- region
losing it's only row: the one that points to .META.
To fix it i
* stopped the HMaster
* changed it's config file to point to /hbase2 instead of /hbase
* started it so that it would generate a ROOT region in the new directory
* stopped it
* changed back to /hbase
* used 'hadoop dfs cp.... ' to copy ROOT's hdfs directory from /hbase2 to
/hbase
* started HMaster back up
apparently it doesn't care what the content of the ROOT region is on startup
On Wed, Sep 22, 2010 at 5:45 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Hi,
>
> after restart of hbase cluster, which was running for a while (and perhaps
> under some stress), my hbase UI fails with the message below. the logs also
> have the same message.
>
> My other clients are spitting similar messages
>
> tried to do compactions, flush on .meta. table, which seem to succeed but
> otherwise hbase is still defunct.
>
> hbase hdfs directory seems to be looking ok .
>
> how can i troubleshoot it further?
>
> it's 0.20.5.
>
> Thanks in advance.
> -Dmitriy
>
>
> HTTP ERROR: 500
>
> Trying to contact region server null for region , row '', but failed
> after 3 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> RequestURI=/master.jsp
> Caused by:
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact region server null for region , row '', but failed after 3
> attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1037)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:75)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:48)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
> at
> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:152)
> at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
>
>
> on the clients :
>
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:698)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)
> ...
>