You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Yabo Xu <ar...@gmail.com> on 2010/03/09 03:21:01 UTC
Table broken: NoServerForRegionException
Hi there:
We have been using HBase for a while in our project, and once in a while we
have encountered the "NoServerForRegionException" exception. Some region of
a certain table seems to be broken, and a simple count will throw the
following exception.
Any idea on how to fix the table, and also anybody knows where the exception
might be from?
We are rushing for a deadline, but are stucking on this issue now. So your
help would be really appreciated!
Arber
---------------------------
NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.
client.RetriesExhaustedException: Trying to contact region server null for
region , row '1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45',
but failed after 5 attempts.
Exceptions:
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in .META. for
regionSummba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in .META. for region
Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in .META. for region
Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in .META. for region
Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in .META. for region
Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
from org/apache/hadoop/hbase/
client/HTable.java:1716:in `hasNext'
from sun.reflect.GeneratedMethodAccessor3:-1:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:73:in `call'
from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
from org/jruby/ast/WhileNode.java:124:in `interpret'
from org/jruby/ast/NewlineNode.java:101:in `interpret'
from org/jruby/ast/BlockNode.java:68:in `interpret'
from org/jruby/internal/runtime/methods/DefaultMethod.java:156:in
`interpretedCall'
from org/jruby/internal/runtime/methods/DefaultMethod.java:133:in `call'
from org/jruby/internal/runtime/methods/DefaultMethod.java:246:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:114:in `call'
... 107 levels...
Re: Table broken: NoServerForRegionException
Posted by Yabo Xu <ar...@gmail.com>.
It seems to be working. Thanks, Ryan!
On Tue, Mar 9, 2010 at 10:22 AM, Ryan Rawson <ry...@gmail.com> wrote:
> Try this in the shell:
>
> flush '.META.'
> // Wait about 20 seconds
> major_compact '.META.'
>
> what version of hbase are you running?
>
> On Mon, Mar 8, 2010 at 6:21 PM, Yabo Xu <ar...@gmail.com> wrote:
> > Hi there:
> >
> > We have been using HBase for a while in our project, and once in a while
> we
> > have encountered the "NoServerForRegionException" exception. Some region
> of
> > a certain table seems to be broken, and a simple count will throw the
> > following exception.
> >
> > Any idea on how to fix the table, and also anybody knows where the
> exception
> > might be from?
> >
> > We are rushing for a deadline, but are stucking on this issue now. So
> your
> > help would be really appreciated!
> >
> > Arber
> >
> > ---------------------------
> > NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.
> > client.RetriesExhaustedException: Trying to contact region server null
> for
> > region , row
> '1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45',
> > but failed after 5 attempts.
> > Exceptions:
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for
> >
> regionSummba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > from org/apache/hadoop/hbase/
> >
> > client/HTable.java:1716:in `hasNext'
> > from sun.reflect.GeneratedMethodAccessor3:-1:in `invoke'
> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> > from java/lang/reflect/Method.java:597:in `invoke'
> > from org/jruby/javasupport/JavaMethod.java:298:in
> > `invokeWithExceptionHandling'
> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> > from org/jruby/runtime/callsite/CachingCallSite.java:73:in `call'
> > from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> > from org/jruby/ast/WhileNode.java:124:in `interpret'
> > from org/jruby/ast/NewlineNode.java:101:in `interpret'
> > from org/jruby/ast/BlockNode.java:68:in `interpret'
> > from org/jruby/internal/runtime/methods/DefaultMethod.java:156:in
> > `interpretedCall'
> > from org/jruby/internal/runtime/methods/DefaultMethod.java:133:in `call'
> > from org/jruby/internal/runtime/methods/DefaultMethod.java:246:in `call'
> > from org/jruby/runtime/callsite/CachingCallSite.java:114:in `call'
> > ... 107 levels...
> >
>
Re: Table broken: NoServerForRegionException
Posted by Ryan Rawson <ry...@gmail.com>.
If you didnt get this resolved, you could try kill -9ing the master
then restarting it.
-ryan
On Tue, Mar 9, 2010 at 11:14 AM, Ted Yu <yu...@gmail.com> wrote:
> Can I apply those two commands to resolve the following (hbase 0.20.1) ?
>
> 2010-03-09 11:06:49,229 DEBUG [pool-1-thread-1] hfile.LruBlockCache(551):
> Cache Stats: Sizes: Total=12.646942MB (13261280), Free=1212.828MB
> (1271742368), Max=1225.475MB (1285003648), Counts: Blocks=43, Access=5141,
> Hit=5098, Miss=43, Evictions=0, Evicted=0, Ratios: Hit
> Ratio=99.16358590126038%, Miss Ratio=0.8364131674170494%, Evicted/Run=NaN
> 2010-03-09 11:07:40,759 ERROR [IPC Server handler 49 on 60020]
> regionserver.HRegionServer(846): Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1893)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2010-03-09 11:07:42,766 ERROR [IPC Server handler 50 on 60020]
> regionserver.HRegionServer(844):
> org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
>
> On Mon, Mar 8, 2010 at 6:22 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Try this in the shell:
>>
>> flush '.META.'
>> // Wait about 20 seconds
>> major_compact '.META.'
>>
>> what version of hbase are you running?
>>
>> On Mon, Mar 8, 2010 at 6:21 PM, Yabo Xu <ar...@gmail.com> wrote:
>> > Hi there:
>> >
>> > We have been using HBase for a while in our project, and once in a while
>> we
>> > have encountered the "NoServerForRegionException" exception. Some region
>> of
>> > a certain table seems to be broken, and a simple count will throw the
>> > following exception.
>> >
>> > Any idea on how to fix the table, and also anybody knows where the
>> exception
>> > might be from?
>> >
>> > We are rushing for a deadline, but are stucking on this issue now. So
>> your
>> > help would be really appreciated!
>> >
>> > Arber
>> >
>> > ---------------------------
>> > NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.
>> > client.RetriesExhaustedException: Trying to contact region server null
>> for
>> > region , row
>> '1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45',
>> > but failed after 5 attempts.
>> > Exceptions:
>> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
>> address
>> > listed in .META. for
>> >
>> regionSummba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
>> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
>> address
>> > listed in .META. for region
>> >
>> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
>> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
>> address
>> > listed in .META. for region
>> >
>> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
>> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
>> address
>> > listed in .META. for region
>> >
>> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
>> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
>> address
>> > listed in .META. for region
>> >
>> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
>> > from org/apache/hadoop/hbase/
>> >
>> > client/HTable.java:1716:in `hasNext'
>> > from sun.reflect.GeneratedMethodAccessor3:-1:in `invoke'
>> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>> > from java/lang/reflect/Method.java:597:in `invoke'
>> > from org/jruby/javasupport/JavaMethod.java:298:in
>> > `invokeWithExceptionHandling'
>> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
>> > from org/jruby/runtime/callsite/CachingCallSite.java:73:in `call'
>> > from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
>> > from org/jruby/ast/WhileNode.java:124:in `interpret'
>> > from org/jruby/ast/NewlineNode.java:101:in `interpret'
>> > from org/jruby/ast/BlockNode.java:68:in `interpret'
>> > from org/jruby/internal/runtime/methods/DefaultMethod.java:156:in
>> > `interpretedCall'
>> > from org/jruby/internal/runtime/methods/DefaultMethod.java:133:in `call'
>> > from org/jruby/internal/runtime/methods/DefaultMethod.java:246:in `call'
>> > from org/jruby/runtime/callsite/CachingCallSite.java:114:in `call'
>> > ... 107 levels...
>> >
>>
>
Re: Table broken: NoServerForRegionException
Posted by Ted Yu <yu...@gmail.com>.
Can I apply those two commands to resolve the following (hbase 0.20.1) ?
2010-03-09 11:06:49,229 DEBUG [pool-1-thread-1] hfile.LruBlockCache(551):
Cache Stats: Sizes: Total=12.646942MB (13261280), Free=1212.828MB
(1271742368), Max=1225.475MB (1285003648), Counts: Blocks=43, Access=5141,
Hit=5098, Miss=43, Evictions=0, Evicted=0, Ratios: Hit
Ratio=99.16358590126038%, Miss Ratio=0.8364131674170494%, Evicted/Run=NaN
2010-03-09 11:07:40,759 ERROR [IPC Server handler 49 on 60020]
regionserver.HRegionServer(846): Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1893)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-03-09 11:07:42,766 ERROR [IPC Server handler 50 on 60020]
regionserver.HRegionServer(844):
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
On Mon, Mar 8, 2010 at 6:22 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Try this in the shell:
>
> flush '.META.'
> // Wait about 20 seconds
> major_compact '.META.'
>
> what version of hbase are you running?
>
> On Mon, Mar 8, 2010 at 6:21 PM, Yabo Xu <ar...@gmail.com> wrote:
> > Hi there:
> >
> > We have been using HBase for a while in our project, and once in a while
> we
> > have encountered the "NoServerForRegionException" exception. Some region
> of
> > a certain table seems to be broken, and a simple count will throw the
> > following exception.
> >
> > Any idea on how to fix the table, and also anybody knows where the
> exception
> > might be from?
> >
> > We are rushing for a deadline, but are stucking on this issue now. So
> your
> > help would be really appreciated!
> >
> > Arber
> >
> > ---------------------------
> > NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.
> > client.RetriesExhaustedException: Trying to contact region server null
> for
> > region , row
> '1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45',
> > but failed after 5 attempts.
> > Exceptions:
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for
> >
> regionSummba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> > listed in .META. for region
> >
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> > from org/apache/hadoop/hbase/
> >
> > client/HTable.java:1716:in `hasNext'
> > from sun.reflect.GeneratedMethodAccessor3:-1:in `invoke'
> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> > from java/lang/reflect/Method.java:597:in `invoke'
> > from org/jruby/javasupport/JavaMethod.java:298:in
> > `invokeWithExceptionHandling'
> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> > from org/jruby/runtime/callsite/CachingCallSite.java:73:in `call'
> > from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> > from org/jruby/ast/WhileNode.java:124:in `interpret'
> > from org/jruby/ast/NewlineNode.java:101:in `interpret'
> > from org/jruby/ast/BlockNode.java:68:in `interpret'
> > from org/jruby/internal/runtime/methods/DefaultMethod.java:156:in
> > `interpretedCall'
> > from org/jruby/internal/runtime/methods/DefaultMethod.java:133:in `call'
> > from org/jruby/internal/runtime/methods/DefaultMethod.java:246:in `call'
> > from org/jruby/runtime/callsite/CachingCallSite.java:114:in `call'
> > ... 107 levels...
> >
>
Re: Table broken: NoServerForRegionException
Posted by Ryan Rawson <ry...@gmail.com>.
Try this in the shell:
flush '.META.'
// Wait about 20 seconds
major_compact '.META.'
what version of hbase are you running?
On Mon, Mar 8, 2010 at 6:21 PM, Yabo Xu <ar...@gmail.com> wrote:
> Hi there:
>
> We have been using HBase for a while in our project, and once in a while we
> have encountered the "NoServerForRegionException" exception. Some region of
> a certain table seems to be broken, and a simple count will throw the
> following exception.
>
> Any idea on how to fix the table, and also anybody knows where the exception
> might be from?
>
> We are rushing for a deadline, but are stucking on this issue now. So your
> help would be really appreciated!
>
> Arber
>
> ---------------------------
> NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.
> client.RetriesExhaustedException: Trying to contact region server null for
> region , row '1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45',
> but failed after 5 attempts.
> Exceptions:
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
> listed in .META. for
> regionSummba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
> listed in .META. for region
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
> listed in .META. for region
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
> listed in .META. for region
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
> listed in .META. for region
> Summba.RAFE.sentence,1267733340|com.ifeng.finance/stock/ts/20100304/1887962.shtml|1|45,1267781626626
> from org/apache/hadoop/hbase/
>
> client/HTable.java:1716:in `hasNext'
> from sun.reflect.GeneratedMethodAccessor3:-1:in `invoke'
> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> from java/lang/reflect/Method.java:597:in `invoke'
> from org/jruby/javasupport/JavaMethod.java:298:in
> `invokeWithExceptionHandling'
> from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:73:in `call'
> from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> from org/jruby/ast/WhileNode.java:124:in `interpret'
> from org/jruby/ast/NewlineNode.java:101:in `interpret'
> from org/jruby/ast/BlockNode.java:68:in `interpret'
> from org/jruby/internal/runtime/methods/DefaultMethod.java:156:in
> `interpretedCall'
> from org/jruby/internal/runtime/methods/DefaultMethod.java:133:in `call'
> from org/jruby/internal/runtime/methods/DefaultMethod.java:246:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:114:in `call'
> ... 107 levels...
>