You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Liu Yan <gz...@gmail.com> on 2009/02/28 17:00:15 UTC
Exception when scanning table
hi,
I tried to scan a table with "scan 'table_name'" command in the hbase shell,
and hit the following exception:
{{{
NativeException: java.lang.RuntimeException:
org.apache.hadoop.hbase.client.ScannerTimeoutException
from org/apache/hadoop/hbase/client/HTable.java:1704:in `hasNext'
from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:250:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
from org/jruby/javasupport/JavaClass.java:416:in `execute'
from org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
`call'
from org/jruby/internal/runtime/methods/DynamicMethod.java:70:in `call'
from org/jruby/runtime/CallSite.java:295:in `call'
from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
from org/jruby/evaluator/ASTInterpreter.java:324:in `evalInternal'
from org/jruby/evaluator/ASTInterpreter.java:1790:in `whileNode'
from org/jruby/evaluator/ASTInterpreter.java:505:in `evalInternal'
from org/jruby/evaluator/ASTInterpreter.java:620:in `blockNode'
from org/jruby/evaluator/ASTInterpreter.java:318:in `evalInternal'
... 121 levels...
from
ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
`call'
from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
from org/jruby/runtime/CallSite.java:298:in `call'
from
ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
`__file__'
from
ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
`__file__'
from
ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
`load'
from org/jruby/Ruby.java:512:in `runScript'
from org/jruby/Ruby.java:432:in `runNormally'
from org/jruby/Ruby.java:312:in `runFromMain'
from org/jruby/Main.java:144:in `run'
from org/jruby/Main.java:89:in `run'
from org/jruby/Main.java:80:in `main'
from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
from (hbase):3:in `binding
}}}
In the regionserver log file, I found this:
{{{
2009-02-28 10:50:48,685 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
1002_profiles,113495530684109376,1235659992234 in 0sec
2009-02-28 10:50:54,779 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer
: org.apache.hadoop.hbase.UnknownScannerException: Name: 310687422691357774
2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call next(310687422691357774, 30) from
10.254.51.127:54821: error
: org.apache.hadoop.hbase.UnknownScannerException: Name: 310687422691357774
org.apache.hadoop.hbase.UnknownScannerException: Name:
310687422691357774 at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
.java:1568)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
2009-02-28 10:51:08,687 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
1002_profiles,266291625891696512,1235713612502
2009-02-28 10:51:08,688 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
2131509311/fetl: no store files to compact
2009-02-28 10:51:08,689 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
2131509311/pre_fetl: no store files to compact
2009-02-28 10:51:08,691 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
2131509311/scored: no store files to compact
2009-02-28 10:51:08,692 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
2131509311/reverse_edge: no store files to compact
2009-02-28 10:51:08,692 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
2131509311/inferred: no store files to compact
}}}
I didn't find any exception in the master log file.
Can any one help?
Regards,
Yan
Re: Exception when scanning table
Posted by Ryan Smith <ry...@gmail.com>.
Yes, 0.18.3 hadoop and 0.18.1 hbase loads the same data just fine. The only
issues I have with 0.18.x is a few map tasks that timeout on one of the many
processing MR jobs, but that is normal, i just have to set the timeout
higher. the data itself loads into hbase tables just fine.
Obviously im not using the same code, but its logically the same, the only
difference is using the new MapReduce API classes in 0.19
As far as ScannerTimeoutException in hbase 0.19.0, ill test it again on
hbase 0.19.1 once its released and let you know.
-alphaomega
On Wed, Mar 4, 2009 at 2:49 AM, stack <st...@duboce.net> wrote:
> Well, it messes up your client, right? Its not completing its scan?
>
> Is your client processing the return from hbase and taking longer than one
> minute to complete? If so, its lease on the remote server will have
> timedout in this time. Otherwise, do any of questions above to Ryan apply
> to you?
>
> Thanks,
> St.Ack
>
> On Tue, Mar 3, 2009 at 10:23 PM, Liu Yan <gz...@gmail.com> wrote:
>
> > Can I just ignore this error? I assume it is only a scanner error. What
> > caused this? Memory, disk space, max-file, I checked all these, but seems
> > very unlikly caused these conditions.
> >
> > Regards,
> > Yan
> >
> > 2009/3/1 Ryan Smith <ry...@gmail.com>
> >
> > > Hi guys,
> > >
> > > http://pastebin.com/m1c7b01da
> > >
> > > I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0.
> I
> > am
> > > loading millions of records in 20GB of data in a mapreduce job. Just
> > > trying
> > > to map (write) one text file to one hbase table. This job would
> complete
> > > fine on 0.18.x This is the first time ive tried to run this on
> 0.19.0.
> > > Ive repeated this test under several 0.19.0 clouds and i keep getting
> > this
> > > ScannerTimeoutException. I am going to test it once again on 0.18.x
> to
> > > make sure that is the problem.
> > >
> > > -alphaomega
> > >
> > >
> > >
> > > On Sun, Mar 1, 2009 at 3:39 AM, Liu Yan <gz...@gmail.com> wrote:
> > >
> > > > I have 2 tables, each with about 350K rows (I did "count" on them).
> > > >
> > > > For the first table, the scan kept running for about 5 hours before
> > > failing
> > > > out. For the second table, it took about 5 minutes to see the same
> > error.
> > > >
> > > > How to increase the timeout, I searched for "timeout" or "scan"
> keyword
> > > > under the ./conf folder, didn't find anything too helpful.
> > > >
> > > > Regards,
> > > > Yan
> > > >
> > > > 2009/3/1 stack <st...@duboce.net>
> > > >
> > > > > Before you hit the exception, had it been scanning fine?
> > > > > The exception from the regionserver is a complaint that an
> > outstanding
> > > > > scanner didn't check in within the configured client timeout. Was
> > > > > something
> > > > > happening on the client? Was the cell being fetched extra large?
> > You
> > > > > could
> > > > > up the timeout and see if that helps.
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > > On Sat, Feb 28, 2009 at 8:00 AM, Liu Yan <gz...@gmail.com>
> > > wrote:
> > > > >
> > > > > > hi,
> > > > > >
> > > > > > I tried to scan a table with "scan 'table_name'" command in the
> > hbase
> > > > > > shell,
> > > > > > and hit the following exception:
> > > > > >
> > > > > > {{{
> > > > > > NativeException: java.lang.RuntimeException:
> > > > > > org.apache.hadoop.hbase.client.ScannerTimeoutException
> > > > > > from org/apache/hadoop/hbase/client/HTable.java:1704:in
> > `hasNext'
> > > > > > from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
> > > > > > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
> > `invoke'
> > > > > > from java/lang/reflect/Method.java:597:in `invoke'
> > > > > > from org/jruby/javasupport/JavaMethod.java:250:in
> > > > > > `invokeWithExceptionHandling'
> > > > > > from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> > > > > > from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > > > > > from
> > > > > org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
> > > > > > `call'
> > > > > > from
> org/jruby/internal/runtime/methods/DynamicMethod.java:70:in
> > > > > `call'
> > > > > > from org/jruby/runtime/CallSite.java:295:in `call'
> > > > > > from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
> > > > > > from org/jruby/evaluator/ASTInterpreter.java:324:in
> > `evalInternal'
> > > > > > from org/jruby/evaluator/ASTInterpreter.java:1790:in
> `whileNode'
> > > > > > from org/jruby/evaluator/ASTInterpreter.java:505:in
> > `evalInternal'
> > > > > > from org/jruby/evaluator/ASTInterpreter.java:620:in
> `blockNode'
> > > > > > from org/jruby/evaluator/ASTInterpreter.java:318:in
> > `evalInternal'
> > > > > > ... 121 levels...
> > > > > > from
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
> > > > > > `call'
> > > > > > from
> org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> > > > > `call'
> > > > > > from
> > org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> > > > > `call'
> > > > > > from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > > > > > from org/jruby/runtime/CallSite.java:298:in `call'
> > > > > > from
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
> > > > > > `__file__'
> > > > > > from
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > > > `__file__'
> > > > > > from
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > > > `load'
> > > > > > from org/jruby/Ruby.java:512:in `runScript'
> > > > > > from org/jruby/Ruby.java:432:in `runNormally'
> > > > > > from org/jruby/Ruby.java:312:in `runFromMain'
> > > > > > from org/jruby/Main.java:144:in `run'
> > > > > > from org/jruby/Main.java:89:in `run'
> > > > > > from org/jruby/Main.java:80:in `main'
> > > > > > from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
> > > > > > from (hbase):3:in `binding
> > > > > > }}}
> > > > > >
> > > > > > In the regionserver log file, I found this:
> > > > > >
> > > > > > {{{
> > > > > > 2009-02-28 10:50:48,685 INFO
> > > > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > > > compaction completed on region
> > > > > > 1002_profiles,113495530684109376,1235659992234 in 0sec
> > > > > > 2009-02-28 10:50:54,779 INFO
> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > > > > > 310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer
> > > > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > > 310687422691357774
> > > > > > 2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer:
> IPC
> > > > > Server
> > > > > > handler 7 on 60020, call next(310687422691357774, 30) from
> > > > > > 10.254.51.127:54821: error
> > > > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > > 310687422691357774
> > > > > > org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > > > 310687422691357774 at
> > > > > >
> > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
> > > > > > .java:1568)
> > > > > > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown
> > > Source)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > > > > >
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > > > > > at
> > > > > >
> > > >
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> > > > > > 2009-02-28 10:51:08,687 INFO
> > > > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > > > starting compaction on region
> > > > > > 1002_profiles,266291625891696512,1235713612502
> > > > > > 2009-02-28 10:51:08,688 DEBUG
> > > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > > 2131509311/fetl: no store files to compact
> > > > > > 2009-02-28 10:51:08,689 DEBUG
> > > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > > 2131509311/pre_fetl: no store files to compact
> > > > > > 2009-02-28 10:51:08,691 DEBUG
> > > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > > 2131509311/scored: no store files to compact
> > > > > > 2009-02-28 10:51:08,692 DEBUG
> > > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > > 2131509311/reverse_edge: no store files to compact
> > > > > > 2009-02-28 10:51:08,692 DEBUG
> > > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > > 2131509311/inferred: no store files to compact
> > > > > > }}}
> > > > > >
> > > > > > I didn't find any exception in the master log file.
> > > > > >
> > > > > > Can any one help?
> > > > > >
> > > > > > Regards,
> > > > > > Yan
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Exception when scanning table
Posted by stack <st...@duboce.net>.
Well, it messes up your client, right? Its not completing its scan?
Is your client processing the return from hbase and taking longer than one
minute to complete? If so, its lease on the remote server will have
timedout in this time. Otherwise, do any of questions above to Ryan apply
to you?
Thanks,
St.Ack
On Tue, Mar 3, 2009 at 10:23 PM, Liu Yan <gz...@gmail.com> wrote:
> Can I just ignore this error? I assume it is only a scanner error. What
> caused this? Memory, disk space, max-file, I checked all these, but seems
> very unlikly caused these conditions.
>
> Regards,
> Yan
>
> 2009/3/1 Ryan Smith <ry...@gmail.com>
>
> > Hi guys,
> >
> > http://pastebin.com/m1c7b01da
> >
> > I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0. I
> am
> > loading millions of records in 20GB of data in a mapreduce job. Just
> > trying
> > to map (write) one text file to one hbase table. This job would complete
> > fine on 0.18.x This is the first time ive tried to run this on 0.19.0.
> > Ive repeated this test under several 0.19.0 clouds and i keep getting
> this
> > ScannerTimeoutException. I am going to test it once again on 0.18.x to
> > make sure that is the problem.
> >
> > -alphaomega
> >
> >
> >
> > On Sun, Mar 1, 2009 at 3:39 AM, Liu Yan <gz...@gmail.com> wrote:
> >
> > > I have 2 tables, each with about 350K rows (I did "count" on them).
> > >
> > > For the first table, the scan kept running for about 5 hours before
> > failing
> > > out. For the second table, it took about 5 minutes to see the same
> error.
> > >
> > > How to increase the timeout, I searched for "timeout" or "scan" keyword
> > > under the ./conf folder, didn't find anything too helpful.
> > >
> > > Regards,
> > > Yan
> > >
> > > 2009/3/1 stack <st...@duboce.net>
> > >
> > > > Before you hit the exception, had it been scanning fine?
> > > > The exception from the regionserver is a complaint that an
> outstanding
> > > > scanner didn't check in within the configured client timeout. Was
> > > > something
> > > > happening on the client? Was the cell being fetched extra large?
> You
> > > > could
> > > > up the timeout and see if that helps.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > > On Sat, Feb 28, 2009 at 8:00 AM, Liu Yan <gz...@gmail.com>
> > wrote:
> > > >
> > > > > hi,
> > > > >
> > > > > I tried to scan a table with "scan 'table_name'" command in the
> hbase
> > > > > shell,
> > > > > and hit the following exception:
> > > > >
> > > > > {{{
> > > > > NativeException: java.lang.RuntimeException:
> > > > > org.apache.hadoop.hbase.client.ScannerTimeoutException
> > > > > from org/apache/hadoop/hbase/client/HTable.java:1704:in
> `hasNext'
> > > > > from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
> > > > > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
> `invoke'
> > > > > from java/lang/reflect/Method.java:597:in `invoke'
> > > > > from org/jruby/javasupport/JavaMethod.java:250:in
> > > > > `invokeWithExceptionHandling'
> > > > > from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> > > > > from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > > > > from
> > > > org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
> > > > > `call'
> > > > > from org/jruby/internal/runtime/methods/DynamicMethod.java:70:in
> > > > `call'
> > > > > from org/jruby/runtime/CallSite.java:295:in `call'
> > > > > from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
> > > > > from org/jruby/evaluator/ASTInterpreter.java:324:in
> `evalInternal'
> > > > > from org/jruby/evaluator/ASTInterpreter.java:1790:in `whileNode'
> > > > > from org/jruby/evaluator/ASTInterpreter.java:505:in
> `evalInternal'
> > > > > from org/jruby/evaluator/ASTInterpreter.java:620:in `blockNode'
> > > > > from org/jruby/evaluator/ASTInterpreter.java:318:in
> `evalInternal'
> > > > > ... 121 levels...
> > > > > from
> > > > >
> > > > >
> > > >
> > >
> >
> ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
> > > > > `call'
> > > > > from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> > > > `call'
> > > > > from
> org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> > > > `call'
> > > > > from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > > > > from org/jruby/runtime/CallSite.java:298:in `call'
> > > > > from
> > > > >
> > > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
> > > > > `__file__'
> > > > > from
> > > > >
> > > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > > `__file__'
> > > > > from
> > > > >
> > > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > > `load'
> > > > > from org/jruby/Ruby.java:512:in `runScript'
> > > > > from org/jruby/Ruby.java:432:in `runNormally'
> > > > > from org/jruby/Ruby.java:312:in `runFromMain'
> > > > > from org/jruby/Main.java:144:in `run'
> > > > > from org/jruby/Main.java:89:in `run'
> > > > > from org/jruby/Main.java:80:in `main'
> > > > > from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
> > > > > from (hbase):3:in `binding
> > > > > }}}
> > > > >
> > > > > In the regionserver log file, I found this:
> > > > >
> > > > > {{{
> > > > > 2009-02-28 10:50:48,685 INFO
> > > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > > compaction completed on region
> > > > > 1002_profiles,113495530684109376,1235659992234 in 0sec
> > > > > 2009-02-28 10:50:54,779 INFO
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > > > > 310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer
> > > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > 310687422691357774
> > > > > 2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > > > Server
> > > > > handler 7 on 60020, call next(310687422691357774, 30) from
> > > > > 10.254.51.127:54821: error
> > > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > 310687422691357774
> > > > > org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > > 310687422691357774 at
> > > > >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
> > > > > .java:1568)
> > > > > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown
> > Source)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> at
> > > > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > > > > at
> > > > >
> > >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> > > > > 2009-02-28 10:51:08,687 INFO
> > > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > > starting compaction on region
> > > > > 1002_profiles,266291625891696512,1235713612502
> > > > > 2009-02-28 10:51:08,688 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > 2131509311/fetl: no store files to compact
> > > > > 2009-02-28 10:51:08,689 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > 2131509311/pre_fetl: no store files to compact
> > > > > 2009-02-28 10:51:08,691 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > 2131509311/scored: no store files to compact
> > > > > 2009-02-28 10:51:08,692 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > 2131509311/reverse_edge: no store files to compact
> > > > > 2009-02-28 10:51:08,692 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > > 2131509311/inferred: no store files to compact
> > > > > }}}
> > > > >
> > > > > I didn't find any exception in the master log file.
> > > > >
> > > > > Can any one help?
> > > > >
> > > > > Regards,
> > > > > Yan
> > > > >
> > > >
> > >
> >
>
Re: Exception when scanning table
Posted by Liu Yan <gz...@gmail.com>.
Can I just ignore this error? I assume it is only a scanner error. What
caused this? Memory, disk space, max-file, I checked all these, but seems
very unlikly caused these conditions.
Regards,
Yan
2009/3/1 Ryan Smith <ry...@gmail.com>
> Hi guys,
>
> http://pastebin.com/m1c7b01da
>
> I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0. I am
> loading millions of records in 20GB of data in a mapreduce job. Just
> trying
> to map (write) one text file to one hbase table. This job would complete
> fine on 0.18.x This is the first time ive tried to run this on 0.19.0.
> Ive repeated this test under several 0.19.0 clouds and i keep getting this
> ScannerTimeoutException. I am going to test it once again on 0.18.x to
> make sure that is the problem.
>
> -alphaomega
>
>
>
> On Sun, Mar 1, 2009 at 3:39 AM, Liu Yan <gz...@gmail.com> wrote:
>
> > I have 2 tables, each with about 350K rows (I did "count" on them).
> >
> > For the first table, the scan kept running for about 5 hours before
> failing
> > out. For the second table, it took about 5 minutes to see the same error.
> >
> > How to increase the timeout, I searched for "timeout" or "scan" keyword
> > under the ./conf folder, didn't find anything too helpful.
> >
> > Regards,
> > Yan
> >
> > 2009/3/1 stack <st...@duboce.net>
> >
> > > Before you hit the exception, had it been scanning fine?
> > > The exception from the regionserver is a complaint that an outstanding
> > > scanner didn't check in within the configured client timeout. Was
> > > something
> > > happening on the client? Was the cell being fetched extra large? You
> > > could
> > > up the timeout and see if that helps.
> > >
> > > St.Ack
> > >
> > >
> > > On Sat, Feb 28, 2009 at 8:00 AM, Liu Yan <gz...@gmail.com>
> wrote:
> > >
> > > > hi,
> > > >
> > > > I tried to scan a table with "scan 'table_name'" command in the hbase
> > > > shell,
> > > > and hit the following exception:
> > > >
> > > > {{{
> > > > NativeException: java.lang.RuntimeException:
> > > > org.apache.hadoop.hbase.client.ScannerTimeoutException
> > > > from org/apache/hadoop/hbase/client/HTable.java:1704:in `hasNext'
> > > > from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
> > > > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> > > > from java/lang/reflect/Method.java:597:in `invoke'
> > > > from org/jruby/javasupport/JavaMethod.java:250:in
> > > > `invokeWithExceptionHandling'
> > > > from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> > > > from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > > > from
> > > org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
> > > > `call'
> > > > from org/jruby/internal/runtime/methods/DynamicMethod.java:70:in
> > > `call'
> > > > from org/jruby/runtime/CallSite.java:295:in `call'
> > > > from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
> > > > from org/jruby/evaluator/ASTInterpreter.java:324:in `evalInternal'
> > > > from org/jruby/evaluator/ASTInterpreter.java:1790:in `whileNode'
> > > > from org/jruby/evaluator/ASTInterpreter.java:505:in `evalInternal'
> > > > from org/jruby/evaluator/ASTInterpreter.java:620:in `blockNode'
> > > > from org/jruby/evaluator/ASTInterpreter.java:318:in `evalInternal'
> > > > ... 121 levels...
> > > > from
> > > >
> > > >
> > >
> >
> ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
> > > > `call'
> > > > from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> > > `call'
> > > > from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> > > `call'
> > > > from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > > > from org/jruby/runtime/CallSite.java:298:in `call'
> > > > from
> > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
> > > > `__file__'
> > > > from
> > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > `__file__'
> > > > from
> > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > `load'
> > > > from org/jruby/Ruby.java:512:in `runScript'
> > > > from org/jruby/Ruby.java:432:in `runNormally'
> > > > from org/jruby/Ruby.java:312:in `runFromMain'
> > > > from org/jruby/Main.java:144:in `run'
> > > > from org/jruby/Main.java:89:in `run'
> > > > from org/jruby/Main.java:80:in `main'
> > > > from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
> > > > from (hbase):3:in `binding
> > > > }}}
> > > >
> > > > In the regionserver log file, I found this:
> > > >
> > > > {{{
> > > > 2009-02-28 10:50:48,685 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > compaction completed on region
> > > > 1002_profiles,113495530684109376,1235659992234 in 0sec
> > > > 2009-02-28 10:50:54,779 INFO
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > > > 310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer
> > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > 310687422691357774
> > > > 2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > > Server
> > > > handler 7 on 60020, call next(310687422691357774, 30) from
> > > > 10.254.51.127:54821: error
> > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > 310687422691357774
> > > > org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > 310687422691357774 at
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
> > > > .java:1568)
> > > > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown
> Source)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > at java.lang.reflect.Method.invoke(Method.java:597) at
> > > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > > > at
> > > >
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> > > > 2009-02-28 10:51:08,687 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > starting compaction on region
> > > > 1002_profiles,266291625891696512,1235713612502
> > > > 2009-02-28 10:51:08,688 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/fetl: no store files to compact
> > > > 2009-02-28 10:51:08,689 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/pre_fetl: no store files to compact
> > > > 2009-02-28 10:51:08,691 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/scored: no store files to compact
> > > > 2009-02-28 10:51:08,692 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/reverse_edge: no store files to compact
> > > > 2009-02-28 10:51:08,692 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/inferred: no store files to compact
> > > > }}}
> > > >
> > > > I didn't find any exception in the master log file.
> > > >
> > > > Can any one help?
> > > >
> > > > Regards,
> > > > Yan
> > > >
> > >
> >
>
Re: Exception when scanning table
Posted by stack <st...@duboce.net>.
Let us know how it did on 0.18.0 Ryan.
Basically, the message is saying that the interval between invocations of
next by the client was longer than the 60 seconds default. Here is the
relevant property:
<property>
<name>hbase.regionserver.lease.period</name>
<value>60000</value>
<description>HRegion server lease period in milliseconds. Default is
60 seconds. Clients must report in within this period else they are
considered dead.</description>
</property>
You could try upping this timeout.
Otherwise, was the server struggling at this time? You could enable metrics
-- the jvm ones in particular -- or enable gc logging by adding jvm parms to
HBASE_OPTS in hbase-env.sh to see if its struggling doing a full gc (See
also end of troubleshooting section for a suggestion that you use a
different garbage collector). Do you see instances of 'slept ten times
longer than expected...' in your logs before the scanner timeout or anything
that might give a clue about the server being hung up in any way?
What about the client. Could it have been stalled by competing processes on
same machine?
I don't think this code changed between 0.18 and 0.19 but if you find that
it works on 0.18, then for sure something is up.
Thanks Ryan,
St.Ack
On Sun, Mar 1, 2009 at 4:49 AM, Ryan Smith <ry...@gmail.com>wrote:
> Hi guys,
>
> http://pastebin.com/m1c7b01da
>
> I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0. I am
> loading millions of records in 20GB of data in a mapreduce job. Just
> trying
> to map (write) one text file to one hbase table. This job would complete
> fine on 0.18.x This is the first time ive tried to run this on 0.19.0.
> Ive repeated this test under several 0.19.0 clouds and i keep getting this
> ScannerTimeoutException. I am going to test it once again on 0.18.x to
> make sure that is the problem.
>
> -alphaomega
>
>
>
> On Sun, Mar 1, 2009 at 3:39 AM, Liu Yan <gz...@gmail.com> wrote:
>
> > I have 2 tables, each with about 350K rows (I did "count" on them).
> >
> > For the first table, the scan kept running for about 5 hours before
> failing
> > out. For the second table, it took about 5 minutes to see the same error.
> >
> > How to increase the timeout, I searched for "timeout" or "scan" keyword
> > under the ./conf folder, didn't find anything too helpful.
> >
> > Regards,
> > Yan
> >
> > 2009/3/1 stack <st...@duboce.net>
> >
> > > Before you hit the exception, had it been scanning fine?
> > > The exception from the regionserver is a complaint that an outstanding
> > > scanner didn't check in within the configured client timeout. Was
> > > something
> > > happening on the client? Was the cell being fetched extra large? You
> > > could
> > > up the timeout and see if that helps.
> > >
> > > St.Ack
> > >
> > >
> > > On Sat, Feb 28, 2009 at 8:00 AM, Liu Yan <gz...@gmail.com>
> wrote:
> > >
> > > > hi,
> > > >
> > > > I tried to scan a table with "scan 'table_name'" command in the hbase
> > > > shell,
> > > > and hit the following exception:
> > > >
> > > > {{{
> > > > NativeException: java.lang.RuntimeException:
> > > > org.apache.hadoop.hbase.client.ScannerTimeoutException
> > > > from org/apache/hadoop/hbase/client/HTable.java:1704:in `hasNext'
> > > > from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
> > > > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> > > > from java/lang/reflect/Method.java:597:in `invoke'
> > > > from org/jruby/javasupport/JavaMethod.java:250:in
> > > > `invokeWithExceptionHandling'
> > > > from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> > > > from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > > > from
> > > org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
> > > > `call'
> > > > from org/jruby/internal/runtime/methods/DynamicMethod.java:70:in
> > > `call'
> > > > from org/jruby/runtime/CallSite.java:295:in `call'
> > > > from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
> > > > from org/jruby/evaluator/ASTInterpreter.java:324:in `evalInternal'
> > > > from org/jruby/evaluator/ASTInterpreter.java:1790:in `whileNode'
> > > > from org/jruby/evaluator/ASTInterpreter.java:505:in `evalInternal'
> > > > from org/jruby/evaluator/ASTInterpreter.java:620:in `blockNode'
> > > > from org/jruby/evaluator/ASTInterpreter.java:318:in `evalInternal'
> > > > ... 121 levels...
> > > > from
> > > >
> > > >
> > >
> >
> ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
> > > > `call'
> > > > from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> > > `call'
> > > > from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> > > `call'
> > > > from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > > > from org/jruby/runtime/CallSite.java:298:in `call'
> > > > from
> > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
> > > > `__file__'
> > > > from
> > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > `__file__'
> > > > from
> > > >
> > > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > > `load'
> > > > from org/jruby/Ruby.java:512:in `runScript'
> > > > from org/jruby/Ruby.java:432:in `runNormally'
> > > > from org/jruby/Ruby.java:312:in `runFromMain'
> > > > from org/jruby/Main.java:144:in `run'
> > > > from org/jruby/Main.java:89:in `run'
> > > > from org/jruby/Main.java:80:in `main'
> > > > from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
> > > > from (hbase):3:in `binding
> > > > }}}
> > > >
> > > > In the regionserver log file, I found this:
> > > >
> > > > {{{
> > > > 2009-02-28 10:50:48,685 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > compaction completed on region
> > > > 1002_profiles,113495530684109376,1235659992234 in 0sec
> > > > 2009-02-28 10:50:54,779 INFO
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > > > 310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer
> > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > 310687422691357774
> > > > 2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > > Server
> > > > handler 7 on 60020, call next(310687422691357774, 30) from
> > > > 10.254.51.127:54821: error
> > > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > 310687422691357774
> > > > org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > > 310687422691357774 at
> > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
> > > > .java:1568)
> > > > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown
> Source)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > at java.lang.reflect.Method.invoke(Method.java:597) at
> > > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > > > at
> > > >
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> > > > 2009-02-28 10:51:08,687 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > starting compaction on region
> > > > 1002_profiles,266291625891696512,1235713612502
> > > > 2009-02-28 10:51:08,688 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/fetl: no store files to compact
> > > > 2009-02-28 10:51:08,689 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/pre_fetl: no store files to compact
> > > > 2009-02-28 10:51:08,691 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/scored: no store files to compact
> > > > 2009-02-28 10:51:08,692 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/reverse_edge: no store files to compact
> > > > 2009-02-28 10:51:08,692 DEBUG
> > > org.apache.hadoop.hbase.regionserver.HStore:
> > > > 2131509311/inferred: no store files to compact
> > > > }}}
> > > >
> > > > I didn't find any exception in the master log file.
> > > >
> > > > Can any one help?
> > > >
> > > > Regards,
> > > > Yan
> > > >
> > >
> >
>
Re: Exception when scanning table
Posted by Ryan Smith <ry...@gmail.com>.
Hi guys,
http://pastebin.com/m1c7b01da
I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0. I am
loading millions of records in 20GB of data in a mapreduce job. Just trying
to map (write) one text file to one hbase table. This job would complete
fine on 0.18.x This is the first time ive tried to run this on 0.19.0.
Ive repeated this test under several 0.19.0 clouds and i keep getting this
ScannerTimeoutException. I am going to test it once again on 0.18.x to
make sure that is the problem.
-alphaomega
On Sun, Mar 1, 2009 at 3:39 AM, Liu Yan <gz...@gmail.com> wrote:
> I have 2 tables, each with about 350K rows (I did "count" on them).
>
> For the first table, the scan kept running for about 5 hours before failing
> out. For the second table, it took about 5 minutes to see the same error.
>
> How to increase the timeout, I searched for "timeout" or "scan" keyword
> under the ./conf folder, didn't find anything too helpful.
>
> Regards,
> Yan
>
> 2009/3/1 stack <st...@duboce.net>
>
> > Before you hit the exception, had it been scanning fine?
> > The exception from the regionserver is a complaint that an outstanding
> > scanner didn't check in within the configured client timeout. Was
> > something
> > happening on the client? Was the cell being fetched extra large? You
> > could
> > up the timeout and see if that helps.
> >
> > St.Ack
> >
> >
> > On Sat, Feb 28, 2009 at 8:00 AM, Liu Yan <gz...@gmail.com> wrote:
> >
> > > hi,
> > >
> > > I tried to scan a table with "scan 'table_name'" command in the hbase
> > > shell,
> > > and hit the following exception:
> > >
> > > {{{
> > > NativeException: java.lang.RuntimeException:
> > > org.apache.hadoop.hbase.client.ScannerTimeoutException
> > > from org/apache/hadoop/hbase/client/HTable.java:1704:in `hasNext'
> > > from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
> > > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> > > from java/lang/reflect/Method.java:597:in `invoke'
> > > from org/jruby/javasupport/JavaMethod.java:250:in
> > > `invokeWithExceptionHandling'
> > > from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> > > from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > > from
> > org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
> > > `call'
> > > from org/jruby/internal/runtime/methods/DynamicMethod.java:70:in
> > `call'
> > > from org/jruby/runtime/CallSite.java:295:in `call'
> > > from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
> > > from org/jruby/evaluator/ASTInterpreter.java:324:in `evalInternal'
> > > from org/jruby/evaluator/ASTInterpreter.java:1790:in `whileNode'
> > > from org/jruby/evaluator/ASTInterpreter.java:505:in `evalInternal'
> > > from org/jruby/evaluator/ASTInterpreter.java:620:in `blockNode'
> > > from org/jruby/evaluator/ASTInterpreter.java:318:in `evalInternal'
> > > ... 121 levels...
> > > from
> > >
> > >
> >
> ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
> > > `call'
> > > from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> > `call'
> > > from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> > `call'
> > > from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > > from org/jruby/runtime/CallSite.java:298:in `call'
> > > from
> > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
> > > `__file__'
> > > from
> > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > `__file__'
> > > from
> > >
> > >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > > `load'
> > > from org/jruby/Ruby.java:512:in `runScript'
> > > from org/jruby/Ruby.java:432:in `runNormally'
> > > from org/jruby/Ruby.java:312:in `runFromMain'
> > > from org/jruby/Main.java:144:in `run'
> > > from org/jruby/Main.java:89:in `run'
> > > from org/jruby/Main.java:80:in `main'
> > > from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
> > > from (hbase):3:in `binding
> > > }}}
> > >
> > > In the regionserver log file, I found this:
> > >
> > > {{{
> > > 2009-02-28 10:50:48,685 INFO
> > org.apache.hadoop.hbase.regionserver.HRegion:
> > > compaction completed on region
> > > 1002_profiles,113495530684109376,1235659992234 in 0sec
> > > 2009-02-28 10:50:54,779 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > > 310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
> > > org.apache.hadoop.hbase.regionserver.HRegionServer
> > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > 310687422691357774
> > > 2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > handler 7 on 60020, call next(310687422691357774, 30) from
> > > 10.254.51.127:54821: error
> > > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> > 310687422691357774
> > > org.apache.hadoop.hbase.UnknownScannerException: Name:
> > > 310687422691357774 at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
> > > .java:1568)
> > > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597) at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > > at
> > >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> > > 2009-02-28 10:51:08,687 INFO
> > org.apache.hadoop.hbase.regionserver.HRegion:
> > > starting compaction on region
> > > 1002_profiles,266291625891696512,1235713612502
> > > 2009-02-28 10:51:08,688 DEBUG
> > org.apache.hadoop.hbase.regionserver.HStore:
> > > 2131509311/fetl: no store files to compact
> > > 2009-02-28 10:51:08,689 DEBUG
> > org.apache.hadoop.hbase.regionserver.HStore:
> > > 2131509311/pre_fetl: no store files to compact
> > > 2009-02-28 10:51:08,691 DEBUG
> > org.apache.hadoop.hbase.regionserver.HStore:
> > > 2131509311/scored: no store files to compact
> > > 2009-02-28 10:51:08,692 DEBUG
> > org.apache.hadoop.hbase.regionserver.HStore:
> > > 2131509311/reverse_edge: no store files to compact
> > > 2009-02-28 10:51:08,692 DEBUG
> > org.apache.hadoop.hbase.regionserver.HStore:
> > > 2131509311/inferred: no store files to compact
> > > }}}
> > >
> > > I didn't find any exception in the master log file.
> > >
> > > Can any one help?
> > >
> > > Regards,
> > > Yan
> > >
> >
>
Re: Exception when scanning table
Posted by Liu Yan <gz...@gmail.com>.
I have 2 tables, each with about 350K rows (I did "count" on them).
For the first table, the scan kept running for about 5 hours before failing
out. For the second table, it took about 5 minutes to see the same error.
How to increase the timeout, I searched for "timeout" or "scan" keyword
under the ./conf folder, didn't find anything too helpful.
Regards,
Yan
2009/3/1 stack <st...@duboce.net>
> Before you hit the exception, had it been scanning fine?
> The exception from the regionserver is a complaint that an outstanding
> scanner didn't check in within the configured client timeout. Was
> something
> happening on the client? Was the cell being fetched extra large? You
> could
> up the timeout and see if that helps.
>
> St.Ack
>
>
> On Sat, Feb 28, 2009 at 8:00 AM, Liu Yan <gz...@gmail.com> wrote:
>
> > hi,
> >
> > I tried to scan a table with "scan 'table_name'" command in the hbase
> > shell,
> > and hit the following exception:
> >
> > {{{
> > NativeException: java.lang.RuntimeException:
> > org.apache.hadoop.hbase.client.ScannerTimeoutException
> > from org/apache/hadoop/hbase/client/HTable.java:1704:in `hasNext'
> > from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> > from java/lang/reflect/Method.java:597:in `invoke'
> > from org/jruby/javasupport/JavaMethod.java:250:in
> > `invokeWithExceptionHandling'
> > from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> > from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > from
> org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
> > `call'
> > from org/jruby/internal/runtime/methods/DynamicMethod.java:70:in
> `call'
> > from org/jruby/runtime/CallSite.java:295:in `call'
> > from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
> > from org/jruby/evaluator/ASTInterpreter.java:324:in `evalInternal'
> > from org/jruby/evaluator/ASTInterpreter.java:1790:in `whileNode'
> > from org/jruby/evaluator/ASTInterpreter.java:505:in `evalInternal'
> > from org/jruby/evaluator/ASTInterpreter.java:620:in `blockNode'
> > from org/jruby/evaluator/ASTInterpreter.java:318:in `evalInternal'
> > ... 121 levels...
> > from
> >
> >
> ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
> > `call'
> > from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> `call'
> > from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> `call'
> > from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > from org/jruby/runtime/CallSite.java:298:in `call'
> > from
> >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
> > `__file__'
> > from
> >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > `__file__'
> > from
> >
> >
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> > `load'
> > from org/jruby/Ruby.java:512:in `runScript'
> > from org/jruby/Ruby.java:432:in `runNormally'
> > from org/jruby/Ruby.java:312:in `runFromMain'
> > from org/jruby/Main.java:144:in `run'
> > from org/jruby/Main.java:89:in `run'
> > from org/jruby/Main.java:80:in `main'
> > from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
> > from (hbase):3:in `binding
> > }}}
> >
> > In the regionserver log file, I found this:
> >
> > {{{
> > 2009-02-28 10:50:48,685 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > compaction completed on region
> > 1002_profiles,113495530684109376,1235659992234 in 0sec
> > 2009-02-28 10:50:54,779 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > 310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer
> > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> 310687422691357774
> > 2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 7 on 60020, call next(310687422691357774, 30) from
> > 10.254.51.127:54821: error
> > : org.apache.hadoop.hbase.UnknownScannerException: Name:
> 310687422691357774
> > org.apache.hadoop.hbase.UnknownScannerException: Name:
> > 310687422691357774 at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
> > .java:1568)
> > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597) at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> > 2009-02-28 10:51:08,687 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > starting compaction on region
> > 1002_profiles,266291625891696512,1235713612502
> > 2009-02-28 10:51:08,688 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore:
> > 2131509311/fetl: no store files to compact
> > 2009-02-28 10:51:08,689 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore:
> > 2131509311/pre_fetl: no store files to compact
> > 2009-02-28 10:51:08,691 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore:
> > 2131509311/scored: no store files to compact
> > 2009-02-28 10:51:08,692 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore:
> > 2131509311/reverse_edge: no store files to compact
> > 2009-02-28 10:51:08,692 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore:
> > 2131509311/inferred: no store files to compact
> > }}}
> >
> > I didn't find any exception in the master log file.
> >
> > Can any one help?
> >
> > Regards,
> > Yan
> >
>
Re: Exception when scanning table
Posted by stack <st...@duboce.net>.
Before you hit the exception, had it been scanning fine?
The exception from the regionserver is a complaint that an outstanding
scanner didn't check in within the configured client timeout. Was something
happening on the client? Was the cell being fetched extra large? You could
up the timeout and see if that helps.
St.Ack
On Sat, Feb 28, 2009 at 8:00 AM, Liu Yan <gz...@gmail.com> wrote:
> hi,
>
> I tried to scan a table with "scan 'table_name'" command in the hbase
> shell,
> and hit the following exception:
>
> {{{
> NativeException: java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.ScannerTimeoutException
> from org/apache/hadoop/hbase/client/HTable.java:1704:in `hasNext'
> from sun.reflect.GeneratedMethodAccessor10:-1:in `invoke'
> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> from java/lang/reflect/Method.java:597:in `invoke'
> from org/jruby/javasupport/JavaMethod.java:250:in
> `invokeWithExceptionHandling'
> from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> from org/jruby/javasupport/JavaClass.java:416:in `execute'
> from org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
> `call'
> from org/jruby/internal/runtime/methods/DynamicMethod.java:70:in `call'
> from org/jruby/runtime/CallSite.java:295:in `call'
> from org/jruby/evaluator/ASTInterpreter.java:646:in `callNode'
> from org/jruby/evaluator/ASTInterpreter.java:324:in `evalInternal'
> from org/jruby/evaluator/ASTInterpreter.java:1790:in `whileNode'
> from org/jruby/evaluator/ASTInterpreter.java:505:in `evalInternal'
> from org/jruby/evaluator/ASTInterpreter.java:620:in `blockNode'
> from org/jruby/evaluator/ASTInterpreter.java:318:in `evalInternal'
> ... 121 levels...
> from
>
> ruby.usr.local.hbase_minus_0_dot_19_dot_0.bin.hirbInvokermethod__32$RUBY$startOpt:-1:in
> `call'
> from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
> from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
> from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> from org/jruby/runtime/CallSite.java:298:in `call'
> from
>
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:429:in
> `__file__'
> from
>
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> `__file__'
> from
>
> ruby/usr/local/hbase_minus_0_dot_19_dot_0/bin//usr/local/hbase/bin/../bin/hirb.rb:-1:in
> `load'
> from org/jruby/Ruby.java:512:in `runScript'
> from org/jruby/Ruby.java:432:in `runNormally'
> from org/jruby/Ruby.java:312:in `runFromMain'
> from org/jruby/Main.java:144:in `run'
> from org/jruby/Main.java:89:in `run'
> from org/jruby/Main.java:80:in `main'
> from /usr/local/hbase/bin/../bin/hirb.rb:334:in `scan'
> from (hbase):3:in `binding
> }}}
>
> In the regionserver log file, I found this:
>
> {{{
> 2009-02-28 10:50:48,685 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region
> 1002_profiles,113495530684109376,1235659992234 in 0sec
> 2009-02-28 10:50:54,779 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 310687422691357774 lease expired2009-02-28 10:50:55,869 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer
> : org.apache.hadoop.hbase.UnknownScannerException: Name: 310687422691357774
> 2009-02-28 10:50:55,871 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020, call next(310687422691357774, 30) from
> 10.254.51.127:54821: error
> : org.apache.hadoop.hbase.UnknownScannerException: Name: 310687422691357774
> org.apache.hadoop.hbase.UnknownScannerException: Name:
> 310687422691357774 at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer
> .java:1568)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597) at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> 2009-02-28 10:51:08,687 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> starting compaction on region
> 1002_profiles,266291625891696512,1235713612502
> 2009-02-28 10:51:08,688 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> 2131509311/fetl: no store files to compact
> 2009-02-28 10:51:08,689 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> 2131509311/pre_fetl: no store files to compact
> 2009-02-28 10:51:08,691 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> 2131509311/scored: no store files to compact
> 2009-02-28 10:51:08,692 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> 2131509311/reverse_edge: no store files to compact
> 2009-02-28 10:51:08,692 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> 2131509311/inferred: no store files to compact
> }}}
>
> I didn't find any exception in the master log file.
>
> Can any one help?
>
> Regards,
> Yan
>