You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vidosh Sahu <vi...@girnarsoft.com> on 2013/02/16 07:03:47 UTC
Regarding ScannerTimeout Exception
Hi,
I am using HBase 0.92.1-cdh4.1.1. There is mapper job which do a lot of
operation. This job is getting failed with the below operation. It looks
like the scanner operation is timed out. How to increase it using Cloudera
Manager Web Console. Can anyone please help me out.
2013-02-16 05:30:45,978 [Thread-158] WARN
org.apache.hadoop.mapred.LocalJobRunner - job_local_0005
org.apache.hadoop.hbase.client.ScannerTimeoutException: 614527ms passed
since the last invocation, timeout is currently set to 60000
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302)
at
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:133)
at
org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:142)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: -5689607602334883791
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2110)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
... 8 more
2013-02-16 05:30:46,016 [communication thread] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2013-02-16 05:30:46,424 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - Job complete: job_local_0005
2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - Counters: 5
2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - FileSystemCounters
2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - FILE_BYTES_READ=3827305
2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - FILE_BYTES_WRITTEN=3967770
2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - Map-Reduce Framework
2013-02-16 05:30:46,432 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - Map input records=2000
2013-02-16 05:30:46,432 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - Spilled Records=0
2013-02-16 05:30:46,433 [pool-1-thread-1] INFO
org.apache.hadoop.mapred.JobClient - Map output records=1040
Thanks,
Vidosh
Re: Regarding ScannerTimeout Exception
Posted by Marcos Ortiz <ml...@uci.cu>.
Regards, Vidosh. My responses in line
On 02/16/2013 01:03 AM, Vidosh Sahu wrote:
> Hi,
>
> I am using HBase 0.92.1-cdh4.1.1. There is mapper job which do a lot of
> operation. This job is getting failed with the below operation. It looks
> like the scanner operation is timed out. How to increase it using Cloudera
> Manager Web Console. Can anyone please help me out.
Well, this questions should be in CDH mailing list (cdh-user@cloudera.org).
Another thing is that Scanner´s performance have seen a lot of improvements
until the stable release (which is 0.94.5 right now, thanks to Lars H.
for the information).
I remembered a great talk by Mikhail Bautin called "Optimizing HBase
scanner performance"
in the last HBaseCon who explained all things done in this particular
topic, essentially:[1]
- HBASE-4433: Avoid extra next if done with row/column
- HBASE-4434: Don't do HFile Scanner next() unless the next KV is needed
- HBASE-2794: Multi-column Bloom Filters
- HBASE-4465: Lazy Seek
- HBASE-4469: Utilize existing ROWCOL Bloom filter
- HBASE-4532: Added a separate ROW-only Bloom filter for DeleteFamily
- HBASE-4585: Seek on deleted KV
- HBASE-4962: Top-of-the-column seek
So, you can see that you should upgrade your HBase installation. CDH 4.2
is out, and CDH 5.0 will be available very soon.[2]
[1] http://www.slideshare.net/cloudera/tag/hbasecon-2012
[2]
http://blog.cloudera.com/blog/2012/11/apache-hbase-assignmentmanager-improvements/
>
> 2013-02-16 05:30:45,978 [Thread-158] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0005
> org.apache.hadoop.hbase.client.ScannerTimeoutException: 614527ms passed
> since the last invocation, timeout is currently set to 60000
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:133)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:142)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
> org.apache.hadoop.hbase.UnknownScannerException: Name: -5689607602334883791
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2110)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
> ... 8 more
> 2013-02-16 05:30:46,016 [communication thread] INFO
> org.apache.hadoop.mapred.LocalJobRunner -
> 2013-02-16 05:30:46,424 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Job complete: job_local_0005
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Counters: 5
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - FileSystemCounters
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - FILE_BYTES_READ=3827305
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - FILE_BYTES_WRITTEN=3967770
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Map-Reduce Framework
> 2013-02-16 05:30:46,432 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Map input records=2000
> 2013-02-16 05:30:46,432 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Spilled Records=0
> 2013-02-16 05:30:46,433 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Map output records=1040
>
>
> Thanks,
> Vidosh
>
--
Marcos Ortiz Valmaseda,
Product Manager && Data Scientist at UCI
Blog: http://marcosluis2186.posterous.com
Twitter: @marcosluis2186 <http://twitter.com/marcosluis2186>
Re: Regarding ScannerTimeout Exception
Posted by Stack <st...@duboce.net>.
What Marcos says.
Also read this:
http://hbase.apache.org/book.html#hbase.client.scanner.caching and what is
happening on your server when you get a timeout? Is it working hard?
Swapping? Processing other stuff? Is there a process running beside it
contending for resources (e.g. mapreduce?).
St.Ack
On Fri, Feb 15, 2013 at 10:03 PM, Vidosh Sahu <vi...@girnarsoft.com> wrote:
> Hi,
>
> I am using HBase 0.92.1-cdh4.1.1. There is mapper job which do a lot of
> operation. This job is getting failed with the below operation. It looks
> like the scanner operation is timed out. How to increase it using Cloudera
> Manager Web Console. Can anyone please help me out.
>
> 2013-02-16 05:30:45,978 [Thread-158] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0005
> org.apache.hadoop.hbase.client.ScannerTimeoutException: 614527ms passed
> since the last invocation, timeout is currently set to 60000
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302)
> at
>
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:133)
> at
>
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:142)
> at
>
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
> org.apache.hadoop.hbase.UnknownScannerException: Name: -5689607602334883791
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2110)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
> at
>
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
> at
>
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
> ... 8 more
> 2013-02-16 05:30:46,016 [communication thread] INFO
> org.apache.hadoop.mapred.LocalJobRunner -
> 2013-02-16 05:30:46,424 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Job complete: job_local_0005
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Counters: 5
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - FileSystemCounters
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - FILE_BYTES_READ=3827305
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - FILE_BYTES_WRITTEN=3967770
> 2013-02-16 05:30:46,431 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Map-Reduce Framework
> 2013-02-16 05:30:46,432 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Map input records=2000
> 2013-02-16 05:30:46,432 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Spilled Records=0
> 2013-02-16 05:30:46,433 [pool-1-thread-1] INFO
> org.apache.hadoop.mapred.JobClient - Map output records=1040
>
>
> Thanks,
> Vidosh
>