You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Nishanth S <ni...@gmail.com> on 2015/01/22 20:52:06 UTC

Hbase Error When running Map reduce

Hi All,
I am running a map reduce job which scans the hbase table for  a particular
time period and then  creates some files from that.The job runs fine for 10
minutes or so and few around 10% of maps get completed succesfully.Here is
the error that I am getting.Can some one help?


15/01/22 19:34:33 INFO mapreduce.TableRecordReaderImpl: recovered from
org.apache.hadoop.hbase.client.ScannerTimeoutException: 559843ms
passed since the last invocation, timeout is currently set to 60000
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:352)
	at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:194)
	at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)
	at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name:
3432603283499371482, already closed?
	at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2973)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:277)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:198)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:336)
	... 13 more

Re: Hbase Error When running Map reduce

Posted by Ted Yu <yu...@gmail.com>.
Nishanth :
Please also tell us which hbase release you're using.
See if there was log similar to the following in region server logs:

2015-01-21 20:07:12,780 WARN  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
72135ms


On Thu, Jan 22, 2015 at 12:05 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:

> Hi Nishanth,
>
> There are too many things that might be causing that problem and might
> depend on your cluster deployment (cluster size, network, spindles, etc.)
> or even related to the key design or how you are using filters in the
> scanner that triggers the timeout. Have you looked into if the regions are
> evenly distributed, if regions are balanced across the cluster and data
> locality is acceptable? If you could share with us more details that would
> be really useful.
>
> thanks.
> esteban.
>
>
>
>
>
> --
> Cloudera, Inc.
>
>
> On Thu, Jan 22, 2015 at 11:52 AM, Nishanth S <ni...@gmail.com>
> wrote:
>
> > Hi All,
> > I am running a map reduce job which scans the hbase table for  a
> particular
> > time period and then  creates some files from that.The job runs fine for
> 10
> > minutes or so and few around 10% of maps get completed succesfully.Here
> is
> > the error that I am getting.Can some one help?
> >
> >
> > 15/01/22 19:34:33 INFO mapreduce.TableRecordReaderImpl: recovered from
> > org.apache.hadoop.hbase.client.ScannerTimeoutException: 559843ms
> > passed since the last invocation, timeout is currently set to 60000
> >         at
> > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:352)
> >         at
> >
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:194)
> >         at
> >
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
> >         at
> >
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)
> >         at
> >
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> >         at
> >
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> >         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >         at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> >         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> >         at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> >         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> > Caused by: org.apache.hadoop.hbase.UnknownScannerException:
> > org.apache.hadoop.hbase.UnknownScannerException: Name:
> > 3432603283499371482, already closed?
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2973)
> >         at
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
> >         at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
> >         at
> > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
> >
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >         at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> >         at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >         at
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> >         at
> >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> >         at
> >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> >         at
> >
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:277)
> >         at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:198)
> >         at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
> >         at
> >
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
> >         at
> >
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
> >         at
> > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:336)
> >         ... 13 more
> >
>

Re: Hbase Error When running Map reduce

Posted by Esteban Gutierrez <es...@cloudera.com>.
Hi Nishanth,

There are too many things that might be causing that problem and might
depend on your cluster deployment (cluster size, network, spindles, etc.)
or even related to the key design or how you are using filters in the
scanner that triggers the timeout. Have you looked into if the regions are
evenly distributed, if regions are balanced across the cluster and data
locality is acceptable? If you could share with us more details that would
be really useful.

thanks.
esteban.





--
Cloudera, Inc.


On Thu, Jan 22, 2015 at 11:52 AM, Nishanth S <ni...@gmail.com>
wrote:

> Hi All,
> I am running a map reduce job which scans the hbase table for  a particular
> time period and then  creates some files from that.The job runs fine for 10
> minutes or so and few around 10% of maps get completed succesfully.Here is
> the error that I am getting.Can some one help?
>
>
> 15/01/22 19:34:33 INFO mapreduce.TableRecordReaderImpl: recovered from
> org.apache.hadoop.hbase.client.ScannerTimeoutException: 559843ms
> passed since the last invocation, timeout is currently set to 60000
>         at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:352)
>         at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:194)
>         at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
>         at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)
>         at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>         at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
> org.apache.hadoop.hbase.UnknownScannerException: Name:
> 3432603283499371482, already closed?
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2973)
>         at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
>         at
> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
>
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>         at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>         at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:277)
>         at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:198)
>         at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
>         at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
>         at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
>         at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:336)
>         ... 13 more
>