You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jonathan Bishop <jb...@gmail.com> on 2012/07/20 18:27:54 UTC
Is this exception caused by an overloaded node?
Hi,
I am running on a cluster where some of the machines are loaded for other
purposes. Occasionally an HBase scan fails with the message below, and I
suspect this is caused by on or more of the region servers being overloaded
with other processes (not hadoop/hbase) and not being able to respond
correctly. Is this possible?
Thanks,
Jon
Exception in thread "main" java.lang.RuntimeException:
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-3573171992963675348' does not exist
at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1402)
at prd.Levelize.readPins(Levelize.java:50)
at prd.Levelize.main(Levelize.java:245)
Caused by: org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-3573171992963675348' does not exist
at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399)
... 2 more
Re: Is this exception caused by an overloaded node?
Posted by Jimmy Xiang <jx...@cloudera.com>.
This exception means the scanner is expired in the region server side.
You can adjust the scanner expiration setting, or make your client fast.
Thanks,
Jimmy
On Fri, Jul 20, 2012 at 9:27 AM, Jonathan Bishop <jb...@gmail.com> wrote:
> Hi,
>
> I am running on a cluster where some of the machines are loaded for other
> purposes. Occasionally an HBase scan fails with the message below, and I
> suspect this is caused by on or more of the region servers being overloaded
> with other processes (not hadoop/hbase) and not being able to respond
> correctly. Is this possible?
>
> Thanks,
>
> Jon
>
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '-3573171992963675348' does not exist
> at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1402)
> at prd.Levelize.readPins(Levelize.java:50)
> at prd.Levelize.main(Levelize.java:245)
> Caused by: org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '-3573171992963675348' does not exist
> at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399)
> ... 2 more