You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Rajeshkumar J <ra...@gmail.com> on 2016/12/21 11:30:17 UTC
Lease exception
Hi,
I have faced below issue in our production cluster
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does
not exist
at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221)
at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:744)
Can any one explain what is lease exception
Thanks,
Rajeshkumar J
Re: Lease exception
Posted by Richard Startin <ri...@outlook.com>.
If your client caching is set to a large value, you will need to do a long scan occasionally, and the rpc itself will be expensive in terms of IO. So it's worth looking at hbase.client.scanner.caching to see if it is too large. If you're scanning the whole table check you aren't churning the block cache.
The XML below looks wrong, was that copied verbatim from your site file?
https://richardstartin.com
> On 21 Dec 2016, at 12:02, Rajeshkumar J <ra...@gmail.com> wrote:
>
> Hi,
>
> Thanks for the reply. I have properties as below
>
> <property>
> <name>hbase.regionserver.lease.period</name>
> <value>900000</value>
> </property>
> <property>
> <name>hbase.rpc.timeout</name>
> <value>900000>/value>
> </property>
>
>
> Correct me If I am wrong.
>
> I know hbase.regionserver.lease.period, which says how long a scanner
> lives between calls to scanner.next().
>
> As far as I understand when scanner.next() is called it will fetch no
> of rows as in *hbase.client.scanner.caching. *When this fetching
> process takes more than lease period it will close the scanner object.
> so this exception occuring?
>
>
> Thanks,
>
> Rajeshkumar J
>
>
>
> On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <richardstartin@outlook.com
>> wrote:
>
>> It means your lease on a region server has expired during a call to
>> resultscanner.next(). This happens on a slow call to next(). You can either
>> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
>> hbase.regionserver.lease.period.
>>
>> https://richardstartin.com
>>
>> On 21 Dec 2016, at 11:30, Rajeshkumar J <rajeshkumarit8292@gmail.com<
>> mailto:rajeshkumarit8292@gmail.com>> wrote:
>>
>> Hi,
>>
>> I have faced below issue in our production cluster
>>
>> org.apache.hadoop.hbase.regionserver.LeaseException:
>> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does
>> not exist
>> at org.apache.hadoop.hbase.regionserver.Leases.
>> removeLease(Leases.java:221)
>> at org.apache.hadoop.hbase.regionserver.Leases.
>> cancelLease(Leases.java:206)
>> at
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> scan(RSRpcServices.java:2491)
>> at
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.
>> callBlockingMethod(ClientProtos.java:32205)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>> at
>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>> at java.lang.Thread.run(Thread.java:744)
>>
>>
>> Can any one explain what is lease exception
>>
>> Thanks,
>> Rajeshkumar J
>>
Re: Lease exception
Posted by Rajeshkumar J <ra...@gmail.com>.
Also how to change this property hbase.client.scanner.timeout.period in
client side as I only know that to change this property in hbase-site.xml
On Thu, Dec 22, 2016 at 3:13 PM, Phil Yang <ud...@gmail.com> wrote:
> https://github.com/apache/hbase/blob/rel/1.1.1/hbase-
> server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.
> java#L2491
>
> There is a TTL for scanners at server, to prevent client don't close the
> scanners and they leak. The TTL is configured by
> hbase.client.scanner.timeout.period at server and refreshed when a scan
> RPC
> request comes . The TTLs of all scanners are managed by Lease. Your error
> happens when server closes a scanner but in Lease it is already expired. So
> I think you can try to increase hbase.client.scanner.timeout.period at
> server or decrease your hbase.client.scanner.timeout.period at client to
> prevent the scanner expired before its scanning done.
> hbase.client.scanner.timeout.period is used both at client and server, may
> be different if you change one of sides.
>
> BTW, I still suggest that you can upgrade your cluster and client. 1.1.1
> has some data-loss bugs on scanning.
>
> Thanks,
> Phil
>
>
> 2016-12-22 17:26 GMT+08:00 Rajeshkumar J <ra...@gmail.com>:
>
> > can you please explain what is the cause of this lease exception and is
> > there any solve this in current version
> >
> > Thanks
> >
> > On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang <ud...@gmail.com> wrote:
> >
> > > In fact at client the rpc timeout of scan request is also
> > > hbase.client.scanner.timeout.period which replaces the
> > > deprecated hbase.regionserver.lease.period.
> > >
> > > Your code that throws LeaseException has been removed by HBASE-16604,
> > maybe
> > > you can try to upgrade your cluster to 1.1.7? Your client can also
> > upgrade
> > > to 1.1.7 which will ignore UnknowScannerException and retry when the
> > lease
> > > is expired at server.
> > >
> > > Thanks,
> > > Phil
> > >
> > >
> > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J <rajeshkumarit8292@gmail.com
> >:
> > >
> > > > Also there is a solution what i have found from hbase user guide that
> > > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > > > period.
> > > > How these two properties plays a part in the above exception. Please
> > can
> > > > anyone explain?
> > > >
> > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > wrote:
> > > >
> > > > > I am using hbase version 1.1.1
> > > > > Also I didn't understand something here. Whenever a scanner.next()
> is
> > > > > called it needs to return rows(based on caching value) within
> leasing
> > > > > period or else scanner client will be closed eventually throwing
> this
> > > > > exception. Correct me as I didn't get the clear understanding of
> this
> > > > issue
> > > > >
> > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > >> Which hbase release are you using ?
> > > > >>
> > > > >> There is heartbeat support when scanning.
> > > > >> Looks like the version you use doesn't have this support.
> > > > >>
> > > > >> Cheers
> > > > >>
> > > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > Hi,
> > > > >> >
> > > > >> > Thanks for the reply. I have properties as below
> > > > >> >
> > > > >> > <property>
> > > > >> > <name>hbase.regionserver.lease.period</name>
> > > > >> > <value>900000</value>
> > > > >> > </property>
> > > > >> > <property>
> > > > >> > <name>hbase.rpc.timeout</name>
> > > > >> > <value>900000>/value>
> > > > >> > </property>
> > > > >> >
> > > > >> >
> > > > >> > Correct me If I am wrong.
> > > > >> >
> > > > >> > I know hbase.regionserver.lease.period, which says how long a
> > > scanner
> > > > >> > lives between calls to scanner.next().
> > > > >> >
> > > > >> > As far as I understand when scanner.next() is called it will
> fetch
> > > no
> > > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > > > >> > process takes more than lease period it will close the scanner
> > > object.
> > > > >> > so this exception occuring?
> > > > >> >
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Rajeshkumar J
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> > > > >> richardstartin@outlook.com
> > > > >> >> wrote:
> > > > >> >
> > > > >> >> It means your lease on a region server has expired during a
> call
> > to
> > > > >> >> resultscanner.next(). This happens on a slow call to next().
> You
> > > can
> > > > >> either
> > > > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> > > > >> >> hbase.regionserver.lease.period.
> > > > >> >>
> > > > >> >> https://richardstartin.com
> > > > >> >>
> > > > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com
> > > > <
> > > > >> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
> > > > >> >>
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> I have faced below issue in our production cluster
> > > > >> >>
> > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > > '166881'
> > > > >> does
> > > > >> >> not exist
> > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > > >> >> removeLease(Leases.java:221)
> > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > > >> >> cancelLease(Leases.java:206)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > > >> >> scan(RSRpcServices.java:2491)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> > > > >> ClientService$2.
> > > > >> >> callBlockingMethod(ClientProtos.java:32205)
> > > > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:
> > 2114)
> > > > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.
> > java:101)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> > > > >> utor.java:130)
> > > > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > > > java:107)
> > > > >> >> at java.lang.Thread.run(Thread.java:744)
> > > > >> >>
> > > > >> >>
> > > > >> >> Can any one explain what is lease exception
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> Rajeshkumar J
> > > > >> >>
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lease exception
Posted by Rajeshkumar J <ra...@gmail.com>.
This is the log i got
2017-01-05 11:41:49,629 DEBUG
[B.defaultRpcServer.handler=15,queue=0,port=16020] ipc.RpcServer:
B.defaultRpcServer.handler=15,queue=0,port=16020: callId: 3 service:
ClientService methodName: Scan size: 23 connection: xx.xx.xx.xx:xxxxx
org.apache.hadoop.hbase.regionserver.LeaseException: lease '706' does not
exist
at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221)
at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:744)
2017-01-05 11:41:49,629 TRACE
[B.defaultRpcServer.handler=18,queue=0,port=16020] ipc.RpcServer: callId: 2
service: ClientService methodName: Scan size: 29 connection:
xx.xx.xx.xx:xxxxx param: scanner_id: 706 number_of_rows: 2147483647
close_scanner: false next_call_seq: 0 client_handles_partials: true
client_handles_heartbeats: true connection: xx.xx.xx.xx:xxxxx, response
scanner_id: 706 more_results: true stale: false more_results_in_region:
false queueTime: 1 processingTime: 60136 totalTime: 60137
I have hbase scanner timeout of 60000 but here total time is greater than
that so I am getting lease exception. can any one suggest me is there any
way to find out why it takes this time.
Thanks
On Thu, Dec 22, 2016 at 3:13 PM, Phil Yang <ud...@gmail.com> wrote:
> https://github.com/apache/hbase/blob/rel/1.1.1/hbase-
> server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.
> java#L2491
>
> There is a TTL for scanners at server, to prevent client don't close the
> scanners and they leak. The TTL is configured by
> hbase.client.scanner.timeout.period at server and refreshed when a scan
> RPC
> request comes . The TTLs of all scanners are managed by Lease. Your error
> happens when server closes a scanner but in Lease it is already expired. So
> I think you can try to increase hbase.client.scanner.timeout.period at
> server or decrease your hbase.client.scanner.timeout.period at client to
> prevent the scanner expired before its scanning done.
> hbase.client.scanner.timeout.period is used both at client and server, may
> be different if you change one of sides.
>
> BTW, I still suggest that you can upgrade your cluster and client. 1.1.1
> has some data-loss bugs on scanning.
>
> Thanks,
> Phil
>
>
> 2016-12-22 17:26 GMT+08:00 Rajeshkumar J <ra...@gmail.com>:
>
> > can you please explain what is the cause of this lease exception and is
> > there any solve this in current version
> >
> > Thanks
> >
> > On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang <ud...@gmail.com> wrote:
> >
> > > In fact at client the rpc timeout of scan request is also
> > > hbase.client.scanner.timeout.period which replaces the
> > > deprecated hbase.regionserver.lease.period.
> > >
> > > Your code that throws LeaseException has been removed by HBASE-16604,
> > maybe
> > > you can try to upgrade your cluster to 1.1.7? Your client can also
> > upgrade
> > > to 1.1.7 which will ignore UnknowScannerException and retry when the
> > lease
> > > is expired at server.
> > >
> > > Thanks,
> > > Phil
> > >
> > >
> > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J <rajeshkumarit8292@gmail.com
> >:
> > >
> > > > Also there is a solution what i have found from hbase user guide that
> > > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > > > period.
> > > > How these two properties plays a part in the above exception. Please
> > can
> > > > anyone explain?
> > > >
> > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > wrote:
> > > >
> > > > > I am using hbase version 1.1.1
> > > > > Also I didn't understand something here. Whenever a scanner.next()
> is
> > > > > called it needs to return rows(based on caching value) within
> leasing
> > > > > period or else scanner client will be closed eventually throwing
> this
> > > > > exception. Correct me as I didn't get the clear understanding of
> this
> > > > issue
> > > > >
> > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > >> Which hbase release are you using ?
> > > > >>
> > > > >> There is heartbeat support when scanning.
> > > > >> Looks like the version you use doesn't have this support.
> > > > >>
> > > > >> Cheers
> > > > >>
> > > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > Hi,
> > > > >> >
> > > > >> > Thanks for the reply. I have properties as below
> > > > >> >
> > > > >> > <property>
> > > > >> > <name>hbase.regionserver.lease.period</name>
> > > > >> > <value>900000</value>
> > > > >> > </property>
> > > > >> > <property>
> > > > >> > <name>hbase.rpc.timeout</name>
> > > > >> > <value>900000>/value>
> > > > >> > </property>
> > > > >> >
> > > > >> >
> > > > >> > Correct me If I am wrong.
> > > > >> >
> > > > >> > I know hbase.regionserver.lease.period, which says how long a
> > > scanner
> > > > >> > lives between calls to scanner.next().
> > > > >> >
> > > > >> > As far as I understand when scanner.next() is called it will
> fetch
> > > no
> > > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > > > >> > process takes more than lease period it will close the scanner
> > > object.
> > > > >> > so this exception occuring?
> > > > >> >
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Rajeshkumar J
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> > > > >> richardstartin@outlook.com
> > > > >> >> wrote:
> > > > >> >
> > > > >> >> It means your lease on a region server has expired during a
> call
> > to
> > > > >> >> resultscanner.next(). This happens on a slow call to next().
> You
> > > can
> > > > >> either
> > > > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> > > > >> >> hbase.regionserver.lease.period.
> > > > >> >>
> > > > >> >> https://richardstartin.com
> > > > >> >>
> > > > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com
> > > > <
> > > > >> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
> > > > >> >>
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> I have faced below issue in our production cluster
> > > > >> >>
> > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > > '166881'
> > > > >> does
> > > > >> >> not exist
> > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > > >> >> removeLease(Leases.java:221)
> > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > > >> >> cancelLease(Leases.java:206)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > > >> >> scan(RSRpcServices.java:2491)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> > > > >> ClientService$2.
> > > > >> >> callBlockingMethod(ClientProtos.java:32205)
> > > > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:
> > 2114)
> > > > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.
> > java:101)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> > > > >> utor.java:130)
> > > > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > > > java:107)
> > > > >> >> at java.lang.Thread.run(Thread.java:744)
> > > > >> >>
> > > > >> >>
> > > > >> >> Can any one explain what is lease exception
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> Rajeshkumar J
> > > > >> >>
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lease exception
Posted by Rajeshkumar J <ra...@gmail.com>.
sorry for the delay. I didn't get the lease concept here whether it is
specific to hbase or like lease in hadoop?
On Thu, Dec 22, 2016 at 3:13 PM, Phil Yang <ud...@gmail.com> wrote:
> https://github.com/apache/hbase/blob/rel/1.1.1/hbase-
> server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.
> java#L2491
>
> There is a TTL for scanners at server, to prevent client don't close the
> scanners and they leak. The TTL is configured by
> hbase.client.scanner.timeout.period at server and refreshed when a scan
> RPC
> request comes . The TTLs of all scanners are managed by Lease. Your error
> happens when server closes a scanner but in Lease it is already expired. So
> I think you can try to increase hbase.client.scanner.timeout.period at
> server or decrease your hbase.client.scanner.timeout.period at client to
> prevent the scanner expired before its scanning done.
> hbase.client.scanner.timeout.period is used both at client and server, may
> be different if you change one of sides.
>
> BTW, I still suggest that you can upgrade your cluster and client. 1.1.1
> has some data-loss bugs on scanning.
>
> Thanks,
> Phil
>
>
> 2016-12-22 17:26 GMT+08:00 Rajeshkumar J <ra...@gmail.com>:
>
> > can you please explain what is the cause of this lease exception and is
> > there any solve this in current version
> >
> > Thanks
> >
> > On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang <ud...@gmail.com> wrote:
> >
> > > In fact at client the rpc timeout of scan request is also
> > > hbase.client.scanner.timeout.period which replaces the
> > > deprecated hbase.regionserver.lease.period.
> > >
> > > Your code that throws LeaseException has been removed by HBASE-16604,
> > maybe
> > > you can try to upgrade your cluster to 1.1.7? Your client can also
> > upgrade
> > > to 1.1.7 which will ignore UnknowScannerException and retry when the
> > lease
> > > is expired at server.
> > >
> > > Thanks,
> > > Phil
> > >
> > >
> > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J <rajeshkumarit8292@gmail.com
> >:
> > >
> > > > Also there is a solution what i have found from hbase user guide that
> > > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > > > period.
> > > > How these two properties plays a part in the above exception. Please
> > can
> > > > anyone explain?
> > > >
> > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > wrote:
> > > >
> > > > > I am using hbase version 1.1.1
> > > > > Also I didn't understand something here. Whenever a scanner.next()
> is
> > > > > called it needs to return rows(based on caching value) within
> leasing
> > > > > period or else scanner client will be closed eventually throwing
> this
> > > > > exception. Correct me as I didn't get the clear understanding of
> this
> > > > issue
> > > > >
> > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > >> Which hbase release are you using ?
> > > > >>
> > > > >> There is heartbeat support when scanning.
> > > > >> Looks like the version you use doesn't have this support.
> > > > >>
> > > > >> Cheers
> > > > >>
> > > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > Hi,
> > > > >> >
> > > > >> > Thanks for the reply. I have properties as below
> > > > >> >
> > > > >> > <property>
> > > > >> > <name>hbase.regionserver.lease.period</name>
> > > > >> > <value>900000</value>
> > > > >> > </property>
> > > > >> > <property>
> > > > >> > <name>hbase.rpc.timeout</name>
> > > > >> > <value>900000>/value>
> > > > >> > </property>
> > > > >> >
> > > > >> >
> > > > >> > Correct me If I am wrong.
> > > > >> >
> > > > >> > I know hbase.regionserver.lease.period, which says how long a
> > > scanner
> > > > >> > lives between calls to scanner.next().
> > > > >> >
> > > > >> > As far as I understand when scanner.next() is called it will
> fetch
> > > no
> > > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > > > >> > process takes more than lease period it will close the scanner
> > > object.
> > > > >> > so this exception occuring?
> > > > >> >
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Rajeshkumar J
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> > > > >> richardstartin@outlook.com
> > > > >> >> wrote:
> > > > >> >
> > > > >> >> It means your lease on a region server has expired during a
> call
> > to
> > > > >> >> resultscanner.next(). This happens on a slow call to next().
> You
> > > can
> > > > >> either
> > > > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> > > > >> >> hbase.regionserver.lease.period.
> > > > >> >>
> > > > >> >> https://richardstartin.com
> > > > >> >>
> > > > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com
> > > > <
> > > > >> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
> > > > >> >>
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> I have faced below issue in our production cluster
> > > > >> >>
> > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > > '166881'
> > > > >> does
> > > > >> >> not exist
> > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > > >> >> removeLease(Leases.java:221)
> > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > > >> >> cancelLease(Leases.java:206)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > > >> >> scan(RSRpcServices.java:2491)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> > > > >> ClientService$2.
> > > > >> >> callBlockingMethod(ClientProtos.java:32205)
> > > > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:
> > 2114)
> > > > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.
> > java:101)
> > > > >> >> at
> > > > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> > > > >> utor.java:130)
> > > > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > > > java:107)
> > > > >> >> at java.lang.Thread.run(Thread.java:744)
> > > > >> >>
> > > > >> >>
> > > > >> >> Can any one explain what is lease exception
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> Rajeshkumar J
> > > > >> >>
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lease exception
Posted by Phil Yang <ud...@gmail.com>.
https://github.com/apache/hbase/blob/rel/1.1.1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L2491
There is a TTL for scanners at server, to prevent client don't close the
scanners and they leak. The TTL is configured by
hbase.client.scanner.timeout.period at server and refreshed when a scan RPC
request comes . The TTLs of all scanners are managed by Lease. Your error
happens when server closes a scanner but in Lease it is already expired. So
I think you can try to increase hbase.client.scanner.timeout.period at
server or decrease your hbase.client.scanner.timeout.period at client to
prevent the scanner expired before its scanning done.
hbase.client.scanner.timeout.period is used both at client and server, may
be different if you change one of sides.
BTW, I still suggest that you can upgrade your cluster and client. 1.1.1
has some data-loss bugs on scanning.
Thanks,
Phil
2016-12-22 17:26 GMT+08:00 Rajeshkumar J <ra...@gmail.com>:
> can you please explain what is the cause of this lease exception and is
> there any solve this in current version
>
> Thanks
>
> On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang <ud...@gmail.com> wrote:
>
> > In fact at client the rpc timeout of scan request is also
> > hbase.client.scanner.timeout.period which replaces the
> > deprecated hbase.regionserver.lease.period.
> >
> > Your code that throws LeaseException has been removed by HBASE-16604,
> maybe
> > you can try to upgrade your cluster to 1.1.7? Your client can also
> upgrade
> > to 1.1.7 which will ignore UnknowScannerException and retry when the
> lease
> > is expired at server.
> >
> > Thanks,
> > Phil
> >
> >
> > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J <ra...@gmail.com>:
> >
> > > Also there is a solution what i have found from hbase user guide that
> > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > > period.
> > > How these two properties plays a part in the above exception. Please
> can
> > > anyone explain?
> > >
> > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > wrote:
> > >
> > > > I am using hbase version 1.1.1
> > > > Also I didn't understand something here. Whenever a scanner.next() is
> > > > called it needs to return rows(based on caching value) within leasing
> > > > period or else scanner client will be closed eventually throwing this
> > > > exception. Correct me as I didn't get the clear understanding of this
> > > issue
> > > >
> > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > >> Which hbase release are you using ?
> > > >>
> > > >> There is heartbeat support when scanning.
> > > >> Looks like the version you use doesn't have this support.
> > > >>
> > > >> Cheers
> > > >>
> > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > Hi,
> > > >> >
> > > >> > Thanks for the reply. I have properties as below
> > > >> >
> > > >> > <property>
> > > >> > <name>hbase.regionserver.lease.period</name>
> > > >> > <value>900000</value>
> > > >> > </property>
> > > >> > <property>
> > > >> > <name>hbase.rpc.timeout</name>
> > > >> > <value>900000>/value>
> > > >> > </property>
> > > >> >
> > > >> >
> > > >> > Correct me If I am wrong.
> > > >> >
> > > >> > I know hbase.regionserver.lease.period, which says how long a
> > scanner
> > > >> > lives between calls to scanner.next().
> > > >> >
> > > >> > As far as I understand when scanner.next() is called it will fetch
> > no
> > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > > >> > process takes more than lease period it will close the scanner
> > object.
> > > >> > so this exception occuring?
> > > >> >
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Rajeshkumar J
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> > > >> richardstartin@outlook.com
> > > >> >> wrote:
> > > >> >
> > > >> >> It means your lease on a region server has expired during a call
> to
> > > >> >> resultscanner.next(). This happens on a slow call to next(). You
> > can
> > > >> either
> > > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> > > >> >> hbase.regionserver.lease.period.
> > > >> >>
> > > >> >> https://richardstartin.com
> > > >> >>
> > > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com
> > > <
> > > >> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
> > > >> >>
> > > >> >> Hi,
> > > >> >>
> > > >> >> I have faced below issue in our production cluster
> > > >> >>
> > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > '166881'
> > > >> does
> > > >> >> not exist
> > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > >> >> removeLease(Leases.java:221)
> > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > >> >> cancelLease(Leases.java:206)
> > > >> >> at
> > > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > >> >> scan(RSRpcServices.java:2491)
> > > >> >> at
> > > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> > > >> ClientService$2.
> > > >> >> callBlockingMethod(ClientProtos.java:32205)
> > > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:
> 2114)
> > > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.
> java:101)
> > > >> >> at
> > > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> > > >> utor.java:130)
> > > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > > java:107)
> > > >> >> at java.lang.Thread.run(Thread.java:744)
> > > >> >>
> > > >> >>
> > > >> >> Can any one explain what is lease exception
> > > >> >>
> > > >> >> Thanks,
> > > >> >> Rajeshkumar J
> > > >> >>
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Lease exception
Posted by Rajeshkumar J <ra...@gmail.com>.
can you please explain what is the cause of this lease exception and is
there any solve this in current version
Thanks
On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang <ud...@gmail.com> wrote:
> In fact at client the rpc timeout of scan request is also
> hbase.client.scanner.timeout.period which replaces the
> deprecated hbase.regionserver.lease.period.
>
> Your code that throws LeaseException has been removed by HBASE-16604, maybe
> you can try to upgrade your cluster to 1.1.7? Your client can also upgrade
> to 1.1.7 which will ignore UnknowScannerException and retry when the lease
> is expired at server.
>
> Thanks,
> Phil
>
>
> 2016-12-22 16:51 GMT+08:00 Rajeshkumar J <ra...@gmail.com>:
>
> > Also there is a solution what i have found from hbase user guide that
> > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > period.
> > How these two properties plays a part in the above exception. Please can
> > anyone explain?
> >
> > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com>
> > wrote:
> >
> > > I am using hbase version 1.1.1
> > > Also I didn't understand something here. Whenever a scanner.next() is
> > > called it needs to return rows(based on caching value) within leasing
> > > period or else scanner client will be closed eventually throwing this
> > > exception. Correct me as I didn't get the clear understanding of this
> > issue
> > >
> > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > >> Which hbase release are you using ?
> > >>
> > >> There is heartbeat support when scanning.
> > >> Looks like the version you use doesn't have this support.
> > >>
> > >> Cheers
> > >>
> > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hi,
> > >> >
> > >> > Thanks for the reply. I have properties as below
> > >> >
> > >> > <property>
> > >> > <name>hbase.regionserver.lease.period</name>
> > >> > <value>900000</value>
> > >> > </property>
> > >> > <property>
> > >> > <name>hbase.rpc.timeout</name>
> > >> > <value>900000>/value>
> > >> > </property>
> > >> >
> > >> >
> > >> > Correct me If I am wrong.
> > >> >
> > >> > I know hbase.regionserver.lease.period, which says how long a
> scanner
> > >> > lives between calls to scanner.next().
> > >> >
> > >> > As far as I understand when scanner.next() is called it will fetch
> no
> > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > >> > process takes more than lease period it will close the scanner
> object.
> > >> > so this exception occuring?
> > >> >
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Rajeshkumar J
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> > >> richardstartin@outlook.com
> > >> >> wrote:
> > >> >
> > >> >> It means your lease on a region server has expired during a call to
> > >> >> resultscanner.next(). This happens on a slow call to next(). You
> can
> > >> either
> > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> > >> >> hbase.regionserver.lease.period.
> > >> >>
> > >> >> https://richardstartin.com
> > >> >>
> > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <
> rajeshkumarit8292@gmail.com
> > <
> > >> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
> > >> >>
> > >> >> Hi,
> > >> >>
> > >> >> I have faced below issue in our production cluster
> > >> >>
> > >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '166881'
> > >> does
> > >> >> not exist
> > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > >> >> removeLease(Leases.java:221)
> > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > >> >> cancelLease(Leases.java:206)
> > >> >> at
> > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > >> >> scan(RSRpcServices.java:2491)
> > >> >> at
> > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> > >> ClientService$2.
> > >> >> callBlockingMethod(ClientProtos.java:32205)
> > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > >> >> at
> > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> > >> utor.java:130)
> > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > java:107)
> > >> >> at java.lang.Thread.run(Thread.java:744)
> > >> >>
> > >> >>
> > >> >> Can any one explain what is lease exception
> > >> >>
> > >> >> Thanks,
> > >> >> Rajeshkumar J
> > >> >>
> > >>
> > >
> > >
> >
>
Re: Lease exception
Posted by Phil Yang <ud...@gmail.com>.
In fact at client the rpc timeout of scan request is also
hbase.client.scanner.timeout.period which replaces the
deprecated hbase.regionserver.lease.period.
Your code that throws LeaseException has been removed by HBASE-16604, maybe
you can try to upgrade your cluster to 1.1.7? Your client can also upgrade
to 1.1.7 which will ignore UnknowScannerException and retry when the lease
is expired at server.
Thanks,
Phil
2016-12-22 16:51 GMT+08:00 Rajeshkumar J <ra...@gmail.com>:
> Also there is a solution what i have found from hbase user guide that
> hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> period.
> How these two properties plays a part in the above exception. Please can
> anyone explain?
>
> On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > I am using hbase version 1.1.1
> > Also I didn't understand something here. Whenever a scanner.next() is
> > called it needs to return rows(based on caching value) within leasing
> > period or else scanner client will be closed eventually throwing this
> > exception. Correct me as I didn't get the clear understanding of this
> issue
> >
> > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Which hbase release are you using ?
> >>
> >> There is heartbeat support when scanning.
> >> Looks like the version you use doesn't have this support.
> >>
> >> Cheers
> >>
> >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> >> wrote:
> >> >
> >> > Hi,
> >> >
> >> > Thanks for the reply. I have properties as below
> >> >
> >> > <property>
> >> > <name>hbase.regionserver.lease.period</name>
> >> > <value>900000</value>
> >> > </property>
> >> > <property>
> >> > <name>hbase.rpc.timeout</name>
> >> > <value>900000>/value>
> >> > </property>
> >> >
> >> >
> >> > Correct me If I am wrong.
> >> >
> >> > I know hbase.regionserver.lease.period, which says how long a scanner
> >> > lives between calls to scanner.next().
> >> >
> >> > As far as I understand when scanner.next() is called it will fetch no
> >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> >> > process takes more than lease period it will close the scanner object.
> >> > so this exception occuring?
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Rajeshkumar J
> >> >
> >> >
> >> >
> >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> >> richardstartin@outlook.com
> >> >> wrote:
> >> >
> >> >> It means your lease on a region server has expired during a call to
> >> >> resultscanner.next(). This happens on a slow call to next(). You can
> >> either
> >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> >> >> hbase.regionserver.lease.period.
> >> >>
> >> >> https://richardstartin.com
> >> >>
> >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <rajeshkumarit8292@gmail.com
> <
> >> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I have faced below issue in our production cluster
> >> >>
> >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881'
> >> does
> >> >> not exist
> >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> >> removeLease(Leases.java:221)
> >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> >> cancelLease(Leases.java:206)
> >> >> at
> >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> >> >> scan(RSRpcServices.java:2491)
> >> >> at
> >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> >> ClientService$2.
> >> >> callBlockingMethod(ClientProtos.java:32205)
> >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> >> >> at
> >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> >> utor.java:130)
> >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> java:107)
> >> >> at java.lang.Thread.run(Thread.java:744)
> >> >>
> >> >>
> >> >> Can any one explain what is lease exception
> >> >>
> >> >> Thanks,
> >> >> Rajeshkumar J
> >> >>
> >>
> >
> >
>
Re: Lease exception
Posted by Rajeshkumar J <ra...@gmail.com>.
Also there is a solution what i have found from hbase user guide that
hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.period.
How these two properties plays a part in the above exception. Please can
anyone explain?
On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <ra...@gmail.com>
wrote:
> I am using hbase version 1.1.1
> Also I didn't understand something here. Whenever a scanner.next() is
> called it needs to return rows(based on caching value) within leasing
> period or else scanner client will be closed eventually throwing this
> exception. Correct me as I didn't get the clear understanding of this issue
>
> On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Which hbase release are you using ?
>>
>> There is heartbeat support when scanning.
>> Looks like the version you use doesn't have this support.
>>
>> Cheers
>>
>> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <ra...@gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > Thanks for the reply. I have properties as below
>> >
>> > <property>
>> > <name>hbase.regionserver.lease.period</name>
>> > <value>900000</value>
>> > </property>
>> > <property>
>> > <name>hbase.rpc.timeout</name>
>> > <value>900000>/value>
>> > </property>
>> >
>> >
>> > Correct me If I am wrong.
>> >
>> > I know hbase.regionserver.lease.period, which says how long a scanner
>> > lives between calls to scanner.next().
>> >
>> > As far as I understand when scanner.next() is called it will fetch no
>> > of rows as in *hbase.client.scanner.caching. *When this fetching
>> > process takes more than lease period it will close the scanner object.
>> > so this exception occuring?
>> >
>> >
>> > Thanks,
>> >
>> > Rajeshkumar J
>> >
>> >
>> >
>> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
>> richardstartin@outlook.com
>> >> wrote:
>> >
>> >> It means your lease on a region server has expired during a call to
>> >> resultscanner.next(). This happens on a slow call to next(). You can
>> either
>> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
>> >> hbase.regionserver.lease.period.
>> >>
>> >> https://richardstartin.com
>> >>
>> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <rajeshkumarit8292@gmail.com<
>> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I have faced below issue in our production cluster
>> >>
>> >> org.apache.hadoop.hbase.regionserver.LeaseException:
>> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881'
>> does
>> >> not exist
>> >> at org.apache.hadoop.hbase.regionserver.Leases.
>> >> removeLease(Leases.java:221)
>> >> at org.apache.hadoop.hbase.regionserver.Leases.
>> >> cancelLease(Leases.java:206)
>> >> at
>> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> >> scan(RSRpcServices.java:2491)
>> >> at
>> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
>> ClientService$2.
>> >> callBlockingMethod(ClientProtos.java:32205)
>> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>> >> at
>> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
>> utor.java:130)
>> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>> >> at java.lang.Thread.run(Thread.java:744)
>> >>
>> >>
>> >> Can any one explain what is lease exception
>> >>
>> >> Thanks,
>> >> Rajeshkumar J
>> >>
>>
>
>
Re: Lease exception
Posted by Rajeshkumar J <ra...@gmail.com>.
I am using hbase version 1.1.1
Also I didn't understand something here. Whenever a scanner.next() is
called it needs to return rows(based on caching value) within leasing
period or else scanner client will be closed eventually throwing this
exception. Correct me as I didn't get the clear understanding of this issue
On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yu...@gmail.com> wrote:
> Which hbase release are you using ?
>
> There is heartbeat support when scanning.
> Looks like the version you use doesn't have this support.
>
> Cheers
>
> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <ra...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > Thanks for the reply. I have properties as below
> >
> > <property>
> > <name>hbase.regionserver.lease.period</name>
> > <value>900000</value>
> > </property>
> > <property>
> > <name>hbase.rpc.timeout</name>
> > <value>900000>/value>
> > </property>
> >
> >
> > Correct me If I am wrong.
> >
> > I know hbase.regionserver.lease.period, which says how long a scanner
> > lives between calls to scanner.next().
> >
> > As far as I understand when scanner.next() is called it will fetch no
> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > process takes more than lease period it will close the scanner object.
> > so this exception occuring?
> >
> >
> > Thanks,
> >
> > Rajeshkumar J
> >
> >
> >
> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> richardstartin@outlook.com
> >> wrote:
> >
> >> It means your lease on a region server has expired during a call to
> >> resultscanner.next(). This happens on a slow call to next(). You can
> either
> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> >> hbase.regionserver.lease.period.
> >>
> >> https://richardstartin.com
> >>
> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <rajeshkumarit8292@gmail.com<
> >> mailto:rajeshkumarit8292@gmail.com>> wrote:
> >>
> >> Hi,
> >>
> >> I have faced below issue in our production cluster
> >>
> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881'
> does
> >> not exist
> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> removeLease(Leases.java:221)
> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> cancelLease(Leases.java:206)
> >> at
> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> >> scan(RSRpcServices.java:2491)
> >> at
> >> org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.
> >> callBlockingMethod(ClientProtos.java:32205)
> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> >> at
> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:130)
> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> >> at java.lang.Thread.run(Thread.java:744)
> >>
> >>
> >> Can any one explain what is lease exception
> >>
> >> Thanks,
> >> Rajeshkumar J
> >>
>
Re: Lease exception
Posted by Ted Yu <yu...@gmail.com>.
Which hbase release are you using ?
There is heartbeat support when scanning.
Looks like the version you use doesn't have this support.
Cheers
> On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <ra...@gmail.com> wrote:
>
> Hi,
>
> Thanks for the reply. I have properties as below
>
> <property>
> <name>hbase.regionserver.lease.period</name>
> <value>900000</value>
> </property>
> <property>
> <name>hbase.rpc.timeout</name>
> <value>900000>/value>
> </property>
>
>
> Correct me If I am wrong.
>
> I know hbase.regionserver.lease.period, which says how long a scanner
> lives between calls to scanner.next().
>
> As far as I understand when scanner.next() is called it will fetch no
> of rows as in *hbase.client.scanner.caching. *When this fetching
> process takes more than lease period it will close the scanner object.
> so this exception occuring?
>
>
> Thanks,
>
> Rajeshkumar J
>
>
>
> On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <richardstartin@outlook.com
>> wrote:
>
>> It means your lease on a region server has expired during a call to
>> resultscanner.next(). This happens on a slow call to next(). You can either
>> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
>> hbase.regionserver.lease.period.
>>
>> https://richardstartin.com
>>
>> On 21 Dec 2016, at 11:30, Rajeshkumar J <rajeshkumarit8292@gmail.com<
>> mailto:rajeshkumarit8292@gmail.com>> wrote:
>>
>> Hi,
>>
>> I have faced below issue in our production cluster
>>
>> org.apache.hadoop.hbase.regionserver.LeaseException:
>> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does
>> not exist
>> at org.apache.hadoop.hbase.regionserver.Leases.
>> removeLease(Leases.java:221)
>> at org.apache.hadoop.hbase.regionserver.Leases.
>> cancelLease(Leases.java:206)
>> at
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> scan(RSRpcServices.java:2491)
>> at
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.
>> callBlockingMethod(ClientProtos.java:32205)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>> at
>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>> at java.lang.Thread.run(Thread.java:744)
>>
>>
>> Can any one explain what is lease exception
>>
>> Thanks,
>> Rajeshkumar J
>>
Re: Lease exception
Posted by Rajeshkumar J <ra...@gmail.com>.
Hi,
Thanks for the reply. I have properties as below
<property>
<name>hbase.regionserver.lease.period</name>
<value>900000</value>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>900000>/value>
</property>
Correct me If I am wrong.
I know hbase.regionserver.lease.period, which says how long a scanner
lives between calls to scanner.next().
As far as I understand when scanner.next() is called it will fetch no
of rows as in *hbase.client.scanner.caching. *When this fetching
process takes more than lease period it will close the scanner object.
so this exception occuring?
Thanks,
Rajeshkumar J
On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <richardstartin@outlook.com
> wrote:
> It means your lease on a region server has expired during a call to
> resultscanner.next(). This happens on a slow call to next(). You can either
> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> hbase.regionserver.lease.period.
>
> https://richardstartin.com
>
> On 21 Dec 2016, at 11:30, Rajeshkumar J <rajeshkumarit8292@gmail.com<
> mailto:rajeshkumarit8292@gmail.com>> wrote:
>
> Hi,
>
> I have faced below issue in our production cluster
>
> org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does
> not exist
> at org.apache.hadoop.hbase.regionserver.Leases.
> removeLease(Leases.java:221)
> at org.apache.hadoop.hbase.regionserver.Leases.
> cancelLease(Leases.java:206)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> scan(RSRpcServices.java:2491)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.
> callBlockingMethod(ClientProtos.java:32205)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:744)
>
>
> Can any one explain what is lease exception
>
> Thanks,
> Rajeshkumar J
>
Re: Lease exception
Posted by Richard Startin <ri...@outlook.com>.
It means your lease on a region server has expired during a call to resultscanner.next(). This happens on a slow call to next(). You can either embrace it or "fix" it by making sure hbase.rpc.timeout exceeds hbase.regionserver.lease.period.
https://richardstartin.com
On 21 Dec 2016, at 11:30, Rajeshkumar J <ra...@gmail.com>> wrote:
Hi,
I have faced below issue in our production cluster
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does
not exist
at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221)
at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:744)
Can any one explain what is lease exception
Thanks,
Rajeshkumar J