You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Alex Baranau <al...@gmail.com> on 2012/02/12 19:41:35 UTC
LeaseException despite high hbase.regionserver.lease.period
Hello,
I'm getting scanner lease exceptions during mapreduce job [1] after running
it for less than 7 minutes. Though I have set
hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
configuration on master and all regionservers and master (and restarted
all). Also set it in job's configuration.
Is there anything else I could do to eliminate the exception?
Thanks,
Alex
[1]
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
ʼ3854333525977315054ʼ does not exist
at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Last
at
java.lang.reflect.Method.invoke(Method.java:597)
4KB
al FAILED at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
Last
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
8KB
All
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.ap
Re: LeaseException despite high hbase.regionserver.lease.period
Posted by Alex Baranau <al...@gmail.com>.
Yeah, looks like the same thing.
The job went well eventually (many mappers, lasted for 13h +, no errors at
all).
Alex
On Sun, Feb 12, 2012 at 4:35 PM, Mikael Sitruk <mi...@gmail.com>wrote:
> I had same wired problem last week when i upgraded from cdh3u0 to cdh3u2.
> I got lease exception even when the hbase.regionserver.lease.period was
> set '300000'. The lease occurred after 1 mins or more.
> Prior the upgrade everything was fine, i was executing with map reduce 8
> slots.
> The lease totally disappeared when i use only 1 slot, i thought there were
> some concurrency problem with MR integration, but this weekend some of the
> machines where restarted and suddenly everything is fine!! I'm back to 8
> slot and the extract are running. (i run several time even without even one
> lease exception)
> In my case NO region server were stuck in transition.
> It is really frustrating that i cannot point on what was the real problem.
> Even log with debug did not point on problems (perhaps because it is also
> missing some debug statement like when a scanner lease is added to the RS)
>
> Mikael.S
>
>
> On Sun, Feb 12, 2012 at 9:55 PM, Alex Baranau <alex.baranov.v@gmail.com
> >wrote:
>
> > Hm, weird. Looks like now it goes well. What I did is once again
> restarted
> > several regionservers (noticed that some where holding regions which were
> > stuck in "in transition" state). And started back the job.
> >
> > Btw, forgot to mention that caching on Scanner was set to only 64 and I
> the
> > estimate for records size is < 100KB. Anyhow, the weird thing was that
> the
> > limits were set to 10min and it started to fail even before 10 min has
> > passed (sometimes after 3-4 minutes).
> >
> > Have to go and can't watch the job to the end, will write whether it
> > succeed or not. If face same problems again will try brutal variant: set
> > caching = 10 (or even 1), set batch = 10 (or even 1).
> >
> > Alex
> >
> > On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <alex.baranov.v@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > 0.90.4-cdh3u2
> > >
> > > Alex
> > >
> > >
> > > On Sun, Feb 12, 2012 at 1:44 PM, <yu...@gmail.com> wrote:
> > >
> > >> Which version of hbase are you using ?
> > >>
> > >> Thanks
> > >>
> > >>
> > >>
> > >> On Feb 12, 2012, at 10:41 AM, Alex Baranau <al...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > I'm getting scanner lease exceptions during mapreduce job [1] after
> > >> running
> > >> > it for less than 7 minutes. Though I have set
> > >> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> > >> > configuration on master and all regionservers and master (and
> > restarted
> > >> > all). Also set it in job's configuration.
> > >> >
> > >> > Is there anything else I could do to eliminate the exception?
> > >> >
> > >> > Thanks,
> > >> > Alex
> > >> >
> > >> > [1]
> > >> >
> > >> > org.apache.hadoop.hbase.regionserver.LeaseException:
> > >> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > >> > ʼ3854333525977315054ʼ does not exist
> > >> > at
> > >> >
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> > >> >
> > >> > at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> > >> >
> > >> > at
> > sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> > >> > Source)
> > >> >
> > >> > at
> > >> >
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >> > Last
> > >> > at
> > >> > java.lang.reflect.Method.invoke(Method.java:597)
> > >> > 4KB
> > >> > al FAILED at
> > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > >> > Last
> > >> > at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > >> > 8KB
> > >> >
> > >> > All
> > >> > at
> > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > >> > Method)
> > >> >
> > >> > at
> > >> >
> > >>
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > >> >
> > >> > at
> > >> >
> > >>
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > >> >
> > >> > at
> > >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > >> >
> > >> > at
> > >> > org.ap
> > >>
> > >
> > >
> >
>
>
>
> --
> Mikael.S
>
Re: LeaseException despite high hbase.regionserver.lease.period
Posted by Mikael Sitruk <mi...@gmail.com>.
I had same wired problem last week when i upgraded from cdh3u0 to cdh3u2.
I got lease exception even when the hbase.regionserver.lease.period was
set '300000'. The lease occurred after 1 mins or more.
Prior the upgrade everything was fine, i was executing with map reduce 8
slots.
The lease totally disappeared when i use only 1 slot, i thought there were
some concurrency problem with MR integration, but this weekend some of the
machines where restarted and suddenly everything is fine!! I'm back to 8
slot and the extract are running. (i run several time even without even one
lease exception)
In my case NO region server were stuck in transition.
It is really frustrating that i cannot point on what was the real problem.
Even log with debug did not point on problems (perhaps because it is also
missing some debug statement like when a scanner lease is added to the RS)
Mikael.S
On Sun, Feb 12, 2012 at 9:55 PM, Alex Baranau <al...@gmail.com>wrote:
> Hm, weird. Looks like now it goes well. What I did is once again restarted
> several regionservers (noticed that some where holding regions which were
> stuck in "in transition" state). And started back the job.
>
> Btw, forgot to mention that caching on Scanner was set to only 64 and I the
> estimate for records size is < 100KB. Anyhow, the weird thing was that the
> limits were set to 10min and it started to fail even before 10 min has
> passed (sometimes after 3-4 minutes).
>
> Have to go and can't watch the job to the end, will write whether it
> succeed or not. If face same problems again will try brutal variant: set
> caching = 10 (or even 1), set batch = 10 (or even 1).
>
> Alex
>
> On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <alex.baranov.v@gmail.com
> >wrote:
>
> > Hi,
> >
> > 0.90.4-cdh3u2
> >
> > Alex
> >
> >
> > On Sun, Feb 12, 2012 at 1:44 PM, <yu...@gmail.com> wrote:
> >
> >> Which version of hbase are you using ?
> >>
> >> Thanks
> >>
> >>
> >>
> >> On Feb 12, 2012, at 10:41 AM, Alex Baranau <al...@gmail.com>
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm getting scanner lease exceptions during mapreduce job [1] after
> >> running
> >> > it for less than 7 minutes. Though I have set
> >> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> >> > configuration on master and all regionservers and master (and
> restarted
> >> > all). Also set it in job's configuration.
> >> >
> >> > Is there anything else I could do to eliminate the exception?
> >> >
> >> > Thanks,
> >> > Alex
> >> >
> >> > [1]
> >> >
> >> > org.apache.hadoop.hbase.regionserver.LeaseException:
> >> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> >> > ʼ3854333525977315054ʼ does not exist
> >> > at
> >> >
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> >> >
> >> > at
> sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> >> > Source)
> >> >
> >> > at
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > Last
> >> > at
> >> > java.lang.reflect.Method.invoke(Method.java:597)
> >> > 4KB
> >> > al FAILED at
> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> >> > Last
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> >> > 8KB
> >> >
> >> > All
> >> > at
> >> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> > Method)
> >> >
> >> > at
> >> >
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> >
> >> > at
> >> >
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >
> >> > at
> >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> >
> >> > at
> >> > org.ap
> >>
> >
> >
>
--
Mikael.S
Re: LeaseException despite high hbase.regionserver.lease.period
Posted by Daniel Iancu <da...@1and1.ro>.
Hi Alex
Do you use a filter for KVs in scanner? How many columns per row can you
have?
Thanks
Daniel
On 02/12/2012 09:55 PM, Alex Baranau wrote:
> forgot to mention that caching on Scanner was set to only 64 and I the
> estimate for records size is< 100KB
Re: LeaseException despite high hbase.regionserver.lease.period
Posted by Alex Baranau <al...@gmail.com>.
Hm, weird. Looks like now it goes well. What I did is once again restarted
several regionservers (noticed that some where holding regions which were
stuck in "in transition" state). And started back the job.
Btw, forgot to mention that caching on Scanner was set to only 64 and I the
estimate for records size is < 100KB. Anyhow, the weird thing was that the
limits were set to 10min and it started to fail even before 10 min has
passed (sometimes after 3-4 minutes).
Have to go and can't watch the job to the end, will write whether it
succeed or not. If face same problems again will try brutal variant: set
caching = 10 (or even 1), set batch = 10 (or even 1).
Alex
On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <al...@gmail.com>wrote:
> Hi,
>
> 0.90.4-cdh3u2
>
> Alex
>
>
> On Sun, Feb 12, 2012 at 1:44 PM, <yu...@gmail.com> wrote:
>
>> Which version of hbase are you using ?
>>
>> Thanks
>>
>>
>>
>> On Feb 12, 2012, at 10:41 AM, Alex Baranau <al...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I'm getting scanner lease exceptions during mapreduce job [1] after
>> running
>> > it for less than 7 minutes. Though I have set
>> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
>> > configuration on master and all regionservers and master (and restarted
>> > all). Also set it in job's configuration.
>> >
>> > Is there anything else I could do to eliminate the exception?
>> >
>> > Thanks,
>> > Alex
>> >
>> > [1]
>> >
>> > org.apache.hadoop.hbase.regionserver.LeaseException:
>> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
>> > ʼ3854333525977315054ʼ does not exist
>> > at
>> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
>> >
>> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
>> > Source)
>> >
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Last
>> > at
>> > java.lang.reflect.Method.invoke(Method.java:597)
>> > 4KB
>> > al FAILED at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> > Last
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>> > 8KB
>> >
>> > All
>> > at
>> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > Method)
>> >
>> > at
>> >
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> >
>> > at
>> >
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >
>> > at
>> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> >
>> > at
>> > org.ap
>>
>
>
Re: LeaseException despite high hbase.regionserver.lease.period
Posted by Alex Baranau <al...@gmail.com>.
Hi,
0.90.4-cdh3u2
Alex
On Sun, Feb 12, 2012 at 1:44 PM, <yu...@gmail.com> wrote:
> Which version of hbase are you using ?
>
> Thanks
>
>
>
> On Feb 12, 2012, at 10:41 AM, Alex Baranau <al...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I'm getting scanner lease exceptions during mapreduce job [1] after
> running
> > it for less than 7 minutes. Though I have set
> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> > configuration on master and all regionservers and master (and restarted
> > all). Also set it in job's configuration.
> >
> > Is there anything else I could do to eliminate the exception?
> >
> > Thanks,
> > Alex
> >
> > [1]
> >
> > org.apache.hadoop.hbase.regionserver.LeaseException:
> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > ʼ3854333525977315054ʼ does not exist
> > at
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
> >
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
> >
> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> > Source)
> >
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > Last
> > at
> > java.lang.reflect.Method.invoke(Method.java:597)
> > 4KB
> > al FAILED at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > Last
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > 8KB
> >
> > All
> > at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >
> > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >
> > at
> > org.ap
>
Re: LeaseException despite high hbase.regionserver.lease.period
Posted by yu...@gmail.com.
Which version of hbase are you using ?
Thanks
On Feb 12, 2012, at 10:41 AM, Alex Baranau <al...@gmail.com> wrote:
> Hello,
>
> I'm getting scanner lease exceptions during mapreduce job [1] after running
> it for less than 7 minutes. Though I have set
> hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase
> configuration on master and all regionservers and master (and restarted
> all). Also set it in job's configuration.
>
> Is there anything else I could do to eliminate the exception?
>
> Thanks,
> Alex
>
> [1]
>
> org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> ʼ3854333525977315054ʼ does not exist
> at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862)
>
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
> Source)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> Last
> at
> java.lang.reflect.Method.invoke(Method.java:597)
> 4KB
> al FAILED at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Last
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> 8KB
>
> All
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>
> at
> org.ap