You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Bing Jiang <ji...@gmail.com> on 2012/12/25 12:32:00 UTC
hbase-0.94.3 region cannot split , which results into regionserver offline？
hi,all
use hbase-0.94.3, We have encounted an error that results into
Regionserver offline.
Stack:
2012-12-25 19:25:04,654 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
10.1.1.34,60020,1356416242185: Uncaught exception in service thread
regionserver60020.cacheFlusher
java.lang.AssertionError: getSplitPoint() called on a region that can't
split!
at
org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
at
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
at java.lang.Thread.run(Thread.java:662)
2012-12-25 19:25:04,654 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
loaded coprocessors are: []
....
Can anyone meet these errors?
Cheers!
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: hbase-0.94.3 region cannot split , which results into regionserver offline？
Posted by lars hofhansl <lh...@yahoo.com>.
Whoa. Unless a custom split policy was used here, this is bad.
________________________________
From: ramkrishna vasudevan <ra...@gmail.com>
To: user@hbase.apache.org
Sent: Tuesday, December 25, 2012 8:28 AM
Subject: Re: hbase-0.94.3 region cannot split , which results into regionserver offline？
The above error is an assertion error. It comes from here
{code}
// Should already be enforced by the split policy!
assert !this.region.getRegionInfo().isMetaRegion();
{code}
So need to see this i feel.
Regards
Ram
On Tue, Dec 25, 2012 at 9:48 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> By looking at the code, we have:
> "Not splittable if we find a reference store file present in the store"...
>
> Not really sure to understand what that mean :(
>
> 2012/12/25, ramkrishna vasudevan <ra...@gmail.com>:
> > Oh, this seems interesting.
> > You have logs before this which says for which region the split started?
> > BTW are you using any custom split policy.
> >
> > Regards
> > Ram
> >
> > On Tue, Dec 25, 2012 at 7:43 PM, <yu...@gmail.com> wrote:
> >
> >> Can you provide more log around the time this happened ?
> >>
> >> Looks like the exception message should be enhanced with region name.
> >>
> >> Cheers
> >>
> >> On Dec 25, 2012, at 3:32 AM, Bing Jiang <ji...@gmail.com>
> wrote:
> >>
> >> > hi,all
> >> > use hbase-0.94.3, We have encounted an error that results into
> >> > Regionserver offline.
> >> >
> >> > Stack:
> >> >
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> >> server
> >> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> >> > regionserver60020.cacheFlusher
> >> > java.lang.AssertionError: getSplitPoint() called on a region that
> can't
> >> > split!
> >> > at
> >> >
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer
> abort:
> >> > loaded coprocessors are: []
> >> > ....
> >> > Can anyone meet these errors?
> >> >
> >> >
> >> > Cheers!
> >> > --
> >> > Bing Jiang
> >> > weibo: http://weibo.com/jiangbinglover
> >> > BLOG: http://blog.sina.com.cn/jiangbinglover
> >> > National Research Center for Intelligent Computing Systems
> >> > Institute of Computing technology
> >> > Graduate University of Chinese Academy of Science
> >>
> >
>
Re: hbase-0.94.3 region cannot split , which results into regionserver offline？
Posted by ramkrishna vasudevan <ra...@gmail.com>.
The above error is an assertion error. It comes from here
{code}
// Should already be enforced by the split policy!
assert !this.region.getRegionInfo().isMetaRegion();
{code}
So need to see this i feel.
Regards
Ram
On Tue, Dec 25, 2012 at 9:48 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> By looking at the code, we have:
> "Not splittable if we find a reference store file present in the store"...
>
> Not really sure to understand what that mean :(
>
> 2012/12/25, ramkrishna vasudevan <ra...@gmail.com>:
> > Oh, this seems interesting.
> > You have logs before this which says for which region the split started?
> > BTW are you using any custom split policy.
> >
> > Regards
> > Ram
> >
> > On Tue, Dec 25, 2012 at 7:43 PM, <yu...@gmail.com> wrote:
> >
> >> Can you provide more log around the time this happened ?
> >>
> >> Looks like the exception message should be enhanced with region name.
> >>
> >> Cheers
> >>
> >> On Dec 25, 2012, at 3:32 AM, Bing Jiang <ji...@gmail.com>
> wrote:
> >>
> >> > hi,all
> >> > use hbase-0.94.3, We have encounted an error that results into
> >> > Regionserver offline.
> >> >
> >> > Stack:
> >> >
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> >> server
> >> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> >> > regionserver60020.cacheFlusher
> >> > java.lang.AssertionError: getSplitPoint() called on a region that
> can't
> >> > split!
> >> > at
> >> >
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer
> abort:
> >> > loaded coprocessors are: []
> >> > ....
> >> > Can anyone meet these errors?
> >> >
> >> >
> >> > Cheers!
> >> > --
> >> > Bing Jiang
> >> > weibo: http://weibo.com/jiangbinglover
> >> > BLOG: http://blog.sina.com.cn/jiangbinglover
> >> > National Research Center for Intelligent Computing Systems
> >> > Institute of Computing technology
> >> > Graduate University of Chinese Academy of Science
> >>
> >
>
Re: hbase-0.94.3 region cannot split , which results into regionserver offline？
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
By looking at the code, we have:
"Not splittable if we find a reference store file present in the store"...
Not really sure to understand what that mean :(
2012/12/25, ramkrishna vasudevan <ra...@gmail.com>:
> Oh, this seems interesting.
> You have logs before this which says for which region the split started?
> BTW are you using any custom split policy.
>
> Regards
> Ram
>
> On Tue, Dec 25, 2012 at 7:43 PM, <yu...@gmail.com> wrote:
>
>> Can you provide more log around the time this happened ?
>>
>> Looks like the exception message should be enhanced with region name.
>>
>> Cheers
>>
>> On Dec 25, 2012, at 3:32 AM, Bing Jiang <ji...@gmail.com> wrote:
>>
>> > hi,all
>> > use hbase-0.94.3, We have encounted an error that results into
>> > Regionserver offline.
>> >
>> > Stack:
>> >
>> > 2012-12-25 19:25:04,654 FATAL
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
>> server
>> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
>> > regionserver60020.cacheFlusher
>> > java.lang.AssertionError: getSplitPoint() called on a region that can't
>> > split!
>> > at
>> > org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
>> > at java.lang.Thread.run(Thread.java:662)
>> > 2012-12-25 19:25:04,654 FATAL
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
>> > loaded coprocessors are: []
>> > ....
>> > Can anyone meet these errors?
>> >
>> >
>> > Cheers!
>> > --
>> > Bing Jiang
>> > weibo: http://weibo.com/jiangbinglover
>> > BLOG: http://blog.sina.com.cn/jiangbinglover
>> > National Research Center for Intelligent Computing Systems
>> > Institute of Computing technology
>> > Graduate University of Chinese Academy of Science
>>
>
Re: hbase-0.94.3 region cannot split , which results into regionserver offline？
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Oh, this seems interesting.
You have logs before this which says for which region the split started?
BTW are you using any custom split policy.
Regards
Ram
On Tue, Dec 25, 2012 at 7:43 PM, <yu...@gmail.com> wrote:
> Can you provide more log around the time this happened ?
>
> Looks like the exception message should be enhanced with region name.
>
> Cheers
>
> On Dec 25, 2012, at 3:32 AM, Bing Jiang <ji...@gmail.com> wrote:
>
> > hi,all
> > use hbase-0.94.3, We have encounted an error that results into
> > Regionserver offline.
> >
> > Stack:
> >
> > 2012-12-25 19:25:04,654 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> server
> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> > regionserver60020.cacheFlusher
> > java.lang.AssertionError: getSplitPoint() called on a region that can't
> > split!
> > at
> > org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
> > at
> >
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
> > at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
> > at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
> > at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
> > at java.lang.Thread.run(Thread.java:662)
> > 2012-12-25 19:25:04,654 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> > loaded coprocessors are: []
> > ....
> > Can anyone meet these errors?
> >
> >
> > Cheers!
> > --
> > Bing Jiang
> > weibo: http://weibo.com/jiangbinglover
> > BLOG: http://blog.sina.com.cn/jiangbinglover
> > National Research Center for Intelligent Computing Systems
> > Institute of Computing technology
> > Graduate University of Chinese Academy of Science
>
Re: hbase-0.94.3 region cannot split , which results into regionserver offline？
Posted by yu...@gmail.com.
Can you provide more log around the time this happened ?
Looks like the exception message should be enhanced with region name.
Cheers
On Dec 25, 2012, at 3:32 AM, Bing Jiang <ji...@gmail.com> wrote:
> hi,all
> use hbase-0.94.3, We have encounted an error that results into
> Regionserver offline.
>
> Stack:
>
> 2012-12-25 19:25:04,654 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> regionserver60020.cacheFlusher
> java.lang.AssertionError: getSplitPoint() called on a region that can't
> split!
> at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
> at java.lang.Thread.run(Thread.java:662)
> 2012-12-25 19:25:04,654 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are: []
> ....
> Can anyone meet these errors?
>
>
> Cheers!
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science