You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Angus He <an...@gmail.com> on 2009/07/27 11:49:30 UTC

Many NotServingRegionException exceptions when inserting data to HBase

Hi All,

We have a cluster with 4 nodes, one for the Master server, and others
for Region Server.

In our case, there are 3 Downloaders crawling some specific web pages
which will be saved in HBase(0.19.3) then.
After running the Downloaders for a while,  we found that there are
tons of Exceptions in the hbase log files like this:

2009-07-27 12:12:25,833 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
of ~1.8m for region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
in 316ms, sequence id=4774267, compaction requested=true
2009-07-27 12:12:25,833 DEBUG
org.apache.hadoop.hbase.regionserver.HStore: closed
1734514356/CF_INFORMATION
2009-07-27 12:12:25,834 DEBUG
org.apache.hadoop.hbase.regionserver.HStore: closed
1734514356/CF_CONTENT
2009-07-27 12:12:25,834 INFO
org.apache.hadoop.hbase.regionserver.HRegion: Closed
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
2009-07-27 12:12:25,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
2009-07-27 12:12:25,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
        at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
        at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
        at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
        at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)


And from then ,  the Downloaders got a big drop in the speed of page crawling.

Any ideas?


-- 
Regards
Angus

Re: Many NotServingRegionException exceptions when inserting data to HBase

Posted by Angus He <an...@gmail.com>.
Hi Ninad,

Thanks for your comments.

We have already committed 3G memory to each of the region servers.
And the following texts are excerpted from hbase-env.sh in our cluster.

# The java implementation to use.  Java 1.6 required.
export JAVA_HOME=/usr/lib/jvm/java-6-sun

# The maximum amount of heap to use, in MB. Default is 1000.
export HBASE_HEAPSIZE=3000

# Extra Java runtime options.  Empty by default.
export HBASE_OPTS="-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-Xloggc:/opt/hbase/logs/gc-hbase.log"


 On Mon, Jul 27, 2009 at 6:03 PM, Ninad Raut<hb...@gmail.com> wrote:
> I think there is a swap happening on regionservers due to less memory...
> increase the allotted memory in hadoop-site.xml for all the slaves.
>
> On Mon, Jul 27, 2009 at 3:19 PM, Angus He <an...@gmail.com> wrote:
>



-- 
Regards
Angus

Re: Many NotServingRegionException exceptions when inserting data to HBase

Posted by Ninad Raut <hb...@gmail.com>.
I think there is a swap happening on regionservers due to less memory...
increase the allotted memory in hadoop-site.xml for all the slaves.

On Mon, Jul 27, 2009 at 3:19 PM, Angus He <an...@gmail.com> wrote:

> Hi All,
>
> We have a cluster with 4 nodes, one for the Master server, and others
> for Region Server.
>
> In our case, there are 3 Downloaders crawling some specific web pages
> which will be saved in HBase(0.19.3) then.
> After running the Downloaders for a while,  we found that there are
> tons of Exceptions in the hbase log files like this:
>
> 2009-07-27 12:12:25,833 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
> of ~1.8m for region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> in 316ms, sequence id=4774267, compaction requested=true
> 2009-07-27 12:12:25,833 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: closed
> 1734514356/CF_INFORMATION
> 2009-07-27 12:12:25,834 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: closed
> 1734514356/CF_CONTENT
> 2009-07-27 12:12:25,834 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Closed
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> 2009-07-27 12:12:25,837 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> 2009-07-27 12:12:25,837 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
> 9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
> 9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
>
> And from then ,  the Downloaders got a big drop in the speed of page
> crawling.
>
> Any ideas?
>
>
> --
> Regards
> Angus
>

Re: Many NotServingRegionException exceptions when inserting data to HBase

Posted by Angus He <an...@gmail.com>.
Yup ,  I found this in the Master log file.

2009-07-27 12:13:10,880 INFO
org.apache.hadoop.hbase.master.ServerManager: Received
MSG_REPORT_SPLIT:
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868:
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
split; daughters:
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248667945476,
webpage,http://www.tianya.cn/techforum/content/50/565730.shtml,1248667945476:
safeMode=false from 192.168.33.5:60020

BTW, There is about a 50-second time lag between Region server and
Master server.


On Tue, Jul 28, 2009 at 11:45 AM, stack<st...@duboce.net> wrote:
> I was looking at this today.
>
> When you start to get the NotServingRegionException, has the region just
> split?  You can tell by looking for "webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868" in the
> master log.  If it has a message about splitting just before you get the
> below NSRE exception, then it seems like its a case of HBASE-1671.
> Hopefully will have a fix this evening.
>
> St.Ack
>
> On Mon, Jul 27, 2009 at 2:49 AM, Angus He <an...@gmail.com> wrote:
>
>> Hi All,
>>
>> We have a cluster with 4 nodes, one for the Master server, and others
>> for Region Server.
>>
>> In our case, there are 3 Downloaders crawling some specific web pages
>> which will be saved in HBase(0.19.3) then.
>> After running the Downloaders for a while,  we found that there are
>> tons of Exceptions in the hbase log files like this:
>>
>> 2009-07-27 12:12:25,833 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
>> of ~1.8m for region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> in 316ms, sequence id=4774267, compaction requested=true
>> 2009-07-27 12:12:25,833 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: closed
>> 1734514356/CF_INFORMATION
>> 2009-07-27 12:12:25,834 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: closed
>> 1734514356/CF_CONTENT
>> 2009-07-27 12:12:25,834 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Closed
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> 2009-07-27<http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868%0A2009-07-27>12:12:25,837 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> 2009-07-27 12:12:25,837 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
>> 9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
>> 9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>>
>>
>> And from then ,  the Downloaders got a big drop in the speed of page
>> crawling.
>>
>> Any ideas?
>>
>>
>> --
>> Regards
>> Angus
>>
>



-- 
Regards
Angus

Re: Many NotServingRegionException exceptions when inserting data to HBase

Posted by stack <st...@duboce.net>.
I was looking at this today.

When you start to get the NotServingRegionException, has the region just
split?  You can tell by looking for "webpage,
http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868" in the
master log.  If it has a message about splitting just before you get the
below NSRE exception, then it seems like its a case of HBASE-1671.
Hopefully will have a fix this evening.

St.Ack

On Mon, Jul 27, 2009 at 2:49 AM, Angus He <an...@gmail.com> wrote:

> Hi All,
>
> We have a cluster with 4 nodes, one for the Master server, and others
> for Region Server.
>
> In our case, there are 3 Downloaders crawling some specific web pages
> which will be saved in HBase(0.19.3) then.
> After running the Downloaders for a while,  we found that there are
> tons of Exceptions in the hbase log files like this:
>
> 2009-07-27 12:12:25,833 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
> of ~1.8m for region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> in 316ms, sequence id=4774267, compaction requested=true
> 2009-07-27 12:12:25,833 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: closed
> 1734514356/CF_INFORMATION
> 2009-07-27 12:12:25,834 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: closed
> 1734514356/CF_CONTENT
> 2009-07-27 12:12:25,834 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Closed
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> 2009-07-27<http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868%0A2009-07-27>12:12:25,837 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> 2009-07-27 12:12:25,837 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
> 9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
> 9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
> closed
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
>
> And from then ,  the Downloaders got a big drop in the speed of page
> crawling.
>
> Any ideas?
>
>
> --
> Regards
> Angus
>