You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Oleg Ruchovets <or...@gmail.com> on 2010/11/08 16:52:10 UTC
Hbase insertion process cause to region server down.
Hi ,
I run map/reduce job which from reducers writes to hbase. I have to write
8-9 million raws.
My environment:
5 servers dual core , 8G ram.
HDFS -- 9TB
hadoop has 1000 heap_size ,
hbase has 2000 heap_size
zookeeper has 1000 heap_size
20 map / 16 reducers while running the job
Allmost all the time get such exception
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1607)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1731)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Thanks in advance.
Re: Hbase insertion process cause to region server down.
Posted by Oleg Ruchovets <or...@gmail.com>.
As Jonathan said it is normal , I still need to do more tests to be sure
that I hasn't lost the inserted data.
I paid attention that I got this exceptions during compaction
2010-11-08 18:28:53,663 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction
started. Attempting to free 52292232 bytes
2010-11-08 18:28:53,666 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction
completed. Freed 52309408 bytes. Priority Sizes: Single=57.505157MB
2010-11-08 18:29:22,111 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region
URLs_sanity,20101107_n/gs/34468.html,1289233112752
2010-11-08 18:29:22,141 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compaction size of globalcounters: 65.5m; Skipped 1 file(s), size: 41262334
2010-11-08 18:29:22,141 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Started compaction of 3 file(s) into
/hbase/URLs_sanity/compaction.dir/613365657, seqid=2855632301
2010-11-08 18:29:22,896 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
URLs_sanity,20101107_gallery.mobile9.com/f/1139457/,1289233750914
2010-11-08 18:29:22,897 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
URLs_sanity,20101107_gallery/1139457/,1289233750914
2010-11-08 18:29:22,897 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region URLs_sanity,20101107_bl/1139457/,1289233750914,
encoded=1345660410
My questions are:
1) what happens with insertion process when the job got NSRE exception?
2) Could compaction cause to additional load to the system? and if yes
what is the recommendation regarding
compaction parameters? In my case reducers inserts to hbase almost 3 hours.
3) In addition such behavior I got only in one machine , but I have 5.
Oleg.
On Tue, Nov 9, 2010 at 6:26 AM, Hari Sreekumar <hs...@clickable.com>wrote:
> Hi Jonathan,
>
> I have faced the exact same issue as Oleg and it seems to
> happen randomly. Sometimes I get this exception and the task fails,
> sometimes I don't and it succeeds. Do you mean that this is normal
> behavior?
>
> hari
>
> On Tue, Nov 9, 2010 at 12:25 AM, Jonathan Gray <jg...@facebook.com> wrote:
>
> > NSRE is normal, this happens when regions move around and your client
> needs
> > to update the location.
> >
> > That seems like an awful lot of mappers/reducers on a 5 server / dual
> core
> > setup... You have only 2 cores per server but you have a DataNode,
> > RegionServer, and 4 map tasks and 3 reduce tasks? That may be over
> > extending things a bit.
> >
> > JG
> >
> > > -----Original Message-----
> > > From: Oleg Ruchovets [mailto:oruchovets@gmail.com]
> > > Sent: Monday, November 08, 2010 7:52 AM
> > > To: user@hbase.apache.org
> > > Subject: Hbase insertion process cause to region server down.
> > >
> > > Hi ,
> > >
> > > I run map/reduce job which from reducers writes to hbase. I have to
> > > write
> > > 8-9 million raws.
> > >
> > > My environment:
> > > 5 servers dual core , 8G ram.
> > > HDFS -- 9TB
> > >
> > > hadoop has 1000 heap_size ,
> > > hbase has 2000 heap_size
> > > zookeeper has 1000 heap_size
> > > 20 map / 16 reducers while running the job
> > >
> > > Allmost all the time get such exception
> > >
> > > org.apache.hadoop.hbase.NotServingRegionException:
> > > org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
> > > 20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337
> > > closed
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
> > > :1607)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
> > > va:1731)
> > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > > rImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > > at
> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
> > > 5)
> > >
> > >
> > >
> > > Thanks in advance.
> >
>
RE: Hbase insertion process cause to region server down.
Posted by Jonathan Gray <jg...@facebook.com>.
Yes, it is normal and expected to get some NSREs if your regions are splitting or there is rebalancing going on.
They should not fail the task unless for some reason the client is never able to locate the region. If this is the case, something is wrong with the cluster.
JG
> -----Original Message-----
> From: Hari Sreekumar [mailto:hsreekumar@clickable.com]
> Sent: Monday, November 08, 2010 8:26 PM
> To: user@hbase.apache.org
> Subject: Re: Hbase insertion process cause to region server down.
>
> Hi Jonathan,
>
> I have faced the exact same issue as Oleg and it seems to
> happen randomly. Sometimes I get this exception and the task fails,
> sometimes I don't and it succeeds. Do you mean that this is normal
> behavior?
>
> hari
>
> On Tue, Nov 9, 2010 at 12:25 AM, Jonathan Gray <jg...@facebook.com>
> wrote:
>
> > NSRE is normal, this happens when regions move around and your client
> needs
> > to update the location.
> >
> > That seems like an awful lot of mappers/reducers on a 5 server / dual
> core
> > setup... You have only 2 cores per server but you have a DataNode,
> > RegionServer, and 4 map tasks and 3 reduce tasks? That may be over
> > extending things a bit.
> >
> > JG
> >
> > > -----Original Message-----
> > > From: Oleg Ruchovets [mailto:oruchovets@gmail.com]
> > > Sent: Monday, November 08, 2010 7:52 AM
> > > To: user@hbase.apache.org
> > > Subject: Hbase insertion process cause to region server down.
> > >
> > > Hi ,
> > >
> > > I run map/reduce job which from reducers writes to hbase. I have to
> > > write
> > > 8-9 million raws.
> > >
> > > My environment:
> > > 5 servers dual core , 8G ram.
> > > HDFS -- 9TB
> > >
> > > hadoop has 1000 heap_size ,
> > > hbase has 2000 heap_size
> > > zookeeper has 1000 heap_size
> > > 20 map / 16 reducers while running the job
> > >
> > > Allmost all the time get such exception
> > >
> > > org.apache.hadoop.hbase.NotServingRegionException:
> > > org.apache.hadoop.hbase.NotServingRegionException: Region
> URLs_sanity,
> > > 20101106_www.homemade-chinese-soups.com/conch-
> soup.html,1289163460337
> > > closed
> > > at
> > >
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
> > > :1607)
> > > at
> > >
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
> > > at
> > >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
> > > va:1731)
> > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > > at
> > >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > > rImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > > at
> > >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
> > > 5)
> > >
> > >
> > >
> > > Thanks in advance.
> >
Re: Hbase insertion process cause to region server down.
Posted by Hari Sreekumar <hs...@clickable.com>.
Hi Jonathan,
I have faced the exact same issue as Oleg and it seems to
happen randomly. Sometimes I get this exception and the task fails,
sometimes I don't and it succeeds. Do you mean that this is normal behavior?
hari
On Tue, Nov 9, 2010 at 12:25 AM, Jonathan Gray <jg...@facebook.com> wrote:
> NSRE is normal, this happens when regions move around and your client needs
> to update the location.
>
> That seems like an awful lot of mappers/reducers on a 5 server / dual core
> setup... You have only 2 cores per server but you have a DataNode,
> RegionServer, and 4 map tasks and 3 reduce tasks? That may be over
> extending things a bit.
>
> JG
>
> > -----Original Message-----
> > From: Oleg Ruchovets [mailto:oruchovets@gmail.com]
> > Sent: Monday, November 08, 2010 7:52 AM
> > To: user@hbase.apache.org
> > Subject: Hbase insertion process cause to region server down.
> >
> > Hi ,
> >
> > I run map/reduce job which from reducers writes to hbase. I have to
> > write
> > 8-9 million raws.
> >
> > My environment:
> > 5 servers dual core , 8G ram.
> > HDFS -- 9TB
> >
> > hadoop has 1000 heap_size ,
> > hbase has 2000 heap_size
> > zookeeper has 1000 heap_size
> > 20 map / 16 reducers while running the job
> >
> > Allmost all the time get such exception
> >
> > org.apache.hadoop.hbase.NotServingRegionException:
> > org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
> > 20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337
> > closed
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
> > :1607)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
> > va:1731)
> > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > rImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
> > 5)
> >
> >
> >
> > Thanks in advance.
>
RE: Hbase insertion process cause to region server down.
Posted by Jonathan Gray <jg...@facebook.com>.
NSRE is normal, this happens when regions move around and your client needs to update the location.
That seems like an awful lot of mappers/reducers on a 5 server / dual core setup... You have only 2 cores per server but you have a DataNode, RegionServer, and 4 map tasks and 3 reduce tasks? That may be over extending things a bit.
JG
> -----Original Message-----
> From: Oleg Ruchovets [mailto:oruchovets@gmail.com]
> Sent: Monday, November 08, 2010 7:52 AM
> To: user@hbase.apache.org
> Subject: Hbase insertion process cause to region server down.
>
> Hi ,
>
> I run map/reduce job which from reducers writes to hbase. I have to
> write
> 8-9 million raws.
>
> My environment:
> 5 servers dual core , 8G ram.
> HDFS -- 9TB
>
> hadoop has 1000 heap_size ,
> hbase has 2000 heap_size
> zookeeper has 1000 heap_size
> 20 map / 16 reducers while running the job
>
> Allmost all the time get such exception
>
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
> 20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337
> closed
> at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
> :1607)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
> va:1731)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
> 5)
>
>
>
> Thanks in advance.