You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dru Jensen <dr...@gmail.com> on 2008/11/20 18:47:43 UTC
rowcounter fails - Region closed
I have a table that has 20+ million rows. I tried to run rowcounter
MR process against it but one of the task attempts fail on the
following exception:
java.io.IOException: java.io.IOException: Region metrics,Game Face|
News,1226620888277 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:
1336)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1175)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
I checked .meta. and I don't see this region online. Should it be?
Any ideas how to fix it?
Thanks,
Dru
Re: rowcounter fails - Region closed
Posted by Dru Jensen <dr...@gmail.com>.
Thanks Stack. Yes, upping the xceivercount fixed the previous issue.
The system is more stable than ever. I used the "spelling" you
provided. Maybe add to FAQ?
grep the master I see the region but unfortunately, I did not have
DEBUG on at the time. I added DEBUG to the log4j settings. After
restarting I saw this:
hbase-root-master-dev-hadoop1.brandaffinity.net.log:2008-11-20
10:28:31,265 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current
assignment of metrics,Game Face|News,1226620888277 is not valid:
serverInfo: address: 10.10.10.51:60020, startcode: 1227205702706,
load: (requests: 4 regions: 1), passed startCode: 1227124620094,
storedInfo.startCode: 1227205702706, unassignedRegions: false,
pendingRegions: false
hbase-root-master-dev-hadoop1.brandaffinity.net.log:2008-11-20
10:28:48,156 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region metrics,Game Face|News,1226620888277 to server
10.10.10.51:60020
I ran the jobcounter again and it was successful. Looks like HBase
recovered on it's own.
Thanks again,
Dru
On Nov 20, 2008, at 10:11 AM, stack wrote:
> Says its 'closed'.
>
> Look in master log for references to this region. See if it was
> ever online. If it was, see which host it was up on; e.g. do
> something like "grep 'metrics,Game Face|News,1226620888277'"
> master.log". Look at its RegionInfo -- should be output in master
> log if you have DEBUG enabled. Look at the encoded name. See that
> the region exists still in hdfs by doing "./bin/hadoop fs -lsr
> HBASE_ROOTDIR/metrics/ENCODED_NAME. Next add '-v metaScanner' to
> weed the metaScanner output when grepping master log to see trail of
> events as they pertain to this particular region. Might give you a
> clue as to what happened. When you scan the .META., does this
> region appear at all? Is there a 'hole' in the .META. where this
> region should be? We may have to reinsert if so. To do this, will
> need old value for HRI.
>
> Did the upping of xceivercount work for your previous issue Dru?
>
> St.Ack
>
>
>
> Dru Jensen wrote:
>> I have a table that has 20+ million rows. I tried to run
>> rowcounter MR process against it but one of the task attempts fail
>> on the following exception:
>>
>> java.io.IOException: java.io.IOException: Region metrics,Game Face|
>> News,1226620888277 closed
>> at
>> org
>> .apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:
>> 1336)
>> at
>> org
>> .apache
>> .hadoop
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:
>> 1175)
>> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>> at
>> sun
>> .reflect
>> .DelegatingMethodAccessorImpl
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.apache.hadoop.hbase.ipc.HbaseRPC
>> $Server.call(HbaseRPC.java:554)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>>
>> I checked .meta. and I don't see this region online. Should it be?
>> Any ideas how to fix it?
>>
>> Thanks,
>> Dru
>>
>>
>
Re: rowcounter fails - Region closed
Posted by stack <st...@duboce.net>.
Says its 'closed'.
Look in master log for references to this region. See if it was ever
online. If it was, see which host it was up on; e.g. do something like
"grep 'metrics,Game Face|News,1226620888277'" master.log". Look at its
RegionInfo -- should be output in master log if you have DEBUG enabled.
Look at the encoded name. See that the region exists still in hdfs by
doing "./bin/hadoop fs -lsr HBASE_ROOTDIR/metrics/ENCODED_NAME. Next
add '-v metaScanner' to weed the metaScanner output when grepping master
log to see trail of events as they pertain to this particular region.
Might give you a clue as to what happened. When you scan the .META.,
does this region appear at all? Is there a 'hole' in the .META. where
this region should be? We may have to reinsert if so. To do this, will
need old value for HRI.
Did the upping of xceivercount work for your previous issue Dru?
St.Ack
Dru Jensen wrote:
> I have a table that has 20+ million rows. I tried to run rowcounter
> MR process against it but one of the task attempts fail on the
> following exception:
>
> java.io.IOException: java.io.IOException: Region metrics,Game
> Face|News,1226620888277 closed
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1336)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1175)
>
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>
> I checked .meta. and I don't see this region online. Should it be?
> Any ideas how to fix it?
>
> Thanks,
> Dru
>
>