You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Rohit Nigam <rn...@decarta.com> on 2011/08/12 19:48:01 UTC
Hbck errors
Hi
We are having a issue while running the job as it hangs forever at 99%
(reducer) inserting into the table , hbase hbck -details finds
inconsistency in the table I am trying to insert data as :--
Chain of regions in table NAM_CLUSTERKEYS3 is broken; edges does not
contain
00000000|00000000000000000USA|0000240b|000000000000000LAKE,|002ff027|000
0000000CONSULTING|006e8f90|000000000000000000NY:0
I tried hbase hbck -fix but does not help.
Tried scanning the '.META' table
scan
'.META.',{STARTROW=>'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000
240b|000000000000000LAKE',LIMIT=>1}
Getting this error:--
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying
to contact region server doop6. for region .META.,,1, row
'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|000000000000000
LAKE', but failed after 7 attempts.
Exceptions:
java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)
at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)
at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)
at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)
at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContainingKey(HFile.j
ava:918)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:
1296)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(Stor
eFileScanner.java:136)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScan
ner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.ja
va:77)
at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1341)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegio
n.java:2269)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(
HRegion.java:1126)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:111
8)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:110
2)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1781)
... 5 more
java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)
at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)
at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)
at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)
at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContaini
How can I fix the problem in the table . Any help would be appreciated.
Re: Hbck errors
Posted by Stack <st...@duboce.net>.
On Fri, Aug 12, 2011 at 10:48 AM, Rohit Nigam <rn...@decarta.com> wrote:
> Tried scanning the '.META' table
> scan
> '.META.',{STARTROW=>'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000
> 240b|000000000000000LAKE',LIMIT=>1}
>
>
>
> Getting this error:--
>
I think you need to add two commas to the end of the key when scanning
.META. to avoid the IllegalArgumentException below.
> How can I fix the problem in the table . Any help would be appreciated.
>
See if you can scan the region as you are doing above. What error do
you get back or are you able to scan?
Next see if an entry for it in .META. (Look at content of meta by
doing '> echo "scan '.META.'" | ./bin/hbase shell &> /tmp/meta.txt'
then review /tmp/meta.txt.
Get this regions encoded name from the above /tmp/meta.txt. See if
its in the filesystem. Grep its encoded name in the master. See if
you can tell story on what happened with this region.
St.Ack