You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Haijun Cao <ha...@ymail.com> on 2009/07/15 00:10:55 UTC
java.io.IOException: HRegionInfo was null or empty in .META.
Hi
I am running hbase PE test, loaded TestTable with 10M records without any problem, later hbase crashed (during another sequentialWrite test), after restart, I can't count TestTable, it always get stuck at 1.1 mil records.
I am wondering if anybody has encountered the same problem (data got corrupted after server crash)? How do you recover? At this point, it is just test data, so it is ok for me to lose some data. Is there a way to drop/repair the bad region (0001192603)? Does hbase have a auto-repair tool like fsck?
Your help is greatly appreciated.
Haijun
Error stack:
Current count: 1100000, row: 0001099999
NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '0001192603', but failed after 5 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext'
from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:298:in `invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
from org/jruby/ast/WhileNode.java:127:in `interpret'
from org/jruby/ast/NewlineNode.java:104:in `interpret'
from org/jruby/ast/BlockNode.java:71:in `interpret'
from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:273:in `cacheAndCall'
from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call'
I scan the .META. table and noticed that region 0001192603 has no info columns, only historian columns. I also check the hadoop file system, region 0001192603 has a oldlogfile.log file in its directory (other regions don't):
Found 3 items
-rw-r--r-- 3 bamboo supergroup 619 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/.regioninfo
drwxr-xr-x - bamboo supergroup 0 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/info
-rw-r--r-- 3 bamboo supergroup 39417817 2009-07-13 16:52 /user/bamboo/hbase/TestTable/954997373/oldlogfile.log
Re: java.io.IOException: HRegionInfo was null or empty in .META.
Posted by stack <st...@duboce.net>.
On Tue, Jul 14, 2009 at 4:09 PM, Ryan Rawson <ry...@gmail.com> wrote:
> I believe that we are making truncate in the shell actually issue the
> major_compact itself too now.
>
It does. In trunk the shell does this on 'drop' of a table:
def drop(tableName)
now = Time.now
@formatter.header()
if @admin.isTableEnabled(tableName)
raise IOError.new("Table " + tableName + " is enabled. Disable it
first")
else
@admin.deleteTable(tableName)
major_compact(HConstants::META_TABLE_NAME);
end
@formatter.footer(now)
end
St.Ack
Re: java.io.IOException: HRegionInfo was null or empty in .META.
Posted by Ryan Rawson <ry...@gmail.com>.
It's a known problem with the .META. table involving old deleted rows
and new rows. There is a brief summary here:
http://issues.apache.org/jira/browse/HBASE-1383
I believe that we are making truncate in the shell actually issue the
major_compact itself too now.
On Tue, Jul 14, 2009 at 3:40 PM, Haijun Cao<ha...@ymail.com> wrote:
> Ryan,
>
> It worked. thank you!
>
> I have used truncate command on TestTable before, but I then decided to drop the table completely and start the test from scratch. This problem seem to happen after the hbase server crashed. I experienced both hbase-1634/HBASE-1638 and this count issue. Running the script in 1638 resolve the NPE problem (Thanks stack), running flush/major_compact meta table resolved the count issue.
>
> For my education, do you mind explaining the cause/fix for this count issue I am seeing?
>
> Thanks
> Haijun
>
>
>
>
> ________________________________
> From: Ryan Rawson <ry...@gmail.com>
> To: hbase-dev@hadoop.apache.org
> Sent: Tuesday, July 14, 2009 3:14:33 PM
> Subject: Re: java.io.IOException: HRegionInfo was null or empty in .META.
>
> Try doing this on the shell:
>
> flush '.META.'
> major_compact '.META.'
>
> did you use 'truncate' command at any point?
>
> On Tue, Jul 14, 2009 at 3:10 PM, Haijun Cao<ha...@ymail.com> wrote:
>>
>> Hi
>>
>> I am running hbase PE test, loaded TestTable with 10M records without any problem, later hbase crashed (during another sequentialWrite test), after restart, I can't count TestTable, it always get stuck at 1.1 mil records.
>>
>>
>> I am wondering if anybody has encountered the same problem (data got corrupted after server crash)? How do you recover? At this point, it is just test data, so it is ok for me to lose some data. Is there a way to drop/repair the bad region (0001192603)? Does hbase have a auto-repair tool like fsck?
>>
>> Your help is greatly appreciated.
>>
>> Haijun
>>
>>
>> Error stack:
>>
>> Current count: 1100000, row: 0001099999
>> NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '0001192603', but failed after 5 attempts.
>> Exceptions:
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>>
>> from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext'
>> from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke'
>> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>> from java/lang/reflect/Method.java:597:in `invoke'
>> from org/jruby/javasupport/JavaMethod.java:298:in `invokeWithExceptionHandling'
>> from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>> from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
>> from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
>> from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
>> from org/jruby/ast/WhileNode.java:127:in `interpret'
>> from org/jruby/ast/NewlineNode.java:104:in `interpret'
>> from org/jruby/ast/BlockNode.java:71:in `interpret'
>> from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
>> from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
>> from org/jruby/runtime/callsite/CachingCallSite.java:273:in `cacheAndCall'
>> from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call'
>>
>>
>> I scan the .META. table and noticed that region 0001192603 has no info columns, only historian columns. I also check the hadoop file system, region 0001192603 has a oldlogfile.log file in its directory (other regions don't):
>>
>> Found 3 items
>> -rw-r--r-- 3 bamboo supergroup 619 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/.regioninfo
>> drwxr-xr-x - bamboo supergroup 0 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/info
>> -rw-r--r-- 3 bamboo supergroup 39417817 2009-07-13 16:52 /user/bamboo/hbase/TestTable/954997373/oldlogfile.log
>>
>>
>>
>
>
>
>
Re: java.io.IOException: HRegionInfo was null or empty in .META.
Posted by Haijun Cao <ha...@ymail.com>.
Ryan,
It worked. thank you!
I have used truncate command on TestTable before, but I then decided to drop the table completely and start the test from scratch. This problem seem to happen after the hbase server crashed. I experienced both hbase-1634/HBASE-1638 and this count issue. Running the script in 1638 resolve the NPE problem (Thanks stack), running flush/major_compact meta table resolved the count issue.
For my education, do you mind explaining the cause/fix for this count issue I am seeing?
Thanks
Haijun
________________________________
From: Ryan Rawson <ry...@gmail.com>
To: hbase-dev@hadoop.apache.org
Sent: Tuesday, July 14, 2009 3:14:33 PM
Subject: Re: java.io.IOException: HRegionInfo was null or empty in .META.
Try doing this on the shell:
flush '.META.'
major_compact '.META.'
did you use 'truncate' command at any point?
On Tue, Jul 14, 2009 at 3:10 PM, Haijun Cao<ha...@ymail.com> wrote:
>
> Hi
>
> I am running hbase PE test, loaded TestTable with 10M records without any problem, later hbase crashed (during another sequentialWrite test), after restart, I can't count TestTable, it always get stuck at 1.1 mil records.
>
>
> I am wondering if anybody has encountered the same problem (data got corrupted after server crash)? How do you recover? At this point, it is just test data, so it is ok for me to lose some data. Is there a way to drop/repair the bad region (0001192603)? Does hbase have a auto-repair tool like fsck?
>
> Your help is greatly appreciated.
>
> Haijun
>
>
> Error stack:
>
> Current count: 1100000, row: 0001099999
> NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '0001192603', but failed after 5 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
>
> from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext'
> from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke'
> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> from java/lang/reflect/Method.java:597:in `invoke'
> from org/jruby/javasupport/JavaMethod.java:298:in `invokeWithExceptionHandling'
> from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
> from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> from org/jruby/ast/WhileNode.java:127:in `interpret'
> from org/jruby/ast/NewlineNode.java:104:in `interpret'
> from org/jruby/ast/BlockNode.java:71:in `interpret'
> from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
> from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:273:in `cacheAndCall'
> from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call'
>
>
> I scan the .META. table and noticed that region 0001192603 has no info columns, only historian columns. I also check the hadoop file system, region 0001192603 has a oldlogfile.log file in its directory (other regions don't):
>
> Found 3 items
> -rw-r--r-- 3 bamboo supergroup 619 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/.regioninfo
> drwxr-xr-x - bamboo supergroup 0 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/info
> -rw-r--r-- 3 bamboo supergroup 39417817 2009-07-13 16:52 /user/bamboo/hbase/TestTable/954997373/oldlogfile.log
>
>
>
Re: java.io.IOException: HRegionInfo was null or empty in .META.
Posted by Ryan Rawson <ry...@gmail.com>.
Try doing this on the shell:
flush '.META.'
major_compact '.META.'
did you use 'truncate' command at any point?
On Tue, Jul 14, 2009 at 3:10 PM, Haijun Cao<ha...@ymail.com> wrote:
>
> Hi
>
> I am running hbase PE test, loaded TestTable with 10M records without any problem, later hbase crashed (during another sequentialWrite test), after restart, I can't count TestTable, it always get stuck at 1.1 mil records.
>
>
> I am wondering if anybody has encountered the same problem (data got corrupted after server crash)? How do you recover? At this point, it is just test data, so it is ok for me to lose some data. Is there a way to drop/repair the bad region (0001192603)? Does hbase have a auto-repair tool like fsck?
>
> Your help is greatly appreciated.
>
> Haijun
>
>
> Error stack:
>
> Current count: 1100000, row: 0001099999
> NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '0001192603', but failed after 5 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
>
> from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext'
> from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke'
> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> from java/lang/reflect/Method.java:597:in `invoke'
> from org/jruby/javasupport/JavaMethod.java:298:in `invokeWithExceptionHandling'
> from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
> from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> from org/jruby/ast/WhileNode.java:127:in `interpret'
> from org/jruby/ast/NewlineNode.java:104:in `interpret'
> from org/jruby/ast/BlockNode.java:71:in `interpret'
> from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
> from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:273:in `cacheAndCall'
> from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call'
>
>
> I scan the .META. table and noticed that region 0001192603 has no info columns, only historian columns. I also check the hadoop file system, region 0001192603 has a oldlogfile.log file in its directory (other regions don't):
>
> Found 3 items
> -rw-r--r-- 3 bamboo supergroup 619 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/.regioninfo
> drwxr-xr-x - bamboo supergroup 0 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/info
> -rw-r--r-- 3 bamboo supergroup 39417817 2009-07-13 16:52 /user/bamboo/hbase/TestTable/954997373/oldlogfile.log
>
>
>