You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by bijieshan <bi...@huawei.com> on 2011/05/03 03:27:41 UTC
Re: HRegionInfo was null or empty in -ROOT-
Thanks Stack.
Sorry for replying you so late. Because I just have several days of rest.
I will notice that I should not attach the logs directly. And I hope I can do the test according to your suggestion immediately. What I'm not sure is whether the env is still kept there :(
This cluster of hbase is 0.90.1.
Thanks,
Jieshan Bean
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月30日 0:05
收件人: user@hbase.apache.org
主题: Re: HRegionInfo was null or empty in -ROOT-
Your logs did not come across. This list does not allow attachments
usually. I would like you to delete all in the -ROOT- table and then
pastebin what is in your master log (Did you say what version of hbase
this?).
Thanks,
St.Ack
2011/4/29 bijieshan <bi...@huawei.com>:
> I could't see the two regions in the UI.
> After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META. table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
> Before the exception occurred, someone else deleted some log files:
>
> /hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
> /hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
> /hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
> /hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
> /hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052
>
> I don't know whether these operations are the direct causation.
>
> 【Error info from the page of master.jsp 】
>
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> 【Error info from hbase shell】
> hbase(main):001:0> list
> TABLE
> 11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
>
>
> hbase(main):002:0>
> hbase(main):003:0* scan '-ROOT-'
> ROW COLUMN+CELL
> .META.,,1 column=info:server, timestamp=1304062076821, value=Linux-2:60020
> .META.,,1 column=info:serverstartcode, timestamp=1304062076821, value=1304060518330
> 1 row(s) in 0.0830 seconds
>
> hbase(main):004:0> scan '.META.'
> ROW COLUMN+CELL
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}
>
>
> All the logs I have attached in the email.
>
> Thanks!
>
> Jieshan Bean
>
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 12:10
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Can you see your two regions in the UI?
>
> What if you restart hbase? Does this fix the issue?
>
> Going by your master log, it seems happy enough, almost as though it
> were a different cluser altogether.
>
> You could try scanning '.META.' to see if you got your two regions back.
>
> You could try restarting the regionserver that is currently hosting .META.?
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks Stack!
>>
>> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>>
>> Hbase > delateall '-ROOT-','.META.,,1'
>>
>> After that, I scan the -ROOT again , the records was indeed deleted.
>>
>> Hbase > scan '-ROOT-'
>> 0 row(s) in 0.0210 seconds.
>>
>> And then, I unassign the META region:
>>
>> Hbase > unassign ".META.,,1"
>>
>> After a long while, I executed list:
>>
>> hbase(main):002:0> list
>> TABLE
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>>
>> Here is some help for this command:
>> List all tables in hbase. Optional regular expression parameter could
>> be used to filter the output. Examples:
>>
>> hbase> list
>> hbase> list 'abc.*'
>>
>> In HMaster logs, there's no effect happened:
>>
>> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>>
>> Any problem exist in my operations?
>>
>> Jieshan Bean
>>
>>
>> -----邮件原件-----
>> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
>> 发送时间: 2011年4月29日 11:41
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> In general, its usually best to let hbase manage all under the /hbase
>> dir in hdfs.
>>
>> What happens if you scan the -ROOT- table in shell?
>>
>> hbase> scan "-ROOT-"
>>
>> Is it empty there?
>>
>> If so, thats not the real issue. The real issue is why .META. is not
>> being deployed.
>>
>> Try removing the entry in -ROOT-
>>
>> hbase> deleteall ".META.,,1"
>>
>> Then, try doing something like unassigning .META.
>>
>> hbase> unassign .META.
>>
>> Tail the master log. See if the above has any effect.
>>
>> St.Ack
>>
>> 2011/4/28 bijieshan <bi...@huawei.com>:
>>> Thanks J-D.
>>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>>> Some one deleted some hdfs files, use the following method:
>>>
>>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>>
>>> I don't know whether these operations can bring any effects.
>>>
>>> You can check the log files from the attachments.
>>>
>>> Jieshan Bean
>>>
>>> -----邮件原件-----
>>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>>> 发送时间: 2011年4月29日 10:53
>>> 收件人: user@hbase.apache.org
>>> 抄送: Chenjian
>>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>>
>>> Last time I heard about this issue, it was because of a DNS problem.
>>> Look at your master and region server logs.
>>>
>>> J-D
>>>
>>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>>> Hi,
>>>>
>>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>>
>>>> Here's the info from the 'list' and 'hbck' tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> hbase(main):001:0> list
>>>> TABLE
>>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>>
>>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>>> Exceptions:
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> HBase hbck tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> --------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Thanks,
>>>>
>>>> Regards,
>>>> Jieshan bean
>>>>
>>>>
>>>
>>
>