You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by bijieshan <bi...@huawei.com> on 2011/04/29 04:49:56 UTC
HRegionInfo was null or empty in -ROOT-
Hi,
When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
Did the -ROOT- table value got corrupted or any other reasons cause this problem?
Here's the info from the 'list' and 'hbck' tool:
--------------------------------------------------------------------------------------------------------------------
hbase(main):001:0> list
TABLE
11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
--------------------------------------------------------------------------------------------------------------------
HBase hbck tool:
--------------------------------------------------------------------------------------------------------------------
11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
--------------------------------------------------------------------------------------------------------------------
Thanks,
Regards,
Jieshan bean
Re: A question about Metrics
Posted by Gaojinchao <ga...@huawei.com>.
Make a issue https://issues.apache.org/jira/browse/HBASE-3848 and submit a patch.
Please review it. Thanks.
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月29日 12:27
收件人: user@hbase.apache.org
主题: Re: A question about Metrics
It looks like a bug. I see same over here. Please file an issue. If
you have a patch too, that'd be great.
St.Ack
On Thu, Apr 28, 2011 at 8:55 PM, Gaojinchao <ga...@huawei.com> wrote:
> request always is zero in WebUI for region server
>
> Metrics request=0.0, regions=36, stores=36, storefiles=148, storefileIndexSize=29, memstoreSize=253, compactionQueueSize=24, flushQueueSize=0, usedHeap=655, maxHeap=8175, blockCacheSize=14230920, blockCacheFree=1700269560, blockCacheCount=21, blockCacheHitCount=2887, blockCacheMissCount=204829, blockCacheEvictedCount=0, blockCacheHitRatio=1, blockCacheHitCachingRatio=99
>
> requests is not zero in WebUI for Hmaster
> requests=15000, regions=35, usedHeap=513, maxHeap=8175
>
> Is there any different for these metrics?
> How do I use it?
> Thanks.
>
>
Re: A question about Metrics
Posted by Stack <st...@duboce.net>.
It looks like a bug. I see same over here. Please file an issue. If
you have a patch too, that'd be great.
St.Ack
On Thu, Apr 28, 2011 at 8:55 PM, Gaojinchao <ga...@huawei.com> wrote:
> request always is zero in WebUI for region server
>
> Metrics request=0.0, regions=36, stores=36, storefiles=148, storefileIndexSize=29, memstoreSize=253, compactionQueueSize=24, flushQueueSize=0, usedHeap=655, maxHeap=8175, blockCacheSize=14230920, blockCacheFree=1700269560, blockCacheCount=21, blockCacheHitCount=2887, blockCacheMissCount=204829, blockCacheEvictedCount=0, blockCacheHitRatio=1, blockCacheHitCachingRatio=99
>
> requests is not zero in WebUI for Hmaster
> requests=15000, regions=35, usedHeap=513, maxHeap=8175
>
> Is there any different for these metrics?
> How do I use it?
> Thanks.
>
>
A question about Metrics
Posted by Gaojinchao <ga...@huawei.com>.
request always is zero in WebUI for region server
Metrics request=0.0, regions=36, stores=36, storefiles=148, storefileIndexSize=29, memstoreSize=253, compactionQueueSize=24, flushQueueSize=0, usedHeap=655, maxHeap=8175, blockCacheSize=14230920, blockCacheFree=1700269560, blockCacheCount=21, blockCacheHitCount=2887, blockCacheMissCount=204829, blockCacheEvictedCount=0, blockCacheHitRatio=1, blockCacheHitCachingRatio=99
requests is not zero in WebUI for Hmaster
requests=15000, regions=35, usedHeap=513, maxHeap=8175
Is there any different for these metrics?
How do I use it?
Thanks.
Re: HRegionInfo was null or empty in -ROOT-
Posted by bijieshan <bi...@huawei.com>.
Thanks Stack.
Sorry for replying you so late. Because I just have several days of rest.
I will notice that I should not attach the logs directly. And I hope I can do the test according to your suggestion immediately. What I'm not sure is whether the env is still kept there :(
This cluster of hbase is 0.90.1.
Thanks,
Jieshan Bean
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月30日 0:05
收件人: user@hbase.apache.org
主题: Re: HRegionInfo was null or empty in -ROOT-
Your logs did not come across. This list does not allow attachments
usually. I would like you to delete all in the -ROOT- table and then
pastebin what is in your master log (Did you say what version of hbase
this?).
Thanks,
St.Ack
2011/4/29 bijieshan <bi...@huawei.com>:
> I could't see the two regions in the UI.
> After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META. table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
> Before the exception occurred, someone else deleted some log files:
>
> /hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
> /hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
> /hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
> /hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
> /hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052
>
> I don't know whether these operations are the direct causation.
>
> 【Error info from the page of master.jsp 】
>
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> 【Error info from hbase shell】
> hbase(main):001:0> list
> TABLE
> 11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
>
>
> hbase(main):002:0>
> hbase(main):003:0* scan '-ROOT-'
> ROW COLUMN+CELL
> .META.,,1 column=info:server, timestamp=1304062076821, value=Linux-2:60020
> .META.,,1 column=info:serverstartcode, timestamp=1304062076821, value=1304060518330
> 1 row(s) in 0.0830 seconds
>
> hbase(main):004:0> scan '.META.'
> ROW COLUMN+CELL
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}
>
>
> All the logs I have attached in the email.
>
> Thanks!
>
> Jieshan Bean
>
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 12:10
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Can you see your two regions in the UI?
>
> What if you restart hbase? Does this fix the issue?
>
> Going by your master log, it seems happy enough, almost as though it
> were a different cluser altogether.
>
> You could try scanning '.META.' to see if you got your two regions back.
>
> You could try restarting the regionserver that is currently hosting .META.?
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks Stack!
>>
>> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>>
>> Hbase > delateall '-ROOT-','.META.,,1'
>>
>> After that, I scan the -ROOT again , the records was indeed deleted.
>>
>> Hbase > scan '-ROOT-'
>> 0 row(s) in 0.0210 seconds.
>>
>> And then, I unassign the META region:
>>
>> Hbase > unassign ".META.,,1"
>>
>> After a long while, I executed list:
>>
>> hbase(main):002:0> list
>> TABLE
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>>
>> Here is some help for this command:
>> List all tables in hbase. Optional regular expression parameter could
>> be used to filter the output. Examples:
>>
>> hbase> list
>> hbase> list 'abc.*'
>>
>> In HMaster logs, there's no effect happened:
>>
>> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>>
>> Any problem exist in my operations?
>>
>> Jieshan Bean
>>
>>
>> -----邮件原件-----
>> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
>> 发送时间: 2011年4月29日 11:41
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> In general, its usually best to let hbase manage all under the /hbase
>> dir in hdfs.
>>
>> What happens if you scan the -ROOT- table in shell?
>>
>> hbase> scan "-ROOT-"
>>
>> Is it empty there?
>>
>> If so, thats not the real issue. The real issue is why .META. is not
>> being deployed.
>>
>> Try removing the entry in -ROOT-
>>
>> hbase> deleteall ".META.,,1"
>>
>> Then, try doing something like unassigning .META.
>>
>> hbase> unassign .META.
>>
>> Tail the master log. See if the above has any effect.
>>
>> St.Ack
>>
>> 2011/4/28 bijieshan <bi...@huawei.com>:
>>> Thanks J-D.
>>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>>> Some one deleted some hdfs files, use the following method:
>>>
>>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>>
>>> I don't know whether these operations can bring any effects.
>>>
>>> You can check the log files from the attachments.
>>>
>>> Jieshan Bean
>>>
>>> -----邮件原件-----
>>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>>> 发送时间: 2011年4月29日 10:53
>>> 收件人: user@hbase.apache.org
>>> 抄送: Chenjian
>>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>>
>>> Last time I heard about this issue, it was because of a DNS problem.
>>> Look at your master and region server logs.
>>>
>>> J-D
>>>
>>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>>> Hi,
>>>>
>>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>>
>>>> Here's the info from the 'list' and 'hbck' tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> hbase(main):001:0> list
>>>> TABLE
>>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>>
>>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>>> Exceptions:
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> HBase hbck tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> --------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Thanks,
>>>>
>>>> Regards,
>>>> Jieshan bean
>>>>
>>>>
>>>
>>
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by Stack <st...@duboce.net>.
Your logs did not come across. This list does not allow attachments
usually. I would like you to delete all in the -ROOT- table and then
pastebin what is in your master log (Did you say what version of hbase
this?).
Thanks,
St.Ack
2011/4/29 bijieshan <bi...@huawei.com>:
> I could't see the two regions in the UI.
> After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META. table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
> Before the exception occurred, someone else deleted some log files:
>
> /hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
> /hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
> /hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
> /hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
> /hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052
>
> I don't know whether these operations are the direct causation.
>
> 【Error info from the page of master.jsp 】
>
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> 【Error info from hbase shell】
> hbase(main):001:0> list
> TABLE
> 11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
>
>
> hbase(main):002:0>
> hbase(main):003:0* scan '-ROOT-'
> ROW COLUMN+CELL
> .META.,,1 column=info:server, timestamp=1304062076821, value=Linux-2:60020
> .META.,,1 column=info:serverstartcode, timestamp=1304062076821, value=1304060518330
> 1 row(s) in 0.0830 seconds
>
> hbase(main):004:0> scan '.META.'
> ROW COLUMN+CELL
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}
>
>
> All the logs I have attached in the email.
>
> Thanks!
>
> Jieshan Bean
>
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 12:10
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Can you see your two regions in the UI?
>
> What if you restart hbase? Does this fix the issue?
>
> Going by your master log, it seems happy enough, almost as though it
> were a different cluser altogether.
>
> You could try scanning '.META.' to see if you got your two regions back.
>
> You could try restarting the regionserver that is currently hosting .META.?
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks Stack!
>>
>> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>>
>> Hbase > delateall '-ROOT-','.META.,,1'
>>
>> After that, I scan the -ROOT again , the records was indeed deleted.
>>
>> Hbase > scan '-ROOT-'
>> 0 row(s) in 0.0210 seconds.
>>
>> And then, I unassign the META region:
>>
>> Hbase > unassign ".META.,,1"
>>
>> After a long while, I executed list:
>>
>> hbase(main):002:0> list
>> TABLE
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>>
>> Here is some help for this command:
>> List all tables in hbase. Optional regular expression parameter could
>> be used to filter the output. Examples:
>>
>> hbase> list
>> hbase> list 'abc.*'
>>
>> In HMaster logs, there's no effect happened:
>>
>> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>>
>> Any problem exist in my operations?
>>
>> Jieshan Bean
>>
>>
>> -----邮件原件-----
>> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
>> 发送时间: 2011年4月29日 11:41
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> In general, its usually best to let hbase manage all under the /hbase
>> dir in hdfs.
>>
>> What happens if you scan the -ROOT- table in shell?
>>
>> hbase> scan "-ROOT-"
>>
>> Is it empty there?
>>
>> If so, thats not the real issue. The real issue is why .META. is not
>> being deployed.
>>
>> Try removing the entry in -ROOT-
>>
>> hbase> deleteall ".META.,,1"
>>
>> Then, try doing something like unassigning .META.
>>
>> hbase> unassign .META.
>>
>> Tail the master log. See if the above has any effect.
>>
>> St.Ack
>>
>> 2011/4/28 bijieshan <bi...@huawei.com>:
>>> Thanks J-D.
>>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>>> Some one deleted some hdfs files, use the following method:
>>>
>>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>>
>>> I don't know whether these operations can bring any effects.
>>>
>>> You can check the log files from the attachments.
>>>
>>> Jieshan Bean
>>>
>>> -----邮件原件-----
>>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>>> 发送时间: 2011年4月29日 10:53
>>> 收件人: user@hbase.apache.org
>>> 抄送: Chenjian
>>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>>
>>> Last time I heard about this issue, it was because of a DNS problem.
>>> Look at your master and region server logs.
>>>
>>> J-D
>>>
>>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>>> Hi,
>>>>
>>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>>
>>>> Here's the info from the 'list' and 'hbck' tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> hbase(main):001:0> list
>>>> TABLE
>>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>>
>>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>>> Exceptions:
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> HBase hbck tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> --------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Thanks,
>>>>
>>>> Regards,
>>>> Jieshan bean
>>>>
>>>>
>>>
>>
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by bijieshan <bi...@huawei.com>.
I could't see the two regions in the UI.
After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META. table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
Before the exception occurred, someone else deleted some log files:
/hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
/hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
/hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
/hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
/hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
/hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
/hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
/hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
/hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
/hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
/hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
/hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052
I don't know whether these operations are the direct causation.
【Error info from the page of master.jsp 】
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts.
Exceptions:
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
【Error info from hbase shell】
hbase(main):001:0> list
TABLE
11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
hbase(main):002:0>
hbase(main):003:0* scan '-ROOT-'
ROW COLUMN+CELL
.META.,,1 column=info:server, timestamp=1304062076821, value=Linux-2:60020
.META.,,1 column=info:serverstartcode, timestamp=1304062076821, value=1304060518330
1 row(s) in 0.0830 seconds
hbase(main):004:0> scan '.META.'
ROW COLUMN+CELL
ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}
All the logs I have attached in the email.
Thanks!
Jieshan Bean
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月29日 12:10
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: HRegionInfo was null or empty in -ROOT-
Can you see your two regions in the UI?
What if you restart hbase? Does this fix the issue?
Going by your master log, it seems happy enough, almost as though it
were a different cluser altogether.
You could try scanning '.META.' to see if you got your two regions back.
You could try restarting the regionserver that is currently hosting .META.?
St.Ack
2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks Stack!
>
> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>
> Hbase > delateall '-ROOT-','.META.,,1'
>
> After that, I scan the -ROOT again , the records was indeed deleted.
>
> Hbase > scan '-ROOT-'
> 0 row(s) in 0.0210 seconds.
>
> And then, I unassign the META region:
>
> Hbase > unassign ".META.,,1"
>
> After a long while, I executed list:
>
> hbase(main):002:0> list
> TABLE
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> Here is some help for this command:
> List all tables in hbase. Optional regular expression parameter could
> be used to filter the output. Examples:
>
> hbase> list
> hbase> list 'abc.*'
>
> In HMaster logs, there's no effect happened:
>
> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>
> Any problem exist in my operations?
>
> Jieshan Bean
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 11:41
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> In general, its usually best to let hbase manage all under the /hbase
> dir in hdfs.
>
> What happens if you scan the -ROOT- table in shell?
>
> hbase> scan "-ROOT-"
>
> Is it empty there?
>
> If so, thats not the real issue. The real issue is why .META. is not
> being deployed.
>
> Try removing the entry in -ROOT-
>
> hbase> deleteall ".META.,,1"
>
> Then, try doing something like unassigning .META.
>
> hbase> unassign .META.
>
> Tail the master log. See if the above has any effect.
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks J-D.
>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>> Some one deleted some hdfs files, use the following method:
>>
>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>
>> I don't know whether these operations can bring any effects.
>>
>> You can check the log files from the attachments.
>>
>> Jieshan Bean
>>
>> -----邮件原件-----
>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>> 发送时间: 2011年4月29日 10:53
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> Last time I heard about this issue, it was because of a DNS problem.
>> Look at your master and region server logs.
>>
>> J-D
>>
>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>> Hi,
>>>
>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>
>>> Here's the info from the 'list' and 'hbck' tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> hbase(main):001:0> list
>>> TABLE
>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>
>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>> Exceptions:
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> --------------------------------------------------------------------------------------------------------------------
>>> HBase hbck tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> --------------------------------------------------------------------------------------------------------------------
>>>
>>> Thanks,
>>>
>>> Regards,
>>> Jieshan bean
>>>
>>>
>>
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by Stack <st...@duboce.net>.
Can you see your two regions in the UI?
What if you restart hbase? Does this fix the issue?
Going by your master log, it seems happy enough, almost as though it
were a different cluser altogether.
You could try scanning '.META.' to see if you got your two regions back.
You could try restarting the regionserver that is currently hosting .META.?
St.Ack
2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks Stack!
>
> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>
> Hbase > delateall '-ROOT-','.META.,,1'
>
> After that, I scan the -ROOT again , the records was indeed deleted.
>
> Hbase > scan '-ROOT-'
> 0 row(s) in 0.0210 seconds.
>
> And then, I unassign the META region:
>
> Hbase > unassign ".META.,,1"
>
> After a long while, I executed list:
>
> hbase(main):002:0> list
> TABLE
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> Here is some help for this command:
> List all tables in hbase. Optional regular expression parameter could
> be used to filter the output. Examples:
>
> hbase> list
> hbase> list 'abc.*'
>
> In HMaster logs, there's no effect happened:
>
> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>
> Any problem exist in my operations?
>
> Jieshan Bean
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 11:41
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> In general, its usually best to let hbase manage all under the /hbase
> dir in hdfs.
>
> What happens if you scan the -ROOT- table in shell?
>
> hbase> scan "-ROOT-"
>
> Is it empty there?
>
> If so, thats not the real issue. The real issue is why .META. is not
> being deployed.
>
> Try removing the entry in -ROOT-
>
> hbase> deleteall ".META.,,1"
>
> Then, try doing something like unassigning .META.
>
> hbase> unassign .META.
>
> Tail the master log. See if the above has any effect.
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks J-D.
>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>> Some one deleted some hdfs files, use the following method:
>>
>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>
>> I don't know whether these operations can bring any effects.
>>
>> You can check the log files from the attachments.
>>
>> Jieshan Bean
>>
>> -----邮件原件-----
>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>> 发送时间: 2011年4月29日 10:53
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> Last time I heard about this issue, it was because of a DNS problem.
>> Look at your master and region server logs.
>>
>> J-D
>>
>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>> Hi,
>>>
>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>
>>> Here's the info from the 'list' and 'hbck' tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> hbase(main):001:0> list
>>> TABLE
>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>
>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>> Exceptions:
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> --------------------------------------------------------------------------------------------------------------------
>>> HBase hbck tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> --------------------------------------------------------------------------------------------------------------------
>>>
>>> Thanks,
>>>
>>> Regards,
>>> Jieshan bean
>>>
>>>
>>
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by bijieshan <bi...@huawei.com>.
Thanks Stack!
When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
Hbase > delateall '-ROOT-','.META.,,1'
After that, I scan the -ROOT again , the records was indeed deleted.
Hbase > scan '-ROOT-'
0 row(s) in 0.0210 seconds.
And then, I unassign the META region:
Hbase > unassign ".META.,,1"
After a long while, I executed list:
hbase(main):002:0> list
TABLE
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
Exceptions:
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:
hbase> list
hbase> list 'abc.*'
In HMaster logs, there's no effect happened:
2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
Any problem exist in my operations?
Jieshan Bean
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月29日 11:41
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: HRegionInfo was null or empty in -ROOT-
In general, its usually best to let hbase manage all under the /hbase
dir in hdfs.
What happens if you scan the -ROOT- table in shell?
hbase> scan "-ROOT-"
Is it empty there?
If so, thats not the real issue. The real issue is why .META. is not
being deployed.
Try removing the entry in -ROOT-
hbase> deleteall ".META.,,1"
Then, try doing something like unassigning .META.
hbase> unassign .META.
Tail the master log. See if the above has any effect.
St.Ack
2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks J-D.
> I think it's not this problem. I have checked the logs, there's no DNS exception.
> Some one deleted some hdfs files, use the following method:
>
> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>
> I don't know whether these operations can bring any effects.
>
> You can check the log files from the attachments.
>
> Jieshan Bean
>
> -----邮件原件-----
> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
> 发送时间: 2011年4月29日 10:53
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Last time I heard about this issue, it was because of a DNS problem.
> Look at your master and region server logs.
>
> J-D
>
> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>> Hi,
>>
>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>
>> Here's the info from the 'list' and 'hbck' tool:
>> --------------------------------------------------------------------------------------------------------------------
>> hbase(main):001:0> list
>> TABLE
>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> --------------------------------------------------------------------------------------------------------------------
>> HBase hbck tool:
>> --------------------------------------------------------------------------------------------------------------------
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> --------------------------------------------------------------------------------------------------------------------
>>
>> Thanks,
>>
>> Regards,
>> Jieshan bean
>>
>>
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by Stack <st...@duboce.net>.
In general, its usually best to let hbase manage all under the /hbase
dir in hdfs.
What happens if you scan the -ROOT- table in shell?
hbase> scan "-ROOT-"
Is it empty there?
If so, thats not the real issue. The real issue is why .META. is not
being deployed.
Try removing the entry in -ROOT-
hbase> deleteall ".META.,,1"
Then, try doing something like unassigning .META.
hbase> unassign .META.
Tail the master log. See if the above has any effect.
St.Ack
2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks J-D.
> I think it's not this problem. I have checked the logs, there's no DNS exception.
> Some one deleted some hdfs files, use the following method:
>
> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>
> I don't know whether these operations can bring any effects.
>
> You can check the log files from the attachments.
>
> Jieshan Bean
>
> -----邮件原件-----
> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
> 发送时间: 2011年4月29日 10:53
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Last time I heard about this issue, it was because of a DNS problem.
> Look at your master and region server logs.
>
> J-D
>
> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>> Hi,
>>
>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>
>> Here's the info from the 'list' and 'hbck' tool:
>> --------------------------------------------------------------------------------------------------------------------
>> hbase(main):001:0> list
>> TABLE
>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> --------------------------------------------------------------------------------------------------------------------
>> HBase hbck tool:
>> --------------------------------------------------------------------------------------------------------------------
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> --------------------------------------------------------------------------------------------------------------------
>>
>> Thanks,
>>
>> Regards,
>> Jieshan bean
>>
>>
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by bijieshan <bi...@huawei.com>.
Thanks J-D.
I think it's not this problem. I have checked the logs, there's no DNS exception.
Some one deleted some hdfs files, use the following method:
./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
I don't know whether these operations can bring any effects.
You can check the log files from the attachments.
Jieshan Bean
-----邮件原件-----
发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
发送时间: 2011年4月29日 10:53
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: HRegionInfo was null or empty in -ROOT-
Last time I heard about this issue, it was because of a DNS problem.
Look at your master and region server logs.
J-D
On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
> Hi,
>
> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>
> Here's the info from the 'list' and 'hbck' tool:
> --------------------------------------------------------------------------------------------------------------------
> hbase(main):001:0> list
> TABLE
> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> --------------------------------------------------------------------------------------------------------------------
> HBase hbck tool:
> --------------------------------------------------------------------------------------------------------------------
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> --------------------------------------------------------------------------------------------------------------------
>
> Thanks,
>
> Regards,
> Jieshan bean
>
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Last time I heard about this issue, it was because of a DNS problem.
Look at your master and region server logs.
J-D
On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
> Hi,
>
> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>
> Here's the info from the 'list' and 'hbck' tool:
> --------------------------------------------------------------------------------------------------------------------
> hbase(main):001:0> list
> TABLE
> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> --------------------------------------------------------------------------------------------------------------------
> HBase hbck tool:
> --------------------------------------------------------------------------------------------------------------------
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> --------------------------------------------------------------------------------------------------------------------
>
> Thanks,
>
> Regards,
> Jieshan bean
>
>