You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by sulabh choudhury <su...@gmail.com> on 2011/04/28 20:59:43 UTC

Configurations for running a M/R job remotely with HBase

I am trying to run a M/R job remotely on a HBAse table.

I have added the conf.set("fs.default.name","hdfs://10.0.0.3:60000") to the
code hence now it does go to the cluster there I see the error

WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000:
readAndProcess threw exception java.io.EOFException. Count of bytes read: 0

jjava.io.EOFException


	at java.io.DataInputStream.readFully(DataInputStream.java:180)
	at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
	at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
	at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
	at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:179)
	at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processHeader(HBaseServer.java:966)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:950)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

and at the client side I see

Exception in thread "main" java.io.IOException: Call to
/10.0.0.3:60000failed on local exception: java.io.IOException:
Connection reset by peer

at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)

at org.apache.hadoop.ipc.Client.call(Client.java:743)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)

at $Proxy4.getProtocolVersion(Unknown Source)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)

at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)

at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)

        etc



I am pretty sure I am missing configuration pieces to achieve the same..What
could be those ?

Re: A question about Metrics

Posted by Gaojinchao <ga...@huawei.com>.
Make a issue https://issues.apache.org/jira/browse/HBASE-3848 and submit a patch.
Please review it. Thanks.


-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月29日 12:27
收件人: user@hbase.apache.org
主题: Re: A question about Metrics

It looks like a bug.  I see same over here.  Please file an issue.  If
you have a patch too, that'd be great.
St.Ack

On Thu, Apr 28, 2011 at 8:55 PM, Gaojinchao <ga...@huawei.com> wrote:
> request always is zero in WebUI for region server
>
> Metrics request=0.0, regions=36, stores=36, storefiles=148, storefileIndexSize=29, memstoreSize=253, compactionQueueSize=24, flushQueueSize=0, usedHeap=655, maxHeap=8175, blockCacheSize=14230920, blockCacheFree=1700269560, blockCacheCount=21, blockCacheHitCount=2887, blockCacheMissCount=204829, blockCacheEvictedCount=0, blockCacheHitRatio=1, blockCacheHitCachingRatio=99
>
> requests is not zero in WebUI for Hmaster
> requests=15000, regions=35, usedHeap=513, maxHeap=8175
>
> Is there any different for these metrics?
> How do I use it?
> Thanks.
>
>

Re: A question about Metrics

Posted by Stack <st...@duboce.net>.
It looks like a bug.  I see same over here.  Please file an issue.  If
you have a patch too, that'd be great.
St.Ack

On Thu, Apr 28, 2011 at 8:55 PM, Gaojinchao <ga...@huawei.com> wrote:
> request always is zero in WebUI for region server
>
> Metrics request=0.0, regions=36, stores=36, storefiles=148, storefileIndexSize=29, memstoreSize=253, compactionQueueSize=24, flushQueueSize=0, usedHeap=655, maxHeap=8175, blockCacheSize=14230920, blockCacheFree=1700269560, blockCacheCount=21, blockCacheHitCount=2887, blockCacheMissCount=204829, blockCacheEvictedCount=0, blockCacheHitRatio=1, blockCacheHitCachingRatio=99
>
> requests is not zero in WebUI for Hmaster
> requests=15000, regions=35, usedHeap=513, maxHeap=8175
>
> Is there any different for these metrics?
> How do I use it?
> Thanks.
>
>

A question about Metrics

Posted by Gaojinchao <ga...@huawei.com>.
request always is zero in WebUI for region server

Metrics request=0.0, regions=36, stores=36, storefiles=148, storefileIndexSize=29, memstoreSize=253, compactionQueueSize=24, flushQueueSize=0, usedHeap=655, maxHeap=8175, blockCacheSize=14230920, blockCacheFree=1700269560, blockCacheCount=21, blockCacheHitCount=2887, blockCacheMissCount=204829, blockCacheEvictedCount=0, blockCacheHitRatio=1, blockCacheHitCachingRatio=99

requests is not zero in WebUI for Hmaster 
requests=15000, regions=35, usedHeap=513, maxHeap=8175

Is there any different for these metrics?
How do I use it? 
Thanks.

Re: HRegionInfo was null or empty in -ROOT-

Posted by bijieshan <bi...@huawei.com>.
Thanks Stack.
Sorry for replying you so late. Because I just have several days of rest.
I will notice that I should not attach the logs directly. And I hope I can do the test according to your suggestion immediately. What I'm not sure is whether the env is still kept there :( 

This cluster of hbase is 0.90.1.

Thanks,
Jieshan Bean

-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月30日 0:05
收件人: user@hbase.apache.org
主题: Re: HRegionInfo was null or empty in -ROOT-

Your logs did not come across. This list does not allow attachments
usually.  I would like you to delete all in the -ROOT- table and then
pastebin what is in your master log (Did you say what version of hbase
this?).

Thanks,
St.Ack


2011/4/29 bijieshan <bi...@huawei.com>:
> I could't see the two regions in the UI.
> After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META. table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
> Before the exception occurred, someone else deleted some log files:
>
> /hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
> /hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
> /hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
> /hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
> /hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052
>
> I don't know whether these operations are the direct causation.
>
> 【Error info from the page of master.jsp 】
>
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> 【Error info from hbase shell】
> hbase(main):001:0> list
> TABLE
> 11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
>
>
> hbase(main):002:0>
> hbase(main):003:0* scan '-ROOT-'
> ROW                                           COLUMN+CELL
>  .META.,,1                                    column=info:server, timestamp=1304062076821, value=Linux-2:60020
>  .META.,,1                                    column=info:serverstartcode, timestamp=1304062076821, value=1304060518330
> 1 row(s) in 0.0830 seconds
>
> hbase(main):004:0> scan '.META.'
> ROW                                           COLUMN+CELL
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}
>
>
> All the logs I have attached in the email.
>
> Thanks!
>
> Jieshan Bean
>
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 12:10
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Can you see your two regions in the UI?
>
> What if you restart hbase?  Does this fix the issue?
>
> Going by your master log, it seems happy enough, almost as though it
> were a different cluser altogether.
>
> You could try scanning '.META.' to see if you got your two regions back.
>
> You could try restarting the regionserver that is currently hosting .META.?
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks Stack!
>>
>> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>>
>> Hbase > delateall '-ROOT-','.META.,,1'
>>
>> After that, I scan the -ROOT again , the records was indeed deleted.
>>
>> Hbase > scan '-ROOT-'
>> 0 row(s) in 0.0210 seconds.
>>
>> And then, I unassign the META region:
>>
>> Hbase > unassign ".META.,,1"
>>
>> After a long while, I executed list:
>>
>> hbase(main):002:0> list
>> TABLE
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>>
>> Here is some help for this command:
>> List all tables in hbase. Optional regular expression parameter could
>> be used to filter the output. Examples:
>>
>>  hbase> list
>>  hbase> list 'abc.*'
>>
>> In HMaster logs, there's no effect happened:
>>
>> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>>
>> Any problem exist in my operations?
>>
>> Jieshan Bean
>>
>>
>> -----邮件原件-----
>> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
>> 发送时间: 2011年4月29日 11:41
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> In general, its usually best to let hbase manage all under the /hbase
>> dir in hdfs.
>>
>> What happens if you scan the -ROOT- table in shell?
>>
>> hbase> scan "-ROOT-"
>>
>> Is it empty there?
>>
>> If so, thats not the real issue.  The real issue is why .META. is not
>> being deployed.
>>
>> Try removing the entry in -ROOT-
>>
>> hbase> deleteall ".META.,,1"
>>
>> Then, try doing something like unassigning .META.
>>
>> hbase> unassign .META.
>>
>> Tail the master log.  See if the above has any effect.
>>
>> St.Ack
>>
>> 2011/4/28 bijieshan <bi...@huawei.com>:
>>> Thanks J-D.
>>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>>> Some one deleted some hdfs files, use the following method:
>>>
>>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>>
>>> I don't know whether these operations can bring any effects.
>>>
>>> You can check the log files from the attachments.
>>>
>>> Jieshan Bean
>>>
>>> -----邮件原件-----
>>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>>> 发送时间: 2011年4月29日 10:53
>>> 收件人: user@hbase.apache.org
>>> 抄送: Chenjian
>>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>>
>>> Last time I heard about this issue, it was because of a DNS problem.
>>> Look at your master and region server logs.
>>>
>>> J-D
>>>
>>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>>> Hi,
>>>>
>>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>>
>>>> Here's the info from the 'list' and 'hbck' tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> hbase(main):001:0> list
>>>> TABLE
>>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>>
>>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>>> Exceptions:
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> HBase hbck tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> --------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Thanks,
>>>>
>>>> Regards,
>>>> Jieshan bean
>>>>
>>>>
>>>
>>
>

Re: HRegionInfo was null or empty in -ROOT-

Posted by Stack <st...@duboce.net>.
Your logs did not come across. This list does not allow attachments
usually.  I would like you to delete all in the -ROOT- table and then
pastebin what is in your master log (Did you say what version of hbase
this?).

Thanks,
St.Ack


2011/4/29 bijieshan <bi...@huawei.com>:
> I could't see the two regions in the UI.
> After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META. table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
> Before the exception occurred, someone else deleted some log files:
>
> /hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
> /hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
> /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
> /hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
> /hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
> /hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
> /hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
> /hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052
>
> I don't know whether these operations are the direct causation.
>
> 【Error info from the page of master.jsp 】
>
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> 【Error info from hbase shell】
> hbase(main):001:0> list
> TABLE
> 11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
>
>
> hbase(main):002:0>
> hbase(main):003:0* scan '-ROOT-'
> ROW                                           COLUMN+CELL
>  .META.,,1                                    column=info:server, timestamp=1304062076821, value=Linux-2:60020
>  .META.,,1                                    column=info:serverstartcode, timestamp=1304062076821, value=1304060518330
> 1 row(s) in 0.0830 seconds
>
> hbase(main):004:0> scan '.META.'
> ROW                                           COLUMN+CELL
>
> ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}
>
>
> All the logs I have attached in the email.
>
> Thanks!
>
> Jieshan Bean
>
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 12:10
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Can you see your two regions in the UI?
>
> What if you restart hbase?  Does this fix the issue?
>
> Going by your master log, it seems happy enough, almost as though it
> were a different cluser altogether.
>
> You could try scanning '.META.' to see if you got your two regions back.
>
> You could try restarting the regionserver that is currently hosting .META.?
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks Stack!
>>
>> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>>
>> Hbase > delateall '-ROOT-','.META.,,1'
>>
>> After that, I scan the -ROOT again , the records was indeed deleted.
>>
>> Hbase > scan '-ROOT-'
>> 0 row(s) in 0.0210 seconds.
>>
>> And then, I unassign the META region:
>>
>> Hbase > unassign ".META.,,1"
>>
>> After a long while, I executed list:
>>
>> hbase(main):002:0> list
>> TABLE
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>> org.apache.hadoop.hbase.TableNotFoundException: .META.
>>
>> Here is some help for this command:
>> List all tables in hbase. Optional regular expression parameter could
>> be used to filter the output. Examples:
>>
>>  hbase> list
>>  hbase> list 'abc.*'
>>
>> In HMaster logs, there's no effect happened:
>>
>> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>>
>> Any problem exist in my operations?
>>
>> Jieshan Bean
>>
>>
>> -----邮件原件-----
>> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
>> 发送时间: 2011年4月29日 11:41
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> In general, its usually best to let hbase manage all under the /hbase
>> dir in hdfs.
>>
>> What happens if you scan the -ROOT- table in shell?
>>
>> hbase> scan "-ROOT-"
>>
>> Is it empty there?
>>
>> If so, thats not the real issue.  The real issue is why .META. is not
>> being deployed.
>>
>> Try removing the entry in -ROOT-
>>
>> hbase> deleteall ".META.,,1"
>>
>> Then, try doing something like unassigning .META.
>>
>> hbase> unassign .META.
>>
>> Tail the master log.  See if the above has any effect.
>>
>> St.Ack
>>
>> 2011/4/28 bijieshan <bi...@huawei.com>:
>>> Thanks J-D.
>>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>>> Some one deleted some hdfs files, use the following method:
>>>
>>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>>
>>> I don't know whether these operations can bring any effects.
>>>
>>> You can check the log files from the attachments.
>>>
>>> Jieshan Bean
>>>
>>> -----邮件原件-----
>>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>>> 发送时间: 2011年4月29日 10:53
>>> 收件人: user@hbase.apache.org
>>> 抄送: Chenjian
>>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>>
>>> Last time I heard about this issue, it was because of a DNS problem.
>>> Look at your master and region server logs.
>>>
>>> J-D
>>>
>>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>>> Hi,
>>>>
>>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>>
>>>> Here's the info from the 'list' and 'hbck' tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> hbase(main):001:0> list
>>>> TABLE
>>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>>
>>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>>> Exceptions:
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> HBase hbck tool:
>>>> --------------------------------------------------------------------------------------------------------------------
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>>> --------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Thanks,
>>>>
>>>> Regards,
>>>> Jieshan bean
>>>>
>>>>
>>>
>>
>

Re: HRegionInfo was null or empty in -ROOT-

Posted by bijieshan <bi...@huawei.com>.
I could't see the two regions in the UI.
After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META. table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
Before the exception occurred, someone else deleted some log files:

/hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
/hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
/hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
/hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
/hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
/hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
/hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
/hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
/hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
/hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
/hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
/hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052

I don't know whether these operations are the direct causation.

【Error info from the page of master.jsp 】

Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts.
Exceptions:
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.

【Error info from hbase shell】
hbase(main):001:0> list
TABLE
11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575

ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}


hbase(main):002:0>
hbase(main):003:0* scan '-ROOT-'
ROW                                           COLUMN+CELL
 .META.,,1                                    column=info:server, timestamp=1304062076821, value=Linux-2:60020
 .META.,,1                                    column=info:serverstartcode, timestamp=1304062076821, value=1304060518330
1 row(s) in 0.0830 seconds

hbase(main):004:0> scan '.META.'
ROW                                           COLUMN+CELL

ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}


All the logs I have attached in the email.

Thanks!

Jieshan Bean



-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月29日 12:10
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: HRegionInfo was null or empty in -ROOT-

Can you see your two regions in the UI?

What if you restart hbase?  Does this fix the issue?

Going by your master log, it seems happy enough, almost as though it
were a different cluser altogether.

You could try scanning '.META.' to see if you got your two regions back.

You could try restarting the regionserver that is currently hosting .META.?

St.Ack

2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks Stack!
>
> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>
> Hbase > delateall '-ROOT-','.META.,,1'
>
> After that, I scan the -ROOT again , the records was indeed deleted.
>
> Hbase > scan '-ROOT-'
> 0 row(s) in 0.0210 seconds.
>
> And then, I unassign the META region:
>
> Hbase > unassign ".META.,,1"
>
> After a long while, I executed list:
>
> hbase(main):002:0> list
> TABLE
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> Here is some help for this command:
> List all tables in hbase. Optional regular expression parameter could
> be used to filter the output. Examples:
>
>  hbase> list
>  hbase> list 'abc.*'
>
> In HMaster logs, there's no effect happened:
>
> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>
> Any problem exist in my operations?
>
> Jieshan Bean
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 11:41
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> In general, its usually best to let hbase manage all under the /hbase
> dir in hdfs.
>
> What happens if you scan the -ROOT- table in shell?
>
> hbase> scan "-ROOT-"
>
> Is it empty there?
>
> If so, thats not the real issue.  The real issue is why .META. is not
> being deployed.
>
> Try removing the entry in -ROOT-
>
> hbase> deleteall ".META.,,1"
>
> Then, try doing something like unassigning .META.
>
> hbase> unassign .META.
>
> Tail the master log.  See if the above has any effect.
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks J-D.
>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>> Some one deleted some hdfs files, use the following method:
>>
>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>
>> I don't know whether these operations can bring any effects.
>>
>> You can check the log files from the attachments.
>>
>> Jieshan Bean
>>
>> -----邮件原件-----
>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>> 发送时间: 2011年4月29日 10:53
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> Last time I heard about this issue, it was because of a DNS problem.
>> Look at your master and region server logs.
>>
>> J-D
>>
>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>> Hi,
>>>
>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>
>>> Here's the info from the 'list' and 'hbck' tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> hbase(main):001:0> list
>>> TABLE
>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>
>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>> Exceptions:
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> --------------------------------------------------------------------------------------------------------------------
>>> HBase hbck tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> --------------------------------------------------------------------------------------------------------------------
>>>
>>> Thanks,
>>>
>>> Regards,
>>> Jieshan bean
>>>
>>>
>>
>

Re: HRegionInfo was null or empty in -ROOT-

Posted by Stack <st...@duboce.net>.
Can you see your two regions in the UI?

What if you restart hbase?  Does this fix the issue?

Going by your master log, it seems happy enough, almost as though it
were a different cluser altogether.

You could try scanning '.META.' to see if you got your two regions back.

You could try restarting the regionserver that is currently hosting .META.?

St.Ack

2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks Stack!
>
> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:
>
> Hbase > delateall '-ROOT-','.META.,,1'
>
> After that, I scan the -ROOT again , the records was indeed deleted.
>
> Hbase > scan '-ROOT-'
> 0 row(s) in 0.0210 seconds.
>
> And then, I unassign the META region:
>
> Hbase > unassign ".META.,,1"
>
> After a long while, I executed list:
>
> hbase(main):002:0> list
> TABLE
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> Here is some help for this command:
> List all tables in hbase. Optional regular expression parameter could
> be used to filter the output. Examples:
>
>  hbase> list
>  hbase> list 'abc.*'
>
> In HMaster logs, there's no effect happened:
>
> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>
> Any problem exist in my operations?
>
> Jieshan Bean
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 11:41
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> In general, its usually best to let hbase manage all under the /hbase
> dir in hdfs.
>
> What happens if you scan the -ROOT- table in shell?
>
> hbase> scan "-ROOT-"
>
> Is it empty there?
>
> If so, thats not the real issue.  The real issue is why .META. is not
> being deployed.
>
> Try removing the entry in -ROOT-
>
> hbase> deleteall ".META.,,1"
>
> Then, try doing something like unassigning .META.
>
> hbase> unassign .META.
>
> Tail the master log.  See if the above has any effect.
>
> St.Ack
>
> 2011/4/28 bijieshan <bi...@huawei.com>:
>> Thanks J-D.
>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>> Some one deleted some hdfs files, use the following method:
>>
>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>
>> I don't know whether these operations can bring any effects.
>>
>> You can check the log files from the attachments.
>>
>> Jieshan Bean
>>
>> -----邮件原件-----
>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>> 发送时间: 2011年4月29日 10:53
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> Last time I heard about this issue, it was because of a DNS problem.
>> Look at your master and region server logs.
>>
>> J-D
>>
>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>>> Hi,
>>>
>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>
>>> Here's the info from the 'list' and 'hbck' tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> hbase(main):001:0> list
>>> TABLE
>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>>
>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>>> Exceptions:
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> --------------------------------------------------------------------------------------------------------------------
>>> HBase hbck tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>>> --------------------------------------------------------------------------------------------------------------------
>>>
>>> Thanks,
>>>
>>> Regards,
>>> Jieshan bean
>>>
>>>
>>
>

Re: HRegionInfo was null or empty in -ROOT-

Posted by bijieshan <bi...@huawei.com>.
Thanks Stack!

When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using as you told:

Hbase > delateall '-ROOT-','.META.,,1'

After that, I scan the -ROOT again , the records was indeed deleted.

Hbase > scan '-ROOT-'
0 row(s) in 0.0210 seconds.

And then, I unassign the META region:

Hbase > unassign ".META.,,1"

After a long while, I executed list:

hbase(main):002:0> list
TABLE

ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
Exceptions:
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.

Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

  hbase> list
  hbase> list 'abc.*'

In HMaster logs, there's no effect happened:

2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1

Any problem exist in my operations? 

Jieshan Bean


-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月29日 11:41
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: HRegionInfo was null or empty in -ROOT-

In general, its usually best to let hbase manage all under the /hbase
dir in hdfs.

What happens if you scan the -ROOT- table in shell?

hbase> scan "-ROOT-"

Is it empty there?

If so, thats not the real issue.  The real issue is why .META. is not
being deployed.

Try removing the entry in -ROOT-

hbase> deleteall ".META.,,1"

Then, try doing something like unassigning .META.

hbase> unassign .META.

Tail the master log.  See if the above has any effect.

St.Ack

2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks J-D.
> I think it's not this problem. I have checked the logs, there's no DNS exception.
> Some one deleted some hdfs files, use the following method:
>
> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>
> I don't know whether these operations can bring any effects.
>
> You can check the log files from the attachments.
>
> Jieshan Bean
>
> -----邮件原件-----
> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
> 发送时间: 2011年4月29日 10:53
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Last time I heard about this issue, it was because of a DNS problem.
> Look at your master and region server logs.
>
> J-D
>
> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>> Hi,
>>
>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>
>> Here's the info from the 'list' and 'hbck' tool:
>> --------------------------------------------------------------------------------------------------------------------
>> hbase(main):001:0> list
>> TABLE
>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> --------------------------------------------------------------------------------------------------------------------
>> HBase hbck tool:
>> --------------------------------------------------------------------------------------------------------------------
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> --------------------------------------------------------------------------------------------------------------------
>>
>> Thanks,
>>
>> Regards,
>> Jieshan bean
>>
>>
>

Re: HRegionInfo was null or empty in -ROOT-

Posted by Stack <st...@duboce.net>.
In general, its usually best to let hbase manage all under the /hbase
dir in hdfs.

What happens if you scan the -ROOT- table in shell?

hbase> scan "-ROOT-"

Is it empty there?

If so, thats not the real issue.  The real issue is why .META. is not
being deployed.

Try removing the entry in -ROOT-

hbase> deleteall ".META.,,1"

Then, try doing something like unassigning .META.

hbase> unassign .META.

Tail the master log.  See if the above has any effect.

St.Ack

2011/4/28 bijieshan <bi...@huawei.com>:
> Thanks J-D.
> I think it's not this problem. I have checked the logs, there's no DNS exception.
> Some one deleted some hdfs files, use the following method:
>
> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>
> I don't know whether these operations can bring any effects.
>
> You can check the log files from the attachments.
>
> Jieshan Bean
>
> -----邮件原件-----
> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
> 发送时间: 2011年4月29日 10:53
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> Last time I heard about this issue, it was because of a DNS problem.
> Look at your master and region server logs.
>
> J-D
>
> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
>> Hi,
>>
>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>
>> Here's the info from the 'list' and 'hbck' tool:
>> --------------------------------------------------------------------------------------------------------------------
>> hbase(main):001:0> list
>> TABLE
>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>>
>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
>> Exceptions:
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> --------------------------------------------------------------------------------------------------------------------
>> HBase hbck tool:
>> --------------------------------------------------------------------------------------------------------------------
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
>> --------------------------------------------------------------------------------------------------------------------
>>
>> Thanks,
>>
>> Regards,
>> Jieshan bean
>>
>>
>

Re: HRegionInfo was null or empty in -ROOT-

Posted by bijieshan <bi...@huawei.com>.
Thanks J-D.
I think it's not this problem. I have checked the logs, there's no DNS exception. 
Some one deleted some hdfs files, use the following method:

./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239

I don't know whether these operations can bring any effects.

You can check the log files from the attachments.

Jieshan Bean

-----邮件原件-----
发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
发送时间: 2011年4月29日 10:53
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: HRegionInfo was null or empty in -ROOT-

Last time I heard about this issue, it was because of a DNS problem.
Look at your master and region server logs.

J-D

On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
> Hi,
>
> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>
> Here's the info from the 'list' and 'hbck' tool:
> --------------------------------------------------------------------------------------------------------------------
> hbase(main):001:0> list
> TABLE
> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> --------------------------------------------------------------------------------------------------------------------
> HBase hbck tool:
> --------------------------------------------------------------------------------------------------------------------
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> --------------------------------------------------------------------------------------------------------------------
>
> Thanks,
>
> Regards,
> Jieshan bean
>
>

Re: HRegionInfo was null or empty in -ROOT-

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Last time I heard about this issue, it was because of a DNS problem.
Look at your master and region server logs.

J-D

On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bi...@huawei.com> wrote:
> Hi,
>
> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>
> Here's the info from the 'list' and 'hbck' tool:
> --------------------------------------------------------------------------------------------------------------------
> hbase(main):001:0> list
> TABLE
> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> --------------------------------------------------------------------------------------------------------------------
> HBase hbck tool:
> --------------------------------------------------------------------------------------------------------------------
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
> --------------------------------------------------------------------------------------------------------------------
>
> Thanks,
>
> Regards,
> Jieshan bean
>
>

HRegionInfo was null or empty in -ROOT-

Posted by bijieshan <bi...@huawei.com>.
Hi,

When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
Did the -ROOT- table value got corrupted or any other reasons cause this problem?

Here's the info from the 'list' and 'hbck' tool:
--------------------------------------------------------------------------------------------------------------------
hbase(main):001:0> list
TABLE
11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575

ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
--------------------------------------------------------------------------------------------------------------------
HBase hbck tool:
--------------------------------------------------------------------------------------------------------------------
11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
--------------------------------------------------------------------------------------------------------------------

Thanks,

Regards,
Jieshan bean

Re: Configurations for running a M/R job remotely with HBase

Posted by sulabh choudhury <su...@gmail.com>.
Cool, that resolved the issue.

On Thu, Apr 28, 2011 at 4:32 PM, Stack <st...@duboce.net> wrote:

> You need to fix "Could not resolve the DNS name of db2.dev.abc.net:60020"
> St.Ack
>
> On Thu, Apr 28, 2011 at 3:46 PM, sulabh choudhury <su...@gmail.com>
> wrote:
> > Thanks for the response.
> > Harsh..I already looked into that document, but for my use-case I need it
> to
> > connect to HBase and should be a HMaster or RegionServer port
> >
> > Bennett...
> > Supplying the "hbase.zookeeper.quorum" it throws an error
> > 11/04/28 14:06:18 ERROR hbase.HServerAddress: Could not resolve the DNS
> name
> > of db2.dev.abc.net:60020
> > 11/04/28 14:06:18 ERROR mapreduce.TableInputFormat:
> > java.lang.IllegalArgumentException: Could not resolve the DNS name of
> > db2.dev.abc.net:60020
> >  at
> >
> org.apache.hadoop.hbase.HServerAddress.checkBindAddressCanBeResolved(HServerAddress.java:105)
> >  at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:66)
> >  at
> >
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82)
> >  at
> >
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:578)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:687)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:589)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:687)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:593)
> >  at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
> >  at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> >  at
> >
> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91)
> >  at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> >  at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> >  at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:882)
> > at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> >  at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
> >  at abc.main(abc.java:194)
> >
> > Exception in thread "main" java.io.IOException: No table was provided.
> > at
> >
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:130)
> >  at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> >  at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
> >  at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
> > at abc.main(abc.java:194)
> >
> >
> > On Thu, Apr 28, 2011 at 12:29 PM, Bennett Andrews <
> > bennett.j.andrews@gmail.com> wrote:
> >
> >> For HBase TableMR you should just need "hbase.zookeeper.quorum" in
> addition
> >> to TaskTracker stuff.
> >>
> >> On Thu, Apr 28, 2011 at 3:15 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> > I believe 60000 is your HBase HMaster IPC port, not the NameNode one.
> >> > What are you attempting to do here?
> >> >
> >> > See this package's docs on proper docs for running regular MR jobs
> >> > using HBase as Sink/Source:
> >> >
> >> >
> >>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html
> >> >
> >> > On Fri, Apr 29, 2011 at 12:29 AM, sulabh choudhury <sulabhc@gmail.com
> >
> >> > wrote:
> >> > > I am trying to run a M/R job remotely on a HBAse table.
> >> > >
> >> > > I have added the conf.set("fs.default.name","hdfs://10.0.0.3:60000
> ")
> >> to
> >> > the
> >> > > code hence now it does go to the cluster there I see the error
> >> > >
> >> > > WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on
> 60000:
> >> > > readAndProcess threw exception java.io.EOFException. Count of bytes
> >> read:
> >> > 0
> >> > >
> >> > > jjava.io.EOFException
> >> > >
> >> > >
> >> > >        at
> java.io.DataInputStream.readFully(DataInputStream.java:180)
> >> > >        at
> >> >
> >>
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
> >> > >        at
> >> > org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
> >> > >        at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
> >> > >        at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
> >> > >        at
> >> >
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:179)
> >> > >        at
> >> >
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
> >> > >        at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processHeader(HBaseServer.java:966)
> >> > >        at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:950)
> >> > >        at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> >> > >        at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> >> > >        at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> > >        at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> > >        at java.lang.Thread.run(Thread.java:662)
> >> > >
> >> > > and at the client side I see
> >> > >
> >> > > Exception in thread "main" java.io.IOException: Call to
> >> > > /10.0.0.3:60000failed on local exception: java.io.IOException:
> >> > > Connection reset by peer
> >> > >
> >> > > at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> >> > >
> >> > > at org.apache.hadoop.ipc.Client.call(Client.java:743)
> >> > >
> >> > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> >> > >
> >> > > at $Proxy4.getProtocolVersion(Unknown Source)
> >> > >
> >> > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> >> > >
> >> > > at
> >> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
> >> > >
> >> > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
> >> > >
> >> > >        etc
> >> > >
> >> > >
> >> > >
> >> > > I am pretty sure I am missing configuration pieces to achieve the
> >> > same..What
> >> > > could be those ?
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Harsh J
> >> >
> >>
> >
> >
> >
> > --
> >
> > --
> > Thanks and Regards,
> > Sulabh Choudhury
> >
>

Re: Configurations for running a M/R job remotely with HBase

Posted by Stack <st...@duboce.net>.
You need to fix "Could not resolve the DNS name of db2.dev.abc.net:60020"
St.Ack

On Thu, Apr 28, 2011 at 3:46 PM, sulabh choudhury <su...@gmail.com> wrote:
> Thanks for the response.
> Harsh..I already looked into that document, but for my use-case I need it to
> connect to HBase and should be a HMaster or RegionServer port
>
> Bennett...
> Supplying the "hbase.zookeeper.quorum" it throws an error
> 11/04/28 14:06:18 ERROR hbase.HServerAddress: Could not resolve the DNS name
> of db2.dev.abc.net:60020
> 11/04/28 14:06:18 ERROR mapreduce.TableInputFormat:
> java.lang.IllegalArgumentException: Could not resolve the DNS name of
> db2.dev.abc.net:60020
>  at
> org.apache.hadoop.hbase.HServerAddress.checkBindAddressCanBeResolved(HServerAddress.java:105)
>  at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:66)
>  at
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82)
>  at
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:578)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:687)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:589)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:687)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:593)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
>  at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
>  at
> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91)
>  at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>  at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>  at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:882)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
>  at abc.main(abc.java:194)
>
> Exception in thread "main" java.io.IOException: No table was provided.
> at
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:130)
>  at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>  at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
>  at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
> at abc.main(abc.java:194)
>
>
> On Thu, Apr 28, 2011 at 12:29 PM, Bennett Andrews <
> bennett.j.andrews@gmail.com> wrote:
>
>> For HBase TableMR you should just need "hbase.zookeeper.quorum" in addition
>> to TaskTracker stuff.
>>
>> On Thu, Apr 28, 2011 at 3:15 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> > I believe 60000 is your HBase HMaster IPC port, not the NameNode one.
>> > What are you attempting to do here?
>> >
>> > See this package's docs on proper docs for running regular MR jobs
>> > using HBase as Sink/Source:
>> >
>> >
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html
>> >
>> > On Fri, Apr 29, 2011 at 12:29 AM, sulabh choudhury <su...@gmail.com>
>> > wrote:
>> > > I am trying to run a M/R job remotely on a HBAse table.
>> > >
>> > > I have added the conf.set("fs.default.name","hdfs://10.0.0.3:60000")
>> to
>> > the
>> > > code hence now it does go to the cluster there I see the error
>> > >
>> > > WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000:
>> > > readAndProcess threw exception java.io.EOFException. Count of bytes
>> read:
>> > 0
>> > >
>> > > jjava.io.EOFException
>> > >
>> > >
>> > >        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> > >        at
>> >
>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
>> > >        at
>> > org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
>> > >        at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
>> > >        at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
>> > >        at
>> > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:179)
>> > >        at
>> > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
>> > >        at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processHeader(HBaseServer.java:966)
>> > >        at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:950)
>> > >        at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
>> > >        at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
>> > >        at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > >        at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > >        at java.lang.Thread.run(Thread.java:662)
>> > >
>> > > and at the client side I see
>> > >
>> > > Exception in thread "main" java.io.IOException: Call to
>> > > /10.0.0.3:60000failed on local exception: java.io.IOException:
>> > > Connection reset by peer
>> > >
>> > > at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
>> > >
>> > > at org.apache.hadoop.ipc.Client.call(Client.java:743)
>> > >
>> > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>> > >
>> > > at $Proxy4.getProtocolVersion(Unknown Source)
>> > >
>> > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
>> > >
>> > > at
>> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
>> > >
>> > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
>> > >
>> > >        etc
>> > >
>> > >
>> > >
>> > > I am pretty sure I am missing configuration pieces to achieve the
>> > same..What
>> > > could be those ?
>> > >
>> >
>> >
>> >
>> > --
>> > Harsh J
>> >
>>
>
>
>
> --
>
> --
> Thanks and Regards,
> Sulabh Choudhury
>

Re: Configurations for running a M/R job remotely with HBase

Posted by sulabh choudhury <su...@gmail.com>.
Thanks for the response.
Harsh..I already looked into that document, but for my use-case I need it to
connect to HBase and should be a HMaster or RegionServer port

Bennett...
Supplying the "hbase.zookeeper.quorum" it throws an error
11/04/28 14:06:18 ERROR hbase.HServerAddress: Could not resolve the DNS name
of db2.dev.abc.net:60020
11/04/28 14:06:18 ERROR mapreduce.TableInputFormat:
java.lang.IllegalArgumentException: Could not resolve the DNS name of
db2.dev.abc.net:60020
 at
org.apache.hadoop.hbase.HServerAddress.checkBindAddressCanBeResolved(HServerAddress.java:105)
 at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:66)
 at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82)
 at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:578)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:687)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:589)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:687)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:593)
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:558)
 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
 at
org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91)
 at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
 at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
 at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:882)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
 at abc.main(abc.java:194)

Exception in thread "main" java.io.IOException: No table was provided.
at
org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:130)
 at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
 at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
at abc.main(abc.java:194)


On Thu, Apr 28, 2011 at 12:29 PM, Bennett Andrews <
bennett.j.andrews@gmail.com> wrote:

> For HBase TableMR you should just need "hbase.zookeeper.quorum" in addition
> to TaskTracker stuff.
>
> On Thu, Apr 28, 2011 at 3:15 PM, Harsh J <ha...@cloudera.com> wrote:
>
> > I believe 60000 is your HBase HMaster IPC port, not the NameNode one.
> > What are you attempting to do here?
> >
> > See this package's docs on proper docs for running regular MR jobs
> > using HBase as Sink/Source:
> >
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html
> >
> > On Fri, Apr 29, 2011 at 12:29 AM, sulabh choudhury <su...@gmail.com>
> > wrote:
> > > I am trying to run a M/R job remotely on a HBAse table.
> > >
> > > I have added the conf.set("fs.default.name","hdfs://10.0.0.3:60000")
> to
> > the
> > > code hence now it does go to the cluster there I see the error
> > >
> > > WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000:
> > > readAndProcess threw exception java.io.EOFException. Count of bytes
> read:
> > 0
> > >
> > > jjava.io.EOFException
> > >
> > >
> > >        at java.io.DataInputStream.readFully(DataInputStream.java:180)
> > >        at
> >
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
> > >        at
> > org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
> > >        at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
> > >        at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
> > >        at
> > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:179)
> > >        at
> > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
> > >        at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processHeader(HBaseServer.java:966)
> > >        at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:950)
> > >        at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> > >        at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> > >        at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > >        at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > >        at java.lang.Thread.run(Thread.java:662)
> > >
> > > and at the client side I see
> > >
> > > Exception in thread "main" java.io.IOException: Call to
> > > /10.0.0.3:60000failed on local exception: java.io.IOException:
> > > Connection reset by peer
> > >
> > > at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> > >
> > > at org.apache.hadoop.ipc.Client.call(Client.java:743)
> > >
> > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > >
> > > at $Proxy4.getProtocolVersion(Unknown Source)
> > >
> > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> > >
> > > at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
> > >
> > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
> > >
> > >        etc
> > >
> > >
> > >
> > > I am pretty sure I am missing configuration pieces to achieve the
> > same..What
> > > could be those ?
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>



-- 

-- 
Thanks and Regards,
Sulabh Choudhury

Re: Configurations for running a M/R job remotely with HBase

Posted by Bennett Andrews <be...@gmail.com>.
For HBase TableMR you should just need "hbase.zookeeper.quorum" in addition
to TaskTracker stuff.

On Thu, Apr 28, 2011 at 3:15 PM, Harsh J <ha...@cloudera.com> wrote:

> I believe 60000 is your HBase HMaster IPC port, not the NameNode one.
> What are you attempting to do here?
>
> See this package's docs on proper docs for running regular MR jobs
> using HBase as Sink/Source:
>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html
>
> On Fri, Apr 29, 2011 at 12:29 AM, sulabh choudhury <su...@gmail.com>
> wrote:
> > I am trying to run a M/R job remotely on a HBAse table.
> >
> > I have added the conf.set("fs.default.name","hdfs://10.0.0.3:60000") to
> the
> > code hence now it does go to the cluster there I see the error
> >
> > WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000:
> > readAndProcess threw exception java.io.EOFException. Count of bytes read:
> 0
> >
> > jjava.io.EOFException
> >
> >
> >        at java.io.DataInputStream.readFully(DataInputStream.java:180)
> >        at
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
> >        at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
> >        at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
> >        at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
> >        at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:179)
> >        at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processHeader(HBaseServer.java:966)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:950)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> >        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >        at java.lang.Thread.run(Thread.java:662)
> >
> > and at the client side I see
> >
> > Exception in thread "main" java.io.IOException: Call to
> > /10.0.0.3:60000failed on local exception: java.io.IOException:
> > Connection reset by peer
> >
> > at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:743)
> >
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> >
> > at $Proxy4.getProtocolVersion(Unknown Source)
> >
> > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> >
> > at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
> >
> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
> >
> >        etc
> >
> >
> >
> > I am pretty sure I am missing configuration pieces to achieve the
> same..What
> > could be those ?
> >
>
>
>
> --
> Harsh J
>

Re: Configurations for running a M/R job remotely with HBase

Posted by Harsh J <ha...@cloudera.com>.
I believe 60000 is your HBase HMaster IPC port, not the NameNode one.
What are you attempting to do here?

See this package's docs on proper docs for running regular MR jobs
using HBase as Sink/Source:
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html

On Fri, Apr 29, 2011 at 12:29 AM, sulabh choudhury <su...@gmail.com> wrote:
> I am trying to run a M/R job remotely on a HBAse table.
>
> I have added the conf.set("fs.default.name","hdfs://10.0.0.3:60000") to the
> code hence now it does go to the cluster there I see the error
>
> WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000:
> readAndProcess threw exception java.io.EOFException. Count of bytes read: 0
>
> jjava.io.EOFException
>
>
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
>        at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
>        at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
>        at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
>        at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:179)
>        at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processHeader(HBaseServer.java:966)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:950)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
>
> and at the client side I see
>
> Exception in thread "main" java.io.IOException: Call to
> /10.0.0.3:60000failed on local exception: java.io.IOException:
> Connection reset by peer
>
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
>
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>
> at $Proxy4.getProtocolVersion(Unknown Source)
>
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
>
> at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
>
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
>
>        etc
>
>
>
> I am pretty sure I am missing configuration pieces to achieve the same..What
> could be those ?
>



-- 
Harsh J