You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Nichole Treadway <kn...@gmail.com> on 2011/03/30 16:21:07 UTC
HRegionInfo was null or empty in -ROOT-
Hi all,
This morning I was restarting my HBase cluster, and I'm finding that the
region servers start up fine, but master always fails with the following
error shown in the web ui:
Problem accessing /master.jsp. Reason:
Trying to contact region server null for region , row '', but
failed after 10 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
...
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server null for region , row '', but failed after 10
attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
...
Here is what I get when I scan -ROOT-:
hbase(main):001:0> scan '-ROOT-'
ROW COLUMN+CELL
.META.,,1 column=info:server,
timestamp=1301428760725, value=sspv2rndhadoop1.gce2000.com:60020
.META.,,1 column=info:serverstartcode,
timestamp=1301428760725, value=1301428748891
Any ideas what's going on / how I can fix this?
Cluster info: 4 nodes, hadoop-0.20-append, hbase-0.90.3, red hat 5.5,
64-bit, jdk1.6; 64 gb ram, 2.8ghz.
Thanks guys
Re: HRegionInfo was null or empty in -ROOT-
Posted by Nichole Treadway <kn...@gmail.com>.
In the master logs, I see messages like I've pasted below. I get the same
behavior even when I stop master and restart it. I see the root and meta
table regions listed in the region server ui's
2011-03-30 13:15:00,675 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address:
node1.abc123.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10
failed; retrying after sleep of 8000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
2011-03-30 13:15:00,675 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:08,679 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:08,681 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address:
node1.abc123.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10
failed; retrying after sleep of 16000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
2011-03-30 13:15:08,681 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:24,685 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:24,688 WARN org.mortbay.log: /master.jsp:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server null for region , row '', but failed after 10 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
...
2011-03-30 13:19:32,344 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-03-30 13:19:32,346 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=4 regions=2 average=0.5 mostloaded=1
leastloaded=1
On Wed, Mar 30, 2011 at 12:19 PM, Stack <st...@duboce.net> wrote:
> Can you check master log and see whats happening around -ROOT-? Why is
> it not being assigned out? What if you kill master process and
> restart it? Does -ROOT- get assigned then?
>
> St.Ack
>
> On Wed, Mar 30, 2011 at 7:21 AM, Nichole Treadway <kn...@gmail.com>
> wrote:
> > Hi all,
> >
> > This morning I was restarting my HBase cluster, and I'm finding that the
> > region servers start up fine, but master always fails with the following
> > error shown in the web ui:
> >
> > Problem accessing /master.jsp. Reason:
> >
> > Trying to contact region server null for region , row '', but
> > failed after 10 attempts.
> > Exceptions:
> > java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> > row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> > .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> > java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> > row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> > .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> > ...
> >
> > Caused by:
> >
> >
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> > contact region server null for region , row '', but failed after 10
> > attempts.
> > Exceptions:
> > java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> > row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> > .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> > java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> > row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> > .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> >
> >
> > ...
> >
> > Here is what I get when I scan -ROOT-:
> >
> >
> > hbase(main):001:0> scan '-ROOT-'
> > ROW COLUMN+CELL
> > .META.,,1 column=info:server,
> > timestamp=1301428760725, value=sspv2rndhadoop1.gce2000.com:60020
> > .META.,,1 column=info:serverstartcode,
> > timestamp=1301428760725, value=1301428748891
> >
> >
> >
> > Any ideas what's going on / how I can fix this?
> >
> >
> > Cluster info: 4 nodes, hadoop-0.20-append, hbase-0.90.3, red hat 5.5,
> > 64-bit, jdk1.6; 64 gb ram, 2.8ghz.
> >
> >
> > Thanks guys
> >
>
Re: HRegionInfo was null or empty in -ROOT-
Posted by Stack <st...@duboce.net>.
Can you check master log and see whats happening around -ROOT-? Why is
it not being assigned out? What if you kill master process and
restart it? Does -ROOT- get assigned then?
St.Ack
On Wed, Mar 30, 2011 at 7:21 AM, Nichole Treadway <kn...@gmail.com> wrote:
> Hi all,
>
> This morning I was restarting my HBase cluster, and I'm finding that the
> region servers start up fine, but master always fails with the following
> error shown in the web ui:
>
> Problem accessing /master.jsp. Reason:
>
> Trying to contact region server null for region , row '', but
> failed after 10 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> ...
>
> Caused by:
>
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact region server null for region , row '', but failed after 10
> attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
>
>
> ...
>
> Here is what I get when I scan -ROOT-:
>
>
> hbase(main):001:0> scan '-ROOT-'
> ROW COLUMN+CELL
> .META.,,1 column=info:server,
> timestamp=1301428760725, value=sspv2rndhadoop1.gce2000.com:60020
> .META.,,1 column=info:serverstartcode,
> timestamp=1301428760725, value=1301428748891
>
>
>
> Any ideas what's going on / how I can fix this?
>
>
> Cluster info: 4 nodes, hadoop-0.20-append, hbase-0.90.3, red hat 5.5,
> 64-bit, jdk1.6; 64 gb ram, 2.8ghz.
>
>
> Thanks guys
>