You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vimal Jain <vk...@gmail.com> on 2014/03/20 12:19:14 UTC
Getting exceptions in fresh installment of hbase
Hi,
I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo distributed
mode and i am getting below exceptions when starting hbase for first time.
Master logs :-
1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/backup-m
asters/hbase.rummycircle.com,60000,1395314029900 already deleted, and this
is not a retry
2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
-ROOT
-,,0 at address=hbase.rummycircle.com,60020,1395313673569;
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.N
otServingRegionException: Region is not online: -ROOT-,,0
3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
.META
.,,1 at address=hbase.rummycircle.com,60020,1395313673569;
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.N
otServingRegionException: Region is not online: .META.,,1
4) ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
Inconsist
ent configuration. Previous configuration for using table name in metrics:
true, new configuration: false
5) Got user-level KeeperEx
ception when processing sessionid:0x144df32705d0000 type:delete cxid:0xc
zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
ers/hbase.rummycircle.com,60000,1395314029900 Error:KeeperErrorCode =
NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
314029900
3rd and 5th ones getting repeated multiple times.
Please help.
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by haosdent <ha...@gmail.com>.
>Zookeeper logs are still appended to "hbase-hadoop-zookeeper-watto.log".
You could change the behaviour of zookeeper log in log4j.properties.
On Mon, Mar 24, 2014 at 4:34 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> One more issue i found is that zookeeper logs are not getting rotated.
> I started cluster 3-4 days back and still only master and region server
> logs are rotated properly.
> Zookeeper logs are still appended to "hbase-hadoop-zookeeper-watto.log".
> Please help on this and also on whether to use hadoop 1.2.1 or hadop 1.1.2
> ?
>
>
> On Sat, Mar 22, 2014 at 11:15 AM, Vimal Jain <vk...@gmail.com> wrote:
>
> > Stack,
> > Thanks a lot for the analysis , so nothing to worry about ?
> > Lars,
> > I did not built hbase from source.
> > I simply downloaded tars from hbase and hadoop download page( hbase from
> > "stable" directory ( i.e 0.94.17 ) and hadoop from "stable1" directory (
> > i.e. 1.2.1) ).
> > Do i need to build from source ?
> > If yes , is there any guidelines in doing same as i have never used Maven
> > ( i use Ant for build and deployment ).
> >
> >
> > On Sat, Mar 22, 2014 at 5:22 AM, lars hofhansl <la...@apache.org> wrote:
> >
> >> Hi Vimal,
> >>
> >> did you rebuild HBase against Hadoop 1.2.1? The released tarballs are
> >> built against Hadoop 1.0.4 and won't work with Hadoop 1.2.x or 2.x.y.
> >>
> >> -- Lars
> >>
> >> ------------------------------
> >> *From:* Vimal Jain <vk...@gmail.com>
> >> *To:* "user@hbase.apache.org" <us...@hbase.apache.org>; lars hofhansl <
> >> larsh@apache.org>
> >> *Sent:* Friday, March 21, 2014 12:08 AM
> >>
> >> *Subject:* Re: Getting exceptions in fresh installment of hbase
> >>
> >> Hi Stack/Lars,
> >> I configured Hbase again in pseudo distributed mode.
> >> Again i got some exceptions which have shared in pastebin
> >>
> >> zookeeper log - http://pastebin.com/6vgbm987
> >> master log - http://pastebin.com/kcjpd1Zp
> >> region log - http://pastebin.com/vmfE2HG2
> >>
> >> There is no error in all 3 hadoop processes.
> >> And i am sure if i restart the hbase , i will get errors like 'Region is
> >> not online: -ROOT-' , 'Region is not online: .META'
> >> This is the pattern i have observed.
> >>
> >> Let me know if you want me to restart the hbase cluster and then i will
> >> attach fresh logs.
> >>
> >> Hbase - 0.94.17 and Hadoop 1.2.1
> >>
> >>
> >> On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org>
> wrote:
> >>
> >> I do this *all* the time, I have never seen an issue like this. So this
> >> is interesting.
> >> Is it possible that ZK happened to have picked a use ephemeral port?
> (but
> >> in that case I would have only expected this to fail once and work the
> next
> >> time).
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >> From: Vimal Jain <vk...@gmail.com>
> >> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> >> Sent: Thursday, March 20, 2014 11:10 PM
> >> Subject: Re: Getting exceptions in fresh installment of hbase
> >>
> >>
> >> Hi Stack,
> >> I will delete everything and reconfigure hbase from scratch.
> >> I will then attach logs in pastebin if i find same issues again.
> >>
> >>
> >> On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
> >>
> >> > Put the full logs up in pastebin. The below seem like symptoms, not
> the
> >> > problem.
> >> > St.Ack
> >> >
> >> >
> >> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com>
> wrote:
> >> >
> >> > > Yes Stack.
> >> > >
> >> > >
> >> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> >> > >
> >> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Hi,
> >> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> >> > > distributed
> >> > > > > mode and i am getting below exceptions when starting hbase for
> >> first
> >> > > > time.
> >> > > > >
> >> > > > >
> >> > > > You started fresh? Fresh HDFS and ZK in particular?
> >> > > > St.Ack
> >> > > >
> >> > > >
> >> > > >
> >> > > > > Master logs :-
> >> > > > >
> >> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> >> Node
> >> > > > > /hbase/backup-m
> >> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already
> deleted,
> >> > and
> >> > > > this
> >> > > > > is not a retry
> >> > > > >
> >> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> >> > verification
> >> > > of
> >> > > > > -ROOT
> >> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> >> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> >> > > > > org.apache.hadoop.hbase.N
> >> > > > > otServingRegionException: Region is not online: -ROOT-,,0
> >> > > > >
> >> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> >> > verification
> >> > > of
> >> > > > > .META
> >> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> >> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> >> > > > > org.apache.hadoop.hbase.N
> >> > > > > otServingRegionException: Region is not online: .META.,,1
> >> > > > >
> >> > > > > 4) ERROR
> >> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> >> > > > > Inconsist
> >> > > > > ent configuration. Previous configuration for using table name
> in
> >> > > > metrics:
> >> > > > > true, new configuration: false
> >> > > > >
> >> > > > > 5) Got user-level KeeperEx
> >> > > > > ception when processing sessionid:0x144df32705d0000 type:delete
> >> > > cxid:0xc
> >> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> >> > > > > ers/hbase.rummycircle.com,60000,1395314029900
> >> Error:KeeperErrorCode
> >> > =
> >> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com
> ,60000,1395
> >> > > > > 314029900
> >> > > > >
> >> > > > > 3rd and 5th ones getting repeated multiple times.
> >> > > > >
> >> > > > > Please help.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Thanks and Regards,
> >>
> >> > > > > Vimal Jain
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Thanks and Regards,
> >> > > Vimal Jain
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks and Regards,
> >> Vimal Jain
> >>
> >>
> >>
> >>
> >> --
> >> Thanks and Regards,
> >> Vimal Jain
> >>
> >>
> >>
> >
> >
> > --
> > Thanks and Regards,
> > Vimal Jain
> >
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Best Regards,
Haosdent Huang
Re: Getting exceptions in fresh installment of hbase
Posted by Stack <st...@duboce.net>.
On Mon, Mar 24, 2014 at 1:34 AM, Vimal Jain <vk...@gmail.com> wrote:
...
> Please help on this and also on whether to use hadoop 1.2.1 or hadop 1.1.2
> ?
>
>
You should use hbase 0.98.x and hadoop 2.x.
St.Ack
Re: Getting exceptions in fresh installment of hbase
Posted by Vimal Jain <vk...@gmail.com>.
Hi,
One more issue i found is that zookeeper logs are not getting rotated.
I started cluster 3-4 days back and still only master and region server
logs are rotated properly.
Zookeeper logs are still appended to "hbase-hadoop-zookeeper-watto.log".
Please help on this and also on whether to use hadoop 1.2.1 or hadop 1.1.2 ?
On Sat, Mar 22, 2014 at 11:15 AM, Vimal Jain <vk...@gmail.com> wrote:
> Stack,
> Thanks a lot for the analysis , so nothing to worry about ?
> Lars,
> I did not built hbase from source.
> I simply downloaded tars from hbase and hadoop download page( hbase from
> "stable" directory ( i.e 0.94.17 ) and hadoop from "stable1" directory (
> i.e. 1.2.1) ).
> Do i need to build from source ?
> If yes , is there any guidelines in doing same as i have never used Maven
> ( i use Ant for build and deployment ).
>
>
> On Sat, Mar 22, 2014 at 5:22 AM, lars hofhansl <la...@apache.org> wrote:
>
>> Hi Vimal,
>>
>> did you rebuild HBase against Hadoop 1.2.1? The released tarballs are
>> built against Hadoop 1.0.4 and won't work with Hadoop 1.2.x or 2.x.y.
>>
>> -- Lars
>>
>> ------------------------------
>> *From:* Vimal Jain <vk...@gmail.com>
>> *To:* "user@hbase.apache.org" <us...@hbase.apache.org>; lars hofhansl <
>> larsh@apache.org>
>> *Sent:* Friday, March 21, 2014 12:08 AM
>>
>> *Subject:* Re: Getting exceptions in fresh installment of hbase
>>
>> Hi Stack/Lars,
>> I configured Hbase again in pseudo distributed mode.
>> Again i got some exceptions which have shared in pastebin
>>
>> zookeeper log - http://pastebin.com/6vgbm987
>> master log - http://pastebin.com/kcjpd1Zp
>> region log - http://pastebin.com/vmfE2HG2
>>
>> There is no error in all 3 hadoop processes.
>> And i am sure if i restart the hbase , i will get errors like 'Region is
>> not online: -ROOT-' , 'Region is not online: .META'
>> This is the pattern i have observed.
>>
>> Let me know if you want me to restart the hbase cluster and then i will
>> attach fresh logs.
>>
>> Hbase - 0.94.17 and Hadoop 1.2.1
>>
>>
>> On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org> wrote:
>>
>> I do this *all* the time, I have never seen an issue like this. So this
>> is interesting.
>> Is it possible that ZK happened to have picked a use ephemeral port? (but
>> in that case I would have only expected this to fail once and work the next
>> time).
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Vimal Jain <vk...@gmail.com>
>> To: "user@hbase.apache.org" <us...@hbase.apache.org>
>> Sent: Thursday, March 20, 2014 11:10 PM
>> Subject: Re: Getting exceptions in fresh installment of hbase
>>
>>
>> Hi Stack,
>> I will delete everything and reconfigure hbase from scratch.
>> I will then attach logs in pastebin if i find same issues again.
>>
>>
>> On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
>>
>> > Put the full logs up in pastebin. The below seem like symptoms, not the
>> > problem.
>> > St.Ack
>> >
>> >
>> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
>> >
>> > > Yes Stack.
>> > >
>> > >
>> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
>> > >
>> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi,
>> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
>> > > distributed
>> > > > > mode and i am getting below exceptions when starting hbase for
>> first
>> > > > time.
>> > > > >
>> > > > >
>> > > > You started fresh? Fresh HDFS and ZK in particular?
>> > > > St.Ack
>> > > >
>> > > >
>> > > >
>> > > > > Master logs :-
>> > > > >
>> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
>> Node
>> > > > > /hbase/backup-m
>> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
>> > and
>> > > > this
>> > > > > is not a retry
>> > > > >
>> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
>> > verification
>> > > of
>> > > > > -ROOT
>> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
>> > > > > org.apache.hadoop.hbase.NotServingRegionException:
>> > > > > org.apache.hadoop.hbase.N
>> > > > > otServingRegionException: Region is not online: -ROOT-,,0
>> > > > >
>> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
>> > verification
>> > > of
>> > > > > .META
>> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
>> > > > > org.apache.hadoop.hbase.NotServingRegionException:
>> > > > > org.apache.hadoop.hbase.N
>> > > > > otServingRegionException: Region is not online: .META.,,1
>> > > > >
>> > > > > 4) ERROR
>> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
>> > > > > Inconsist
>> > > > > ent configuration. Previous configuration for using table name in
>> > > > metrics:
>> > > > > true, new configuration: false
>> > > > >
>> > > > > 5) Got user-level KeeperEx
>> > > > > ception when processing sessionid:0x144df32705d0000 type:delete
>> > > cxid:0xc
>> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
>> > > > > ers/hbase.rummycircle.com,60000,1395314029900
>> Error:KeeperErrorCode
>> > =
>> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
>> > > > > 314029900
>> > > > >
>> > > > > 3rd and 5th ones getting repeated multiple times.
>> > > > >
>> > > > > Please help.
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Thanks and Regards,
>>
>> > > > > Vimal Jain
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thanks and Regards,
>> > > Vimal Jain
>> > >
>> >
>>
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>>
>>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Stack <st...@duboce.net>.
On Mon, Mar 24, 2014 at 10:40 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Lars,Stack,
> I have started building Hbase from source using the guideline provided by
> you.
> My question is i have a working pseudo distributed mode cluster on
> production with hbase 0.94.7 and hadoop 1.1.2 .I did not build hbase from
> source for this.I simply downloaded the tars from stable directories.
> Also i copied hadoop-core jar from hadoop base directory into hbase lib
> directory and common-collection from hadoop lib directory to hbase lib
> directory.
> Wouldn't this solve the mismatch problem ?
>
>
>
That'll probably work. In the versions matrix,
http://hbase.apache.org/book.html#hadoop, it claims we support this
combination. You might want to follow Lars advice though since he knows
0.94 best and recompile.
St.Ack
Re: Getting exceptions in fresh installment of hbase
Posted by Vimal Jain <vk...@gmail.com>.
Hi Lars,Stack,
I have started building Hbase from source using the guideline provided by
you.
My question is i have a working pseudo distributed mode cluster on
production with hbase 0.94.7 and hadoop 1.1.2 .I did not build hbase from
source for this.I simply downloaded the tars from stable directories.
Also i copied hadoop-core jar from hadoop base directory into hbase lib
directory and common-collection from hadoop lib directory to hbase lib
directory.
Wouldn't this solve the mismatch problem ?
On Tue, Mar 25, 2014 at 1:49 AM, Stack <st...@duboce.net> wrote:
> On Mon, Mar 24, 2014 at 11:49 AM, lars hofhansl <la...@apache.org> wrote:
>
> > HBase is a client to HDFS. 0.94 defaults to 1.0.4, 0.98 and a build
> > against 2.2.0, and 1.2.1 (I think).
> > HDFS clients are not compatible across versions, and hence HBase should
> be
> > built from source against your specific version of Hadoop.
> >
> >
> Some notes on getting 0.94 running on hadoop2 are here:
> http://hbase.apache.org/book.html#ftn.d248e643
>
> St.Ack
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Stack <st...@duboce.net>.
On Mon, Mar 24, 2014 at 11:49 AM, lars hofhansl <la...@apache.org> wrote:
> HBase is a client to HDFS. 0.94 defaults to 1.0.4, 0.98 and a build
> against 2.2.0, and 1.2.1 (I think).
> HDFS clients are not compatible across versions, and hence HBase should be
> built from source against your specific version of Hadoop.
>
>
Some notes on getting 0.94 running on hadoop2 are here:
http://hbase.apache.org/book.html#ftn.d248e643
St.Ack
Re: Getting exceptions in fresh installment of hbase
Posted by lars hofhansl <la...@apache.org>.
HBase is a client to HDFS. 0.94 defaults to 1.0.4, 0.98 and a build against 2.2.0, and 1.2.1 (I think).
HDFS clients are not compatible across versions, and hence HBase should be built from source against your specific version of Hadoop.
-- Lars
________________________________
From: Vimal Jain <vk...@gmail.com>
To: lars hofhansl <la...@apache.org>
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Friday, March 21, 2014 10:45 PM
Subject: Re: Getting exceptions in fresh installment of hbase
Stack,
Thanks a lot for the analysis , so nothing to worry about ?
Lars,
I did not built hbase from source.
I simply downloaded tars from hbase and hadoop download page( hbase from
"stable" directory ( i.e 0.94.17 ) and hadoop from "stable1" directory (
i.e. 1.2.1) ).
Do i need to build from source ?
If yes , is there any guidelines in doing same as i have never used Maven (
i use Ant for build and deployment ).
On Sat, Mar 22, 2014 at 5:22 AM, lars hofhansl <la...@apache.org> wrote:
> Hi Vimal,
>
> did you rebuild HBase against Hadoop 1.2.1? The released tarballs are
> built against Hadoop 1.0.4 and won't work with Hadoop 1.2.x or 2.x.y.
>
> -- Lars
>
> ------------------------------
> *From:* Vimal Jain <vk...@gmail.com>
> *To:* "user@hbase.apache.org" <us...@hbase.apache.org>; lars hofhansl <
> larsh@apache.org>
> *Sent:* Friday, March 21, 2014 12:08 AM
>
> *Subject:* Re: Getting exceptions in fresh installment of hbase
>
> Hi Stack/Lars,
> I configured Hbase again in pseudo distributed mode.
> Again i got some exceptions which have shared in pastebin
>
> zookeeper log - http://pastebin.com/6vgbm987
> master log - http://pastebin.com/kcjpd1Zp
> region log - http://pastebin.com/vmfE2HG2
>
> There is no error in all 3 hadoop processes.
> And i am sure if i restart the hbase , i will get errors like 'Region is
> not online: -ROOT-' , 'Region is not online: .META'
> This is the pattern i have observed.
>
> Let me know if you want me to restart the hbase cluster and then i will
> attach fresh logs.
>
> Hbase - 0.94.17 and Hadoop 1.2.1
>
>
> On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org> wrote:
>
> I do this *all* the time, I have never seen an issue like this. So this is
> interesting.
> Is it possible that ZK happened to have picked a use ephemeral port? (but
> in that case I would have only expected this to fail once and work the next
> time).
>
> -- Lars
>
>
>
> ________________________________
> From: Vimal Jain <vk...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Thursday, March 20, 2014 11:10 PM
> Subject: Re: Getting exceptions in fresh installment of hbase
>
>
> Hi Stack,
> I will delete everything and reconfigure hbase from scratch.
> I will then attach logs in pastebin if i find same issues again.
>
>
> On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
>
> > Put the full logs up in pastebin. The below seem like symptoms, not the
> > problem.
> > St.Ack
> >
> >
> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
> >
> > > Yes Stack.
> > >
> > >
> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> > > distributed
> > > > > mode and i am getting below exceptions when starting hbase for
> first
> > > > time.
> > > > >
> > > > >
> > > > You started fresh? Fresh HDFS and ZK in particular?
> > > > St.Ack
> > > >
> > > >
> > > >
> > > > > Master logs :-
> > > > >
> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> Node
> > > > > /hbase/backup-m
> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
> > and
> > > > this
> > > > > is not a retry
> > > > >
> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> > verification
> > > of
> > > > > -ROOT
> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > > org.apache.hadoop.hbase.N
> > > > > otServingRegionException: Region is not online: -ROOT-,,0
> > > > >
> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> > verification
> > > of
> > > > > .META
> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > > org.apache.hadoop.hbase.N
> > > > > otServingRegionException: Region is not online: .META.,,1
> > > > >
> > > > > 4) ERROR
> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> > > > > Inconsist
> > > > > ent configuration. Previous configuration for using table name in
> > > > metrics:
> > > > > true, new configuration: false
> > > > >
> > > > > 5) Got user-level KeeperEx
> > > > > ception when processing sessionid:0x144df32705d0000 type:delete
> > > cxid:0xc
> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> > > > > ers/hbase.rummycircle.com,60000,1395314029900
> Error:KeeperErrorCode
> > =
> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> > > > > 314029900
> > > > >
> > > > > 3rd and 5th ones getting repeated multiple times.
> > > > >
> > > > > Please help.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
>
> > > > > Vimal Jain
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Vimal Jain
> > >
> >
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Vimal Jain <vk...@gmail.com>.
Stack,
Thanks a lot for the analysis , so nothing to worry about ?
Lars,
I did not built hbase from source.
I simply downloaded tars from hbase and hadoop download page( hbase from
"stable" directory ( i.e 0.94.17 ) and hadoop from "stable1" directory (
i.e. 1.2.1) ).
Do i need to build from source ?
If yes , is there any guidelines in doing same as i have never used Maven (
i use Ant for build and deployment ).
On Sat, Mar 22, 2014 at 5:22 AM, lars hofhansl <la...@apache.org> wrote:
> Hi Vimal,
>
> did you rebuild HBase against Hadoop 1.2.1? The released tarballs are
> built against Hadoop 1.0.4 and won't work with Hadoop 1.2.x or 2.x.y.
>
> -- Lars
>
> ------------------------------
> *From:* Vimal Jain <vk...@gmail.com>
> *To:* "user@hbase.apache.org" <us...@hbase.apache.org>; lars hofhansl <
> larsh@apache.org>
> *Sent:* Friday, March 21, 2014 12:08 AM
>
> *Subject:* Re: Getting exceptions in fresh installment of hbase
>
> Hi Stack/Lars,
> I configured Hbase again in pseudo distributed mode.
> Again i got some exceptions which have shared in pastebin
>
> zookeeper log - http://pastebin.com/6vgbm987
> master log - http://pastebin.com/kcjpd1Zp
> region log - http://pastebin.com/vmfE2HG2
>
> There is no error in all 3 hadoop processes.
> And i am sure if i restart the hbase , i will get errors like 'Region is
> not online: -ROOT-' , 'Region is not online: .META'
> This is the pattern i have observed.
>
> Let me know if you want me to restart the hbase cluster and then i will
> attach fresh logs.
>
> Hbase - 0.94.17 and Hadoop 1.2.1
>
>
> On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org> wrote:
>
> I do this *all* the time, I have never seen an issue like this. So this is
> interesting.
> Is it possible that ZK happened to have picked a use ephemeral port? (but
> in that case I would have only expected this to fail once and work the next
> time).
>
> -- Lars
>
>
>
> ________________________________
> From: Vimal Jain <vk...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Thursday, March 20, 2014 11:10 PM
> Subject: Re: Getting exceptions in fresh installment of hbase
>
>
> Hi Stack,
> I will delete everything and reconfigure hbase from scratch.
> I will then attach logs in pastebin if i find same issues again.
>
>
> On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
>
> > Put the full logs up in pastebin. The below seem like symptoms, not the
> > problem.
> > St.Ack
> >
> >
> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
> >
> > > Yes Stack.
> > >
> > >
> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> > > distributed
> > > > > mode and i am getting below exceptions when starting hbase for
> first
> > > > time.
> > > > >
> > > > >
> > > > You started fresh? Fresh HDFS and ZK in particular?
> > > > St.Ack
> > > >
> > > >
> > > >
> > > > > Master logs :-
> > > > >
> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> Node
> > > > > /hbase/backup-m
> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
> > and
> > > > this
> > > > > is not a retry
> > > > >
> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> > verification
> > > of
> > > > > -ROOT
> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > > org.apache.hadoop.hbase.N
> > > > > otServingRegionException: Region is not online: -ROOT-,,0
> > > > >
> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> > verification
> > > of
> > > > > .META
> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > > org.apache.hadoop.hbase.N
> > > > > otServingRegionException: Region is not online: .META.,,1
> > > > >
> > > > > 4) ERROR
> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> > > > > Inconsist
> > > > > ent configuration. Previous configuration for using table name in
> > > > metrics:
> > > > > true, new configuration: false
> > > > >
> > > > > 5) Got user-level KeeperEx
> > > > > ception when processing sessionid:0x144df32705d0000 type:delete
> > > cxid:0xc
> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> > > > > ers/hbase.rummycircle.com,60000,1395314029900
> Error:KeeperErrorCode
> > =
> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> > > > > 314029900
> > > > >
> > > > > 3rd and 5th ones getting repeated multiple times.
> > > > >
> > > > > Please help.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
>
> > > > > Vimal Jain
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Vimal Jain
> > >
> >
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by lars hofhansl <la...@apache.org>.
Hi Vimal,
did you rebuild HBase against Hadoop 1.2.1? The released tarballs are built against Hadoop 1.0.4 and won't work with Hadoop 1.2.x or 2.x.y.
-- Lars
________________________________
From: Vimal Jain <vk...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Sent: Friday, March 21, 2014 12:08 AM
Subject: Re: Getting exceptions in fresh installment of hbase
Hi Stack/Lars,
I configured Hbase again in pseudo distributed mode.
Again i got some exceptions which have shared in pastebin
zookeeper log - http://pastebin.com/6vgbm987
master log - http://pastebin.com/kcjpd1Zp
region log - http://pastebin.com/vmfE2HG2
There is no error in all 3 hadoop processes.
And i am sure if i restart the hbase , i will get errors like 'Region is not online: -ROOT-' , 'Region is not online: .META'
This is the pattern i have observed.
Let me know if you want me to restart the hbase cluster and then i will attach fresh logs.
Hbase - 0.94.17 and Hadoop 1.2.1
On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org> wrote:
I do this *all* the time, I have never seen an issue like this. So this is interesting.
>Is it possible that ZK happened to have picked a use ephemeral port? (but in that case I would have only expected this to fail once and work the next time).
>
>-- Lars
>
>
>
>________________________________
> From: Vimal Jain <vk...@gmail.com>
>To: "user@hbase.apache.org" <us...@hbase.apache.org>
>Sent: Thursday, March 20, 2014 11:10 PM
>Subject: Re: Getting exceptions in fresh installment of hbase
>
>
>
>Hi Stack,
>I will delete everything and reconfigure hbase from scratch.
>I will then attach logs in pastebin if i find same issues again.
>
>
>On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
>
>> Put the full logs up in pastebin. The below seem like symptoms, not the
>> problem.
>> St.Ack
>>
>>
>> On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
>>
>> > Yes Stack.
>> >
>> >
>> > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
>> >
>> > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com> wrote:
>> > >
>> > > > Hi,
>> > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
>> > distributed
>> > > > mode and i am getting below exceptions when starting hbase for first
>> > > time.
>> > > >
>> > > >
>> > > You started fresh? Fresh HDFS and ZK in particular?
>> > > St.Ack
>> > >
>> > >
>> > >
>> > > > Master logs :-
>> > > >
>> > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
>> > > > /hbase/backup-m
>> > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
>> and
>> > > this
>> > > > is not a retry
>> > > >
>> > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
>> verification
>> > of
>> > > > -ROOT
>> > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
>> > > > org.apache.hadoop.hbase.NotServingRegionException:
>> > > > org.apache.hadoop.hbase.N
>> > > > otServingRegionException: Region is not online: -ROOT-,,0
>> > > >
>> > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
>> verification
>> > of
>> > > > .META
>> > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
>> > > > org.apache.hadoop.hbase.NotServingRegionException:
>> > > > org.apache.hadoop.hbase.N
>> > > > otServingRegionException: Region is not online: .META.,,1
>> > > >
>> > > > 4) ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
>> > > > Inconsist
>> > > > ent configuration. Previous configuration for using table name in
>> > > metrics:
>> > > > true, new configuration: false
>> > > >
>> > > > 5) Got user-level KeeperEx
>> > > > ception when processing sessionid:0x144df32705d0000 type:delete
>> > cxid:0xc
>> > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
>> > > > ers/hbase.rummycircle.com,60000,1395314029900 Error:KeeperErrorCode
>> =
>> > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
>> > > > 314029900
>> > > >
>> > > > 3rd and 5th ones getting repeated multiple times.
>> > > >
>> > > > Please help.
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Thanks and Regards,
>
>> > > > Vimal Jain
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Thanks and Regards,
>> > Vimal Jain
>> >
>>
>
>
>
>--
>Thanks and Regards,
>Vimal Jain
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Stack <st...@duboce.net>.
Your cluster seems fine.
Item #1 above should probably not be WARN. It is over-sharing on what zk
is up to during startup (#5 is related). #2 and #3 are just our logging
that root and meta are not where zk says they are so we reassign which
later in the log you'll see succeeds; this is just how we do startup (zk
has last locations of these regions. When the master comes up it probes to
see if these locations are still good...). #4 is complaining about a
change in a metric after startup. See
http://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.html#710
It explains why we log as ERROR. Agree it disturbing. File an issue
if
you are not doing anything here in this regard.
Thanks,
St.Ack
On Fri, Mar 21, 2014 at 12:23 AM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I could not stop myself from restarting the hbase cluster. :(
> My predictions were correct
> I restarted cluster and now I am getting those exceptions.
>
> logs after restarting hbase cluster.
>
> Zookeeper - http://pastebin.com/UBbbiERk
> Master - http://pastebin.com/6GZH5hZK
> Region - http://pastebin.com/Fi38vUeS
>
>
> Please help in understanding this and let me know if there is any
> confusion.
>
>
> On Fri, Mar 21, 2014 at 12:38 PM, Vimal Jain <vk...@gmail.com> wrote:
>
> > Hi Stack/Lars,
> > I configured Hbase again in pseudo distributed mode.
> > Again i got some exceptions which have shared in pastebin
> >
> > zookeeper log - http://pastebin.com/6vgbm987
> > master log - http://pastebin.com/kcjpd1Zp
> > region log - http://pastebin.com/vmfE2HG2
> >
> > There is no error in all 3 hadoop processes.
> > And i am sure if i restart the hbase , i will get errors like 'Region is
> > not online: -ROOT-' , 'Region is not online: .META'
> > This is the pattern i have observed.
> >
> > Let me know if you want me to restart the hbase cluster and then i will
> > attach fresh logs.
> >
> > Hbase - 0.94.17 and Hadoop 1.2.1
> >
> >
> > On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org>
> wrote:
> >
> >> I do this *all* the time, I have never seen an issue like this. So this
> >> is interesting.
> >> Is it possible that ZK happened to have picked a use ephemeral port?
> (but
> >> in that case I would have only expected this to fail once and work the
> next
> >> time).
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >> From: Vimal Jain <vk...@gmail.com>
> >> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> >> Sent: Thursday, March 20, 2014 11:10 PM
> >> Subject: Re: Getting exceptions in fresh installment of hbase
> >>
> >>
> >> Hi Stack,
> >> I will delete everything and reconfigure hbase from scratch.
> >> I will then attach logs in pastebin if i find same issues again.
> >>
> >>
> >> On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
> >>
> >> > Put the full logs up in pastebin. The below seem like symptoms, not
> the
> >> > problem.
> >> > St.Ack
> >> >
> >> >
> >> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com>
> wrote:
> >> >
> >> > > Yes Stack.
> >> > >
> >> > >
> >> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> >> > >
> >> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Hi,
> >> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> >> > > distributed
> >> > > > > mode and i am getting below exceptions when starting hbase for
> >> first
> >> > > > time.
> >> > > > >
> >> > > > >
> >> > > > You started fresh? Fresh HDFS and ZK in particular?
> >> > > > St.Ack
> >> > > >
> >> > > >
> >> > > >
> >> > > > > Master logs :-
> >> > > > >
> >> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> >> Node
> >> > > > > /hbase/backup-m
> >> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already
> deleted,
> >> > and
> >> > > > this
> >> > > > > is not a retry
> >> > > > >
> >> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> >> > verification
> >> > > of
> >> > > > > -ROOT
> >> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> >> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> >> > > > > org.apache.hadoop.hbase.N
> >> > > > > otServingRegionException: Region is not online: -ROOT-,,0
> >> > > > >
> >> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> >> > verification
> >> > > of
> >> > > > > .META
> >> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> >> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> >> > > > > org.apache.hadoop.hbase.N
> >> > > > > otServingRegionException: Region is not online: .META.,,1
> >> > > > >
> >> > > > > 4) ERROR
> >> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> >> > > > > Inconsist
> >> > > > > ent configuration. Previous configuration for using table name
> in
> >> > > > metrics:
> >> > > > > true, new configuration: false
> >> > > > >
> >> > > > > 5) Got user-level KeeperEx
> >> > > > > ception when processing sessionid:0x144df32705d0000 type:delete
> >> > > cxid:0xc
> >> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> >> > > > > ers/hbase.rummycircle.com,60000,1395314029900
> >> Error:KeeperErrorCode
> >> > =
> >> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com
> ,60000,1395
> >> > > > > 314029900
> >> > > > >
> >> > > > > 3rd and 5th ones getting repeated multiple times.
> >> > > > >
> >> > > > > Please help.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Thanks and Regards,
> >>
> >> > > > > Vimal Jain
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Thanks and Regards,
> >> > > Vimal Jain
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks and Regards,
> >> Vimal Jain
> >>
> >
> >
> >
> > --
> > Thanks and Regards,
> > Vimal Jain
> >
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
Re: Getting exceptions in fresh installment of hbase
Posted by Vimal Jain <vk...@gmail.com>.
Hi,
I could not stop myself from restarting the hbase cluster. :(
My predictions were correct
I restarted cluster and now I am getting those exceptions.
logs after restarting hbase cluster.
Zookeeper - http://pastebin.com/UBbbiERk
Master - http://pastebin.com/6GZH5hZK
Region - http://pastebin.com/Fi38vUeS
Please help in understanding this and let me know if there is any confusion.
On Fri, Mar 21, 2014 at 12:38 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Stack/Lars,
> I configured Hbase again in pseudo distributed mode.
> Again i got some exceptions which have shared in pastebin
>
> zookeeper log - http://pastebin.com/6vgbm987
> master log - http://pastebin.com/kcjpd1Zp
> region log - http://pastebin.com/vmfE2HG2
>
> There is no error in all 3 hadoop processes.
> And i am sure if i restart the hbase , i will get errors like 'Region is
> not online: -ROOT-' , 'Region is not online: .META'
> This is the pattern i have observed.
>
> Let me know if you want me to restart the hbase cluster and then i will
> attach fresh logs.
>
> Hbase - 0.94.17 and Hadoop 1.2.1
>
>
> On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org> wrote:
>
>> I do this *all* the time, I have never seen an issue like this. So this
>> is interesting.
>> Is it possible that ZK happened to have picked a use ephemeral port? (but
>> in that case I would have only expected this to fail once and work the next
>> time).
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Vimal Jain <vk...@gmail.com>
>> To: "user@hbase.apache.org" <us...@hbase.apache.org>
>> Sent: Thursday, March 20, 2014 11:10 PM
>> Subject: Re: Getting exceptions in fresh installment of hbase
>>
>>
>> Hi Stack,
>> I will delete everything and reconfigure hbase from scratch.
>> I will then attach logs in pastebin if i find same issues again.
>>
>>
>> On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
>>
>> > Put the full logs up in pastebin. The below seem like symptoms, not the
>> > problem.
>> > St.Ack
>> >
>> >
>> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
>> >
>> > > Yes Stack.
>> > >
>> > >
>> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
>> > >
>> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi,
>> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
>> > > distributed
>> > > > > mode and i am getting below exceptions when starting hbase for
>> first
>> > > > time.
>> > > > >
>> > > > >
>> > > > You started fresh? Fresh HDFS and ZK in particular?
>> > > > St.Ack
>> > > >
>> > > >
>> > > >
>> > > > > Master logs :-
>> > > > >
>> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
>> Node
>> > > > > /hbase/backup-m
>> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
>> > and
>> > > > this
>> > > > > is not a retry
>> > > > >
>> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
>> > verification
>> > > of
>> > > > > -ROOT
>> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
>> > > > > org.apache.hadoop.hbase.NotServingRegionException:
>> > > > > org.apache.hadoop.hbase.N
>> > > > > otServingRegionException: Region is not online: -ROOT-,,0
>> > > > >
>> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
>> > verification
>> > > of
>> > > > > .META
>> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
>> > > > > org.apache.hadoop.hbase.NotServingRegionException:
>> > > > > org.apache.hadoop.hbase.N
>> > > > > otServingRegionException: Region is not online: .META.,,1
>> > > > >
>> > > > > 4) ERROR
>> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
>> > > > > Inconsist
>> > > > > ent configuration. Previous configuration for using table name in
>> > > > metrics:
>> > > > > true, new configuration: false
>> > > > >
>> > > > > 5) Got user-level KeeperEx
>> > > > > ception when processing sessionid:0x144df32705d0000 type:delete
>> > > cxid:0xc
>> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
>> > > > > ers/hbase.rummycircle.com,60000,1395314029900
>> Error:KeeperErrorCode
>> > =
>> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
>> > > > > 314029900
>> > > > >
>> > > > > 3rd and 5th ones getting repeated multiple times.
>> > > > >
>> > > > > Please help.
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Thanks and Regards,
>>
>> > > > > Vimal Jain
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thanks and Regards,
>> > > Vimal Jain
>> > >
>> >
>>
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Vimal Jain <vk...@gmail.com>.
Hi Stack/Lars,
I configured Hbase again in pseudo distributed mode.
Again i got some exceptions which have shared in pastebin
zookeeper log - http://pastebin.com/6vgbm987
master log - http://pastebin.com/kcjpd1Zp
region log - http://pastebin.com/vmfE2HG2
There is no error in all 3 hadoop processes.
And i am sure if i restart the hbase , i will get errors like 'Region is
not online: -ROOT-' , 'Region is not online: .META'
This is the pattern i have observed.
Let me know if you want me to restart the hbase cluster and then i will
attach fresh logs.
Hbase - 0.94.17 and Hadoop 1.2.1
On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <la...@apache.org> wrote:
> I do this *all* the time, I have never seen an issue like this. So this is
> interesting.
> Is it possible that ZK happened to have picked a use ephemeral port? (but
> in that case I would have only expected this to fail once and work the next
> time).
>
> -- Lars
>
>
>
> ________________________________
> From: Vimal Jain <vk...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Thursday, March 20, 2014 11:10 PM
> Subject: Re: Getting exceptions in fresh installment of hbase
>
>
> Hi Stack,
> I will delete everything and reconfigure hbase from scratch.
> I will then attach logs in pastebin if i find same issues again.
>
>
> On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
>
> > Put the full logs up in pastebin. The below seem like symptoms, not the
> > problem.
> > St.Ack
> >
> >
> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
> >
> > > Yes Stack.
> > >
> > >
> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> > > distributed
> > > > > mode and i am getting below exceptions when starting hbase for
> first
> > > > time.
> > > > >
> > > > >
> > > > You started fresh? Fresh HDFS and ZK in particular?
> > > > St.Ack
> > > >
> > > >
> > > >
> > > > > Master logs :-
> > > > >
> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> Node
> > > > > /hbase/backup-m
> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
> > and
> > > > this
> > > > > is not a retry
> > > > >
> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> > verification
> > > of
> > > > > -ROOT
> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > > org.apache.hadoop.hbase.N
> > > > > otServingRegionException: Region is not online: -ROOT-,,0
> > > > >
> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> > verification
> > > of
> > > > > .META
> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > > org.apache.hadoop.hbase.N
> > > > > otServingRegionException: Region is not online: .META.,,1
> > > > >
> > > > > 4) ERROR
> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> > > > > Inconsist
> > > > > ent configuration. Previous configuration for using table name in
> > > > metrics:
> > > > > true, new configuration: false
> > > > >
> > > > > 5) Got user-level KeeperEx
> > > > > ception when processing sessionid:0x144df32705d0000 type:delete
> > > cxid:0xc
> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> > > > > ers/hbase.rummycircle.com,60000,1395314029900
> Error:KeeperErrorCode
> > =
> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> > > > > 314029900
> > > > >
> > > > > 3rd and 5th ones getting repeated multiple times.
> > > > >
> > > > > Please help.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
>
> > > > > Vimal Jain
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Vimal Jain
> > >
> >
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by lars hofhansl <la...@apache.org>.
I do this *all* the time, I have never seen an issue like this. So this is interesting.
Is it possible that ZK happened to have picked a use ephemeral port? (but in that case I would have only expected this to fail once and work the next time).
-- Lars
________________________________
From: Vimal Jain <vk...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Thursday, March 20, 2014 11:10 PM
Subject: Re: Getting exceptions in fresh installment of hbase
Hi Stack,
I will delete everything and reconfigure hbase from scratch.
I will then attach logs in pastebin if i find same issues again.
On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
> Put the full logs up in pastebin. The below seem like symptoms, not the
> problem.
> St.Ack
>
>
> On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
>
> > Yes Stack.
> >
> >
> > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> > distributed
> > > > mode and i am getting below exceptions when starting hbase for first
> > > time.
> > > >
> > > >
> > > You started fresh? Fresh HDFS and ZK in particular?
> > > St.Ack
> > >
> > >
> > >
> > > > Master logs :-
> > > >
> > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> > > > /hbase/backup-m
> > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
> and
> > > this
> > > > is not a retry
> > > >
> > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> verification
> > of
> > > > -ROOT
> > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > org.apache.hadoop.hbase.N
> > > > otServingRegionException: Region is not online: -ROOT-,,0
> > > >
> > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> verification
> > of
> > > > .META
> > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > org.apache.hadoop.hbase.N
> > > > otServingRegionException: Region is not online: .META.,,1
> > > >
> > > > 4) ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> > > > Inconsist
> > > > ent configuration. Previous configuration for using table name in
> > > metrics:
> > > > true, new configuration: false
> > > >
> > > > 5) Got user-level KeeperEx
> > > > ception when processing sessionid:0x144df32705d0000 type:delete
> > cxid:0xc
> > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> > > > ers/hbase.rummycircle.com,60000,1395314029900 Error:KeeperErrorCode
> =
> > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> > > > 314029900
> > > >
> > > > 3rd and 5th ones getting repeated multiple times.
> > > >
> > > > Please help.
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks and Regards,
> > > > Vimal Jain
> > > >
> > >
> >
> >
> >
> > --
> > Thanks and Regards,
> > Vimal Jain
> >
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Vimal Jain <vk...@gmail.com>.
Hi Stack,
I will delete everything and reconfigure hbase from scratch.
I will then attach logs in pastebin if i find same issues again.
On Fri, Mar 21, 2014 at 11:20 AM, Stack <st...@duboce.net> wrote:
> Put the full logs up in pastebin. The below seem like symptoms, not the
> problem.
> St.Ack
>
>
> On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
>
> > Yes Stack.
> >
> >
> > On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> > distributed
> > > > mode and i am getting below exceptions when starting hbase for first
> > > time.
> > > >
> > > >
> > > You started fresh? Fresh HDFS and ZK in particular?
> > > St.Ack
> > >
> > >
> > >
> > > > Master logs :-
> > > >
> > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> > > > /hbase/backup-m
> > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted,
> and
> > > this
> > > > is not a retry
> > > >
> > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> verification
> > of
> > > > -ROOT
> > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > org.apache.hadoop.hbase.N
> > > > otServingRegionException: Region is not online: -ROOT-,,0
> > > >
> > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed
> verification
> > of
> > > > .META
> > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> > > > org.apache.hadoop.hbase.NotServingRegionException:
> > > > org.apache.hadoop.hbase.N
> > > > otServingRegionException: Region is not online: .META.,,1
> > > >
> > > > 4) ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> > > > Inconsist
> > > > ent configuration. Previous configuration for using table name in
> > > metrics:
> > > > true, new configuration: false
> > > >
> > > > 5) Got user-level KeeperEx
> > > > ception when processing sessionid:0x144df32705d0000 type:delete
> > cxid:0xc
> > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> > > > ers/hbase.rummycircle.com,60000,1395314029900 Error:KeeperErrorCode
> =
> > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> > > > 314029900
> > > >
> > > > 3rd and 5th ones getting repeated multiple times.
> > > >
> > > > Please help.
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks and Regards,
> > > > Vimal Jain
> > > >
> > >
> >
> >
> >
> > --
> > Thanks and Regards,
> > Vimal Jain
> >
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Stack <st...@duboce.net>.
Put the full logs up in pastebin. The below seem like symptoms, not the
problem.
St.Ack
On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <vk...@gmail.com> wrote:
> Yes Stack.
>
>
> On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com> wrote:
> >
> > > Hi,
> > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo
> distributed
> > > mode and i am getting below exceptions when starting hbase for first
> > time.
> > >
> > >
> > You started fresh? Fresh HDFS and ZK in particular?
> > St.Ack
> >
> >
> >
> > > Master logs :-
> > >
> > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> > > /hbase/backup-m
> > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted, and
> > this
> > > is not a retry
> > >
> > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification
> of
> > > -ROOT
> > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> > > org.apache.hadoop.hbase.NotServingRegionException:
> > > org.apache.hadoop.hbase.N
> > > otServingRegionException: Region is not online: -ROOT-,,0
> > >
> > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification
> of
> > > .META
> > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> > > org.apache.hadoop.hbase.NotServingRegionException:
> > > org.apache.hadoop.hbase.N
> > > otServingRegionException: Region is not online: .META.,,1
> > >
> > > 4) ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> > > Inconsist
> > > ent configuration. Previous configuration for using table name in
> > metrics:
> > > true, new configuration: false
> > >
> > > 5) Got user-level KeeperEx
> > > ception when processing sessionid:0x144df32705d0000 type:delete
> cxid:0xc
> > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> > > ers/hbase.rummycircle.com,60000,1395314029900 Error:KeeperErrorCode =
> > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> > > 314029900
> > >
> > > 3rd and 5th ones getting repeated multiple times.
> > >
> > > Please help.
> > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Vimal Jain
> > >
> >
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
Re: Getting exceptions in fresh installment of hbase
Posted by Vimal Jain <vk...@gmail.com>.
Yes Stack.
On Fri, Mar 21, 2014 at 9:22 AM, Stack <st...@duboce.net> wrote:
> On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com> wrote:
>
> > Hi,
> > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo distributed
> > mode and i am getting below exceptions when starting hbase for first
> time.
> >
> >
> You started fresh? Fresh HDFS and ZK in particular?
> St.Ack
>
>
>
> > Master logs :-
> >
> > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> > /hbase/backup-m
> > asters/hbase.rummycircle.com,60000,1395314029900 already deleted, and
> this
> > is not a retry
> >
> > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
> > -ROOT
> > -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> > org.apache.hadoop.hbase.NotServingRegionException:
> > org.apache.hadoop.hbase.N
> > otServingRegionException: Region is not online: -ROOT-,,0
> >
> > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
> > .META
> > .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> > org.apache.hadoop.hbase.NotServingRegionException:
> > org.apache.hadoop.hbase.N
> > otServingRegionException: Region is not online: .META.,,1
> >
> > 4) ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> > Inconsist
> > ent configuration. Previous configuration for using table name in
> metrics:
> > true, new configuration: false
> >
> > 5) Got user-level KeeperEx
> > ception when processing sessionid:0x144df32705d0000 type:delete cxid:0xc
> > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> > ers/hbase.rummycircle.com,60000,1395314029900 Error:KeeperErrorCode =
> > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> > 314029900
> >
> > 3rd and 5th ones getting repeated multiple times.
> >
> > Please help.
> >
> >
> >
> > --
> > Thanks and Regards,
> > Vimal Jain
> >
>
--
Thanks and Regards,
Vimal Jain
Re: Getting exceptions in fresh installment of hbase
Posted by Stack <st...@duboce.net>.
On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo distributed
> mode and i am getting below exceptions when starting hbase for first time.
>
>
You started fresh? Fresh HDFS and ZK in particular?
St.Ack
> Master logs :-
>
> 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/backup-m
> asters/hbase.rummycircle.com,60000,1395314029900 already deleted, and this
> is not a retry
>
> 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
> -ROOT
> -,,0 at address=hbase.rummycircle.com,60020,1395313673569;
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.N
> otServingRegionException: Region is not online: -ROOT-,,0
>
> 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
> .META
> .,,1 at address=hbase.rummycircle.com,60020,1395313673569;
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.N
> otServingRegionException: Region is not online: .META.,,1
>
> 4) ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
> Inconsist
> ent configuration. Previous configuration for using table name in metrics:
> true, new configuration: false
>
> 5) Got user-level KeeperEx
> ception when processing sessionid:0x144df32705d0000 type:delete cxid:0xc
> zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast
> ers/hbase.rummycircle.com,60000,1395314029900 Error:KeeperErrorCode =
> NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395
> 314029900
>
> 3rd and 5th ones getting repeated multiple times.
>
> Please help.
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>