You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Marco Gallotta <ma...@gallotta.co.za> on 2012/06/21 02:17:29 UTC

Hbase MasterNotRunningException

Hi there

I am getting a MasterNotRunningException after trying to setup an
Hbase cluster using whirr.

Here's my properties file (pretty standard from the docs, except that
I reduced the number of slaves to 1 for testing):

whirr.cluster-name=hbase
whirr.instance-templates=1
zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,1
hadoop-datanode+hadoop-tasktracker+hbase-regionserver
whirr.provider=aws-ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.hardware-id=m1.xlarge
whirr.image-id=us-east-1/ami-da0cf8b3 # Ubuntu 10.04
whirr.location-id=us-east-1

Then I run the following on one of the instances:

$ cd /usr/local/hbase-0.89.20100924/
$ bin/hbase shell
hbase(main):001:0> status
ERROR: org.apache.hadoop.hbase.MasterNotRunningException: null
...

Any suggestions on what I'm doing wrong?

A couple other questions:
- Is it alright to use small instances during testing? Why doesn't it
work at all on micro instances? I get errors about no hardware
matches.
- Is it alright using Ubuntu 12.04? I had the same issue as above,
after getting whirr to install JDK 7 and use the right paths.

Regards
Marco

-- 
Marco Gallotta
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313

Re: Hbase MasterNotRunningException

Posted by Andrei Savu <sa...@gmail.com>.
It looks like the ZooKeeper server is not available when the HBase master
tries to start.

Unfortunately there is no easy workaround that we can implement now.
Something else you can try is to use a dedicated VM for ZooKeeper & add
some sleep statements in the HBase configure scripts.

-- Andrei Savu / axemblr.com / Tools for Clouds

On Thu, Jun 21, 2012 at 3:28 AM, Marco Gallotta <ma...@gallotta.co.za>wrote:

> Here are some logs: http://pastebin.com/GxDpwEmM
>
> Some extracts:
>
> 2012-06-20 23:54:11,604 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x0 for server null, unexpected error, closing socket connection and
> attempting reconnect
> java.net.ConnectException: Connection refused
>
> 10-6-63-150.ec2.internal,60020,1340236451506>Failed to set watcher on
> ZNode /hbase/master
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> On Wed, Jun 20, 2012 at 5:17 PM, Marco Gallotta <ma...@gallotta.co.za>
> wrote:
> > Hi there
> >
> > I am getting a MasterNotRunningException after trying to setup an
> > Hbase cluster using whirr.
> >
> > Here's my properties file (pretty standard from the docs, except that
> > I reduced the number of slaves to 1 for testing):
> >
> > whirr.cluster-name=hbase
> > whirr.instance-templates=1
> > zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,1
> > hadoop-datanode+hadoop-tasktracker+hbase-regionserver
> > whirr.provider=aws-ec2
> > whirr.identity=${env:AWS_ACCESS_KEY_ID}
> > whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
> > whirr.hardware-id=m1.xlarge
> > whirr.image-id=us-east-1/ami-da0cf8b3 # Ubuntu 10.04
> > whirr.location-id=us-east-1
> >
> > Then I run the following on one of the instances:
> >
> > $ cd /usr/local/hbase-0.89.20100924/
> > $ bin/hbase shell
> > hbase(main):001:0> status
> > ERROR: org.apache.hadoop.hbase.MasterNotRunningException: null
> > ...
> >
> > Any suggestions on what I'm doing wrong?
> >
> > A couple other questions:
> > - Is it alright to use small instances during testing? Why doesn't it
> > work at all on micro instances? I get errors about no hardware
> > matches.
> > - Is it alright using Ubuntu 12.04? I had the same issue as above,
> > after getting whirr to install JDK 7 and use the right paths.
> >
> > Regards
> > Marco
> >
> > --
> > Marco Gallotta
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta | twitter.com/marcog
> > marco@gallotta.co.za | +1 (650) 417-3313
>
>
>
> --
> Marco Gallotta
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>

Re: Hbase MasterNotRunningException

Posted by Marco Gallotta <ma...@gallotta.co.za>.
Here are some logs: http://pastebin.com/GxDpwEmM

Some extracts:

2012-06-20 23:54:11,604 WARN org.apache.zookeeper.ClientCnxn: Session
0x0 for server null, unexpected error, closing socket connection and
attempting reconnect
java.net.ConnectException: Connection refused

10-6-63-150.ec2.internal,60020,1340236451506>Failed to set watcher on
ZNode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master

On Wed, Jun 20, 2012 at 5:17 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> Hi there
>
> I am getting a MasterNotRunningException after trying to setup an
> Hbase cluster using whirr.
>
> Here's my properties file (pretty standard from the docs, except that
> I reduced the number of slaves to 1 for testing):
>
> whirr.cluster-name=hbase
> whirr.instance-templates=1
> zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,1
> hadoop-datanode+hadoop-tasktracker+hbase-regionserver
> whirr.provider=aws-ec2
> whirr.identity=${env:AWS_ACCESS_KEY_ID}
> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
> whirr.hardware-id=m1.xlarge
> whirr.image-id=us-east-1/ami-da0cf8b3 # Ubuntu 10.04
> whirr.location-id=us-east-1
>
> Then I run the following on one of the instances:
>
> $ cd /usr/local/hbase-0.89.20100924/
> $ bin/hbase shell
> hbase(main):001:0> status
> ERROR: org.apache.hadoop.hbase.MasterNotRunningException: null
> ...
>
> Any suggestions on what I'm doing wrong?
>
> A couple other questions:
> - Is it alright to use small instances during testing? Why doesn't it
> work at all on micro instances? I get errors about no hardware
> matches.
> - Is it alright using Ubuntu 12.04? I had the same issue as above,
> after getting whirr to install JDK 7 and use the right paths.
>
> Regards
> Marco
>
> --
> Marco Gallotta
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313



-- 
Marco Gallotta
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313

Re: Hbase MasterNotRunningException

Posted by Andrew Purtell <ap...@apache.org>.
Why use HBase 0.89 instead of 0.94? Or even 0.92.1? I think we would
be interested if this happened with a current release, and also
suspect it won't be an issue with those.

On Fri, Jun 22, 2012 at 2:36 AM, Andrei Savu <sa...@gmail.com> wrote:
> I am out of ideas here. I will perform some more testing over the next
> couple of days and get back to you.
>
> -- Andrei Savu / axemblr.com / Tools for Clouds
>
> On Fri, Jun 22, 2012 at 3:37 AM, Marco Gallotta <ma...@gallotta.co.za>
> wrote:
>>
>> On Thu, Jun 21, 2012 at 4:30 PM, Andrei Savu <sa...@gmail.com>
>> wrote:
>> > Looks like a timing issue to me. Can you retry with an Ubuntu 10.04 LTS
>> > ami?
>>
>> That was on 10.04.
>>
>> > t1.micro is just too small to run a bunch of Java daemons. From my
>> > experience I think you need at least m1.small.
>>
>> Ok. m1.small works on head, but not in the latest release (only
>> m1.xlarge and up work) so I'm using head for now.
>>
>> > It looks like the ZooKeeper server is not available when the HBase
>> > master
>> > tries to start.
>> >
>> > Unfortunately there is no easy workaround that we can implement now.
>> > Something else you can try is to use a dedicated VM for ZooKeeper & add
>> > some
>> > sleep statements in the HBase configure scripts.
>>
>> I tried this. Change the template definition to:
>>
>> whirr.instance-templates=1 zookeeper,1
>> hadoop-namenode+hadoop-jobtracker+hbase-master,1
>> hadoop-datanode+hadoop-tasktracker+hbase-regionserver
>>
>> and added a sleep of 60 seconds before configuring hbase. That didn't
>> work. I've also tried starting hbase master manually and that throws
>> the errors below. Main extract is: java.lang.ClassNotFoundException:
>> org.apache.commons.configuration.Configuration
>>
>> http://pastebin.com/ufRcdPee
>>
>> Appreciate the help.
>>
>> Marco
>>
>> --
>> Marco Gallotta
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> marco@gallotta.co.za | +1 (650) 417-3313
>
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)

Re: Hbase MasterNotRunningException

Posted by Andrei Savu <sa...@gmail.com>.
I am out of ideas here. I will perform some more testing over the next
couple of days and get back to you.

-- Andrei Savu / axemblr.com / Tools for Clouds

On Fri, Jun 22, 2012 at 3:37 AM, Marco Gallotta <ma...@gallotta.co.za>wrote:

> On Thu, Jun 21, 2012 at 4:30 PM, Andrei Savu <sa...@gmail.com>
> wrote:
> > Looks like a timing issue to me. Can you retry with an Ubuntu 10.04 LTS
> ami?
>
> That was on 10.04.
>
> > t1.micro is just too small to run a bunch of Java daemons. From my
> > experience I think you need at least m1.small.
>
> Ok. m1.small works on head, but not in the latest release (only
> m1.xlarge and up work) so I'm using head for now.
>
> > It looks like the ZooKeeper server is not available when the HBase master
> > tries to start.
> >
> > Unfortunately there is no easy workaround that we can implement now.
> > Something else you can try is to use a dedicated VM for ZooKeeper & add
> some
> > sleep statements in the HBase configure scripts.
>
> I tried this. Change the template definition to:
>
> whirr.instance-templates=1 zookeeper,1
> hadoop-namenode+hadoop-jobtracker+hbase-master,1
> hadoop-datanode+hadoop-tasktracker+hbase-regionserver
>
> and added a sleep of 60 seconds before configuring hbase. That didn't
> work. I've also tried starting hbase master manually and that throws
> the errors below. Main extract is: java.lang.ClassNotFoundException:
> org.apache.commons.configuration.Configuration
>
> http://pastebin.com/ufRcdPee
>
> Appreciate the help.
>
> Marco
>
> --
> Marco Gallotta
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>

Re: Hbase MasterNotRunningException

Posted by Marco Gallotta <ma...@gallotta.co.za>.
On Thu, Jun 21, 2012 at 4:30 PM, Andrei Savu <sa...@gmail.com> wrote:
> Looks like a timing issue to me. Can you retry with an Ubuntu 10.04 LTS ami?

That was on 10.04.

> t1.micro is just too small to run a bunch of Java daemons. From my
> experience I think you need at least m1.small.

Ok. m1.small works on head, but not in the latest release (only
m1.xlarge and up work) so I'm using head for now.

> It looks like the ZooKeeper server is not available when the HBase master
> tries to start.
>
> Unfortunately there is no easy workaround that we can implement now.
> Something else you can try is to use a dedicated VM for ZooKeeper & add some
> sleep statements in the HBase configure scripts.

I tried this. Change the template definition to:

whirr.instance-templates=1 zookeeper,1
hadoop-namenode+hadoop-jobtracker+hbase-master,1
hadoop-datanode+hadoop-tasktracker+hbase-regionserver

and added a sleep of 60 seconds before configuring hbase. That didn't
work. I've also tried starting hbase master manually and that throws
the errors below. Main extract is: java.lang.ClassNotFoundException:
org.apache.commons.configuration.Configuration

http://pastebin.com/ufRcdPee

Appreciate the help.

Marco

-- 
Marco Gallotta
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313

Re: Hbase MasterNotRunningException

Posted by Andrei Savu <sa...@gmail.com>.
On Thu, Jun 21, 2012 at 3:17 AM, Marco Gallotta <ma...@gallotta.co.za>wrote:

> Any suggestions on what I'm doing wrong?
>

Looks like a timing issue to me. Can you retry with an Ubuntu 10.04 LTS ami?

See: http://cloud.ubuntu.com/ami/


>
> A couple other questions:
> - Is it alright to use small instances during testing? Why doesn't it
> work at all on micro instances? I get errors about no hardware
> matches.
>

t1.micro is just too small to run a bunch of Java daemons. From my
experience I think you need at least m1.small.


>  - Is it alright using Ubuntu 12.04? I had the same issue as above,
> after getting whirr to install JDK 7 and use the right paths.
>

We have tested on Ubuntu 10.04 LTS with JDK 6 (OpenJDK).

-- Andrei Savu / axemblr.com / Tools for Clouds