You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hari Sreekumar <hs...@clickable.com> on 2010/11/21 19:22:05 UTC
Question about Zookeeper quorum
Hi,
Is it necessary that all RegionServers must also be part of the ZK
Quorum? I have a 4 node cluster, with node hadoop1 being master and hadoop2,
hadoop3 and ejabber being the slaves (Both in case of hadoop and for HBase).
When I keep only 3 nodes in the zookeeper.quorum property:
<name>hbase.zookeeper.quorum</name>
<value>hadoop1,hadoop2,hadoop3</value>
I get this exception for all tasks that run on ejabber(the 4th node):
2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server localhost/127.0.0.1:2181
2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-11-21 23:35:47,925 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase
-- check quorum servers, currently=localhost:2181
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
When I add ejabber also to the ZK quorum and restart HBase, I don't get this
exception. My understanding was that a small cluster like mine should only
need one ZK machine.
Thanks,
Hari
Re: which HBase version to use?
Posted by Gary Helmling <gh...@gmail.com>.
Friso,
If you're running the SU HBase on cdh3b3 hadoop, make sure the SU branch
includes the patch for HBASE-3194 or be sure to apply it yourself. Without
it you'll get compilation errors due to the security changes in cdh3b3.
Gary
On Tue, Nov 23, 2010 at 12:26 AM, Friso van Vollenhoven <
fvanvollenhoven@xebia.com> wrote:
> Hi All,
> Thanks for all the feedback. Because I need a 'works right now' version, I
> am going to go for 0.89-<somtehing> with some patches applied (basically
> SU's version on top of CDH3b3 Hadoop), with a planned upgrade path to CDH3
> when it reaches b4 or final (or any state that I have time for to test on
> our dev boxes).
>
> Thanks
> Friso
>
>
> On 23 nov 2010, at 02:30, Todd Lipcon wrote:
>
> > On Mon, Nov 22, 2010 at 3:44 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> >>> On Mon, 11/22/10, Todd Lipcon <to...@cloudera.com> wrote:
> >>> Once 0.90 is released, we plan on spending a week or two to suss
> >>> out any possible integration issues, and then release CDH3b4
> >>> including 0.90.
> >>
> >> I'm sure that will make everyone happy. :-) Glad to hear the projected
> time
> >> between releases will be short. That was my concern.
> >>
> >>
> > Yes, a reasonably stable HBase 0.90 is one of the primary gating
> functions
> > for the next beta. The qualification of "reasonably stable" is sometimes
> > hard to quantify, but in general my rubric has been to run a small test
> > cluster under heavy load for 24+ hours with a configuration that stresses
> > splits, compactions, and flushes, and occasionally kill -9 one of the
> nodes.
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>
Re: which HBase version to use?
Posted by Friso van Vollenhoven <fv...@xebia.com>.
Hi All,
Thanks for all the feedback. Because I need a 'works right now' version, I am going to go for 0.89-<somtehing> with some patches applied (basically SU's version on top of CDH3b3 Hadoop), with a planned upgrade path to CDH3 when it reaches b4 or final (or any state that I have time for to test on our dev boxes).
Thanks
Friso
On 23 nov 2010, at 02:30, Todd Lipcon wrote:
> On Mon, Nov 22, 2010 at 3:44 PM, Andrew Purtell <ap...@apache.org> wrote:
>
>>> On Mon, 11/22/10, Todd Lipcon <to...@cloudera.com> wrote:
>>> Once 0.90 is released, we plan on spending a week or two to suss
>>> out any possible integration issues, and then release CDH3b4
>>> including 0.90.
>>
>> I'm sure that will make everyone happy. :-) Glad to hear the projected time
>> between releases will be short. That was my concern.
>>
>>
> Yes, a reasonably stable HBase 0.90 is one of the primary gating functions
> for the next beta. The qualification of "reasonably stable" is sometimes
> hard to quantify, but in general my rubric has been to run a small test
> cluster under heavy load for 24+ hours with a configuration that stresses
> splits, compactions, and flushes, and occasionally kill -9 one of the nodes.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: which HBase version to use?
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Nov 22, 2010 at 3:44 PM, Andrew Purtell <ap...@apache.org> wrote:
> > On Mon, 11/22/10, Todd Lipcon <to...@cloudera.com> wrote:
> > Once 0.90 is released, we plan on spending a week or two to suss
> > out any possible integration issues, and then release CDH3b4
> > including 0.90.
>
> I'm sure that will make everyone happy. :-) Glad to hear the projected time
> between releases will be short. That was my concern.
>
>
Yes, a reasonably stable HBase 0.90 is one of the primary gating functions
for the next beta. The qualification of "reasonably stable" is sometimes
hard to quantify, but in general my rubric has been to run a small test
cluster under heavy load for 24+ hours with a configuration that stresses
splits, compactions, and flushes, and occasionally kill -9 one of the nodes.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: which HBase version to use?
Posted by Andrew Purtell <ap...@apache.org>.
> On Mon, 11/22/10, Todd Lipcon <to...@cloudera.com> wrote:
> Once 0.90 is released, we plan on spending a week or two to suss
> out any possible integration issues, and then release CDH3b4
> including 0.90.
I'm sure that will make everyone happy. :-) Glad to hear the projected time between releases will be short. That was my concern.
> Right now I believe the version in CDH3b3 is more stable
> than the 0.90rc.
Yes I want to clarify that I believe this to be the case as well.
Best regards,
- Andy
Re: which HBase version to use?
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Nov 22, 2010 at 10:55 AM, Andrew Purtell <ap...@apache.org>wrote:
> Right now you can't get 0.90 from CDH3. It is an 0.89-<mumble>. It will not
> be a better choice than 0.90 once 0.90 is released.
>
> We are looking at deploying CDH3B3 plus a custom RPM built in house that
> updates the CDH3B3 HBase package to 0.90.
>
>
Once 0.90 is released, we plan on spending a week or two to suss out any
possible integration issues, and then release CDH3b4 including 0.90.
Right now I believe the version in CDH3b3 is more stable than the 0.90rc.
This is from the testing I've done in-house with our clusters - assignment
issues cropped up in 0.90 which I haven't seen in 0.89.20100924. The next rc
or 0.90.0 should be better, and then we'll update CDH as well.
> I'm not sure we forgo support for our ops team just because we do that. I
> can't comment as an authority on Cloudera's business of course since I do
> not work from them, but as a customer I expect by the letter of the contract
> we cannot (and would not anyway) ask for support for HBase if using a custom
> version, but e.g. any HDFS related trouble is independent and we would be
> very unhappy if told otherwise.
>
Not going to go into specifics of our contracts on a list like this ;-) But
as an engineer I'm always happy to try to help other engineers if they can
explain their problem well - it increases the quality of the software for
everyone, and really isn't that what we're all here for?
-Todd
>
> --- On Mon, 11/22/10, Michael Segel <mi...@hotmail.com> wrote:
>
> > From: Michael Segel <mi...@hotmail.com>
> > Subject: RE: which HBase version to use?
> > To: user@hbase.apache.org
> > Date: Monday, November 22, 2010, 9:15 AM
> >
> > Friso,
> >
> > I don't disagree with Ryan, however, I think you have to
> > determine which makes the most sense.
> >
> > Going with CDH3 you get the RPMs, and you know that
> > everything in CDH3 works together.
> > Essentially one stop shopping and you can purchase
> > production support.
> >
> > If you're looking at a pure Dev cluster, maybe mixing and
> > matching makes sense.
> >
> > If you are going to go with CDH3, skip b2 and go with b3.
> > HBase is more stable.
> > If you're going to use HBase from ASF, then why not also
> > get everything from ASF?
> >
> > The reason we chose Cloudera (not that I'm giving them a
> > free plug. :-) is that Cloudera sells support which is
> > something my client wanted from the start.
> > If you are going to go w Cloudera, they support HBase in
> > CDH3 and you don't want to use ASF because it could violate
> > your support contract.
> >
> > HTH
> >
> > -Mike
> >
> >
> > > Date: Mon, 22 Nov 2010 01:24:30 -0800
> > > Subject: Re: which HBase version to use?
> > > From: ryanobjc@gmail.com
> > > To: user@hbase.apache.org
> > >
> > > I have to recommend ASF 0.90, which is in release
> > candidate mode right
> > > now. You'll want to run it on top of CDH3 b2 or
> > b3, but that is up to
> > > you to decide. The hbase team here at
> > Stumbleupon has more time in on
> > > b2.
> > >
> > > -ryan
> > >
> > > On Mon, Nov 22, 2010 at 1:19 AM, Friso van
> > Vollenhoven
> > > <fv...@xebia.com>
> > wrote:
> > > > Hi list,
> > > >
> > > > I have the opportunity to reinstall a cluster
> > form scratch. I use Hadoop and HBase (not yet any of the
> > other tools, like Pig, Hive, Avro, Thrift, etc.). Now, I
> > wonder what versions to use. CDH3 is nice, because it comes
> > with RPMs out of the box, so the operations people now what
> > to do with it (of course, we can build our own for versions
> > that don't have RPMs). It does appear, however, that CDH is
> > mostly focused on a very good Hadoop / HDFS / MR version and
> > you're better of with the ASF HBase release right now. And
> > there is also the version that SU provides on githug, which
> > has the advantage of being heavily used in a production
> > environment by people who know what they're doing.
> > > >
> > > > Any advise on this anyone?
> > > >
> > > >
> > > > Thanks,
> > > > Friso
> > > >
> > > >
> >
> >
> >
> >
>
>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
RE: which HBase version to use?
Posted by Andrew Purtell <ap...@apache.org>.
Right now you can't get 0.90 from CDH3. It is an 0.89-<mumble>. It will not be a better choice than 0.90 once 0.90 is released.
We are looking at deploying CDH3B3 plus a custom RPM built in house that updates the CDH3B3 HBase package to 0.90.
I'm not sure we forgo support for our ops team just because we do that. I can't comment as an authority on Cloudera's business of course since I do not work from them, but as a customer I expect by the letter of the contract we cannot (and would not anyway) ask for support for HBase if using a custom version, but e.g. any HDFS related trouble is independent and we would be very unhappy if told otherwise.
Best regards,
- Andy
--- On Mon, 11/22/10, Michael Segel <mi...@hotmail.com> wrote:
> From: Michael Segel <mi...@hotmail.com>
> Subject: RE: which HBase version to use?
> To: user@hbase.apache.org
> Date: Monday, November 22, 2010, 9:15 AM
>
> Friso,
>
> I don't disagree with Ryan, however, I think you have to
> determine which makes the most sense.
>
> Going with CDH3 you get the RPMs, and you know that
> everything in CDH3 works together.
> Essentially one stop shopping and you can purchase
> production support.
>
> If you're looking at a pure Dev cluster, maybe mixing and
> matching makes sense.
>
> If you are going to go with CDH3, skip b2 and go with b3.
> HBase is more stable.
> If you're going to use HBase from ASF, then why not also
> get everything from ASF?
>
> The reason we chose Cloudera (not that I'm giving them a
> free plug. :-) is that Cloudera sells support which is
> something my client wanted from the start.
> If you are going to go w Cloudera, they support HBase in
> CDH3 and you don't want to use ASF because it could violate
> your support contract.
>
> HTH
>
> -Mike
>
>
> > Date: Mon, 22 Nov 2010 01:24:30 -0800
> > Subject: Re: which HBase version to use?
> > From: ryanobjc@gmail.com
> > To: user@hbase.apache.org
> >
> > I have to recommend ASF 0.90, which is in release
> candidate mode right
> > now. You'll want to run it on top of CDH3 b2 or
> b3, but that is up to
> > you to decide. The hbase team here at
> Stumbleupon has more time in on
> > b2.
> >
> > -ryan
> >
> > On Mon, Nov 22, 2010 at 1:19 AM, Friso van
> Vollenhoven
> > <fv...@xebia.com>
> wrote:
> > > Hi list,
> > >
> > > I have the opportunity to reinstall a cluster
> form scratch. I use Hadoop and HBase (not yet any of the
> other tools, like Pig, Hive, Avro, Thrift, etc.). Now, I
> wonder what versions to use. CDH3 is nice, because it comes
> with RPMs out of the box, so the operations people now what
> to do with it (of course, we can build our own for versions
> that don't have RPMs). It does appear, however, that CDH is
> mostly focused on a very good Hadoop / HDFS / MR version and
> you're better of with the ASF HBase release right now. And
> there is also the version that SU provides on githug, which
> has the advantage of being heavily used in a production
> environment by people who know what they're doing.
> > >
> > > Any advise on this anyone?
> > >
> > >
> > > Thanks,
> > > Friso
> > >
> > >
>
>
>
>
RE: which HBase version to use?
Posted by Michael Segel <mi...@hotmail.com>.
Friso,
I don't disagree with Ryan, however, I think you have to determine which makes the most sense.
Going with CDH3 you get the RPMs, and you know that everything in CDH3 works together.
Essentially one stop shopping and you can purchase production support.
If you're looking at a pure Dev cluster, maybe mixing and matching makes sense.
If you are going to go with CDH3, skip b2 and go with b3. HBase is more stable.
If you're going to use HBase from ASF, then why not also get everything from ASF?
The reason we chose Cloudera (not that I'm giving them a free plug. :-) is that Cloudera sells support which is something my client wanted from the start.
If you are going to go w Cloudera, they support HBase in CDH3 and you don't want to use ASF because it could violate your support contract.
HTH
-Mike
> Date: Mon, 22 Nov 2010 01:24:30 -0800
> Subject: Re: which HBase version to use?
> From: ryanobjc@gmail.com
> To: user@hbase.apache.org
>
> I have to recommend ASF 0.90, which is in release candidate mode right
> now. You'll want to run it on top of CDH3 b2 or b3, but that is up to
> you to decide. The hbase team here at Stumbleupon has more time in on
> b2.
>
> -ryan
>
> On Mon, Nov 22, 2010 at 1:19 AM, Friso van Vollenhoven
> <fv...@xebia.com> wrote:
> > Hi list,
> >
> > I have the opportunity to reinstall a cluster form scratch. I use Hadoop and HBase (not yet any of the other tools, like Pig, Hive, Avro, Thrift, etc.). Now, I wonder what versions to use. CDH3 is nice, because it comes with RPMs out of the box, so the operations people now what to do with it (of course, we can build our own for versions that don't have RPMs). It does appear, however, that CDH is mostly focused on a very good Hadoop / HDFS / MR version and you're better of with the ASF HBase release right now. And there is also the version that SU provides on githug, which has the advantage of being heavily used in a production environment by people who know what they're doing.
> >
> > Any advise on this anyone?
> >
> >
> > Thanks,
> > Friso
> >
> >
Re: which HBase version to use?
Posted by Ryan Rawson <ry...@gmail.com>.
I have to recommend ASF 0.90, which is in release candidate mode right
now. You'll want to run it on top of CDH3 b2 or b3, but that is up to
you to decide. The hbase team here at Stumbleupon has more time in on
b2.
-ryan
On Mon, Nov 22, 2010 at 1:19 AM, Friso van Vollenhoven
<fv...@xebia.com> wrote:
> Hi list,
>
> I have the opportunity to reinstall a cluster form scratch. I use Hadoop and HBase (not yet any of the other tools, like Pig, Hive, Avro, Thrift, etc.). Now, I wonder what versions to use. CDH3 is nice, because it comes with RPMs out of the box, so the operations people now what to do with it (of course, we can build our own for versions that don't have RPMs). It does appear, however, that CDH is mostly focused on a very good Hadoop / HDFS / MR version and you're better of with the ASF HBase release right now. And there is also the version that SU provides on githug, which has the advantage of being heavily used in a production environment by people who know what they're doing.
>
> Any advise on this anyone?
>
>
> Thanks,
> Friso
>
>
which HBase version to use?
Posted by Friso van Vollenhoven <fv...@xebia.com>.
Hi list,
I have the opportunity to reinstall a cluster form scratch. I use Hadoop and HBase (not yet any of the other tools, like Pig, Hive, Avro, Thrift, etc.). Now, I wonder what versions to use. CDH3 is nice, because it comes with RPMs out of the box, so the operations people now what to do with it (of course, we can build our own for versions that don't have RPMs). It does appear, however, that CDH is mostly focused on a very good Hadoop / HDFS / MR version and you're better of with the ASF HBase release right now. And there is also the version that SU provides on githug, which has the advantage of being heavily used in a production environment by people who know what they're doing.
Any advise on this anyone?
Thanks,
Friso
Re: Question about Zookeeper quorum
Posted by Lars George <la...@gmail.com>.
Glad to hear that solved it :)
HBaseConfiguration actually reads both as it simply extends the Hadoop
Configuration class. Best of both worlds.
Lars
On Mon, Nov 22, 2010 at 1:02 PM, Hari Sreekumar
<hs...@clickable.com> wrote:
> Hi Lars,
>
> Aaah, Thanks man, it works now! I was initiating the default config
> all the while! Printing out the config turns was really helpful. So
> getConf() reads the hadoop default and core-site.xml files, and
> HBaseConfiguration() reads hbase properties.
>
> Thanks a lot,
> Hari
>
> On Mon, Nov 22, 2010 at 4:41 PM, Lars George <la...@gmail.com> wrote:
>
>> Hi Hari,
>>
>> getConf() returns a Hadoop configuration instance, that does not
>> include HBase's. You need to call
>>
>> Configuration conf = HBaseConfiguration.create();
>> Job job = new Job(conf);
>>
>> for it to be read.
>>
>> Lars
>>
>>
>> On Mon, Nov 22, 2010 at 11:32 AM, Hari Sreekumar
>> <hs...@clickable.com> wrote:
>> > Hi,
>> > I tried using this code to read configuration (I am using Toolrunner) :
>> >
>> > public int run(String[] args) throws Exception {
>> > String fileName = args[0];
>> > Path inputPath = new Path(args[0]);
>> >
>> > Job job = new Job(getConf());
>> > System.out.println("Conf_hbase: " +
>> > getConf().get("hbase.zookeeper.quorum"));
>> > System.out.println("Conf_hadoop: " +
>> > getConf().get("dfs.replication"));
>> >
>> > job.setJarByClass(BulkUpload.class);
>> > FileInputFormat.setInputPaths(job, inputPath);
>> > job.setJobName(NAME + "_" + fileName);
>> >
>> > job.setMapperClass(BulkUploadMapper.class);
>> > job.setInputFormatClass(TextInputFormat.class);
>> >
>> > TableMapReduceUtil.initTableReducerJob("CustomerData", null, job);
>> > //System.out.println("")
>> > job.setNumReduceTasks(0);
>> >
>> > boolean success = job.waitForCompletion(true);
>> > return success ? 0 : 1;
>> > }
>> >
>> > Is this code right to get the configuration? Because I am getting both
>> > dfs.replication and hbase.zookeeper.quorum as null this way. But when I
>> > change dfs.replication in my config file, I do see a change in
>> replication
>> > after upload into HDFS.
>> >
>> > On Mon, Nov 22, 2010 at 2:36 PM, Hari Sreekumar <
>> hsreekumar@clickable.com>wrote:
>> >
>> >> Hi Lars,
>> >>
>> >> I start them through HBase implicitly. I'll try printing the
>> config
>> >> values and post.
>> >>
>> >> thanks,
>> >> Hari
>> >>
>> >>
>> >> On Mon, Nov 22, 2010 at 1:58 PM, Lars George <lars.george@gmail.com
>> >wrote:
>> >>
>> >>> Hi Hari,
>> >>>
>> >>> Are you starting them yourself or have HBase start them for you
>> >>> implicitly?
>> >>>
>> >>> Lars
>> >>>
>> >>> On Nov 22, 2010, at 6:19, Hari Sreekumar <hs...@clickable.com>
>> >>> wrote:
>> >>>
>> >>> > Hey Lars,
>> >>> >
>> >>> > I have HQuorumPeer running on all nodes that I specify in my
>> >>> > hbase-site file. One thing I wanted to clarify.. what is the default
>> >>> value
>> >>> > of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in
>> my
>> >>> > hbase-env.sh file.
>> >>> >
>> >>> > thanks,
>> >>> > hari
>> >>> >
>> >>> > On Mon, Nov 22, 2010 at 10:39 AM, Lars George <lars.george@gmail.com
>> >
>> >>> wrote:
>> >>> >
>> >>> >> Hi Hari,
>> >>> >>
>> >>> >> On which of these for machines do you have a ZooKeeper daemon
>> running
>> >>> as
>> >>> >> well?
>> >>> >>
>> >>> >> Lars
>> >>> >>
>> >>> >> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
>> >>> >> <hs...@clickable.com> wrote:
>> >>> >>> Hi,
>> >>> >>>
>> >>> >>> But it is reading settings from hbase-site.xml. If it was not
>> >>> reading
>> >>> >> my
>> >>> >>> changes, the problem wouldn't have gotten fixed when I add ejabber
>> to
>> >>> the
>> >>> >>> quroum right? After all, it is responding to changes I make in my
>> xml
>> >>> >> file.
>> >>> >>> What else can be the issue here?
>> >>> >>>
>> >>> >>> hari
>> >>> >>>
>> >>> >>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <
>> lars.george@gmail.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>>> Hi Hari,
>> >>> >>>>
>> >>> >>>> You are missing the quorum setting. It seems the hbase-site.xml is
>> >>> >> missing
>> >>> >>>> from the classpath on the clients. Did you pack it into the jar?
>> >>> >>>>
>> >>> >>>> And yes, even one ZK server is fine in such a small cluster.
>> >>> >>>>
>> >>> >>>> You can see it is trying to connect to localhost which is the
>> default
>> >>> if
>> >>> >>>> the site file is missing.
>> >>> >>>>
>> >>> >>>> Regards,
>> >>> >>>> Lars
>> >>> >>>>
>> >>> >>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <
>> hsreekumar@clickable.com>
>> >>> >>>> wrote:
>> >>> >>>>
>> >>> >>>>> Hi,
>> >>> >>>>> Is it necessary that all RegionServers must also be part of the
>> ZK
>> >>> >>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master
>> and
>> >>> >>>> hadoop2,
>> >>> >>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and
>> for
>> >>> >>>> HBase).
>> >>> >>>>>
>> >>> >>>>> When I keep only 3 nodes in the zookeeper.quorum property:
>> >>> >>>>> <name>hbase.zookeeper.quorum</name>
>> >>> >>>>> <value>hadoop1,hadoop2,hadoop3</value>
>> >>> >>>>>
>> >>> >>>>> I get this exception for all tasks that run on ejabber(the 4th
>> >>> node):
>> >>> >>>>>
>> >>> >>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
>> >>> >> Attempting
>> >>> >>>>> connection to server localhost/127.0.0.1:2181
>> >>> >>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
>> >>> >> Exception
>> >>> >>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
>> >>> >>>>> java.net.ConnectException: Connection refused
>> >>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> >>> >>>>> at
>> >>> >>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>> >>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>> >>> Ignoring
>> >>> >>>>> exception during shutdown input
>> >>> >>>>> java.nio.channels.ClosedChannelException
>> >>> >>>>> at
>> >>> >>>>>
>> >>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>> >>> >>>>> at
>> >>> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >>> >>>>> at
>> >>> >>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>> >>> Ignoring
>> >>> >>>>> exception during shutdown output
>> >>> >>>>> java.nio.channels.ClosedChannelException
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>> >>> >>>>> at
>> >>> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>
>> >>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> >>> >>>>> at
>> >>> >>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >>> >>>>> 2010-11-21 23:35:47,925 WARN
>> >>> >>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to
>> create
>> >>> >>>> /hbase
>> >>> >>>>> -- check quorum servers, currently=localhost:2181
>> >>> >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>> >>>>> KeeperErrorCode = ConnectionLoss for /hbase
>> >>> >>>>> at
>> >>> >>>>>
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>> >>> >>>>> at
>> >>> >>>>>
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> >>> >>>>> at
>> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>> >>> >>>>> at
>> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>> >>> >>>>>
>> >>> >>>>> When I add ejabber also to the ZK quorum and restart HBase, I
>> don't
>> >>> >> get
>> >>> >>>> this
>> >>> >>>>> exception. My understanding was that a small cluster like mine
>> >>> should
>> >>> >>>> only
>> >>> >>>>> need one ZK machine.
>> >>> >>>>>
>> >>> >>>>> Thanks,
>> >>> >>>>> Hari
>> >>> >>>>
>> >>> >>>
>> >>> >>
>> >>>
>> >>
>> >>
>> >
>>
>
Re: Question about Zookeeper quorum
Posted by Hari Sreekumar <hs...@clickable.com>.
Hi Lars,
Aaah, Thanks man, it works now! I was initiating the default config
all the while! Printing out the config turns was really helpful. So
getConf() reads the hadoop default and core-site.xml files, and
HBaseConfiguration() reads hbase properties.
Thanks a lot,
Hari
On Mon, Nov 22, 2010 at 4:41 PM, Lars George <la...@gmail.com> wrote:
> Hi Hari,
>
> getConf() returns a Hadoop configuration instance, that does not
> include HBase's. You need to call
>
> Configuration conf = HBaseConfiguration.create();
> Job job = new Job(conf);
>
> for it to be read.
>
> Lars
>
>
> On Mon, Nov 22, 2010 at 11:32 AM, Hari Sreekumar
> <hs...@clickable.com> wrote:
> > Hi,
> > I tried using this code to read configuration (I am using Toolrunner) :
> >
> > public int run(String[] args) throws Exception {
> > String fileName = args[0];
> > Path inputPath = new Path(args[0]);
> >
> > Job job = new Job(getConf());
> > System.out.println("Conf_hbase: " +
> > getConf().get("hbase.zookeeper.quorum"));
> > System.out.println("Conf_hadoop: " +
> > getConf().get("dfs.replication"));
> >
> > job.setJarByClass(BulkUpload.class);
> > FileInputFormat.setInputPaths(job, inputPath);
> > job.setJobName(NAME + "_" + fileName);
> >
> > job.setMapperClass(BulkUploadMapper.class);
> > job.setInputFormatClass(TextInputFormat.class);
> >
> > TableMapReduceUtil.initTableReducerJob("CustomerData", null, job);
> > //System.out.println("")
> > job.setNumReduceTasks(0);
> >
> > boolean success = job.waitForCompletion(true);
> > return success ? 0 : 1;
> > }
> >
> > Is this code right to get the configuration? Because I am getting both
> > dfs.replication and hbase.zookeeper.quorum as null this way. But when I
> > change dfs.replication in my config file, I do see a change in
> replication
> > after upload into HDFS.
> >
> > On Mon, Nov 22, 2010 at 2:36 PM, Hari Sreekumar <
> hsreekumar@clickable.com>wrote:
> >
> >> Hi Lars,
> >>
> >> I start them through HBase implicitly. I'll try printing the
> config
> >> values and post.
> >>
> >> thanks,
> >> Hari
> >>
> >>
> >> On Mon, Nov 22, 2010 at 1:58 PM, Lars George <lars.george@gmail.com
> >wrote:
> >>
> >>> Hi Hari,
> >>>
> >>> Are you starting them yourself or have HBase start them for you
> >>> implicitly?
> >>>
> >>> Lars
> >>>
> >>> On Nov 22, 2010, at 6:19, Hari Sreekumar <hs...@clickable.com>
> >>> wrote:
> >>>
> >>> > Hey Lars,
> >>> >
> >>> > I have HQuorumPeer running on all nodes that I specify in my
> >>> > hbase-site file. One thing I wanted to clarify.. what is the default
> >>> value
> >>> > of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in
> my
> >>> > hbase-env.sh file.
> >>> >
> >>> > thanks,
> >>> > hari
> >>> >
> >>> > On Mon, Nov 22, 2010 at 10:39 AM, Lars George <lars.george@gmail.com
> >
> >>> wrote:
> >>> >
> >>> >> Hi Hari,
> >>> >>
> >>> >> On which of these for machines do you have a ZooKeeper daemon
> running
> >>> as
> >>> >> well?
> >>> >>
> >>> >> Lars
> >>> >>
> >>> >> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
> >>> >> <hs...@clickable.com> wrote:
> >>> >>> Hi,
> >>> >>>
> >>> >>> But it is reading settings from hbase-site.xml. If it was not
> >>> reading
> >>> >> my
> >>> >>> changes, the problem wouldn't have gotten fixed when I add ejabber
> to
> >>> the
> >>> >>> quroum right? After all, it is responding to changes I make in my
> xml
> >>> >> file.
> >>> >>> What else can be the issue here?
> >>> >>>
> >>> >>> hari
> >>> >>>
> >>> >>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <
> lars.george@gmail.com>
> >>> >> wrote:
> >>> >>>
> >>> >>>> Hi Hari,
> >>> >>>>
> >>> >>>> You are missing the quorum setting. It seems the hbase-site.xml is
> >>> >> missing
> >>> >>>> from the classpath on the clients. Did you pack it into the jar?
> >>> >>>>
> >>> >>>> And yes, even one ZK server is fine in such a small cluster.
> >>> >>>>
> >>> >>>> You can see it is trying to connect to localhost which is the
> default
> >>> if
> >>> >>>> the site file is missing.
> >>> >>>>
> >>> >>>> Regards,
> >>> >>>> Lars
> >>> >>>>
> >>> >>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <
> hsreekumar@clickable.com>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> Hi,
> >>> >>>>> Is it necessary that all RegionServers must also be part of the
> ZK
> >>> >>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master
> and
> >>> >>>> hadoop2,
> >>> >>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and
> for
> >>> >>>> HBase).
> >>> >>>>>
> >>> >>>>> When I keep only 3 nodes in the zookeeper.quorum property:
> >>> >>>>> <name>hbase.zookeeper.quorum</name>
> >>> >>>>> <value>hadoop1,hadoop2,hadoop3</value>
> >>> >>>>>
> >>> >>>>> I get this exception for all tasks that run on ejabber(the 4th
> >>> node):
> >>> >>>>>
> >>> >>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
> >>> >> Attempting
> >>> >>>>> connection to server localhost/127.0.0.1:2181
> >>> >>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
> >>> >> Exception
> >>> >>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> >>> >>>>> java.net.ConnectException: Connection refused
> >>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>> >>>>> at
> >>> >>>>>
> >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >>> >>>>> at
> >>> >>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> >>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
> >>> Ignoring
> >>> >>>>> exception during shutdown input
> >>> >>>>> java.nio.channels.ClosedChannelException
> >>> >>>>> at
> >>> >>>>>
> >>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> >>> >>>>> at
> >>> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >>> >>>>> at
> >>> >>>>>
> >>> >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >>> >>>>> at
> >>> >>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
> >>> Ignoring
> >>> >>>>> exception during shutdown output
> >>> >>>>> java.nio.channels.ClosedChannelException
> >>> >>>>> at
> >>> >>>>>
> >>> >>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> >>> >>>>> at
> >>> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> >>> >>>>> at
> >>> >>>>>
> >>> >>
> >>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> >>> >>>>> at
> >>> >>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >>> >>>>> 2010-11-21 23:35:47,925 WARN
> >>> >>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to
> create
> >>> >>>> /hbase
> >>> >>>>> -- check quorum servers, currently=localhost:2181
> >>> >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>> >>>>> KeeperErrorCode = ConnectionLoss for /hbase
> >>> >>>>> at
> >>> >>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> >>> >>>>> at
> >>> >>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> >>> >>>>> at
> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> >>> >>>>> at
> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
> >>> >>>>>
> >>> >>>>> When I add ejabber also to the ZK quorum and restart HBase, I
> don't
> >>> >> get
> >>> >>>> this
> >>> >>>>> exception. My understanding was that a small cluster like mine
> >>> should
> >>> >>>> only
> >>> >>>>> need one ZK machine.
> >>> >>>>>
> >>> >>>>> Thanks,
> >>> >>>>> Hari
> >>> >>>>
> >>> >>>
> >>> >>
> >>>
> >>
> >>
> >
>
Re: Question about Zookeeper quorum
Posted by Lars George <la...@gmail.com>.
Hi Hari,
getConf() returns a Hadoop configuration instance, that does not
include HBase's. You need to call
Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf);
for it to be read.
Lars
On Mon, Nov 22, 2010 at 11:32 AM, Hari Sreekumar
<hs...@clickable.com> wrote:
> Hi,
> I tried using this code to read configuration (I am using Toolrunner) :
>
> public int run(String[] args) throws Exception {
> String fileName = args[0];
> Path inputPath = new Path(args[0]);
>
> Job job = new Job(getConf());
> System.out.println("Conf_hbase: " +
> getConf().get("hbase.zookeeper.quorum"));
> System.out.println("Conf_hadoop: " +
> getConf().get("dfs.replication"));
>
> job.setJarByClass(BulkUpload.class);
> FileInputFormat.setInputPaths(job, inputPath);
> job.setJobName(NAME + "_" + fileName);
>
> job.setMapperClass(BulkUploadMapper.class);
> job.setInputFormatClass(TextInputFormat.class);
>
> TableMapReduceUtil.initTableReducerJob("CustomerData", null, job);
> //System.out.println("")
> job.setNumReduceTasks(0);
>
> boolean success = job.waitForCompletion(true);
> return success ? 0 : 1;
> }
>
> Is this code right to get the configuration? Because I am getting both
> dfs.replication and hbase.zookeeper.quorum as null this way. But when I
> change dfs.replication in my config file, I do see a change in replication
> after upload into HDFS.
>
> On Mon, Nov 22, 2010 at 2:36 PM, Hari Sreekumar <hs...@clickable.com>wrote:
>
>> Hi Lars,
>>
>> I start them through HBase implicitly. I'll try printing the config
>> values and post.
>>
>> thanks,
>> Hari
>>
>>
>> On Mon, Nov 22, 2010 at 1:58 PM, Lars George <la...@gmail.com>wrote:
>>
>>> Hi Hari,
>>>
>>> Are you starting them yourself or have HBase start them for you
>>> implicitly?
>>>
>>> Lars
>>>
>>> On Nov 22, 2010, at 6:19, Hari Sreekumar <hs...@clickable.com>
>>> wrote:
>>>
>>> > Hey Lars,
>>> >
>>> > I have HQuorumPeer running on all nodes that I specify in my
>>> > hbase-site file. One thing I wanted to clarify.. what is the default
>>> value
>>> > of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my
>>> > hbase-env.sh file.
>>> >
>>> > thanks,
>>> > hari
>>> >
>>> > On Mon, Nov 22, 2010 at 10:39 AM, Lars George <la...@gmail.com>
>>> wrote:
>>> >
>>> >> Hi Hari,
>>> >>
>>> >> On which of these for machines do you have a ZooKeeper daemon running
>>> as
>>> >> well?
>>> >>
>>> >> Lars
>>> >>
>>> >> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
>>> >> <hs...@clickable.com> wrote:
>>> >>> Hi,
>>> >>>
>>> >>> But it is reading settings from hbase-site.xml. If it was not
>>> reading
>>> >> my
>>> >>> changes, the problem wouldn't have gotten fixed when I add ejabber to
>>> the
>>> >>> quroum right? After all, it is responding to changes I make in my xml
>>> >> file.
>>> >>> What else can be the issue here?
>>> >>>
>>> >>> hari
>>> >>>
>>> >>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>>> Hi Hari,
>>> >>>>
>>> >>>> You are missing the quorum setting. It seems the hbase-site.xml is
>>> >> missing
>>> >>>> from the classpath on the clients. Did you pack it into the jar?
>>> >>>>
>>> >>>> And yes, even one ZK server is fine in such a small cluster.
>>> >>>>
>>> >>>> You can see it is trying to connect to localhost which is the default
>>> if
>>> >>>> the site file is missing.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Lars
>>> >>>>
>>> >>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Hi,
>>> >>>>> Is it necessary that all RegionServers must also be part of the ZK
>>> >>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master and
>>> >>>> hadoop2,
>>> >>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and for
>>> >>>> HBase).
>>> >>>>>
>>> >>>>> When I keep only 3 nodes in the zookeeper.quorum property:
>>> >>>>> <name>hbase.zookeeper.quorum</name>
>>> >>>>> <value>hadoop1,hadoop2,hadoop3</value>
>>> >>>>>
>>> >>>>> I get this exception for all tasks that run on ejabber(the 4th
>>> node):
>>> >>>>>
>>> >>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
>>> >> Attempting
>>> >>>>> connection to server localhost/127.0.0.1:2181
>>> >>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
>>> >> Exception
>>> >>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
>>> >>>>> java.net.ConnectException: Connection refused
>>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> >>>>> at
>>> >>>>>
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>> >>>>> at
>>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>>> Ignoring
>>> >>>>> exception during shutdown input
>>> >>>>> java.nio.channels.ClosedChannelException
>>> >>>>> at
>>> >>>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>> >>>>> at
>>> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>> >>>>> at
>>> >>>>>
>>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>> >>>>> at
>>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>>> Ignoring
>>> >>>>> exception during shutdown output
>>> >>>>> java.nio.channels.ClosedChannelException
>>> >>>>> at
>>> >>>>>
>>> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>>> >>>>> at
>>> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>> >>>>> at
>>> >>>>>
>>> >>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>> >>>>> at
>>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>> >>>>> 2010-11-21 23:35:47,925 WARN
>>> >>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>>> >>>> /hbase
>>> >>>>> -- check quorum servers, currently=localhost:2181
>>> >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> >>>>> KeeperErrorCode = ConnectionLoss for /hbase
>>> >>>>> at
>>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>> >>>>> at
>>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>>> >>>>> at
>>> >>>>>
>>> >>>>
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
>>> >>>>> at
>>> >>>>>
>>> >>>>
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
>>> >>>>> at
>>> >>>>>
>>> >>>>
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>>> >>>>>
>>> >>>>> When I add ejabber also to the ZK quorum and restart HBase, I don't
>>> >> get
>>> >>>> this
>>> >>>>> exception. My understanding was that a small cluster like mine
>>> should
>>> >>>> only
>>> >>>>> need one ZK machine.
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Hari
>>> >>>>
>>> >>>
>>> >>
>>>
>>
>>
>
Re: Question about Zookeeper quorum
Posted by Hari Sreekumar <hs...@clickable.com>.
Hi,
I tried using this code to read configuration (I am using Toolrunner) :
public int run(String[] args) throws Exception {
String fileName = args[0];
Path inputPath = new Path(args[0]);
Job job = new Job(getConf());
System.out.println("Conf_hbase: " +
getConf().get("hbase.zookeeper.quorum"));
System.out.println("Conf_hadoop: " +
getConf().get("dfs.replication"));
job.setJarByClass(BulkUpload.class);
FileInputFormat.setInputPaths(job, inputPath);
job.setJobName(NAME + "_" + fileName);
job.setMapperClass(BulkUploadMapper.class);
job.setInputFormatClass(TextInputFormat.class);
TableMapReduceUtil.initTableReducerJob("CustomerData", null, job);
//System.out.println("")
job.setNumReduceTasks(0);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
Is this code right to get the configuration? Because I am getting both
dfs.replication and hbase.zookeeper.quorum as null this way. But when I
change dfs.replication in my config file, I do see a change in replication
after upload into HDFS.
On Mon, Nov 22, 2010 at 2:36 PM, Hari Sreekumar <hs...@clickable.com>wrote:
> Hi Lars,
>
> I start them through HBase implicitly. I'll try printing the config
> values and post.
>
> thanks,
> Hari
>
>
> On Mon, Nov 22, 2010 at 1:58 PM, Lars George <la...@gmail.com>wrote:
>
>> Hi Hari,
>>
>> Are you starting them yourself or have HBase start them for you
>> implicitly?
>>
>> Lars
>>
>> On Nov 22, 2010, at 6:19, Hari Sreekumar <hs...@clickable.com>
>> wrote:
>>
>> > Hey Lars,
>> >
>> > I have HQuorumPeer running on all nodes that I specify in my
>> > hbase-site file. One thing I wanted to clarify.. what is the default
>> value
>> > of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my
>> > hbase-env.sh file.
>> >
>> > thanks,
>> > hari
>> >
>> > On Mon, Nov 22, 2010 at 10:39 AM, Lars George <la...@gmail.com>
>> wrote:
>> >
>> >> Hi Hari,
>> >>
>> >> On which of these for machines do you have a ZooKeeper daemon running
>> as
>> >> well?
>> >>
>> >> Lars
>> >>
>> >> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
>> >> <hs...@clickable.com> wrote:
>> >>> Hi,
>> >>>
>> >>> But it is reading settings from hbase-site.xml. If it was not
>> reading
>> >> my
>> >>> changes, the problem wouldn't have gotten fixed when I add ejabber to
>> the
>> >>> quroum right? After all, it is responding to changes I make in my xml
>> >> file.
>> >>> What else can be the issue here?
>> >>>
>> >>> hari
>> >>>
>> >>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com>
>> >> wrote:
>> >>>
>> >>>> Hi Hari,
>> >>>>
>> >>>> You are missing the quorum setting. It seems the hbase-site.xml is
>> >> missing
>> >>>> from the classpath on the clients. Did you pack it into the jar?
>> >>>>
>> >>>> And yes, even one ZK server is fine in such a small cluster.
>> >>>>
>> >>>> You can see it is trying to connect to localhost which is the default
>> if
>> >>>> the site file is missing.
>> >>>>
>> >>>> Regards,
>> >>>> Lars
>> >>>>
>> >>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>> Is it necessary that all RegionServers must also be part of the ZK
>> >>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master and
>> >>>> hadoop2,
>> >>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and for
>> >>>> HBase).
>> >>>>>
>> >>>>> When I keep only 3 nodes in the zookeeper.quorum property:
>> >>>>> <name>hbase.zookeeper.quorum</name>
>> >>>>> <value>hadoop1,hadoop2,hadoop3</value>
>> >>>>>
>> >>>>> I get this exception for all tasks that run on ejabber(the 4th
>> node):
>> >>>>>
>> >>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
>> >> Attempting
>> >>>>> connection to server localhost/127.0.0.1:2181
>> >>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
>> >> Exception
>> >>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
>> >>>>> java.net.ConnectException: Connection refused
>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>>>> at
>> >>>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> >>>>> at
>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>> Ignoring
>> >>>>> exception during shutdown input
>> >>>>> java.nio.channels.ClosedChannelException
>> >>>>> at
>> >>>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>> >>>>> at
>> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >>>>> at
>> >>>>>
>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >>>>> at
>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>> Ignoring
>> >>>>> exception during shutdown output
>> >>>>> java.nio.channels.ClosedChannelException
>> >>>>> at
>> >>>>>
>> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>> >>>>> at
>> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> >>>>> at
>> >>>>>
>> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> >>>>> at
>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >>>>> 2010-11-21 23:35:47,925 WARN
>> >>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>> >>>> /hbase
>> >>>>> -- check quorum servers, currently=localhost:2181
>> >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>>>> KeeperErrorCode = ConnectionLoss for /hbase
>> >>>>> at
>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>> >>>>> at
>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>> >>>>> at
>> >>>>>
>> >>>>
>> >>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
>> >>>>> at
>> >>>>>
>> >>>>
>> >>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
>> >>>>> at
>> >>>>>
>> >>>>
>> >>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>> >>>>>
>> >>>>> When I add ejabber also to the ZK quorum and restart HBase, I don't
>> >> get
>> >>>> this
>> >>>>> exception. My understanding was that a small cluster like mine
>> should
>> >>>> only
>> >>>>> need one ZK machine.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Hari
>> >>>>
>> >>>
>> >>
>>
>
>
Re: Question about Zookeeper quorum
Posted by Hari Sreekumar <hs...@clickable.com>.
Hi Lars,
I start them through HBase implicitly. I'll try printing the config
values and post.
thanks,
Hari
On Mon, Nov 22, 2010 at 1:58 PM, Lars George <la...@gmail.com> wrote:
> Hi Hari,
>
> Are you starting them yourself or have HBase start them for you implicitly?
>
> Lars
>
> On Nov 22, 2010, at 6:19, Hari Sreekumar <hs...@clickable.com> wrote:
>
> > Hey Lars,
> >
> > I have HQuorumPeer running on all nodes that I specify in my
> > hbase-site file. One thing I wanted to clarify.. what is the default
> value
> > of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my
> > hbase-env.sh file.
> >
> > thanks,
> > hari
> >
> > On Mon, Nov 22, 2010 at 10:39 AM, Lars George <la...@gmail.com>
> wrote:
> >
> >> Hi Hari,
> >>
> >> On which of these for machines do you have a ZooKeeper daemon running as
> >> well?
> >>
> >> Lars
> >>
> >> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
> >> <hs...@clickable.com> wrote:
> >>> Hi,
> >>>
> >>> But it is reading settings from hbase-site.xml. If it was not reading
> >> my
> >>> changes, the problem wouldn't have gotten fixed when I add ejabber to
> the
> >>> quroum right? After all, it is responding to changes I make in my xml
> >> file.
> >>> What else can be the issue here?
> >>>
> >>> hari
> >>>
> >>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com>
> >> wrote:
> >>>
> >>>> Hi Hari,
> >>>>
> >>>> You are missing the quorum setting. It seems the hbase-site.xml is
> >> missing
> >>>> from the classpath on the clients. Did you pack it into the jar?
> >>>>
> >>>> And yes, even one ZK server is fine in such a small cluster.
> >>>>
> >>>> You can see it is trying to connect to localhost which is the default
> if
> >>>> the site file is missing.
> >>>>
> >>>> Regards,
> >>>> Lars
> >>>>
> >>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>> Is it necessary that all RegionServers must also be part of the ZK
> >>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master and
> >>>> hadoop2,
> >>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and for
> >>>> HBase).
> >>>>>
> >>>>> When I keep only 3 nodes in the zookeeper.quorum property:
> >>>>> <name>hbase.zookeeper.quorum</name>
> >>>>> <value>hadoop1,hadoop2,hadoop3</value>
> >>>>>
> >>>>> I get this exception for all tasks that run on ejabber(the 4th node):
> >>>>>
> >>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
> >> Attempting
> >>>>> connection to server localhost/127.0.0.1:2181
> >>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
> >> Exception
> >>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> >>>>> java.net.ConnectException: Connection refused
> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>> at
> >>>>>
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >>>>> at
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> >>>>> exception during shutdown input
> >>>>> java.nio.channels.ClosedChannelException
> >>>>> at
> >>>>>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> >>>>> at
> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >>>>> at
> >>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >>>>> at
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> >>>>> exception during shutdown output
> >>>>> java.nio.channels.ClosedChannelException
> >>>>> at
> >>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> >>>>> at
> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> >>>>> at
> >>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> >>>>> at
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >>>>> 2010-11-21 23:35:47,925 WARN
> >>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
> >>>> /hbase
> >>>>> -- check quorum servers, currently=localhost:2181
> >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>> KeeperErrorCode = ConnectionLoss for /hbase
> >>>>> at
> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> >>>>> at
> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> >>>>> at
> >>>>>
> >>>>
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> >>>>> at
> >>>>>
> >>>>
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> >>>>> at
> >>>>>
> >>>>
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
> >>>>>
> >>>>> When I add ejabber also to the ZK quorum and restart HBase, I don't
> >> get
> >>>> this
> >>>>> exception. My understanding was that a small cluster like mine should
> >>>> only
> >>>>> need one ZK machine.
> >>>>>
> >>>>> Thanks,
> >>>>> Hari
> >>>>
> >>>
> >>
>
Re: Question about Zookeeper quorum
Posted by Lars George <la...@gmail.com>.
Hi Hari,
Are you starting them yourself or have HBase start them for you implicitly?
Lars
On Nov 22, 2010, at 6:19, Hari Sreekumar <hs...@clickable.com> wrote:
> Hey Lars,
>
> I have HQuorumPeer running on all nodes that I specify in my
> hbase-site file. One thing I wanted to clarify.. what is the default value
> of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my
> hbase-env.sh file.
>
> thanks,
> hari
>
> On Mon, Nov 22, 2010 at 10:39 AM, Lars George <la...@gmail.com> wrote:
>
>> Hi Hari,
>>
>> On which of these for machines do you have a ZooKeeper daemon running as
>> well?
>>
>> Lars
>>
>> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
>> <hs...@clickable.com> wrote:
>>> Hi,
>>>
>>> But it is reading settings from hbase-site.xml. If it was not reading
>> my
>>> changes, the problem wouldn't have gotten fixed when I add ejabber to the
>>> quroum right? After all, it is responding to changes I make in my xml
>> file.
>>> What else can be the issue here?
>>>
>>> hari
>>>
>>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com>
>> wrote:
>>>
>>>> Hi Hari,
>>>>
>>>> You are missing the quorum setting. It seems the hbase-site.xml is
>> missing
>>>> from the classpath on the clients. Did you pack it into the jar?
>>>>
>>>> And yes, even one ZK server is fine in such a small cluster.
>>>>
>>>> You can see it is trying to connect to localhost which is the default if
>>>> the site file is missing.
>>>>
>>>> Regards,
>>>> Lars
>>>>
>>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> Is it necessary that all RegionServers must also be part of the ZK
>>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master and
>>>> hadoop2,
>>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and for
>>>> HBase).
>>>>>
>>>>> When I keep only 3 nodes in the zookeeper.quorum property:
>>>>> <name>hbase.zookeeper.quorum</name>
>>>>> <value>hadoop1,hadoop2,hadoop3</value>
>>>>>
>>>>> I get this exception for all tasks that run on ejabber(the 4th node):
>>>>>
>>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
>> Attempting
>>>>> connection to server localhost/127.0.0.1:2181
>>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
>> Exception
>>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
>>>>> java.net.ConnectException: Connection refused
>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>>>>> exception during shutdown input
>>>>> java.nio.channels.ClosedChannelException
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>>>> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>> at
>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>>>>> exception during shutdown output
>>>>> java.nio.channels.ClosedChannelException
>>>>> at
>>>>>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>>>>> at
>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>> at
>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>> 2010-11-21 23:35:47,925 WARN
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>>>> /hbase
>>>>> -- check quorum servers, currently=localhost:2181
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>>>>> at
>>>>>
>>>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
>>>>> at
>>>>>
>>>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
>>>>> at
>>>>>
>>>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>>>>>
>>>>> When I add ejabber also to the ZK quorum and restart HBase, I don't
>> get
>>>> this
>>>>> exception. My understanding was that a small cluster like mine should
>>>> only
>>>>> need one ZK machine.
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>
>>>
>>
Re: Question about Zookeeper quorum
Posted by Ted Yu <yu...@gmail.com>.
The default is true:
bin/zookeepers.sh:if [ "$HBASE_MANAGES_ZK" = "" ]; then
bin/zookeepers.sh: HBASE_MANAGES_ZK=true
On Sun, Nov 21, 2010 at 9:19 PM, Hari Sreekumar <hs...@clickable.com>wrote:
> Hey Lars,
>
> I have HQuorumPeer running on all nodes that I specify in my
> hbase-site file. One thing I wanted to clarify.. what is the default value
> of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my
> hbase-env.sh file.
>
> thanks,
> hari
>
> On Mon, Nov 22, 2010 at 10:39 AM, Lars George <la...@gmail.com>
> wrote:
>
> > Hi Hari,
> >
> > On which of these for machines do you have a ZooKeeper daemon running as
> > well?
> >
> > Lars
> >
> > On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
> > <hs...@clickable.com> wrote:
> > > Hi,
> > >
> > > But it is reading settings from hbase-site.xml. If it was not
> reading
> > my
> > > changes, the problem wouldn't have gotten fixed when I add ejabber to
> the
> > > quroum right? After all, it is responding to changes I make in my xml
> > file.
> > > What else can be the issue here?
> > >
> > > hari
> > >
> > > On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com>
> > wrote:
> > >
> > >> Hi Hari,
> > >>
> > >> You are missing the quorum setting. It seems the hbase-site.xml is
> > missing
> > >> from the classpath on the clients. Did you pack it into the jar?
> > >>
> > >> And yes, even one ZK server is fine in such a small cluster.
> > >>
> > >> You can see it is trying to connect to localhost which is the default
> if
> > >> the site file is missing.
> > >>
> > >> Regards,
> > >> Lars
> > >>
> > >> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> > Is it necessary that all RegionServers must also be part of the
> ZK
> > >> > Quorum? I have a 4 node cluster, with node hadoop1 being master and
> > >> hadoop2,
> > >> > hadoop3 and ejabber being the slaves (Both in case of hadoop and for
> > >> HBase).
> > >> >
> > >> > When I keep only 3 nodes in the zookeeper.quorum property:
> > >> > <name>hbase.zookeeper.quorum</name>
> > >> > <value>hadoop1,hadoop2,hadoop3</value>
> > >> >
> > >> > I get this exception for all tasks that run on ejabber(the 4th
> node):
> > >> >
> > >> > 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
> > Attempting
> > >> > connection to server localhost/127.0.0.1:2181
> > >> > 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
> > Exception
> > >> > closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> > >> > java.net.ConnectException: Connection refused
> > >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >> > at
> > >> >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> > >> > at
> > >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> > >> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> > >> > exception during shutdown input
> > >> > java.nio.channels.ClosedChannelException
> > >> > at
> > >> >
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> > >> > at
> > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > >> > at
> > >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > >> > at
> > >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
> Ignoring
> > >> > exception during shutdown output
> > >> > java.nio.channels.ClosedChannelException
> > >> > at
> > >> >
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> > >> > at
> > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > >> > at
> > >> >
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > >> > at
> > >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > >> > 2010-11-21 23:35:47,925 WARN
> > >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
> > >> /hbase
> > >> > -- check quorum servers, currently=localhost:2181
> > >> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > >> > KeeperErrorCode = ConnectionLoss for /hbase
> > >> > at
> > >> > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > >> > at
> > >> > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > >> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> > >> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> > >> > at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> > >> > at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> > >> > at
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
> > >> >
> > >> > When I add ejabber also to the ZK quorum and restart HBase, I don't
> > get
> > >> this
> > >> > exception. My understanding was that a small cluster like mine
> should
> > >> only
> > >> > need one ZK machine.
> > >> >
> > >> > Thanks,
> > >> > Hari
> > >>
> > >
> >
>
Re: Question about Zookeeper quorum
Posted by Hari Sreekumar <hs...@clickable.com>.
Hey Lars,
I have HQuorumPeer running on all nodes that I specify in my
hbase-site file. One thing I wanted to clarify.. what is the default value
of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my
hbase-env.sh file.
thanks,
hari
On Mon, Nov 22, 2010 at 10:39 AM, Lars George <la...@gmail.com> wrote:
> Hi Hari,
>
> On which of these for machines do you have a ZooKeeper daemon running as
> well?
>
> Lars
>
> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
> <hs...@clickable.com> wrote:
> > Hi,
> >
> > But it is reading settings from hbase-site.xml. If it was not reading
> my
> > changes, the problem wouldn't have gotten fixed when I add ejabber to the
> > quroum right? After all, it is responding to changes I make in my xml
> file.
> > What else can be the issue here?
> >
> > hari
> >
> > On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com>
> wrote:
> >
> >> Hi Hari,
> >>
> >> You are missing the quorum setting. It seems the hbase-site.xml is
> missing
> >> from the classpath on the clients. Did you pack it into the jar?
> >>
> >> And yes, even one ZK server is fine in such a small cluster.
> >>
> >> You can see it is trying to connect to localhost which is the default if
> >> the site file is missing.
> >>
> >> Regards,
> >> Lars
> >>
> >> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
> >> wrote:
> >>
> >> > Hi,
> >> > Is it necessary that all RegionServers must also be part of the ZK
> >> > Quorum? I have a 4 node cluster, with node hadoop1 being master and
> >> hadoop2,
> >> > hadoop3 and ejabber being the slaves (Both in case of hadoop and for
> >> HBase).
> >> >
> >> > When I keep only 3 nodes in the zookeeper.quorum property:
> >> > <name>hbase.zookeeper.quorum</name>
> >> > <value>hadoop1,hadoop2,hadoop3</value>
> >> >
> >> > I get this exception for all tasks that run on ejabber(the 4th node):
> >> >
> >> > 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
> Attempting
> >> > connection to server localhost/127.0.0.1:2181
> >> > 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
> Exception
> >> > closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> >> > java.net.ConnectException: Connection refused
> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >> > at
> >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >> > at
> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> >> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> >> > exception during shutdown input
> >> > java.nio.channels.ClosedChannelException
> >> > at
> >> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> >> > at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> >> > at
> >> >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> >> > at
> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> >> > exception during shutdown output
> >> > java.nio.channels.ClosedChannelException
> >> > at
> >> >
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> >> > at
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> >> > at
> >> >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> >> > at
> >> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> >> > 2010-11-21 23:35:47,925 WARN
> >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
> >> /hbase
> >> > -- check quorum servers, currently=localhost:2181
> >> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> > KeeperErrorCode = ConnectionLoss for /hbase
> >> > at
> >> > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> >> > at
> >> > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> >> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> >> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
> >> >
> >> > When I add ejabber also to the ZK quorum and restart HBase, I don't
> get
> >> this
> >> > exception. My understanding was that a small cluster like mine should
> >> only
> >> > need one ZK machine.
> >> >
> >> > Thanks,
> >> > Hari
> >>
> >
>
Re: Question about Zookeeper quorum
Posted by Lars George <la...@gmail.com>.
Hi Hari,
On which of these for machines do you have a ZooKeeper daemon running as well?
Lars
On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
<hs...@clickable.com> wrote:
> Hi,
>
> But it is reading settings from hbase-site.xml. If it was not reading my
> changes, the problem wouldn't have gotten fixed when I add ejabber to the
> quroum right? After all, it is responding to changes I make in my xml file.
> What else can be the issue here?
>
> hari
>
> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com> wrote:
>
>> Hi Hari,
>>
>> You are missing the quorum setting. It seems the hbase-site.xml is missing
>> from the classpath on the clients. Did you pack it into the jar?
>>
>> And yes, even one ZK server is fine in such a small cluster.
>>
>> You can see it is trying to connect to localhost which is the default if
>> the site file is missing.
>>
>> Regards,
>> Lars
>>
>> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
>> wrote:
>>
>> > Hi,
>> > Is it necessary that all RegionServers must also be part of the ZK
>> > Quorum? I have a 4 node cluster, with node hadoop1 being master and
>> hadoop2,
>> > hadoop3 and ejabber being the slaves (Both in case of hadoop and for
>> HBase).
>> >
>> > When I keep only 3 nodes in the zookeeper.quorum property:
>> > <name>hbase.zookeeper.quorum</name>
>> > <value>hadoop1,hadoop2,hadoop3</value>
>> >
>> > I get this exception for all tasks that run on ejabber(the 4th node):
>> >
>> > 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> > connection to server localhost/127.0.0.1:2181
>> > 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn: Exception
>> > closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
>> > java.net.ConnectException: Connection refused
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> > at
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> > exception during shutdown input
>> > java.nio.channels.ClosedChannelException
>> > at
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> > exception during shutdown output
>> > java.nio.channels.ClosedChannelException
>> > at
>> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > 2010-11-21 23:35:47,925 WARN
>> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>> /hbase
>> > -- check quorum servers, currently=localhost:2181
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase
>> > at
>> > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>> > at
>> > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>> > at
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
>> > at
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
>> > at
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>> >
>> > When I add ejabber also to the ZK quorum and restart HBase, I don't get
>> this
>> > exception. My understanding was that a small cluster like mine should
>> only
>> > need one ZK machine.
>> >
>> > Thanks,
>> > Hari
>>
>
Re: Question about Zookeeper quorum
Posted by Marcus Chou <ch...@gmail.com>.
Perhaps not.
The problem is that your tasks on ejabber were assuming that the zookeeper
was running locally, and your master node did read settings from
hbase-site.xml, then it would call the HQuorumPeer on ejabber to run. So
that your problem was fixed.
While if ejabber was not added to the quorum, the HQuorumPeer would not run
on ejabber, thus lead to the problem.
I think there must be some inconsistency in settings on ejabber.
On 22 November 2010 12:51, Hari Sreekumar <hs...@clickable.com> wrote:
> Hi,
>
> But it is reading settings from hbase-site.xml. If it was not reading my
> changes, the problem wouldn't have gotten fixed when I add ejabber to the
> quroum right? After all, it is responding to changes I make in my xml file.
> What else can be the issue here?
>
> hari
>
> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com>
> wrote:
>
> > Hi Hari,
> >
> > You are missing the quorum setting. It seems the hbase-site.xml is
> missing
> > from the classpath on the clients. Did you pack it into the jar?
> >
> > And yes, even one ZK server is fine in such a small cluster.
> >
> > You can see it is trying to connect to localhost which is the default if
> > the site file is missing.
> >
> > Regards,
> > Lars
> >
> > On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
> > wrote:
> >
> > > Hi,
> > > Is it necessary that all RegionServers must also be part of the ZK
> > > Quorum? I have a 4 node cluster, with node hadoop1 being master and
> > hadoop2,
> > > hadoop3 and ejabber being the slaves (Both in case of hadoop and for
> > HBase).
> > >
> > > When I keep only 3 nodes in the zookeeper.quorum property:
> > > <name>hbase.zookeeper.quorum</name>
> > > <value>hadoop1,hadoop2,hadoop3</value>
> > >
> > > I get this exception for all tasks that run on ejabber(the 4th node):
> > >
> > > 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
> Attempting
> > > connection to server localhost/127.0.0.1:2181
> > > 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn: Exception
> > > closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> > > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> > > exception during shutdown input
> > > java.nio.channels.ClosedChannelException
> > > at
> > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> > > at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> > > exception during shutdown output
> > > java.nio.channels.ClosedChannelException
> > > at
> > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> > > at
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > > at
> > >
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > > 2010-11-21 23:35:47,925 WARN
> > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
> > /hbase
> > > -- check quorum servers, currently=localhost:2181
> > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > KeeperErrorCode = ConnectionLoss for /hbase
> > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> > > at
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> > > at
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> > > at
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
> > >
> > > When I add ejabber also to the ZK quorum and restart HBase, I don't get
> > this
> > > exception. My understanding was that a small cluster like mine should
> > only
> > > need one ZK machine.
> > >
> > > Thanks,
> > > Hari
> >
>
--
Best regards,
Yue Zhou(Marcus),
System Research Group,
Parallel Processing Institute, Fudan University
Re: Question about Zookeeper quorum
Posted by Hari Sreekumar <hs...@clickable.com>.
Hi,
But it is reading settings from hbase-site.xml. If it was not reading my
changes, the problem wouldn't have gotten fixed when I add ejabber to the
quroum right? After all, it is responding to changes I make in my xml file.
What else can be the issue here?
hari
On Mon, Nov 22, 2010 at 12:54 AM, Lars George <la...@gmail.com> wrote:
> Hi Hari,
>
> You are missing the quorum setting. It seems the hbase-site.xml is missing
> from the classpath on the clients. Did you pack it into the jar?
>
> And yes, even one ZK server is fine in such a small cluster.
>
> You can see it is trying to connect to localhost which is the default if
> the site file is missing.
>
> Regards,
> Lars
>
> On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com>
> wrote:
>
> > Hi,
> > Is it necessary that all RegionServers must also be part of the ZK
> > Quorum? I have a 4 node cluster, with node hadoop1 being master and
> hadoop2,
> > hadoop3 and ejabber being the slaves (Both in case of hadoop and for
> HBase).
> >
> > When I keep only 3 nodes in the zookeeper.quorum property:
> > <name>hbase.zookeeper.quorum</name>
> > <value>hadoop1,hadoop2,hadoop3</value>
> >
> > I get this exception for all tasks that run on ejabber(the 4th node):
> >
> > 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn: Attempting
> > connection to server localhost/127.0.0.1:2181
> > 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> > exception during shutdown input
> > java.nio.channels.ClosedChannelException
> > at
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> > exception during shutdown output
> > java.nio.channels.ClosedChannelException
> > at
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> > 2010-11-21 23:35:47,925 WARN
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
> /hbase
> > -- check quorum servers, currently=localhost:2181
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
> >
> > When I add ejabber also to the ZK quorum and restart HBase, I don't get
> this
> > exception. My understanding was that a small cluster like mine should
> only
> > need one ZK machine.
> >
> > Thanks,
> > Hari
>
Re: Question about Zookeeper quorum
Posted by Lars George <la...@gmail.com>.
Hi Hari,
You are missing the quorum setting. It seems the hbase-site.xml is missing from the classpath on the clients. Did you pack it into the jar?
And yes, even one ZK server is fine in such a small cluster.
You can see it is trying to connect to localhost which is the default if the site file is missing.
Regards,
Lars
On Nov 21, 2010, at 19:22, Hari Sreekumar <hs...@clickable.com> wrote:
> Hi,
> Is it necessary that all RegionServers must also be part of the ZK
> Quorum? I have a 4 node cluster, with node hadoop1 being master and hadoop2,
> hadoop3 and ejabber being the slaves (Both in case of hadoop and for HBase).
>
> When I keep only 3 nodes in the zookeeper.quorum property:
> <name>hbase.zookeeper.quorum</name>
> <value>hadoop1,hadoop2,hadoop3</value>
>
> I get this exception for all tasks that run on ejabber(the 4th node):
>
> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server localhost/127.0.0.1:2181
> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-11-21 23:35:47,925 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase
> -- check quorum servers, currently=localhost:2181
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>
> When I add ejabber also to the ZK quorum and restart HBase, I don't get this
> exception. My understanding was that a small cluster like mine should only
> need one ZK machine.
>
> Thanks,
> Hari
Re: Question about Zookeeper quorum
Posted by Andrew McCall <an...@goroam.net>.
Not all region servers need to be part of the quorum.
HBase is trying to connect to 127.0.0.1 for zookeeper. What is the value of hbase.zookeeper.quorum from your HBase configuration?
Andrew
On 21 Nov 2010, at 18:22, Hari Sreekumar wrote:
> Hi,
> Is it necessary that all RegionServers must also be part of the ZK
> Quorum? I have a 4 node cluster, with node hadoop1 being master and hadoop2,
> hadoop3 and ejabber being the slaves (Both in case of hadoop and for HBase).
>
> When I keep only 3 nodes in the zookeeper.quorum property:
> <name>hbase.zookeeper.quorum</name>
> <value>hadoop1,hadoop2,hadoop3</value>
>
> I get this exception for all tasks that run on ejabber(the 4th node):
>
> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server localhost/127.0.0.1:2181
> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7c2e1f1f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-11-21 23:35:47,925 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase
> -- check quorum servers, currently=localhost:2181
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>
> When I add ejabber also to the ZK quorum and restart HBase, I don't get this
> exception. My understanding was that a small cluster like mine should only
> need one ZK machine.
>
> Thanks,
> Hari