You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Yair Even-Zohar <ya...@audiencescience.com> on 2009/09/01 12:37:06 UTC
problem running Hbase & Katta together on EC2
Hbase runs perfectly well as long as it manages zookeeper Katta runs
well on EC2
However, when running katta (which is based on internal zookeeper
management) I'm getting some zookeeper errors from Hbase (below):
Looking at the errors, it seems that hbase is not familiar with Katta's
zookeeper so I tried to use the same zookeeper jar but got the same
errors.
I found hbase -1311 and 1315 that are related but that's supposed to be
resolved.
Has anybody seen this behavior?
Thanks
-Yair
For the master I get:
2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server
domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at
sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
For the regionServers I get:
2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at
sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at
sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-01 06:12:34,572 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
watcher on ZNode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
Re: problem running Hbase & Katta together on EC2
Posted by Andrew Purtell <ap...@apache.org>.
Another possibility is the security groups up on EC2 are not set up to allow the ZK client port (2181).
- Andy
________________________________
From: Andrew Purtell <ap...@apache.org>
To: hbase-user@hadoop.apache.org
Sent: Tuesday, September 1, 2009 9:47:38 AM
Subject: Re: problem running Hbase & Katta together on EC2
I think Yair is right. I've seen this same thing when the major versions of ZK differ on the client and server side.
- Andy
________________________________
From: Jean-Daniel Cryans <jd...@apache.org>
To: hbase-user@hadoop.apache.org
Sent: Tuesday, September 1, 2009 3:45:20 AM
Subject: Re: problem running Hbase & Katta together on EC2
Yair,
Unless you are either using a very old or very new version of
zookeeper, it should be ok since they don't change their RPC. The ZK
shipped with HBase only has addons to make it easier to start a quorum
peer without a zoo.cfg
Those exceptions are very generic, they only mean that the processes
are unable to connect to your quorum.
While developing HBase to work with ZK I found that it's very easy to
run out of client connections. Try setting maxClientCnxns to 0 or add
machines to the quorum. That's a limit on a socket level so all
processes on a same machine share it.
J-D
On Tue, Sep 1, 2009 at 6:37 AM, Yair
Even-Zohar<ya...@audiencescience.com> wrote:
> Hbase runs perfectly well as long as it manages zookeeper Katta runs
> well on EC2
>
>
>
> However, when running katta (which is based on internal zookeeper
> management) I'm getting some zookeeper errors from Hbase (below):
>
>
>
> Looking at the errors, it seems that hbase is not familiar with Katta's
> zookeeper so I tried to use the same zookeeper jar but got the same
> errors.
>
> I found hbase -1311 and 1315 that are related but that's supposed to be
> resolved.
>
>
>
> Has anybody seen this behavior?
>
>
>
> Thanks
>
> -Yair
>
>
>
>
>
> For the master I get:
>
> 2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
> sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
>
> 2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is false
>
> 2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server
> domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
>
> 2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
>
>
>
>
> For the regionServers I get:
>
> 2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,572 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
> watcher on ZNode /hbase/master
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
>
>
>
>
Re: problem running Hbase & Katta together on EC2
Posted by Andrew Purtell <ap...@apache.org>.
I think Yair is right. I've seen this same thing when the major versions of ZK differ on the client and server side.
- Andy
________________________________
From: Jean-Daniel Cryans <jd...@apache.org>
To: hbase-user@hadoop.apache.org
Sent: Tuesday, September 1, 2009 3:45:20 AM
Subject: Re: problem running Hbase & Katta together on EC2
Yair,
Unless you are either using a very old or very new version of
zookeeper, it should be ok since they don't change their RPC. The ZK
shipped with HBase only has addons to make it easier to start a quorum
peer without a zoo.cfg
Those exceptions are very generic, they only mean that the processes
are unable to connect to your quorum.
While developing HBase to work with ZK I found that it's very easy to
run out of client connections. Try setting maxClientCnxns to 0 or add
machines to the quorum. That's a limit on a socket level so all
processes on a same machine share it.
J-D
On Tue, Sep 1, 2009 at 6:37 AM, Yair
Even-Zohar<ya...@audiencescience.com> wrote:
> Hbase runs perfectly well as long as it manages zookeeper Katta runs
> well on EC2
>
>
>
> However, when running katta (which is based on internal zookeeper
> management) I'm getting some zookeeper errors from Hbase (below):
>
>
>
> Looking at the errors, it seems that hbase is not familiar with Katta's
> zookeeper so I tried to use the same zookeeper jar but got the same
> errors.
>
> I found hbase -1311 and 1315 that are related but that's supposed to be
> resolved.
>
>
>
> Has anybody seen this behavior?
>
>
>
> Thanks
>
> -Yair
>
>
>
>
>
> For the master I get:
>
> 2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
> sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
>
> 2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is false
>
> 2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server
> domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
>
> 2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
>
>
>
>
> For the regionServers I get:
>
> 2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,572 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
> watcher on ZNode /hbase/master
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
>
>
>
>
Re: problem running Hbase & Katta together on EC2
Posted by Jean-Daniel Cryans <jd...@apache.org>.
There is indeed a mix up. I talked about zoo.cfg since I thought you
were starting the quorum yourself but you aren't since Katta's doing
it.
Because Katta handles ZK in your setup, any config passed to HBase
that are related to handling ZK quorum peers are ineffective. Setting
the maxClientCnxns in HBase in your case won't change anything, you
have to set it in Katta (I don't know where). That is, that config is
passed to the class booting the ZK processes so if HBase doesn't
handle ZK it won't be used.
Clearer? ;)
J-D
On Tue, Sep 1, 2009 at 9:53 AM, Yair
Even-Zohar<ya...@audiencescience.com> wrote:
> Now I'm completely baffled. I'm using hbase 0.20 rc2, I thought zoo.cfg is now gone.
>
> The quorum is handled by Katta. That is, once I run Katta is handles zookeeper internally.
> I thought that all the ZK related properties (e.g. zk client/server ports etc,) are passed via hbase-site (or hbase-defualt).
>
> Is my assumption wrong?
>
> Thanks
> -Yair
>
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Tuesday, September 01, 2009 3:32 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: problem running Hbase & Katta together on EC2
>
> Is your ZK quorum handled by HBase or you started it yourself? In the
> later case, this configuration goes into your own zoo.cfg since this
> is a quorum peer conf.
>
> J-D
>
> On Tue, Sep 1, 2009 at 7:51 AM, Yair
> Even-Zohar<ya...@audiencescience.com> wrote:
>> Unfortunately, that's not the solution in my case.
>> I set maxClientCnxns to 0 in hbase-site.xml and restarted the regionservers & master but the errors are exactly the same.
>>
>>
>> -----Original Message-----
>> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
>> Sent: Tuesday, September 01, 2009 1:45 PM
>> To: hbase-user@hadoop.apache.org
>> Subject: Re: problem running Hbase & Katta together on EC2
>>
>> Yair,
>>
>> Unless you are either using a very old or very new version of
>> zookeeper, it should be ok since they don't change their RPC. The ZK
>> shipped with HBase only has addons to make it easier to start a quorum
>> peer without a zoo.cfg
>>
>> Those exceptions are very generic, they only mean that the processes
>> are unable to connect to your quorum.
>>
>> While developing HBase to work with ZK I found that it's very easy to
>> run out of client connections. Try setting maxClientCnxns to 0 or add
>> machines to the quorum. That's a limit on a socket level so all
>> processes on a same machine share it.
>>
>> J-D
>>
>> On Tue, Sep 1, 2009 at 6:37 AM, Yair
>> Even-Zohar<ya...@audiencescience.com> wrote:
>>> Hbase runs perfectly well as long as it manages zookeeper Katta runs
>>> well on EC2
>>>
>>>
>>>
>>> However, when running katta (which is based on internal zookeeper
>>> management) I'm getting some zookeeper errors from Hbase (below):
>>>
>>>
>>>
>>> Looking at the errors, it seems that hbase is not familiar with Katta's
>>> zookeeper so I tried to use the same zookeeper jar but got the same
>>> errors.
>>>
>>> I found hbase -1311 and 1315 that are related but that's supposed to be
>>> resolved.
>>>
>>>
>>>
>>> Has anybody seen this behavior?
>>>
>>>
>>>
>>> Thanks
>>>
>>> -Yair
>>>
>>>
>>>
>>>
>>>
>>> For the master I get:
>>>
>>> 2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>> client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
>>> sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
>>>
>>> 2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
>>> zookeeper.disableAutoWatchReset is false
>>>
>>> 2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
>>> connection to server
>>> domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
>>>
>>> 2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
>>>
>>> java.net.ConnectException: Connection refused
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>>>
>>> 2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>>> exception during shutdown input
>>>
>>> java.nio.channels.ClosedChannelException
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>>
>>> at
>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>>
>>>
>>>
>>>
>>>
>>> For the regionServers I get:
>>>
>>> 2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
>>>
>>> java.net.ConnectException: Connection refused
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>>>
>>> 2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>>> exception during shutdown input
>>>
>>> java.nio.channels.ClosedChannelException
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>>
>>> at
>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>>
>>> 2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>>> exception during shutdown output
>>>
>>> java.nio.channels.ClosedChannelException
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>>>
>>> at
>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>>
>>> 2009-09-01 06:12:34,572 WARN
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
>>> watcher on ZNode /hbase/master
>>>
>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>
>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
>>>
>>>
>>>
>>>
>>
>
RE: problem running Hbase & Katta together on EC2
Posted by Yair Even-Zohar <ya...@audiencescience.com>.
Now I'm completely baffled. I'm using hbase 0.20 rc2, I thought zoo.cfg is now gone.
The quorum is handled by Katta. That is, once I run Katta is handles zookeeper internally.
I thought that all the ZK related properties (e.g. zk client/server ports etc,) are passed via hbase-site (or hbase-defualt).
Is my assumption wrong?
Thanks
-Yair
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Tuesday, September 01, 2009 3:32 PM
To: hbase-user@hadoop.apache.org
Subject: Re: problem running Hbase & Katta together on EC2
Is your ZK quorum handled by HBase or you started it yourself? In the
later case, this configuration goes into your own zoo.cfg since this
is a quorum peer conf.
J-D
On Tue, Sep 1, 2009 at 7:51 AM, Yair
Even-Zohar<ya...@audiencescience.com> wrote:
> Unfortunately, that's not the solution in my case.
> I set maxClientCnxns to 0 in hbase-site.xml and restarted the regionservers & master but the errors are exactly the same.
>
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Tuesday, September 01, 2009 1:45 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: problem running Hbase & Katta together on EC2
>
> Yair,
>
> Unless you are either using a very old or very new version of
> zookeeper, it should be ok since they don't change their RPC. The ZK
> shipped with HBase only has addons to make it easier to start a quorum
> peer without a zoo.cfg
>
> Those exceptions are very generic, they only mean that the processes
> are unable to connect to your quorum.
>
> While developing HBase to work with ZK I found that it's very easy to
> run out of client connections. Try setting maxClientCnxns to 0 or add
> machines to the quorum. That's a limit on a socket level so all
> processes on a same machine share it.
>
> J-D
>
> On Tue, Sep 1, 2009 at 6:37 AM, Yair
> Even-Zohar<ya...@audiencescience.com> wrote:
>> Hbase runs perfectly well as long as it manages zookeeper Katta runs
>> well on EC2
>>
>>
>>
>> However, when running katta (which is based on internal zookeeper
>> management) I'm getting some zookeeper errors from Hbase (below):
>>
>>
>>
>> Looking at the errors, it seems that hbase is not familiar with Katta's
>> zookeeper so I tried to use the same zookeeper jar but got the same
>> errors.
>>
>> I found hbase -1311 and 1315 that are related but that's supposed to be
>> resolved.
>>
>>
>>
>> Has anybody seen this behavior?
>>
>>
>>
>> Thanks
>>
>> -Yair
>>
>>
>>
>>
>>
>> For the master I get:
>>
>> 2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
>> client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
>> sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
>>
>> 2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
>> zookeeper.disableAutoWatchReset is false
>>
>> 2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> connection to server
>> domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
>>
>> 2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>>
>> 2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown input
>>
>> java.nio.channels.ClosedChannelException
>>
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>
>> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>
>>
>>
>>
>>
>> For the regionServers I get:
>>
>> 2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>>
>> 2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown input
>>
>> java.nio.channels.ClosedChannelException
>>
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>
>> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>
>> 2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown output
>>
>> java.nio.channels.ClosedChannelException
>>
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>>
>> at
>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>
>> 2009-09-01 06:12:34,572 WARN
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
>> watcher on ZNode /hbase/master
>>
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
>>
>>
>>
>>
>
Re: problem running Hbase & Katta together on EC2
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Is your ZK quorum handled by HBase or you started it yourself? In the
later case, this configuration goes into your own zoo.cfg since this
is a quorum peer conf.
J-D
On Tue, Sep 1, 2009 at 7:51 AM, Yair
Even-Zohar<ya...@audiencescience.com> wrote:
> Unfortunately, that's not the solution in my case.
> I set maxClientCnxns to 0 in hbase-site.xml and restarted the regionservers & master but the errors are exactly the same.
>
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Tuesday, September 01, 2009 1:45 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: problem running Hbase & Katta together on EC2
>
> Yair,
>
> Unless you are either using a very old or very new version of
> zookeeper, it should be ok since they don't change their RPC. The ZK
> shipped with HBase only has addons to make it easier to start a quorum
> peer without a zoo.cfg
>
> Those exceptions are very generic, they only mean that the processes
> are unable to connect to your quorum.
>
> While developing HBase to work with ZK I found that it's very easy to
> run out of client connections. Try setting maxClientCnxns to 0 or add
> machines to the quorum. That's a limit on a socket level so all
> processes on a same machine share it.
>
> J-D
>
> On Tue, Sep 1, 2009 at 6:37 AM, Yair
> Even-Zohar<ya...@audiencescience.com> wrote:
>> Hbase runs perfectly well as long as it manages zookeeper Katta runs
>> well on EC2
>>
>>
>>
>> However, when running katta (which is based on internal zookeeper
>> management) I'm getting some zookeeper errors from Hbase (below):
>>
>>
>>
>> Looking at the errors, it seems that hbase is not familiar with Katta's
>> zookeeper so I tried to use the same zookeeper jar but got the same
>> errors.
>>
>> I found hbase -1311 and 1315 that are related but that's supposed to be
>> resolved.
>>
>>
>>
>> Has anybody seen this behavior?
>>
>>
>>
>> Thanks
>>
>> -Yair
>>
>>
>>
>>
>>
>> For the master I get:
>>
>> 2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
>> client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
>> sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
>>
>> 2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
>> zookeeper.disableAutoWatchReset is false
>>
>> 2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> connection to server
>> domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
>>
>> 2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>>
>> 2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown input
>>
>> java.nio.channels.ClosedChannelException
>>
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>
>> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>
>>
>>
>>
>>
>> For the regionServers I get:
>>
>> 2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>>
>> 2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown input
>>
>> java.nio.channels.ClosedChannelException
>>
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>>
>> at
>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>
>> 2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
>> exception during shutdown output
>>
>> java.nio.channels.ClosedChannelException
>>
>> at
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>>
>> at
>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>>
>> 2009-09-01 06:12:34,572 WARN
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
>> watcher on ZNode /hbase/master
>>
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
>>
>>
>>
>>
>
RE: problem running Hbase & Katta together on EC2
Posted by Yair Even-Zohar <ya...@audiencescience.com>.
Unfortunately, that's not the solution in my case.
I set maxClientCnxns to 0 in hbase-site.xml and restarted the regionservers & master but the errors are exactly the same.
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Tuesday, September 01, 2009 1:45 PM
To: hbase-user@hadoop.apache.org
Subject: Re: problem running Hbase & Katta together on EC2
Yair,
Unless you are either using a very old or very new version of
zookeeper, it should be ok since they don't change their RPC. The ZK
shipped with HBase only has addons to make it easier to start a quorum
peer without a zoo.cfg
Those exceptions are very generic, they only mean that the processes
are unable to connect to your quorum.
While developing HBase to work with ZK I found that it's very easy to
run out of client connections. Try setting maxClientCnxns to 0 or add
machines to the quorum. That's a limit on a socket level so all
processes on a same machine share it.
J-D
On Tue, Sep 1, 2009 at 6:37 AM, Yair
Even-Zohar<ya...@audiencescience.com> wrote:
> Hbase runs perfectly well as long as it manages zookeeper Katta runs
> well on EC2
>
>
>
> However, when running katta (which is based on internal zookeeper
> management) I'm getting some zookeeper errors from Hbase (below):
>
>
>
> Looking at the errors, it seems that hbase is not familiar with Katta's
> zookeeper so I tried to use the same zookeeper jar but got the same
> errors.
>
> I found hbase -1311 and 1315 that are related but that's supposed to be
> resolved.
>
>
>
> Has anybody seen this behavior?
>
>
>
> Thanks
>
> -Yair
>
>
>
>
>
> For the master I get:
>
> 2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
> sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
>
> 2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is false
>
> 2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server
> domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
>
> 2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
>
>
>
>
> For the regionServers I get:
>
> 2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,572 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
> watcher on ZNode /hbase/master
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
>
>
>
>
Re: problem running Hbase & Katta together on EC2
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yair,
Unless you are either using a very old or very new version of
zookeeper, it should be ok since they don't change their RPC. The ZK
shipped with HBase only has addons to make it easier to start a quorum
peer without a zoo.cfg
Those exceptions are very generic, they only mean that the processes
are unable to connect to your quorum.
While developing HBase to work with ZK I found that it's very easy to
run out of client connections. Try setting maxClientCnxns to 0 or add
machines to the quorum. That's a limit on a socket level so all
processes on a same machine share it.
J-D
On Tue, Sep 1, 2009 at 6:37 AM, Yair
Even-Zohar<ya...@audiencescience.com> wrote:
> Hbase runs perfectly well as long as it manages zookeeper Katta runs
> well on EC2
>
>
>
> However, when running katta (which is based on internal zookeeper
> management) I'm getting some zookeeper errors from Hbase (below):
>
>
>
> Looking at the errors, it seems that hbase is not familiar with Katta's
> zookeeper so I tried to use the same zookeeper jar but got the same
> errors.
>
> I found hbase -1311 and 1315 that are related but that's supposed to be
> resolved.
>
>
>
> Has anybody seen this behavior?
>
>
>
> Thanks
>
> -Yair
>
>
>
>
>
> For the master I get:
>
> 2009-09-01 06:14:30,584 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, host=domU-12-31-39-01-A4-52.compute-1.internal:2181
> sessionTimeout=60000 watcher=Thread[Thread-0,5,main]
>
> 2009-09-01 06:14:30,586 INFO org.apache.zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is false
>
> 2009-09-01 06:14:30,593 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server
> domU-12-31-39-01-A4-52.compute-1.internal/10.255.171.160:2181
>
> 2009-09-01 06:14:30,599 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@d05c13
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:14:30,600 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
>
>
>
>
> For the regionServers I get:
>
> 2009-09-01 06:12:34,454 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@a613f8
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
>
> 2009-09-01 06:12:34,459 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,460 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
>
> java.nio.channels.ClosedChannelException
>
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>
> at
> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
>
> 2009-09-01 06:12:34,572 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set
> watcher on ZNode /hbase/master
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
>
>
>
>
RE: [Katta-developer] problem running Hbase & Katta together on EC2
Posted by Yair Even-Zohar <ya...@audiencescience.com>.
How do I configure katta to allow external connections?
My EC2 security group allows all zk ports 2181, 2888, 3888
I'm running katta and hbase on the same macines. Master Hbase with
Master katta, Slave Hbase (regionservers) with slave katta
When I execute the command you have suggested, I get a reply on the
server port but not on the client port on both the master and slaves.
Notice however that the katta "slaves" can communicate with the master
(I see this using katta listNodes command)
[root@domU-12-31-39-07-51-C3 conf]# echo stat | nc
domU-12-31-39-07-50-01.compute-1.internal 2888
Zookeeper version: 3.1.1-755636, built on 03/18/2009 16:52 GMT
Clients:
/10.209.82.49:36168[1](queued=0,recved=0,sent=0)
/10.208.35.47:55836[1](queued=0,recved=53,sent=53)
/10.209.82.49:37714[1](queued=0,recved=53,sent=53)
/10.209.83.239:56033[1](queued=0,recved=63,sent=65)
Latency min/avg/max: 0/0/12
Received: 192
Sent: 194
Outstanding: 0
Zxid: 0x14
Mode: standalone
Node count: 15
________________________________
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Tuesday, September 01, 2009 7:18 PM
To: katta-developer@lists.sourceforge.net
Cc: hbase-user@hadoop.apache.org
Subject: Re: [Katta-developer] problem running Hbase & Katta together on
EC2
It looks like hbase is looking at the wrong port, wrong host or like
katta is not configured to allow external connections.
I don't think that this is a bug as much as an error in configuration.
Make sure that you can talk to ZK from every place that needs it (as in,
every hbase host). Use the
echo stat | nc $zk_host $zk_port
trick to find out if you can reach zookeeper.
On Tue, Sep 1, 2009 at 3:37 AM, Yair Even-Zohar
<ya...@audiencescience.com> wrote:
However, when running katta (which is based on internal zookeeper
management) I'm getting some zookeeper errors from Hbase (below):
Looking at the errors, it seems that hbase is not familiar with Katta's
zookeeper so I tried to use the same zookeeper jar but got the same
errors.
I found hbase -1311 and 1315 that are related but that's supposed to be
resolved.
--
Ted Dunning, CTO
DeepDyve