You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2009/08/27 16:26:59 UTC
[jira] Created: (HBASE-1800) Too many ZK connections
Too many ZK connections
-----------------------
Key: HBASE-1800
URL: https://issues.apache.org/jira/browse/HBASE-1800
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Fix For: 0.20.1, 0.21.0
Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
{code}
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>30</value>
<description>Property from ZooKeeper's config zoo.cfg.
Limit on number of concurrent connections (at the socket level) that a
single client, identified by IP address, may make to a single member of
the ZooKeeper ensemble. Set high to avoid zk connection issues running
standalone and pseudo-distributed.
</description>
</property>
{code}
If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1800) Too many ZK connections
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749295#action_12749295 ]
stack commented on HBASE-1800:
------------------------------
Just to say that I do not like static maps. Inevitably they cause grief because their initialization is effectively outside of your control done at class-loading time. They become a problem when you want to reinit without a restart of the jvm (e.g. deploy, undeploy, then redeploy of a web app that has an HTable instance).
That said, we already have a static map of HCMs so you are coming in under its cover. Lets fix both later. For 0.20.0, lets bring in this patch.
We also need to do the idea (I think it a Ryan idea from a while back) where the hbase zk client is dumb doing read-only operations. Is this possible? Does the client need to have watches in zk? (This again is for later).
Otherwise the patch looks good J-D. I'm +1 to commit assuming all unit tests pass.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1800-v1.patch, HBASE-1800-v2.patch, HBASE-1800-v3.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1800) Too many ZK connections
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748392#action_12748392 ]
Jean-Daniel Cryans commented on HBASE-1800:
-------------------------------------------
Unless in the client you pass the real same HBC yes.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.1, 0.21.0
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1800) Too many ZK connections
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749325#action_12749325 ]
Jean-Daniel Cryans commented on HBASE-1800:
-------------------------------------------
For the moment we use watches to make sure that when our session expires we renew it, I don't think there's any other way to do that.
All tests pass, I will commit.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1800-v1.patch, HBASE-1800-v2.patch, HBASE-1800-v3.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1800) Too many ZK connections
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-1800.
---------------------------------------
Resolution: Fixed
Assignee: Jean-Daniel Cryans
Hadoop Flags: [Reviewed]
Committed to branch and trunk, this should help fixing a lot of problems new users were getting and provide faster HTable instantiations.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1800-v1.patch, HBASE-1800-v2.patch, HBASE-1800-v3.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1800) Too many ZK connections
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1800:
--------------------------------------
Attachment: HBASE-1800-v2.patch
Different approach, I found that playing in the lower levels could mess when a client would close a connection while sharing it with a region server and other stuff like that.
In this patch:
- I created a new inner class called ClientZKWatcher in HCM that manages a connection to ZK.
- I added a multiton of ClientZKWatcher keyed by hbase.zookeeper.quorum.
- I added a way to clear all connections.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1800-v1.patch, HBASE-1800-v2.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1800) Too many ZK connections
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1800:
----------------------------------
Fix Version/s: (was: 0.20.1)
0.20.0
Let's pull this in to 0.20 also. New users who set up single node ZK installs will hit this.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1800-v1.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1800) Too many ZK connections
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748517#action_12748517 ]
Jean-Daniel Cryans commented on HBASE-1800:
-------------------------------------------
Yeah Ryan was asking the same since he uses 4 ZK quorums for his clients. I was thinking that instead we could use a multiton keyed by hbase.zookeeper.quorum.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.1, 0.21.0
>
> Attachments: HBASE-1800-v1.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1800) Too many ZK connections
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1800:
--------------------------------------
Attachment: HBASE-1800-v1.patch
This patch adds a singleton for ZKW. Has the advantage that whatever happens outside of ZK we will only have 1 connection per JVM. Without it openplaces' unit tests needed more than 100 client connections, now the default works just fine and there is less INFO log from ZK.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.1, 0.21.0
>
> Attachments: HBASE-1800-v1.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1800) Too many ZK connections
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1800:
--------------------------------------
Attachment: HBASE-1800-v3.patch
Adding a test for multiple ZK quorums. Would love to get a review.
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1800-v1.patch, HBASE-1800-v2.patch, HBASE-1800-v3.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1800) Too many ZK connections
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748389#action_12748389 ]
stack commented on HBASE-1800:
------------------------------
Even if you pass in same HBaseConfiguration each time?
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.1, 0.21.0
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1800) Too many ZK connections
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748515#action_12748515 ]
Nitay Joffe commented on HBASE-1800:
------------------------------------
Overall this looks great JD. Only question I have is it looks like if you call it with two different HBC's you'll still get the same ZKW. Shouldn't it be a map similar to the HConnectionManager?
> Too many ZK connections
> -----------------------
>
> Key: HBASE-1800
> URL: https://issues.apache.org/jira/browse/HBASE-1800
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.1, 0.21.0
>
> Attachments: HBASE-1800-v1.patch
>
>
> Currently we open tons of new connections to Zookeeper, like every time we instantiate a new HTable. There is a maximum number of client connections as described here:
> {code}
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>30</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> Limit on number of concurrent connections (at the socket level) that a
> single client, identified by IP address, may make to a single member of
> the ZooKeeper ensemble. Set high to avoid zk connection issues running
> standalone and pseudo-distributed.
> </description>
> </property>
> {code}
> If you hit that max number, ZK will just refuse your connections. Suppose you have 4 maps running on a server hosting a RS, you may actually lose your connection in the RS and eventually hit a session timeout. Maybe we should singleton ZKW?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.