You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Philippe (JIRA)" <ji...@apache.org> on 2011/06/08 16:46:58 UTC
[jira] [Created] (HBASE-3962)
HConnectionManager.getConnection(HBaseConfiguration) returns new connection
in default HTable constructor
HConnectionManager.getConnection(HBaseConfiguration) returns new connection in default HTable constructor
---------------------------------------------------------------------------------------------------------
Key: HBASE-3962
URL: https://issues.apache.org/jira/browse/HBASE-3962
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.90.1
Reporter: Philippe
The HBase instance are currently indexed by Configuration, which since HBASE-1976 does not have any other equivalence that the object equivalence.
So, everytime a new configuration is passed to the method a new connection is created.
If we create many HTable connections with the same configuration, there is no problem:
HBaseConfiguration config = HBaseConfiguration.create();
HTable table 1 = new HTable(config, "table1"); // init connection
HTable table 2 = new HTable(config, "table2"); // re-use connection
HTable table 3 = new HTable(config, "table3"); // re-use connection
However, if we call the default constructor, or re-call HBaseConfiguration.create();, we will pass a new instance of the configuration to the constructor. This will cause many connections to be created:
HTable table 1 = new HTable("table1"); // init connection
HTable table 2 = new HTable("table2"); // init new connection
HTable table 3 = new HTable("table3"); // init new connection
I know connection should be pooled, but sometimes we have to create a new connection, and without having access to a previously instanced configuration object.
Since zookeeper has a max client connection (default was 30, now is 10), after creating 30 instances of HTable, we can no longer access to the database.
In addition to this, the HBASE_INSTANCES map does not close the connection when removing the eldest entry. So if we have a larger maxConnection value than the hard-coded MAX_CACHED_HBASE_INSTANCES variable, connections will remain but won't be closed. MAX_CACHED_HBASE_INSTANCES should actually be set from the hbase.zookeeper.property.maxClientCnxns parameter (value + 1).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3962)
HConnectionManager.getConnection(HBaseConfiguration) returns new connection
in default HTable constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl resolved HBASE-3962.
----------------------------------
Resolution: Fixed
Since we have this solved in 0.92+ I am closing this.
Please reopen if you disagree.
> HConnectionManager.getConnection(HBaseConfiguration) returns new connection in default HTable constructor
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3962
> URL: https://issues.apache.org/jira/browse/HBASE-3962
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.90.1
> Reporter: Philippe
>
> The HBase instance are currently indexed by Configuration, which since HBASE-1976 does not have any other equivalence that the object equivalence.
> So, everytime a new configuration is passed to the method a new connection is created.
> If we create many HTable connections with the same configuration, there is no problem:
> HBaseConfiguration config = HBaseConfiguration.create();
> HTable table 1 = new HTable(config, "table1"); // init connection
> HTable table 2 = new HTable(config, "table2"); // re-use connection
> HTable table 3 = new HTable(config, "table3"); // re-use connection
> However, if we call the default constructor, or re-call HBaseConfiguration.create();, we will pass a new instance of the configuration to the constructor. This will cause many connections to be created:
> HTable table 1 = new HTable("table1"); // init connection
> HTable table 2 = new HTable("table2"); // init new connection
> HTable table 3 = new HTable("table3"); // init new connection
> I know connection should be pooled, but sometimes we have to create a new connection, and without having access to a previously instanced configuration object.
> Since zookeeper has a max client connection (default was 30, now is 10), after creating 30 instances of HTable, we can no longer access to the database.
> In addition to this, the HBASE_INSTANCES map does not close the connection when removing the eldest entry. So if we have a larger maxConnection value than the hard-coded MAX_CACHED_HBASE_INSTANCES variable, connections will remain but won't be closed. MAX_CACHED_HBASE_INSTANCES should actually be set from the hbase.zookeeper.property.maxClientCnxns parameter (value + 1).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3962)
HConnectionManager.getConnection(HBaseConfiguration) returns new connection
in default HTable constructor
Posted by "Philippe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047262#comment-13047262 ]
Philippe commented on HBASE-3962:
---------------------------------
Hi,
Thank you very much for your message. I did check the code source from the latest release, but not of the trunk actually. It seems you covered all the points I mentioned. Great !
However, I’m not really sure about a couple of things:
a) If I use the ‘same’ configuration each time I create a table, I’ll use the same connection then. I have a server that connects to HBase. Isn’t there a performance issue if the server will use only one connection to the quorum? As the quorum allows me to MAX_CLIENT_CNXNS connections, should we use as many connections as possible instead of just one for the same configuration?
b) The number of cached HConnection are now related to the number of connections allowed by the zookeeper ensemble. But if I pass different configuration, can’t they supposed to use different zookeeper quorum (I guess that may be an extremely rare case)? If so, is there really a point to limit the number of cached HConnections?
> HConnectionManager.getConnection(HBaseConfiguration) returns new connection in default HTable constructor
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3962
> URL: https://issues.apache.org/jira/browse/HBASE-3962
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.90.1
> Reporter: Philippe
>
> The HBase instance are currently indexed by Configuration, which since HBASE-1976 does not have any other equivalence that the object equivalence.
> So, everytime a new configuration is passed to the method a new connection is created.
> If we create many HTable connections with the same configuration, there is no problem:
> HBaseConfiguration config = HBaseConfiguration.create();
> HTable table 1 = new HTable(config, "table1"); // init connection
> HTable table 2 = new HTable(config, "table2"); // re-use connection
> HTable table 3 = new HTable(config, "table3"); // re-use connection
> However, if we call the default constructor, or re-call HBaseConfiguration.create();, we will pass a new instance of the configuration to the constructor. This will cause many connections to be created:
> HTable table 1 = new HTable("table1"); // init connection
> HTable table 2 = new HTable("table2"); // init new connection
> HTable table 3 = new HTable("table3"); // init new connection
> I know connection should be pooled, but sometimes we have to create a new connection, and without having access to a previously instanced configuration object.
> Since zookeeper has a max client connection (default was 30, now is 10), after creating 30 instances of HTable, we can no longer access to the database.
> In addition to this, the HBASE_INSTANCES map does not close the connection when removing the eldest entry. So if we have a larger maxConnection value than the hard-coded MAX_CACHED_HBASE_INSTANCES variable, connections will remain but won't be closed. MAX_CACHED_HBASE_INSTANCES should actually be set from the hbase.zookeeper.property.maxClientCnxns parameter (value + 1).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3962)
HConnectionManager.getConnection(HBaseConfiguration) returns new connection
in default HTable constructor
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046138#comment-13046138 ]
Ted Yu commented on HBASE-3962:
-------------------------------
In trunk, HConnectionManager.getConnection() constructs HConnectionKey from Configuration. Meaning, the identity of Configuration has been redefined.
Also, connection is closed in finalizer.
> HConnectionManager.getConnection(HBaseConfiguration) returns new connection in default HTable constructor
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3962
> URL: https://issues.apache.org/jira/browse/HBASE-3962
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.90.1
> Reporter: Philippe
>
> The HBase instance are currently indexed by Configuration, which since HBASE-1976 does not have any other equivalence that the object equivalence.
> So, everytime a new configuration is passed to the method a new connection is created.
> If we create many HTable connections with the same configuration, there is no problem:
> HBaseConfiguration config = HBaseConfiguration.create();
> HTable table 1 = new HTable(config, "table1"); // init connection
> HTable table 2 = new HTable(config, "table2"); // re-use connection
> HTable table 3 = new HTable(config, "table3"); // re-use connection
> However, if we call the default constructor, or re-call HBaseConfiguration.create();, we will pass a new instance of the configuration to the constructor. This will cause many connections to be created:
> HTable table 1 = new HTable("table1"); // init connection
> HTable table 2 = new HTable("table2"); // init new connection
> HTable table 3 = new HTable("table3"); // init new connection
> I know connection should be pooled, but sometimes we have to create a new connection, and without having access to a previously instanced configuration object.
> Since zookeeper has a max client connection (default was 30, now is 10), after creating 30 instances of HTable, we can no longer access to the database.
> In addition to this, the HBASE_INSTANCES map does not close the connection when removing the eldest entry. So if we have a larger maxConnection value than the hard-coded MAX_CACHED_HBASE_INSTANCES variable, connections will remain but won't be closed. MAX_CACHED_HBASE_INSTANCES should actually be set from the hbase.zookeeper.property.maxClientCnxns parameter (value + 1).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3962)
HConnectionManager.getConnection(HBaseConfiguration) returns new connection
in default HTable constructor
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047441#comment-13047441 ]
Ted Yu commented on HBASE-3962:
-------------------------------
For a, I don't think there is performance impact. Once 0.92 branch is made, we can see some real performance numbers.
For b, you can use different zookeeper quorum.
> HConnectionManager.getConnection(HBaseConfiguration) returns new connection in default HTable constructor
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3962
> URL: https://issues.apache.org/jira/browse/HBASE-3962
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.90.1
> Reporter: Philippe
>
> The HBase instance are currently indexed by Configuration, which since HBASE-1976 does not have any other equivalence that the object equivalence.
> So, everytime a new configuration is passed to the method a new connection is created.
> If we create many HTable connections with the same configuration, there is no problem:
> HBaseConfiguration config = HBaseConfiguration.create();
> HTable table 1 = new HTable(config, "table1"); // init connection
> HTable table 2 = new HTable(config, "table2"); // re-use connection
> HTable table 3 = new HTable(config, "table3"); // re-use connection
> However, if we call the default constructor, or re-call HBaseConfiguration.create();, we will pass a new instance of the configuration to the constructor. This will cause many connections to be created:
> HTable table 1 = new HTable("table1"); // init connection
> HTable table 2 = new HTable("table2"); // init new connection
> HTable table 3 = new HTable("table3"); // init new connection
> I know connection should be pooled, but sometimes we have to create a new connection, and without having access to a previously instanced configuration object.
> Since zookeeper has a max client connection (default was 30, now is 10), after creating 30 instances of HTable, we can no longer access to the database.
> In addition to this, the HBASE_INSTANCES map does not close the connection when removing the eldest entry. So if we have a larger maxConnection value than the hard-coded MAX_CACHED_HBASE_INSTANCES variable, connections will remain but won't be closed. MAX_CACHED_HBASE_INSTANCES should actually be set from the hbase.zookeeper.property.maxClientCnxns parameter (value + 1).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira