You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2011/04/12 22:38:05 UTC
[jira] [Created] (HBASE-3773) Set ZK max connections much higher in
0.90
Set ZK max connections much higher in 0.90
------------------------------------------
Key: HBASE-3773
URL: https://issues.apache.org/jira/browse/HBASE-3773
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.2
Reporter: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.90.3
I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019059#comment-13019059 ]
Jean-Daniel Cryans commented on HBASE-3773:
-------------------------------------------
I can try it.
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025921#comment-13025921 ]
Jean-Daniel Cryans commented on HBASE-3773:
-------------------------------------------
I thought that's what it was doing before? I haven't changed that here either.
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019177#comment-13019177 ]
stack commented on HBASE-3773:
------------------------------
Nice work proofing out which higher number to use.
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027264#comment-13027264 ]
Hudson commented on HBASE-3773:
-------------------------------
Integrated in HBase-TRUNK #1888 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1888/])
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019072#comment-13019072 ]
Jean-Daniel Cryans commented on HBASE-3773:
-------------------------------------------
I tried creating as many connections as I could in the shell on my MBP using a line like this:
bq. 3000.times { HTable.isTableEnabled("t") }
During that time I also monitored the total number of threads used by ZK and by the shell. What happens is that the server doesn't create new threads, only the client does, and it eventually dies of OOME unable to create new threads.
I wouldn't mind setting the max connections lower, like to a number just under MBP's default max number of threads, so that we would hit the ConnectionLoss before OOME.
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019036#comment-13019036 ]
stack commented on HBASE-3773:
------------------------------
OK. Will be interesting to see what kinda issues folks run into when they have 32k connections to an ensemble..... Hopefully they will level off well before we hit this big number.
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "Karthick Sankarachary (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025920#comment-13025920 ]
Karthick Sankarachary commented on HBASE-3773:
----------------------------------------------
Just FYI, HBASE-3777 attempts to address the underlying ZK connection overload issue, and one of the things that it does is set the max connections per IP (viz., HConnectionManager#MAX_CACHED_HBASE_INSTANCES) to the value specified in ""hbase.zookeeper.property.maxClientCnxns". Please let me know if that's the way to go.
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3773) Set ZK max connections much higher
in 0.90
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-3773.
---------------------------------------
Resolution: Fixed
Assignee: Jean-Daniel Cryans
Release Note: The max connections per IP for ZK is now set at 2000, but only for 0.90
Committed to branch after making sure it's working as advertised.
> Set ZK max connections much higher in 0.90
> ------------------------------------------
>
> Key: HBASE-3773
> URL: https://issues.apache.org/jira/browse/HBASE-3773
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.3
>
>
> I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.
> For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira