You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "jack levin (Created) (JIRA)" <ji...@apache.org> on 2011/11/02 23:53:32 UTC
[jira] [Created] (HBASE-4733) Rest client does not close zookeeper
sessions (leaking sessions for each GET or PUT)
Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)
------------------------------------------------------------------------------------
Key: HBASE-4733
URL: https://issues.apache.org/jira/browse/HBASE-4733
Project: HBase
Issue Type: Bug
Components: rest
Affects Versions: 0.90.4
Environment: Fedora 13.
Reporter: jack levin
Fix For: 0.90.4
This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:
2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:
2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4733) Rest client does not close
zookeeper sessions (leaking sessions for each GET or PUT)
Posted by "Andrew Purtell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144289#comment-13144289 ]
Andrew Purtell commented on HBASE-4733:
---------------------------------------
bq. FYI this patch applies well to 0.90.4, those folks that use it in production should be able to fix their REST issue right away.
Yes
> Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)
> ------------------------------------------------------------------------------------
>
> Key: HBASE-4733
> URL: https://issues.apache.org/jira/browse/HBASE-4733
> Project: HBase
> Issue Type: Bug
> Components: rest
> Affects Versions: 0.90.4
> Environment: Fedora 13.
> Reporter: jack levin
> Labels: newbie
> Fix For: 0.90.4
>
> Attachments: my.patch
>
>
> This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:
> 2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
> 2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
> 2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
> 2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
> 2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
> 2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4733) Rest client does not close zookeeper
sessions (leaking sessions for each GET or PUT)
Posted by "jack levin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
jack levin updated HBASE-4733:
------------------------------
Attachment: my.patch
The Patch.
> Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)
> ------------------------------------------------------------------------------------
>
> Key: HBASE-4733
> URL: https://issues.apache.org/jira/browse/HBASE-4733
> Project: HBase
> Issue Type: Bug
> Components: rest
> Affects Versions: 0.90.4
> Environment: Fedora 13.
> Reporter: jack levin
> Labels: newbie
> Fix For: 0.90.4
>
> Attachments: my.patch
>
>
> This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:
> 2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
> 2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
> 2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
> 2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
> 2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
> 2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4733) Rest client does not close zookeeper
sessions (leaking sessions for each GET or PUT)
Posted by "Andrew Purtell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-4733:
----------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Dup of HBASE-4684.
> Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)
> ------------------------------------------------------------------------------------
>
> Key: HBASE-4733
> URL: https://issues.apache.org/jira/browse/HBASE-4733
> Project: HBase
> Issue Type: Bug
> Components: rest
> Affects Versions: 0.90.4
> Environment: Fedora 13.
> Reporter: jack levin
> Labels: newbie
> Fix For: 0.90.4
>
> Attachments: my.patch
>
>
> This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:
> 2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
> 2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
> 2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
> 2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
> 2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
> 2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4733) Rest client does not close
zookeeper sessions (leaking sessions for each GET or PUT)
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142667#comment-13142667 ]
stack commented on HBASE-4733:
------------------------------
Is this same as "HBASE-4684 REST server is leaking ZK connections in 0.90"
> Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)
> ------------------------------------------------------------------------------------
>
> Key: HBASE-4733
> URL: https://issues.apache.org/jira/browse/HBASE-4733
> Project: HBase
> Issue Type: Bug
> Components: rest
> Affects Versions: 0.90.4
> Environment: Fedora 13.
> Reporter: jack levin
> Labels: newbie
> Fix For: 0.90.4
>
> Attachments: my.patch
>
>
> This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:
> 2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
> 2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
> 2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
> 2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
> 2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
> 2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4733) Rest client does not close
zookeeper sessions (leaking sessions for each GET or PUT)
Posted by "jack levin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144264#comment-13144264 ]
jack levin commented on HBASE-4733:
-----------------------------------
FYI this patch applies well to 0.90.4, those folks that use it in production should be able to fix their REST issue right away. Works well in our cluster.
> Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)
> ------------------------------------------------------------------------------------
>
> Key: HBASE-4733
> URL: https://issues.apache.org/jira/browse/HBASE-4733
> Project: HBase
> Issue Type: Bug
> Components: rest
> Affects Versions: 0.90.4
> Environment: Fedora 13.
> Reporter: jack levin
> Labels: newbie
> Fix For: 0.90.4
>
> Attachments: my.patch
>
>
> This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:
> 2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
> 2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
> 2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
> 2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
> 2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
> 2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4733) Rest client does not close zookeeper
sessions (leaking sessions for each GET or PUT)
Posted by "jack levin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
jack levin updated HBASE-4733:
------------------------------
Labels: newbie (was: )
Status: Patch Available (was: Open)
--- src/main/java/org/apache/hadoop/hbase/rest/TableResource.java
+++ src/main/java/org/apache/hadoop/hbase/rest/TableResource.java
@@ -37,6 +37,7 @@
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HTableDescriptor;
+import org.apache.hadoop.hbase.client.HConnectionManager;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
@@ -135,6 +136,7 @@
try {
HBaseAdmin admin = new HBaseAdmin(servlet.getConfiguration());
HTableDescriptor htd = admin.getTableDescriptor(Bytes.toBytes(table));
+ HConnectionManager.deleteConnection(admin.getConfiguration(), false);
for (HColumnDescriptor hcd: htd.getFamilies()) {
for (Map.Entry<ImmutableBytesWritable, ImmutableBytesWritable> e:
hcd.getValues().entrySet()) {
> Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)
> ------------------------------------------------------------------------------------
>
> Key: HBASE-4733
> URL: https://issues.apache.org/jira/browse/HBASE-4733
> Project: HBase
> Issue Type: Bug
> Components: rest
> Affects Versions: 0.90.4
> Environment: Fedora 13.
> Reporter: jack levin
> Labels: newbie
> Fix For: 0.90.4
>
>
> This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:
> 2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
> 2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> 2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
> 2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
> 2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
> 2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
> After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
> 2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
> 2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
> 2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
> 2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
> 2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
> 2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
> 2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
> 2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira