You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@continuum.apache.org by "Brett Porter (JIRA)" <ji...@codehaus.org> on 2012/11/22 09:53:13 UTC
[jira] (CONTINUUM-2693) File handle leak with TCP connections in
CLOSE_WAIT when using distributed builds
Brett Porter created CONTINUUM-2693:
---------------------------------------
Summary: File handle leak with TCP connections in CLOSE_WAIT when using distributed builds
Key: CONTINUUM-2693
URL: https://jira.codehaus.org/browse/CONTINUUM-2693
Project: Continuum
Issue Type: Bug
Affects Versions: 1.4.1
Reporter: Brett Porter
If you generate a lot of requests using the XMLRPC layer, such as when distributed builds are active, it is possible for connections to stay in the {{CLOSE_WAIT}} state until they are garbage collected. On a busy server with the default file limits, this can result in a {{IOException: Too many open files}} error.
The workaround is to increase the limit of the files for the user running Continuum. However, the XMLRPC layer should manage the connections better to avoid them getting into this state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (CONTINUUM-2693) File handle leak with TCP connections in
CLOSE_WAIT when using distributed builds
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/CONTINUUM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=314521#comment-314521 ]
Brett Porter commented on CONTINUUM-2693:
-----------------------------------------
went with option #2. The JDK URL connection could well have had other impacts.
While just setting the connection manager to close after each request, went with the multi-threaded manager which should manage the connections efficiently, even if a couple remain open.
Note that there are other occasions where sockets are left open (e.g. AbstractContinuumProjectBuilder, which has its own httpclient v4). Unifying the HTTP clients used would be beneficial in future, but is best evaluated if the XMLRPC layer gets replaced with something else.
> File handle leak with TCP connections in CLOSE_WAIT when using distributed builds
> ---------------------------------------------------------------------------------
>
> Key: CONTINUUM-2693
> URL: https://jira.codehaus.org/browse/CONTINUUM-2693
> Project: Continuum
> Issue Type: Bug
> Affects Versions: 1.4.1
> Reporter: Brett Porter
> Assignee: Brett Porter
> Fix For: 1.4.1
>
>
> If you generate a lot of requests using the XMLRPC layer, such as when distributed builds are active, it is possible for connections to stay in the {{CLOSE_WAIT}} state until they are garbage collected. On a busy server with the default file limits, this can result in a {{IOException: Too many open files}} error.
> The workaround is to increase the limit of the files for the user running Continuum. However, the XMLRPC layer should manage the connections better to avoid them getting into this state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (CONTINUUM-2693) File handle leak with TCP connections in
CLOSE_WAIT when using distributed builds
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/CONTINUUM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Porter reassigned CONTINUUM-2693:
---------------------------------------
Assignee: Brett Porter
> File handle leak with TCP connections in CLOSE_WAIT when using distributed builds
> ---------------------------------------------------------------------------------
>
> Key: CONTINUUM-2693
> URL: https://jira.codehaus.org/browse/CONTINUUM-2693
> Project: Continuum
> Issue Type: Bug
> Affects Versions: 1.4.1
> Reporter: Brett Porter
> Assignee: Brett Porter
> Fix For: 1.4.1
>
>
> If you generate a lot of requests using the XMLRPC layer, such as when distributed builds are active, it is possible for connections to stay in the {{CLOSE_WAIT}} state until they are garbage collected. On a busy server with the default file limits, this can result in a {{IOException: Too many open files}} error.
> The workaround is to increase the limit of the files for the user running Continuum. However, the XMLRPC layer should manage the connections better to avoid them getting into this state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (CONTINUUM-2693) File handle leak with TCP connections in
CLOSE_WAIT when using distributed builds
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/CONTINUUM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=314149#comment-314149 ]
Brett Porter commented on CONTINUUM-2693:
-----------------------------------------
I traced this into the way the XMLRPC is set up.
Atlassian XMLRPC Binder sets up Apache XMLRPC Client using the Commons HTTP Client 3.1 transport. HTTP Client is using the default {{SimpleHttpConnectionManager}}. These are all hard coded.
Because the binder is not reused, a new XMLRPC client is set up for each call. The XMLRPC client calls {{releaseConnection}} on the HTTP Client, which leaves the physical connection open for reuse - but with nothing set up to reuse it or terminate it, it is left open until GC happens on the socket. This is often quick, but some connections make it into the more permanent area of heap.
See: http://fuyun.org/2009/09/connection-close-in-httpclient/
Medium term, the XMLRPC should be replaced with JAX-RS.
Potential short term solutions:
# adjust the binder to use the JDK URL Connection instead of the HTTP Client
# try and adjust the HTTP Client to use a multi-threaded connection manager, and then reuse the binder/xmlrpc-client/http-client
# configure HTTP Client's background job to cleanup idle connections
This needs further investigation.
> File handle leak with TCP connections in CLOSE_WAIT when using distributed builds
> ---------------------------------------------------------------------------------
>
> Key: CONTINUUM-2693
> URL: https://jira.codehaus.org/browse/CONTINUUM-2693
> Project: Continuum
> Issue Type: Bug
> Affects Versions: 1.4.1
> Reporter: Brett Porter
> Fix For: 1.4.1
>
>
> If you generate a lot of requests using the XMLRPC layer, such as when distributed builds are active, it is possible for connections to stay in the {{CLOSE_WAIT}} state until they are garbage collected. On a busy server with the default file limits, this can result in a {{IOException: Too many open files}} error.
> The workaround is to increase the limit of the files for the user running Continuum. However, the XMLRPC layer should manage the connections better to avoid them getting into this state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (CONTINUUM-2693) File handle leak with TCP connections in
CLOSE_WAIT when using distributed builds
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/CONTINUUM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Porter updated CONTINUUM-2693:
------------------------------------
Fix Version/s: 1.4.1
> File handle leak with TCP connections in CLOSE_WAIT when using distributed builds
> ---------------------------------------------------------------------------------
>
> Key: CONTINUUM-2693
> URL: https://jira.codehaus.org/browse/CONTINUUM-2693
> Project: Continuum
> Issue Type: Bug
> Affects Versions: 1.4.1
> Reporter: Brett Porter
> Fix For: 1.4.1
>
>
> If you generate a lot of requests using the XMLRPC layer, such as when distributed builds are active, it is possible for connections to stay in the {{CLOSE_WAIT}} state until they are garbage collected. On a busy server with the default file limits, this can result in a {{IOException: Too many open files}} error.
> The workaround is to increase the limit of the files for the user running Continuum. However, the XMLRPC layer should manage the connections better to avoid them getting into this state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (CONTINUUM-2693) File handle leak with TCP connections in
CLOSE_WAIT when using distributed builds
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/CONTINUUM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Porter closed CONTINUUM-2693.
-----------------------------------
Resolution: Fixed
> File handle leak with TCP connections in CLOSE_WAIT when using distributed builds
> ---------------------------------------------------------------------------------
>
> Key: CONTINUUM-2693
> URL: https://jira.codehaus.org/browse/CONTINUUM-2693
> Project: Continuum
> Issue Type: Bug
> Affects Versions: 1.4.1
> Reporter: Brett Porter
> Assignee: Brett Porter
> Fix For: 1.4.1
>
>
> If you generate a lot of requests using the XMLRPC layer, such as when distributed builds are active, it is possible for connections to stay in the {{CLOSE_WAIT}} state until they are garbage collected. On a busy server with the default file limits, this can result in a {{IOException: Too many open files}} error.
> The workaround is to increase the limit of the files for the user running Continuum. However, the XMLRPC layer should manage the connections better to avoid them getting into this state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira