You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by "Jeff Peterson (JIRA)" <ji...@apache.org> on 2006/07/27 00:27:13 UTC
[jira] Created: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
---------------------------------------------------------------------------------------------------------------------
Key: AXIS2-935
URL: http://issues.apache.org/jira/browse/AXIS2-935
Project: Apache Axis 2.0 (Axis2)
Issue Type: Bug
Components: core
Affects Versions: 1.0
Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
Reporter: Jeff Peterson
Priority: Critical
It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12424901 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
I have found another possible cause. Looking through the ServiceClient.java code I noticed that complete() is not being called on the operation clients (mepClient).
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Priority: Critical
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Haroon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741314#action_12741314 ]
Haroon commented on AXIS2-935:
------------------------------
We are facing the same issue with Axis2-1.2 version when we use Axis2 as our Webservices client to make WS calls.
Exception stack trace is : org.apache.axis2.AxisFault: Too many open files
Caused by: org.apache.axis2.AxisFault: Too many open files
Caused by: java.net.SocketException: Too many open files
I saw this post on AXIS2-2931 on the patch for this fix. And it is mentioned that this patch can be seamlessly applied in axis2-1.3 and 1.4
<https://issues.apache.org/jira/browse/AXIS2-2931?focusedCommentId=12676837&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12676837>
However, since we are on Axis2-1.2 version, could you please let us know the approach for fixing this issue, with minimum changes to the list of axis2 jars?
I'm doubtful, if I take the fix from <http://www.intalio.org/public/maven2/org/apache/axis2/axis2-kernel/1.3i3/> and replace axis2-kernel-1.2.jar with axis2-kernel-1.3i3.jar, what other dependent jars I might need to upgrade?
Any sort of help/guidelines on applying the proper fix to axis2-1.2 version is appreciated.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assignee: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Mike Baroukh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528669 ]
Mike Baroukh commented on AXIS2-935:
------------------------------------
I just wanted to add a comment :
I have an application with 2 WebService. One with Axis1 and one with Axis2.
With Axis1, if I make many synchronous request, only one connection to the server is opened.
With Axis2, on connection is opened per request and they stay at CLOSE_WAT status.
(So, I have TooManyOpenFiles).
For now, I disabled HTTP 1.1...
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assignee: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Oleg Kalnichevski (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12444911 ]
Oleg Kalnichevski commented on AXIS2-935:
-----------------------------------------
Folks,
I may not know the full extent and all the details of this particular problem, so take my comment for what it is worth.
The described problem is not that infrequent in the HttpClient world. Per default HttpClient always attempts to keep HTTP connections alive for further re-use and may keep them in the connection pool indefinitely. So, some precautions must be taken in order to ensure proper release of network sockets.
(1) _Always_ shut down the multithreaded connection manager when it is no longer needed. It may take a quite while on some JVMs before it gets garbage collected along with all the connections and respective network sockets. If multithreaded connection managers are created frequently and are not cleanly shut down, one can easily end up with hundreds of connections in the "CLOSE_WAIT" state.
(2) Calling #closeIdleConnections() once is a while is a good idea, especially when anticipating (or coming back from) a long period of network inactivity.
Oleg
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Updated: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=all ]
Jeff Peterson updated AXIS2-935:
--------------------------------
Attachment: AXIS2-935.patch
Attached "AXIS2-935.patch"
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Updated: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Davanum Srinivas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=all ]
Davanum Srinivas updated AXIS2-935:
-----------------------------------
Priority: Critical (was: Blocker)
Not a blocker.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12436737 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
I think one of the key causes of this issue is a lack of proper cleanup/garbage collection. As a result, it is exhibited mostly in situations where the app server has a large heap and a lax garbage collection policy. Try increasing your app servers heap to something large (i was using 2GB).
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Deepal Jayasinghe (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12436797 ]
Deepal Jayasinghe commented on AXIS2-935:
-----------------------------------------
I went through your patch and found that you are keeping operationClient reference, AFAIK it is not the way of solving this problem. What if someone tries to invoke two invocations with the same service client ?
Therefore I think we need to come up with , good approach :)
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Updated: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Davanum Srinivas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=all ]
Davanum Srinivas updated AXIS2-935:
-----------------------------------
Priority: Blocker (was: Critical)
Blocker?
-- dims
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Simon So (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475234 ]
Simon So commented on AXIS2-935:
--------------------------------
While the patch didn't get in, if I:
1. set REUSE_HTTP_CLIENT == Boolean.TRUE
2. call operationclient.complete()
Then the CLOSE_WAIT issue will go away.
Listing the file descriptor which is in CLOSE_WAIT,
lsof | grep CLOSE_WAIT | wc -l
The number of CLOSE_WAITs hardly went up to 3. Mostly 0.
Before using 1 & 2 together, my program will run out of descriptors eventually.
Sorry I don't have a unit test to prove. It works for my environment.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Simon So (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475444 ]
Simon So commented on AXIS2-935:
--------------------------------
I backtracked.
It didn't work.
If REUSE_HTTP_CLIENT, is there a way to get rid of the CLOSE_WAIT other than garbage collect?
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Updated: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Deepal Jayasinghe (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=all ]
Deepal Jayasinghe updated AXIS2-935:
------------------------------------
Priority: Blocker (was: Critical)
I think we need to fix this before 1.1
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Assigned: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Deepal Jayasinghe (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=all ]
Deepal Jayasinghe reassigned AXIS2-935:
---------------------------------------
Assignee: Deepal Jayasinghe
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Resolved: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Davanum Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Davanum Srinivas resolved AXIS2-935.
------------------------------------
Resolution: Fixed
Added a AUTO_RELEASE_CONNECTION flag (see svn revision 537188). Please see integration\test\org\apache\axis2\async\AsyncService2Test.java on how to set up the MultiThreadedConnectionManager.
thanks,
dims
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Oleg Kalnichevski (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12445196 ]
Oleg Kalnichevski commented on AXIS2-935:
-----------------------------------------
Deepal,
The re-use of HttpClient instance(s) is in fact highly recommended. So, you should probably keep the option.
When REUSE_HTTP_CLIENT is set to false, always shut down the connection manager associated with the HttpClient instances before dereferencing it
MultiThreadedHttpConnectionManager connManager =
(MultiThreadedHttpConnectionManager) httpclient.getHttpConnectionManager();
connManager.shutdown();
This should take care of the problem.
Oleg
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "robert lazarski (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12436653 ]
robert lazarski commented on AXIS2-935:
---------------------------------------
I can't reproduce this via 'netstat -anc | grep CLOSE_WAIT' on linux. Recently another user claimed this was occurring:
http://marc.theaimsgroup.com/?l=axis-user&m=115868315001899&w=2
Here's the client I'm using for just a typical service with MR RawXMLINOutMessageReceiver .
package client;
import java.io.StringWriter;
import javax.xml.stream.XMLOutputFactory;
import org.apache.axiom.om.OMAbstractFactory;
import org.apache.axiom.om.OMElement;
import org.apache.axiom.om.OMFactory;
import org.apache.axiom.om.OMNamespace;
import org.apache.axis2.addressing.EndpointReference;
import org.apache.axis2.client.Options;
import org.apache.axis2.client.ServiceClient;
// jira imports
import org.apache.axis2.transport.http.HTTPConstants;
import org.apache.axis2.Constants;
import org.apache.axiom.soap.SOAPEnvelope;
import org.apache.axiom.soap.SOAPFactory;
import org.apache.axis2.context.MessageContext;
import org.apache.axis2.client.OperationClient;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamWriter;
import javax.xml.namespace.QName;
import org.apache.axis2.wsdl.WSDLConstants;
public class TestCW {
/** Access point inside the servlet container. **/
private static EndpointReference targetEPR =
new EndpointReference(
"http://localhost:8080/axis2/services/SpringAwareService");
/**
* Simple axis2 client.
*
* @param args Main
*/
public static void main(String[] args) {
try {
ServiceClient serviceClient = new ServiceClient();
Options options = new Options();
options.setTo(targetEPR);
/* code to test close_wait jira axis2-935 */
options.setTransportInProtocol(Constants.TRANSPORT_HTTP);
options.setProperty(HTTPConstants.SO_TIMEOUT,1000);
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.FALSE);
options.setProperty(HTTPConstants.CONNECTION_TIMEOUT, 5000);
MessageContext requetMessageContext = new MessageContext();
requetMessageContext.setEnvelope(getRequestEnvelope());
OperationClient opClient = serviceClient.createClient(ServiceClient.ANON_OUT_IN_OP);
opClient.addMessageContext(requetMessageContext);
opClient.setOptions(options);
/* end extra jira code */
opClient.execute(true);
SOAPEnvelope result = opClient.getMessageContext(
WSDLConstants.MESSAGE_LABEL_IN_VALUE).getEnvelope();
printResult(result);
} catch (Exception ex) {
ex.printStackTrace();
}
}
public static SOAPEnvelope getRequestEnvelope() {
SOAPFactory fac = OMAbstractFactory.getSOAP11Factory();
SOAPEnvelope envelope = fac.getDefaultEnvelope();
OMNamespace omNs = fac.createOMNamespace(
"http://springExample.org/example1", "example1");
OMElement method = fac.createOMElement("getValue", omNs);
OMElement value = fac.createOMElement("Text", omNs);
value.addChild(fac.createOMText(value, "Some String "));
method.addChild(value);
envelope.getBody().setFirstChild(method);
return envelope;
}
public static void printResult(SOAPEnvelope result) {
try {
if (result != null) {
//OMElement resultOM = result.getBody().getFirstChildWithName(new QName("example1"));
OMElement resultOM = result.getBody();
StringWriter writer = new StringWriter();
resultOM.serialize(XMLOutputFactory.newInstance()
.createXMLStreamWriter(writer));
writer.flush();
System.out.println("Response: " + writer.toString());
} else
System.out.println("Result is null");
} catch (Exception e) {
e.printStackTrace();
}
}
}
If there is some way to trigger the issue please comment.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Rigadon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664613#action_12664613 ]
Rigadon commented on AXIS2-935:
-------------------------------
I have a similar issue, I think Guillaume's comments above are very pertinent. In particular, for point 3, the process is something like:
AbstractHTTPSender.processResponse(HttpMethodBase httpMethod, MessageContext msgContext)
This sets the InputStream in the OperationContext:
OperationContext opContext = msgContext.getOperationContext();
if (opContext != null) {
opContext.setProperty(MessageContext.TRANSPORT_IN, in);
}
This then returns control back to HTTPSender.sendViaPost(MessageContext msgContext, URL url, String soapActionString) which calls cleanup(msgContext, postMethod);
private void cleanup(MessageContext msgContext, HttpMethod method) {
if (msgContext.isPropertyTrue(HTTPConstants.AUTO_RELEASE_CONNECTION)) {
method.releaseConnection();
}
}
Where the connection is released if HTTPConstants.AUTO_RELEASE_CONNECTION is set to TRUE. Which is fine, except that the stream is subsequently read by the process, and so fails with an IOException "Attempted read on closed stream".
We have tried (and failed) to code our way around this bug, but to no avail. Our latest solution is to go back to HTTP 1.0, I'll let y'all know how we get on :)
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assignee: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Oleg Kalnichevski (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12425509 ]
Oleg Kalnichevski commented on AXIS2-935:
-----------------------------------------
> When clean() is called in CommonsHttpTransportSender, I fetch the client, get the connection manager, and call closeIdleConnections(0)
As far as HttpClient is concerned, the patch looks good. HttpConnectionManager#closeIdleConnections() is the recommended way to go about persistent connections that are no longer needed.
Oleg
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Davanum Srinivas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12444905 ]
Davanum Srinivas commented on AXIS2-935:
----------------------------------------
Deepal,
Are u able to recreate this problem? Am not sure i agree with the priority if we can't recreate the problem.
-- dims
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12423877 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
I''ve done some more digging through the code and I am no longer convinced that this is an issue with the SOAPOverHTTPSender class. Perhaps there is another culprit...
My MEP looks like the following:
Service1 Service2
| OUT/IN Sync |
| ------------------------->|
| <------------------------ |
| |
| IN |
| <------------------------ |
| |
| IN |
| <------------------------ |
....
| |
| IN |
| <------------------------ |
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Priority: Critical
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12424935 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
I have finally been able to track down what I believe to be the root cause of this issue. The problem originates from how the commons-httpclient deals with http connections. By default it attempts to pool them within a "HttpConnectionManager."
Here's basically what I did:
First, I tweaked some code (as mentioned above) so that complete() was being called by ServiceClient on the various operation clients. This allowed the cleanup operation to propigate through to CommonsHttpTransportSender. Which, in turn, caused releaseConnection() to be called on the HttpMethod. I did some testing and noticed that this had no effect on the number of CLOSE_WAIT connections.
After more research I found out that HttpClient attempts to pool http connections for the purpose of connection reuse. The object which manages this pool is an instance of HttpConnectionManager which can be accessed via httpClient.getHttpConnectionManager(). So, basically, the problem boils down to these connection pools not being cleaned up properly. They are allowed to go out of scope and continue to live until garbage collection is triggered.
There are a couple solutions: the "Connection: close" header could be sent to disable connection pooling altogether or the pools can be purged as part of the cleanup() routine. I chose the latter for my test.
In order to purge the pool, I registered the HttpClient (although I probably should have done the HttpConnectionManager), in the message context properties. When clean() is called in CommonsHttpTransportSender, I fetch the client, get the connection manager, and call closeIdleConnections(0). Testing indicates that this has solved the issue.
I'll see if I can cleanup my working copy and post a patch.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Priority: Critical
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Updated: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Deepal Jayasinghe (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=all ]
Deepal Jayasinghe updated AXIS2-935:
------------------------------------
Priority: Critical (was: Blocker)
I dont think this a blocker for 1.1, so I am reducing the priority
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12423880 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
I forgot to mention, I noticed another interesting symptom. If I force the app server to perform garbage collection most of the connections in the CLOSE_WAIT state get cleaned up.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Priority: Critical
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Deepal Jayasinghe (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12445079 ]
Deepal Jayasinghe commented on AXIS2-935:
-----------------------------------------
Well httpclient will be reuse only if someone set the REUSE_HTTP_CLIENT flag. And the default case it wont happen , Axis2 will always create a new HTTPClient and uses that.
I was able to re-create the problem when I uses that property.
What if we remove REUSE_HTTP_CLIENT and create HTTPClient for each invocation ? I was able to send more than 1000 req w.o having any problem.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Davanum Srinivas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12445182 ]
Davanum Srinivas commented on AXIS2-935:
----------------------------------------
If they use REUSE_HTTP_CLIENT, they can access the HTTPClient and tweak whatever they need to tweak. So let's leave it alone.
thanks,
dims
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Blocker
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "guillaume cottenceau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513543 ]
guillaume cottenceau commented on AXIS2-935:
--------------------------------------------
I think it is probably not resolved. Several observations:
1. Deepal: REUSE_HTTP_CLIENT is *very* important, please don't remove it. In case of HTTPS connection, reusing an existing connection dramatically improves the performance because it saves a potentially rather long SSL handshake. I have the experience of something like 10 seconds cut down to less than 1 second on subsequent requests.
2. In recent version 1.2 of axis2, this problem is still there (of course). With REUSE_HTTP_CLIENT, a server responding with 502 bad gateway blocks my axis sending process after only two send attempts (probably because httpclient uses maximum 2 connections per host by default). In my opinion, this is a high priority problem which would advocate a new stable sub release. As far as I can tell, it is the code generation from WSDL which is at fault (I use adb): when invoking a SOAP request, the following line of code from the stub handles releasing the connection:
_messageContext.getTransportOut().getSender().cleanup(_messageContext);
however, this code is not executed in case an AxisFault exception is catched, which is the case with HTTP 502. The cleanup code would be a good candidate for being executed within a finally block instead of within the try block.
3. Revision 537188 seems to be a good effort towards the right direction, though:
3.a. it seems that it doesn't make sense to not activate AUTO_RELEASE_CONNECTION; if you read httpclient tutorial - http://jakarta.apache.org/commons/httpclient/tutorial.html - you'll see that invoking HttpMethod#releaseConnection "is a crucial step to keep things flowing. We must tell HttpClient that we are done with the connection and that it can now be reused. Without doing this HttpClient will wait indefinitely for a connection to free up so that it can be reused."
3.b. as far as I can read from axis2 1.2 code, HTTPSender's part of revision 537188 means calling method.releaseConnection() prior to parsing the response input stream (AbstractHTTPSender#processResponse merely sets the response input stream in the MessageContext but it's not parsed at that step), so I think the logic might be flawed - this is actually confirmed by the last comment ([03/Jul/07 11:08 PM]) from issue 2593)
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assignee: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12425034 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
By the way, that patch is against version 1.0 in svn (http://svn.apache.org/repos/asf/webservices/axis2/tags/java/v1.0/).
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Alexis Midon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676359#action_12676359 ]
Alexis Midon commented on AXIS2-935:
------------------------------------
I have the exact same issue with the default settings (axis 1.3): unclosed connection and too many open files.
I tried the options REUSE_HTTP_CLIENT and AUTO_RELEASE_CONNECTION and drew the exact same conclusions as Guillaume (see his comment above):
. REUSE_HTTP_CLIENT starves the pool and invocations hang out
. AUTO_RELEASE_CONNECTION release the connection, but before the SOAP handler had a chance to read the http input stream :(
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: https://issues.apache.org/jira/browse/AXIS2-935
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assignee: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12436890 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
The patch was for illustrative purposes. I think I said somewhere above that there is probably a better way of doing it. I am certainly not as familiar with the code as you, so please use what ever method you think is best.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-935) Connections are not properly released
causing them to stick in the "CLOSE_WAIT" state and cause "Too many files
open"
Posted by "Jeff Peterson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/AXIS2-935?page=comments#action_12445214 ]
Jeff Peterson commented on AXIS2-935:
-------------------------------------
I'm pretty sure that I did not have REUSE_HTTP_CLIENT set, unless its the default.
I still think this is a garbage collection/cleanup related issue. Even if you dont intend to reuse the HttpClient, the act of creating a HttpClient object also creates a new HttpConnectionManager for that client instance. The sockets associated with the HttpConnectionManager do not get cleaned up until garbage connection is triggered.
So, in the non-reuse case, lots of http client objects are created (one per request). Each one has its own HttpConnectionManager, which is never explicitly cleaned up. Instead, it is up to the container to garbage collect them and free their sockets. Unfortunately, if the container is slow to get around to garbage collection, this can leave a lot of sockets open.
> Connections are not properly released causing them to stick in the "CLOSE_WAIT" state and cause "Too many files open"
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-935
> URL: http://issues.apache.org/jira/browse/AXIS2-935
> Project: Apache Axis 2.0 (Axis2)
> Issue Type: Bug
> Components: kernel
> Affects Versions: 1.0
> Environment: Weblogic 8.1.2, Solaris 8 (dont ask!)
> Reporter: Jeff Peterson
> Assigned To: Deepal Jayasinghe
> Priority: Critical
> Attachments: AXIS2-935.patch
>
>
> It appears that org.apache.axis2.transport.http.SOAPOverHTTPSender does not properly cleanup the http post connection after payload has been sent.
> The symptoms of this problem show up in `netstat` as literally hundreds of connections in the "CLOSE_WAIT" state. On a busy server this can cause the application to report exceptions with the message "Too many open files."
> I looked in the source code for axis2 1.0 and discovered that in SOAPOverHTTPSender.java the releaseConnection() is never called on the postMethod variable. This seems to be a likely candidate for the source of the issue.
> See SOAPOverHTTPSender.java:97-118
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org