You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by "Gerry Holly (JIRA)" <ji...@apache.org> on 2007/07/10 04:51:04 UTC
[jira] Created: (AXIS2-2931) CLONE -Web Service request loop causes
many TIME_WAIT Connections & "BindException: Address already in use:
connect"
CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
--------------------------------------------------------------------------------------------------------------------
Key: AXIS2-2931
URL: https://issues.apache.org/jira/browse/AXIS2-2931
Project: Axis 2.0 (Axis2)
Issue Type: Bug
Affects Versions: 1.1.1, nightly
Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
Reporter: Gerry Holly
Assignee: Davanum Srinivas
Priority: Critical
I am performing the following:
- Executing a tight loop calling a singel web service
- Using the same Stub over and over
- Creating the Stub with a custom ConfigurationContext (see below)
- Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
java.net.BindException: Address already in use: connect
It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
Here is the code I am using to create the Stub:
HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
connectionManagerParams.setTcpNoDelay(true);
connectionManagerParams.setStaleCheckingEnabled(true);
MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
connectionManager.setParams(connectionManagerParams);
HttpClient httpClient = new HttpClient(connectionManager);
ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
stub = new MyServiceStub(configurationContext, target);
boolean success = true;
while(success) {
success = stub.performService(records);
stub.cleanup();
// System.gc();
}
Here is the complete stack trace:
Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
at com.ws.client.MyClient.main(MyClient.java:88)
Caused by: java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-2931) CLONE -Web Service request loop
causes many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Alexis Midon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676360#action_12676360 ]
Alexis Midon commented on AXIS2-2931:
-------------------------------------
I have the exact same issue with the default settings (axis 1.3): unclosed connection and too many open files.
I tried the options REUSE_HTTP_CLIENT and AUTO_RELEASE_CONNECTION and drew the exact same conclusions as Guillaume [1]
. REUSE_HTTP_CLIENT starves the pool and invocations hang out
. AUTO_RELEASE_CONNECTION release the connection, but before the SOAP handler had a chance to read the http input stream :(
[1] https://issues.apache.org/jira/browse/AXIS2-935?focusedCommentId=12513543#action_12513543
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AXIS2-2931) CLONE -Web Service request loop causes
many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Alexis Midon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexis Midon updated AXIS2-2931:
--------------------------------
Attachment: 0001-AXIS2-2931-AXIS2-3478-Release-the-HTTP-connection.patch
Here is attached a patch for AXIS2-2931 & AXIS2-3478.
## What's in there?
The patch consists in (always) releasing the HTTP connection to avoid connection leak (default behavior), or pool starvation if a connection mangaer is shared across multiple invocations.
The connection can be released safely because the response body is now fully read and wrapped in a ByteArrayInputStream. This avoids passing the connection stream directly and prevents reading from a close stream. (AXIS2-3478)
I don't think that loading the response in memory should be considered as a performance issue, because an xml parser will have to do it at some point anyway.
In that context, the option AUTO_RELEASE_CONNECTION does not male sense anymore.
## Where to apply the patch?
This patch can be seamlessly applied in axis2-1.3 and 1.4. But I guess you guys are working on 1.5 now.
So simply apply it:
http://svn.eu.apache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/http
## How to test it?
. activate the log category org.apache.commons.httpclient.HttpConnection to DEBUG.
. invoke some service with the ServiceClient
. you will get 2 consecutive log messages saying:
"enter HttpConnection.releaseConnection()"
"Releasing connection back to connection manager."
. commit ;)
## Misc
The commit is also available at github [1].
A jar is ready for testing in this public Intalio repo [2].
[1] http://github.com/intalio/axis2/commit/c2a0b92b643713a5efaf18436616d465e2182a23
[2] http://www.intalio.org/public/maven2/org/apache/axis2/axis2-kernel/1.3i3/
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
> Attachments: 0001-AXIS2-2931-AXIS2-3478-Release-the-HTTP-connection.patch
>
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AXIS2-2931) CLONE -Web Service request loop
causes many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Alexis Midon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677625#action_12677625 ]
Alexis Midon commented on AXIS2-2931:
-------------------------------------
a related thread on axis2 ml:
http://markmail.org/thread/voabzcl6u2pck74n
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
> Attachments: 0001-AXIS2-2931-AXIS2-3478-Release-the-HTTP-connection.patch
>
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AXIS2-2931) CLONE -Web Service request loop
causes many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Deepal Jayasinghe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541901 ]
Deepal Jayasinghe commented on AXIS2-2931:
------------------------------------------
Please test with Axis2 1.3 , I think we have fixed the issue in 1.3 release.
Thanks
Deepal
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-2931) CLONE -Web Service request loop
causes many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Tobias Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542809 ]
Tobias Jordan commented on AXIS2-2931:
--------------------------------------
I am sorry, but the issue still exists in 1.3. I tried to set the AUTO_RELEASE_CONNECTION option in a blocking operations client (together with REUSE_HTTP_CLIENT). This results in 'Attempted read on closed stream' exceptions.
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-2931) CLONE -Web Service request loop
causes many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Gerry Holly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12511321 ]
Gerry Holly commented on AXIS2-2931:
------------------------------------
Hello,
It seems the AUTO_RELEASE_CONNECTION solution only covers non-blocking operations. What about blocking operations?
In a blocking scenario, when "handleResponse" is invoked in HTTPSender, it stores a reference to the InputStream in the message context so it can be later processed by the application. Your fix renders this InputStream unreadable before the OutInAxisOperationClient gets a chance to process the SOAP message (through TransportUtils).
This is the exception I get after implementing your HTTPConstants.java and HTTPSender.java (not using the nightly build):
Caused by: org.apache.axis2.AxisFault: Attempted read on closed stream.
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:51)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:130)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:77)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:356)
at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
... 7 more
Caused by: java.io.IOException: Attempted read on closed stream.
at org.apache.commons.httpclient.AutoCloseInputStream.isReadAllowed(AutoCloseInputStream.java:165)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:106)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.apache.axis2.builder.BuilderUtil.getReader(BuilderUtil.java:191)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:41)
... 11 more
Gerry
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Assignee: Davanum Srinivas
> Priority: Critical
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-2931) CLONE -Web Service request loop
causes many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Andreas Veithen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677308#action_12677308 ]
Andreas Veithen commented on AXIS2-2931:
----------------------------------------
Reading the entire response into memory is a performance issue for two reasons:
* Axis2 would no longer be able to start processing the response before the entire message has been received.
* The raw content of the entire message would have to be kept in memory. The argument that "an xml parser will have to do it at some point anyway" is not valid. Even if the full Axiom tree of the message is built (which is not necessarily the case), there is no justification for having two copies of the message in memory (one in the form of a byte array, the other in the form of an Axiom tree).
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
> Attachments: 0001-AXIS2-2931-AXIS2-3478-Release-the-HTTP-connection.patch
>
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AXIS2-2931) CLONE -Web Service request loop
causes many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Haroon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741318#action_12741318 ]
Haroon commented on AXIS2-2931:
-------------------------------
We are facing a similar issue with Axis2-1.2 version when we use Axis2 as our Webservices client to make WS calls.
Exception stack trace is : org.apache.axis2.AxisFault: Too many open files
Caused by: org.apache.axis2.AxisFault: Too many open files
Caused by: java.net.SocketException: Too many open files
I saw this post here on AXIS2-2931 about the patch for this fix. And it is mentioned that it can be seamlessly applied in axis2-1.3 and 1.4
<https://issues.apache.org/jira/browse/AXIS2-2931?focusedCommentId=12676837&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12676837>
However, since we are on Axis2-1.2 version, could you please let us know the approach for fixing this issue, with minimum changes to the list of axis2 jars?
I'm doubtful, if I take the fix from <http://www.intalio.org/public/maven2/org/apache/axis2/axis2-kernel/1.3i3/> and replace axis2-kernel-1.2.jar with axis2-kernel-1.3i3.jar, what other dependent jars I might need to upgrade?
Any sort of help/guidelines on applying the proper fix to axis2-1.2 version is appreciated.
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
> Attachments: 0001-AXIS2-2931-AXIS2-3478-Release-the-HTTP-connection.patch
>
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AXIS2-2931) CLONE -Web Service request loop causes
many TIME_WAIT Connections & "BindException: Address already in use:
connect"
Posted by "Davanum Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Davanum Srinivas updated AXIS2-2931:
------------------------------------
Assignee: (was: Davanum Srinivas)
> CLONE -Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: AXIS2-2931
> URL: https://issues.apache.org/jira/browse/AXIS2-2931
> Project: Axis 2.0 (Axis2)
> Issue Type: Bug
> Affects Versions: 1.1.1, nightly
> Environment: Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.
> Reporter: Gerry Holly
> Priority: Critical
>
> I am performing the following:
> - Executing a tight loop calling a singel web service
> - Using the same Stub over and over
> - Creating the Stub with a custom ConfigurationContext (see below)
> - Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)
> After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):
> java.net.BindException: Address already in use: connect
> It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.
> If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.
> Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.
> Here is the code I am using to create the Stub:
> HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
> connectionManagerParams.setTcpNoDelay(true);
> connectionManagerParams.setStaleCheckingEnabled(true);
>
> MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
> connectionManager.setParams(connectionManagerParams);
> HttpClient httpClient = new HttpClient(connectionManager);
>
> ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
> configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
> configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
>
> stub = new MyServiceStub(configurationContext, target);
>
> boolean success = true;
> while(success) {
> success = stub.performService(records);
> stub.cleanup();
> // System.gc();
> }
> Here is the complete stack trace:
> Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
> at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
> at com.ws.client.MyClient.main(MyClient.java:88)
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org