RE: Connection reuse change in 2.7.8?

["Warren, Jared S" <js...@jcp.com>]

I wanted to update this thread with my learnings, in case they might benefit someone else down the road (or in case this teases anyone's memory for a better fix).  We've learned two things:

1.In Oracle JRE 1.7.0_51-b13, when HTTP KeepAlive is enabled by users java.net.HttpURLConnection and javax.net.ssl.HttpsURLConnection (both subclasses of java.net.URLConnection), and the server terminates the HTTP connection (as it is allowed to do as per RFC), the JRE inappropriately continues to use the disconnected socket (when that socket is in what appears to be one of the FIN_WAIT states).  In doing so, the JRE "waits" on the socket send (that will never return a response) until the read timeout occurs, when the socket timeout exception is thrown.  This functionality works correctly in the IBM JRE under WebSphere, but not in the Oracle JRE under WebLogic.
2.In the same Oracle JRE release, when the "Connection: " header field on an HTTP or HTTPS request is set to "close" (rather than Keep-Alive), as per RFC, subclasses of URLConnection continue to attempt to reuse the same underlying socket, even though they should not.

Our workaround is to set the "http.keepAlive" system property to "false" to disable keep alive on all connections.  This is not an acceptable long-term workaround, as the additional time and resource overhead to setup and breakdown a connection on every request is unacceptable - we'll have to keep working on something to get this working correctly.

-----Original Message-----
From: Warren, Jared S [mailto:jswarren@jcp.com]
Sent: Wednesday, July 30, 2014 7:04 PM
To: users@cxf.apache.org
Subject: RE: Connection reuse change in 2.7.8?

It's been an interesting day....

I have changed the KeepAlive policy to "close" (using the Java code manipulation method, as below).

After this change, on a local development environment (Win 7 64-bit, Tomcat 6.0, jdk 1.7.0_51) everything works as expected, and a wireshark trace shows that each connection is getting setup and broken down as expected.

However; when I deploy that to the test environment (RHEL 6.4, WebLogic 12.1, jdk 1.7.0_51), every second request fails.  When I put a trace on it, I can see that the first request happens as expected.  After the first request completes successfully, the server sends a FIN, which the client (where CXF is running) then ACKs.    CXF then attempts to re-use the same (already FIN-ACK'd) connection and receives a RST.  This appears to happen several times until eventually the "javax.xml.ws.WebServiceException: Could not send Message" exception with a root cause of "Caused by: java.net.SocketTimeoutException: Read time out after xyz millis" occurs.

Any advice appreciated!





....
httpClientPolicy.setConnection(ConnectionType.CLOSE);
conduit.setClient(httpClientPolicy);

-----Original Message-----
From: Warren, Jared S [mailto:jswarren@jcp.com]
Sent: Wednesday, July 30, 2014 1:13 PM
To: users@cxf.apache.org
Subject: RE: Connection reuse change in 2.7.8?

As I investigate: it doesn't look like we specify any keepalive parameters in our configuration (either the xml configuration, wsdl or Java code).

Does anyone know if the default keepalive behavior changed between 2.2.10 and 2.7.8?

-----Original Message-----
From: Warren, Jared S [mailto:jswarren@jcp.com]
Sent: Wednesday, July 30, 2014 12:40 PM
To: users@cxf.apache.org
Subject: Connection reuse change in 2.7.8?

Hey all -

We recently deployed a package that included several library upgrades, including an update from CXF 2.2.10 to 2.7.8 (it also included a migration from one application container to another - so there is quite a lot of noise in the system in terms of changes).  In this case, CXF is the consumer of a SOAP service hosted on another platform.  Nothing has changed wrt the service provider.

After the upgrade deployment, we started seeing relatively frequent "javax.xml.ws.WebServiceException: Could not send Message" exceptions with a root cause of "Caused by: java.net.SocketTimeoutException: Read time out after xyz millis".

When we revert our deployment to the 2.2.10-based release, the problem goes away.

When we performed a network trace to watch what's happening, we can see that the client is reusing the same TCP/IP connection for multiple requests (a good thing).  After many uses over a period of time, the service provider stack sends a FIN.  The client (CXF) continues attempting to send requests despite the FIN.  The service provider sends a RESET for each of those attempts.

Eventually, CXF times out and throws the exception above to the client code.

When we run the same trace on the 2.2.10 based infrastructure, we do not see any TCP/IP connection reuse.

I'm off to read CXF connection reuse documentation, but I thought I'd ask here first, "Any suggestions on a fix?"

Thanks
jared
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.