Socket issues with 2.6.1 and JDK 7

["Canning, Charles" <cc...@stubhub.com>]

Hi,

We are using the 2.6.1 WebClient for REST calls and we are running into issues where the sockets try to connect to 0.0.0.0 (anywhere address) after the client has been running for a while. After some analysis of the symptoms and going through the source for the sun.net and cxf code, we believe it is related to this Oracle document

https://docs.oracle.com/javase/7/docs/technotes/guides/net/http-keepalive.html

And around properly supporting 4xx/5xx status codes and the internal keep-alive cache.

Here is what I think is the relevant excerpt:

What's new in Tiger?

When the application encounters a HTTP 400 or 500 response, it may ignore the IOException and then may issue another HTTP request. In this case, the underlying TCP connection won't be Kept-Alive because the response body is still there to be consumed, so the socket connection is not cleared, therefore not available for reuse. What the application needs to do is call HttpURLConnection.getErrorStream() after catching the IOException , read the response body, then close the stream. However, some existing applications are not doing this. As a result, they do not benefit from persistent connections. To address this problem, we have introduced a workaround.

The workaround involves buffering the response body if the response is >=400, up to a certain amount and within a time limit, thus freeing up the underlying socket connection for reuse. The rationale behind this is that when the server responds with a >=400 error (client error or server error. One example is "404: File Not Found" error), the server usually sends a small response body to explain whom to contact and what to do to recover.

Several new Sun implementation specific properties are introduced to help clean up the connections after error response from the server.

The major one is:

sun.net.http.errorstream.enableBuffering=<boolean>
default: false

With the above system property set to true (default is false), when the response code is >=400, the HTTP handler will try to buffer the response body. Thus freeing up the underlying socket connection for reuse. Thus, even if the application doesn't call getErrorStream(), read the response body, and then call close(), the underlying socket connection may still be kept-alive and reused.

The following two system properties provide further control to the error stream buffering behavior:

sun.net.http.errorstream.timeout=<int> in millisecond
default: 300 millisecond

sun.net.http.errorstream.bufferSize=<int> in bytes
default: 4096 bytes

What can you do to help with Keep-Alive?

Do not abandon a connection by ignoring the response body. Doing so may results in idle TCP connections. That needs to be garbage collected when they are no longer referenced.

If getInputStream() successfully returns, read the entire response body.

When calling getInputStream() from HttpURLConnection, if an IOException occurs, catch the exception and call getErrorStream() to get the response body (if there is any).

Reading the response body cleans up the connection even if you are not interested in the response content itself. But if the response body is long and you are not interested in the rest of it after seeing the beginning, you can close the InputStream. But you need to be aware that more data could be on its way. Thus the connection may not be cleared for reuse.

Here's a code example that complies to the above recommendation:

try {
        URL a = new URL(args[0]);
        URLConnection urlc = a.openConnection();
        is = conn.getInputStream();
        int ret = 0;
        while ((ret = is.read(buf)) > 0) {
          processBuf(buf);
        }
        // close the inputstream
        is.close();
} catch (IOException e) {
        try {
                respCode = ((HttpURLConnection)conn).getResponseCode();
                es = ((HttpURLConnection)conn).getErrorStream();
                int ret = 0;
                // read the response body
                while ((ret = es.read(buf)) > 0) {
                        processBuf(buf);
                }
                // close the errorstream
                es.close();
        } catch(IOException ex) {
                // deal with the exception
        }
}


My question(s) are:


  *   Does CXF WebClient handle this?
     *   If so, which versions?
     *   Where is the code that handles this located
  *   Are users of WebClient supposed to manage this?
  *   Have you seen this error before?

Any help/pointers would be greatly appreciated.

Thanks,
Chuck

Re: Socket issues with 2.6.1 and JDK 7

[Sergey Beryozkin <sb...@gmail.com>]

Hi

Thanks for this analysis.
CXF 2.6.x is not supported any longer but CXF 2.7.x still is,
It is a bit late so I only checked the trunk (CXF 3.1.0-SNAPSHOT), can 
you please check

URLConnectionHttpConduit.getInputStream() ? It appears to be doing the 
right code

In CXF 2.7.x this method is probably directly in HttpConduit...

Besides that I guess WebClient can auto-close InputStream it reads. 
There's a property for it, "response.stream.auto.close". Or if you read 
InputStream directly then it needs to be closed manually

Cheers, Sergey


On 13/04/15 16:55, Canning, Charles wrote:
> Hi,
>
> We are using the 2.6.1 WebClient for REST calls and we are running into issues where the sockets try to connect to 0.0.0.0 (anywhere address) after the client has been running for a while. After some analysis of the symptoms and going through the source for the sun.net and cxf code, we believe it is related to this Oracle document
>
> https://docs.oracle.com/javase/7/docs/technotes/guides/net/http-keepalive.html
>
> And around properly supporting 4xx/5xx status codes and the internal keep-alive cache.
>
> Here is what I think is the relevant excerpt:
>
> What's new in Tiger?
>
> When the application encounters a HTTP 400 or 500 response, it may ignore the IOException and then may issue another HTTP request. In this case, the underlying TCP connection won't be Kept-Alive because the response body is still there to be consumed, so the socket connection is not cleared, therefore not available for reuse. What the application needs to do is call HttpURLConnection.getErrorStream() after catching the IOException , read the response body, then close the stream. However, some existing applications are not doing this. As a result, they do not benefit from persistent connections. To address this problem, we have introduced a workaround.
>
> The workaround involves buffering the response body if the response is >=400, up to a certain amount and within a time limit, thus freeing up the underlying socket connection for reuse. The rationale behind this is that when the server responds with a >=400 error (client error or server error. One example is "404: File Not Found" error), the server usually sends a small response body to explain whom to contact and what to do to recover.
>
> Several new Sun implementation specific properties are introduced to help clean up the connections after error response from the server.
>
> The major one is:
>
> sun.net.http.errorstream.enableBuffering=<boolean>
> default: false
>
> With the above system property set to true (default is false), when the response code is >=400, the HTTP handler will try to buffer the response body. Thus freeing up the underlying socket connection for reuse. Thus, even if the application doesn't call getErrorStream(), read the response body, and then call close(), the underlying socket connection may still be kept-alive and reused.
>
> The following two system properties provide further control to the error stream buffering behavior:
>
> sun.net.http.errorstream.timeout=<int> in millisecond
> default: 300 millisecond
>
> sun.net.http.errorstream.bufferSize=<int> in bytes
> default: 4096 bytes
>
> What can you do to help with Keep-Alive?
>
> Do not abandon a connection by ignoring the response body. Doing so may results in idle TCP connections. That needs to be garbage collected when they are no longer referenced.
>
> If getInputStream() successfully returns, read the entire response body.
>
> When calling getInputStream() from HttpURLConnection, if an IOException occurs, catch the exception and call getErrorStream() to get the response body (if there is any).
>
> Reading the response body cleans up the connection even if you are not interested in the response content itself. But if the response body is long and you are not interested in the rest of it after seeing the beginning, you can close the InputStream. But you need to be aware that more data could be on its way. Thus the connection may not be cleared for reuse.
>
> Here's a code example that complies to the above recommendation:
>
> try {
>          URL a = new URL(args[0]);
>          URLConnection urlc = a.openConnection();
>          is = conn.getInputStream();
>          int ret = 0;
>          while ((ret = is.read(buf)) > 0) {
>            processBuf(buf);
>          }
>          // close the inputstream
>          is.close();
> } catch (IOException e) {
>          try {
>                  respCode = ((HttpURLConnection)conn).getResponseCode();
>                  es = ((HttpURLConnection)conn).getErrorStream();
>                  int ret = 0;
>                  // read the response body
>                  while ((ret = es.read(buf)) > 0) {
>                          processBuf(buf);
>                  }
>                  // close the errorstream
>                  es.close();
>          } catch(IOException ex) {
>                  // deal with the exception
>          }
> }
>
>
> My question(s) are:
>
>
>    *   Does CXF WebClient handle this?
>       *   If so, which versions?
>       *   Where is the code that handles this located
>    *   Are users of WebClient supposed to manage this?
>    *   Have you seen this error before?
>
> Any help/pointers would be greatly appreciated.
>
> Thanks,
> Chuck
>