MultiThreadedHttpConnectionManager and CLOSE_WAIT

[Niranjan Kundapur <ni...@air2web.com>]

Hello,

I am new user of HTTPClient. What I like about it is that it has a
MultiThreadedHttpConnectionManager. However, I am not sure if I am using it
in the correct way (Code included below). There is a static instance of
HttpClient, which is created with a MultiThreadedHttpConnectionManager. The
HttpClient instance is shared by multiple threads. 
The problem I see is a broken pipe exception when the following steps occur:
         1. Some requests have gone through. Connections are being reused.
         2. After a period of inactivity, connections are in CLOSE_WAIT on
the client side and FIN_WAIT on the server side. 
         3. Another request(s) comes in. 
         4. The connection pool does not know that the underlying connection
is in CLOSE_WAIT and tries to reuse it.
         5. Broken pipe exception occurs.
Work around: Test if exception is broken pipe and retry immediately. Retry
for a max number of times so that there is no infinite loop.

Questions:
----------
1) Is this usage of HttpClient correct?
2) Is there some way for the connection pool to know that the connection is
not usable? I read some other mails that seem to suggest that I need to wait
for a redesign. Is this available?
3) When I do netstat, I see lot of connections to the server in CLOSE_WAIT
state. These were used by HttpClient previously and then were timed out by
the server. Is the HttpClient not closing sockets explicitly whenever it
knows that a connection is in CLOSE_WAIT? Or is my usage incorrect?

Thanks,
Niranjan

---------------------

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpConnectionManager;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;

public class HTTPHelper2 
{
    private static final Logger LOG =
LogManager.getLogger(HTTPHelper2.class);
    private static final String
PROPERTY_httpHelper_maxConnectionsPerHost="httpHelper.maxConnectionsPerHost"
;
    private static final String
PROPERTY_httpHelper_numRetriesUponBrokenPipe="httpHelper.numRetriesUponBroke
nPipe";
    private static final String
PROPERTY_httpHelper_brokenPipeExceptionMsgFragment="httpHelper.brokenPipeExc
eptionMsgFragment";

    static private HttpConnectionManager HTTP_CONN_MGR =
initHttpConnectionManager();
    static private HttpClient HTTP_CLIENT = initHttpClient();

    public static byte[] connectToURL(String theUrlString, boolean
connectViaPost) throws Exception
    {
        int
numRetriesUponBrokenPipe=getIntProperty(PROPERTY_httpHelper_numRetriesUponBr
okenPipe);
        String
brokenPipeExceptionMsg=getProperty(PROPERTY_httpHelper_brokenPipeExceptionMs
gFragment,"broken pipe");
        for (int i=0; i<numRetriesUponBrokenPipe; i++)
        {
            if (i>0)
            {
                LOG.debug("Retrying connection to ["+theUrlString+"] upon
broken pipe. i="+i);
            }
            try
            {
                return _connectToURL(theUrlString, connectViaPost);
            }
            catch(Exception e)
            {
                if (e.getMessage()!=null &&
e.getMessage().toLowerCase().indexOf(brokenPipeExceptionMsg)==-1 )
                {
                    LOG.error("HTTPHelper2.connectToURL: Error occurred and
fyi i="+i, e);
                    throw e;
                }
            }
        } //for
        throw new Exception("HTTPHelper2.connectToURL: Exceeded the maximum
number of retries, which is ["+numRetriesUponBrokenPipe+"],  upon broken
pipe.");
    }

    static private byte[] _connectToURL(String theUrlString, boolean
connectViaPost) throws Exception
    {
        HttpClient httpClient = getHttpClient();
        GetMethod method = null;
        try
        {

            method = new GetMethod(theUrlString);

            httpClient.executeMethod(method);

            byte[] bytes = method.getResponseBody();

            return bytes;
        }
        catch (Exception e)
        {
            throw e;
        }
        finally
        {
            // always release the connection after we're done 
            method.releaseConnection();
            LOG.debug(" - connection released");
        }
    }

    static private HttpClient getHttpClient()
    {
        return HTTP_CLIENT;
    }

    static private HttpConnectionManager initHttpConnectionManager()
    {
        MultiThreadedHttpConnectionManager conMgr=new
MultiThreadedHttpConnectionManager();
 
conMgr.setMaxConnectionsPerHost(getIntProperty(PROPERTY_httpHelper_maxConnec
tionsPerHost));  //default is 2 as per RFC 2616 section 8.1.4
        return conMgr;
    }

    static private HttpClient initHttpClient()
    {
        return new HttpClient(HTTP_CONN_MGR);
    }
}

Re: MultiThreadedHttpConnectionManager and CLOSE_WAIT

[Michael Becke <be...@u.washington.edu>]

Hi Niranjan,

This problem should be cleared up with the latest nightly build of 
HttpClient.  Quite a bit of work has been put in recently to fix 
problems with connection timeout and host management.

> Questions:
> ----------
> 1) Is this usage of HttpClient correct?

Yes, what you are doing is fine.

> 2) Is there some way for the connection pool to know that the connection is
> not usable? I read some other mails that seem to suggest that I need to wait
> for a redesign. Is this available?

As previously mentioned, this has recently been fixed.  Connection are 
now tested before reuse and re-opened when necessary.

> 3) When I do netstat, I see lot of connections to the server in CLOSE_WAIT
> state. These were used by HttpClient previously and then were timed out by
> the server. Is the HttpClient not closing sockets explicitly whenever it
> knows that a connection is in CLOSE_WAIT? Or is my usage incorrect?

The connections are being closed, but something about the underlying 
socket implementation keeps the connection open for a period of time. 
Take a look at:

http://www.developerweb.net/sock-faq/flatfaq.php
and
http://www.developerweb.net/sock-faq/detail.php?id=13

Mike