You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Ken DeLong <ke...@babycenter.com> on 2008/07/01 02:50:31 UTC
RE: Too Many Connections
- Switching to the MultiThreadedConnectionManager didn't fix the problem (curiously, the "connection locked" messages went away, tho, but the job still failed)
- turning on httpclient.wire=DEBUG caused it to slow down so much that the problem went away.
Why is it not reusing the connection?
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, June 27, 2008 1:25 PM
To: HttpClient User Discussion
Subject: RE: Too Many Connections
On Fri, 2008-06-27 at 11:25 -0700, Ken DeLong wrote:
> All of our connections are to the exact same host - in fact, the sending and receiving processes are on the same actual server. The same dns name is used for each request (we use the dns name of the machine, not the string "localhost").
>
Ken,
Try disabling the stale connection check. It can happen, especially with TLS/SSL, that perfectly good connections are reported as stale. You can get a better insight as to why connections get dropped by turning on the context logging. For details see
http://hc.apache.org/httpclient-3.x/logging.html
Oleg
> We'll try the multithreaded connection manager next. (I was pretty sure the calling app was single threaded but I'm not 100% sure).
>
> Thanks!
> Ken
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, June 27, 2008 11:20 AM
> To: HttpClient User Discussion
> Subject: RE: Too Many Connections
>
> On Fri, 2008-06-27 at 09:08 -0700, Ken DeLong wrote:
> > That didn't help. We changed to
> >
> > private HttpClient client = new HttpClient(new
> > SimpleHttpConnectionManager());
> >
>
> Ken,
>
> SimpleHttpConnectionManager re-uses the underlying connection object
> but it cannot reuse the network socket when connecting to a different
> host. Ii needs to close the old socket and open a new one every time
> it connects to a different host. If you would like to keep alive
> connections to multiple hosts, consider using
> MultiThreadedHttpConnectionManager
>
> Regarding connections in TIME_WAIT state please see
>
> http://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagem
> entQuestions
>
> Oleg
>
>
> > But we still see the same behavior.
> >
> > Turning on the debug logs shows messages like
> >
> > $ fgrep 'Connection is locked. Call to releaseConnection() ignored.'
> > server.log | wc -l
> > 176
> >
> > $ fgrep 'Releasing connection back to connection manager.'
> > server.log
> > | wc -l
> > 264
> >
> >
> > What would cause the connection to be locked when
> > releaseConnection() is called?
> >
> >
> >
> > -----Original Message-----
> > From: Pete Keyes [mailto:PKeyes@starbucks.com]
> > Sent: Thursday, June 26, 2008 3:27 PM
> > To: HttpClient User Discussion
> > Cc: Stephen Hiley
> > Subject: RE: Too Many Connections
> >
> > You need to use either of these:
> > - org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
> > - org.apache.commons.httpclient.SimpleHttpConnectionManager
> > There are examples of both in the HttpClient site.
> >
> > In addition to that you may need to tune the O/S to formally release
> > the CLOSE_WAIT files sooner. There is a delay between when an
> > application closes a socket and when the O/S can formally release
> > that socket to avoid orphan packets.
> >
> > See the documentation for your O/S on setting that value.
> >
> >
> > ...Pete
> > Starbucks Coffee Co.
> > 2601 Utah Ave S.
> > Seattle, WA. 98134
> > (w) 206-318-5933
> > (c) 206-914-4134
> >
> > -----Original Message-----
> > From: Ken DeLong [mailto:ken.delong@babycenter.com]
> > Sent: Thursday, June 26, 2008 3:17 PM
> > To: httpclient-users@hc.apache.org
> > Cc: Stephen Hiley
> > Subject: Too Many Connections
> >
> > We have a method in our app that calls httpclient like this:
> >
> > private HttpClient client = new HttpClient();
> >
> > public String send(String urlString, String xml)
> > {
> > PostMethod postMethod = new PostMethod(urlString);
> >
> > // Put the XML in the request
> > try
> > {
> > RequestEntity entity = new
> > StringRequestEntity(xml, "text/xml", "UTF-8");
> > postMethod.setRequestEntity(entity);
> > }
> > catch(UnsupportedEncodingException e1)
> > {
> > logger.warn("Couldn't encode the xml?", e1);
> > return "" +
> > IContentPublishingService.ACTION_ERROR;
> > }
> >
> > // Send the data and get the response
> > String response = null;
> > try
> > {
> > client.executeMethod(postMethod);
> > byte[] responseBody =
> > postMethod.getResponseBody();
> > response = new String(responseBody);
> > }
> > catch(HttpException e)
> > {
> > logger.warn("Fatal protocol violation: " + e.getMessage());
> > return "" +
> > IContentPublishingService.ACTION_ERROR;
> > }
> > catch(IOException e)
> > {
> > logger.warn("Fatal transport error: " + e.getMessage());
> > return "" +
> > IContentPublishingService.ACTION_ERROR;
> > }
> > finally
> > {
> > // Release the connection.
> > postMethod.releaseConnection();
> > }
> >
> > return response;
> > }
> >
> >
> > This method is called rapidly by a single thread in our application.
> > After 1500 calls or so, we start to see java.net.SocketException,
> > too many open files. Netstat reveals that there are many sockets in
> > CLOSE_WAIT and TIME_WAIT states. It seems that httpclient is not
> > reusing the connection managed by the HttpClient instance, and
> > instead is closing the connection and creating a new one. This
> > appears to be in conflict with all the documentation that we could find.
> >
> > An interesting wrinkle is that the sending and receiving side of the
> > connection are both on the same machine. The URL is constructed
> > using the machine's dns name, not "localhost".
> >
> > --------------------------------------------------------------------
> > --
> > --
> > --
> > Kenneth DeLong
> > Software Architect
> > Engineering
> > BabyCenter, LLC
> >
> > kdelong@babycenter.com
> > p: 415.344.7616
> > AIM: kenwdelong
> >
> > http://www.babycenter.com
> >
> >
> >
> > This email message is for the sole use of the intended recipient(s)
> > and may contain confidential and privileged information. Any
> > unauthorized review, use, disclosure or distribution is prohibited.
> > If you are not the intended recipient, please contact the sender by
> > reply email and destroy all copies of the original message. If you
> > are the intended recipient, please be advised that the content of
> > this message is subject to access, review and disclosure by the
> > sender's Email System Administrator.
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Too Many Connections
Posted by Ken DeLong <ke...@babycenter.com>.
I'm sorry. It turns out that the 3rd-party app we are using is also using myriads of HTTP connections and in fact it's THEIR code (non-httpclient) that is throwing the error. We found it by completely commenting out all of our (httpclient) code and the problem still manifests.
Sorry for the wild goose chase.
-----Original Message-----
From: Ken DeLong [mailto:ken.delong@babycenter.com]
Sent: Monday, June 30, 2008 5:51 PM
To: HttpClient User Discussion
Cc: Stephen Hiley
Subject: RE: Too Many Connections
- Switching to the MultiThreadedConnectionManager didn't fix the problem (curiously, the "connection locked" messages went away, tho, but the job still failed)
- turning on httpclient.wire=DEBUG caused it to slow down so much that the problem went away.
Why is it not reusing the connection?
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, June 27, 2008 1:25 PM
To: HttpClient User Discussion
Subject: RE: Too Many Connections
On Fri, 2008-06-27 at 11:25 -0700, Ken DeLong wrote:
> All of our connections are to the exact same host - in fact, the sending and receiving processes are on the same actual server. The same dns name is used for each request (we use the dns name of the machine, not the string "localhost").
>
Ken,
Try disabling the stale connection check. It can happen, especially with TLS/SSL, that perfectly good connections are reported as stale. You can get a better insight as to why connections get dropped by turning on the context logging. For details see
http://hc.apache.org/httpclient-3.x/logging.html
Oleg
> We'll try the multithreaded connection manager next. (I was pretty sure the calling app was single threaded but I'm not 100% sure).
>
> Thanks!
> Ken
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, June 27, 2008 11:20 AM
> To: HttpClient User Discussion
> Subject: RE: Too Many Connections
>
> On Fri, 2008-06-27 at 09:08 -0700, Ken DeLong wrote:
> > That didn't help. We changed to
> >
> > private HttpClient client = new HttpClient(new
> > SimpleHttpConnectionManager());
> >
>
> Ken,
>
> SimpleHttpConnectionManager re-uses the underlying connection object
> but it cannot reuse the network socket when connecting to a different
> host. Ii needs to close the old socket and open a new one every time
> it connects to a different host. If you would like to keep alive
> connections to multiple hosts, consider using
> MultiThreadedHttpConnectionManager
>
> Regarding connections in TIME_WAIT state please see
>
> http://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagem
> entQuestions
>
> Oleg
>
>
> > But we still see the same behavior.
> >
> > Turning on the debug logs shows messages like
> >
> > $ fgrep 'Connection is locked. Call to releaseConnection() ignored.'
> > server.log | wc -l
> > 176
> >
> > $ fgrep 'Releasing connection back to connection manager.'
> > server.log
> > | wc -l
> > 264
> >
> >
> > What would cause the connection to be locked when
> > releaseConnection() is called?
> >
> >
> >
> > -----Original Message-----
> > From: Pete Keyes [mailto:PKeyes@starbucks.com]
> > Sent: Thursday, June 26, 2008 3:27 PM
> > To: HttpClient User Discussion
> > Cc: Stephen Hiley
> > Subject: RE: Too Many Connections
> >
> > You need to use either of these:
> > - org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
> > - org.apache.commons.httpclient.SimpleHttpConnectionManager
> > There are examples of both in the HttpClient site.
> >
> > In addition to that you may need to tune the O/S to formally release
> > the CLOSE_WAIT files sooner. There is a delay between when an
> > application closes a socket and when the O/S can formally release
> > that socket to avoid orphan packets.
> >
> > See the documentation for your O/S on setting that value.
> >
> >
> > ...Pete
> > Starbucks Coffee Co.
> > 2601 Utah Ave S.
> > Seattle, WA. 98134
> > (w) 206-318-5933
> > (c) 206-914-4134
> >
> > -----Original Message-----
> > From: Ken DeLong [mailto:ken.delong@babycenter.com]
> > Sent: Thursday, June 26, 2008 3:17 PM
> > To: httpclient-users@hc.apache.org
> > Cc: Stephen Hiley
> > Subject: Too Many Connections
> >
> > We have a method in our app that calls httpclient like this:
> >
> > private HttpClient client = new HttpClient();
> >
> > public String send(String urlString, String xml)
> > {
> > PostMethod postMethod = new PostMethod(urlString);
> >
> > // Put the XML in the request
> > try
> > {
> > RequestEntity entity = new
> > StringRequestEntity(xml, "text/xml", "UTF-8");
> > postMethod.setRequestEntity(entity);
> > }
> > catch(UnsupportedEncodingException e1)
> > {
> > logger.warn("Couldn't encode the xml?", e1);
> > return "" +
> > IContentPublishingService.ACTION_ERROR;
> > }
> >
> > // Send the data and get the response
> > String response = null;
> > try
> > {
> > client.executeMethod(postMethod);
> > byte[] responseBody =
> > postMethod.getResponseBody();
> > response = new String(responseBody);
> > }
> > catch(HttpException e)
> > {
> > logger.warn("Fatal protocol violation: " + e.getMessage());
> > return "" +
> > IContentPublishingService.ACTION_ERROR;
> > }
> > catch(IOException e)
> > {
> > logger.warn("Fatal transport error: " + e.getMessage());
> > return "" +
> > IContentPublishingService.ACTION_ERROR;
> > }
> > finally
> > {
> > // Release the connection.
> > postMethod.releaseConnection();
> > }
> >
> > return response;
> > }
> >
> >
> > This method is called rapidly by a single thread in our application.
> > After 1500 calls or so, we start to see java.net.SocketException,
> > too many open files. Netstat reveals that there are many sockets in
> > CLOSE_WAIT and TIME_WAIT states. It seems that httpclient is not
> > reusing the connection managed by the HttpClient instance, and
> > instead is closing the connection and creating a new one. This
> > appears to be in conflict with all the documentation that we could find.
> >
> > An interesting wrinkle is that the sending and receiving side of the
> > connection are both on the same machine. The URL is constructed
> > using the machine's dns name, not "localhost".
> >
> > --------------------------------------------------------------------
> > --
> > --
> > --
> > Kenneth DeLong
> > Software Architect
> > Engineering
> > BabyCenter, LLC
> >
> > kdelong@babycenter.com
> > p: 415.344.7616
> > AIM: kenwdelong
> >
> > http://www.babycenter.com
> >
> >
> >
> > This email message is for the sole use of the intended recipient(s)
> > and may contain confidential and privileged information. Any
> > unauthorized review, use, disclosure or distribution is prohibited.
> > If you are not the intended recipient, please contact the sender by
> > reply email and destroy all copies of the original message. If you
> > are the intended recipient, please be advised that the content of
> > this message is subject to access, review and disclosure by the
> > sender's Email System Administrator.
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org