You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Zhaohua Meng <zm...@cgsh.com> on 2005/11/03 00:03:03 UTC
HttpClient: can it cause "too many open files" ?
HttpClient gugus,
I have a HttpClientEngine (see code listing at the bottom ) which wraps
commons HttpClient. It is used as following in my code deployed on SuSe
Linux.
for (int i=0; i<requests.length; i++) }
HttpClientEngine engine = new HttpClientEngine ();
response = engine.doPost(requests[i]);
//code to process response
}
In real situation t he request loop can be as big as 500+. After the code
running for a while I got "too many open files" exception. I know in Linux
a stream is considered a file. Please note that
"method.releaseConnection()" is called before doPost() returns a response
object. My questions are following:
1. Will the following code work? In other words, can HttpClient object be
reused to post with different HttpBaseMethod?
HttpClientEngine engine = new HttpClientEngine ();
for (int i=0; i<requests.length; i++) }
response = engine.doPost(requests[i]);
//code to process response
}
2. Will my code leave some streams open thus causes "too many open files"?
3. Besides "method.releaseConnection()", what other methods should I call
to close all streams?
Thanks,
Zhaohua
public class HttpClientEngine {
private HttpClient httpClient; //this is the commons HttpClient
private HttpResponse response; //my proprietary class
private HttpRequest request; //my proprietary class
private URL url;
private Credentials credentials;
public HttpClientEngine() {
httpClient = new HttpClient();
}
public HttpResponse doPost(HttpRequest request) {
return
doPost(request,request.getURL(),request.getCredentials());
}
public HttpResponse doPost(HttpRequest request, URL url) {
return doPost(request,url,request.getCredentials());
}
private static Logger logger = Logger.getInstance();
public HttpResponse doPost(HttpRequest request, URL url,
Credentials credentials) {
this.request = request;
this.url = url;
this.credentials = credentials;
return service(request,new PostMethod(url.toString()));
}
public HttpResponse doGet(HttpRequest request) {
return
doGet(request,request.getURL(),request.getCredentials());
}
public HttpResponse doGet(HttpRequest request, URL url) {
return doGet(request,url,request.getCredentials());
}
public HttpResponse doGet(HttpRequest request, URL url,
Credentials credentials) {
this.request = request;
this.url = url;
this.credentials = credentials;
return service(request,new GetMethod(url.toString()));
}
private HttpResponse service(HttpRequest request, HttpMethodBase
method) {
HttpHeader[] requestHeaders = request.getHeaders();
if ( requestHeaders != null) {
for (int i=0; i<requestHeaders.length; i++) {
//debug("header:
["+requestHeaders[i].getName()+"]["+requestHeaders[i].getValue()+"]");
method.setRequestHeader(requestHeaders[i].getName(),requestHeaders[i].getValue());
}
}
if (credentials != null) {
UsernamePasswordCredentials login =
new
UsernamePasswordCredentials(credentials.getName(),credentials.getPassword());
AuthScope auth= new AuthScope(url.getHost(),
url.getPort(), null);
httpClient.getState().setCredentials(auth, login);
method.setDoAuthentication(true);
}
if (method instanceof PostMethod) {
try {
((PostMethod)method).setRequestEntity(new
ByteArrayRequestEntity(request.getBody()));
} catch (Exception e) {
method.releaseConnection();
return null;
}
}
try {
int statusCode = httpClient.executeMethod(method);
Header[] responseHeaders =
method.getResponseHeaders();
HttpHeader[] headers = null;
if (responseHeaders != null) {
headers = new
HttpHeader[responseHeaders.length];
for (int i=0; i<responseHeaders.length;
i++) {
headers[i] = new HttpHeader();
headers[i].setName(responseHeaders[i].getName());
headers[i].setValue(responseHeaders[i].getValue());
}
}
response =
handler.process(method.getResponseBody(),statusCode, headers);
} catch (HttpException e) {
throw new RuntimeException(e.getMessage());
} catch (IOException e) {
throw new RuntimeException(e.getMessage());
} finally {
method.releaseConnection();
}
return response;
}
private HttpResponseHandler handler;
public void setHttpResponseHandler(HttpResponseHandler handler) {
this.handler = handler;
}
public static void debug(Object obj) {
System.out.println(HttpClientEngine.class.getName()+"::"+obj.toString());
}
Zhaohua Meng
WebSphere Consultant
This message is being sent from a law firm and may contain confidential or privileged information. If you are not the intended recipient, please advise the sender immediately by reply e-mail and delete this message and any attachments without retaining a copy.
Re: HttpClient: can it cause "too many open files" ?
Posted by Zhaohua Meng <zm...@cgsh.com>.
Oleg,
You saved my day.
Thank you very much.
Zhaohua Meng
WebSphere Consultant
"Oleg Kalnichevski" <ol...@apache.org>
11/03/2005 04:13 AM
Please respond to
"Jakarta Commons Users List" <co...@jakarta.apache.org>
To
commons-user@jakarta.apache.org
cc
Subject
Re: HttpClient: can it cause "too many open files" ?
On Wed, Nov 02, 2005 at 06:03:03PM -0500, Zhaohua Meng wrote:
> HttpClient gugus,
>
>
> I have a HttpClientEngine (see code listing at the bottom ) which wraps
> commons HttpClient. It is used as following in my code deployed on SuSe
> Linux.
>
> for (int i=0; i<requests.length; i++) }
> HttpClientEngine engine = new HttpClientEngine ();
> response = engine.doPost(requests[i]);
> //code to process response
> }
>
> In real situation t he request loop can be as big as 500+. After the
code
> running for a while I got "too many open files" exception. I know in
Linux
> a stream is considered a file. Please note that
> "method.releaseConnection()" is called before doPost() returns a
response
> object. My questions are following:
>
> 1. Will the following code work? In other words, can HttpClient object
be
> reused to post with different HttpBaseMethod?
>
> HttpClientEngine engine = new HttpClientEngine ();
> for (int i=0; i<requests.length; i++) }
> response = engine.doPost(requests[i]);
> //code to process response
> }
>
Actually it is strongly recommended to reuse the HttpClient instance.
For details see the HttpClient optimization guide:
http://jakarta.apache.org/commons/httpclient/performance.html
> 2. Will my code leave some streams open thus causes "too many open
files"?
>
Yes it will. HttpMethod#releaseConnection() call does not close the
underlying socket if the connection can be kept alive. This method
merely returns connection back to the connection manager
>
> 3. Besides "method.releaseConnection()", what other methods should I
call
> to close all streams?
>
MultiThreadedHttpConnectionManager#shutdown()
SimpleHttpConnectionManager#dhutdown()
> Thanks,
> Zhaohua
>
>
> public class HttpClientEngine {
> private HttpClient httpClient; //this is the commons HttpClient
> private HttpResponse response; //my proprietary class
> private HttpRequest request; //my proprietary class
> private URL url;
> private Credentials credentials;
> public HttpClientEngine() {
> httpClient = new HttpClient();
> }
> public HttpResponse doPost(HttpRequest request) {
> return
> doPost(request,request.getURL(),request.getCredentials());
> }
> public HttpResponse doPost(HttpRequest request, URL url) {
> return doPost(request,url,request.getCredentials());
> }
> private static Logger logger = Logger.getInstance();
> public HttpResponse doPost(HttpRequest request, URL url,
> Credentials credentials) {
> this.request = request;
> this.url = url;
> this.credentials = credentials;
> return service(request,new PostMethod(url.toString()));
> }
>
> public HttpResponse doGet(HttpRequest request) {
> return
> doGet(request,request.getURL(),request.getCredentials());
> }
> public HttpResponse doGet(HttpRequest request, URL url) {
> return doGet(request,url,request.getCredentials());
> }
> public HttpResponse doGet(HttpRequest request, URL url,
> Credentials credentials) {
> this.request = request;
> this.url = url;
> this.credentials = credentials;
> return service(request,new GetMethod(url.toString()));
> }
>
>
> private HttpResponse service(HttpRequest request, HttpMethodBase
> method) {
> HttpHeader[] requestHeaders = request.getHeaders();
> if ( requestHeaders != null) {
> for (int i=0; i<requestHeaders.length; i++) {
> //debug("header:
> ["+requestHeaders[i].getName()+"]["+requestHeaders[i].getValue()+"]");
>
>
method.setRequestHeader(requestHeaders[i].getName(),requestHeaders[i].getValue());
> }
> }
> if (credentials != null) {
> UsernamePasswordCredentials login =
> new
>
UsernamePasswordCredentials(credentials.getName(),credentials.getPassword());
>
> AuthScope auth= new AuthScope(url.getHost(),
> url.getPort(), null);
> httpClient.getState().setCredentials(auth, login);
> method.setDoAuthentication(true);
> }
> if (method instanceof PostMethod) {
> try {
> ((PostMethod)method).setRequestEntity(new
> ByteArrayRequestEntity(request.getBody()));
> } catch (Exception e) {
> method.releaseConnection();
> return null;
> }
> }
> try {
> int statusCode =
httpClient.executeMethod(method);
> Header[] responseHeaders =
> method.getResponseHeaders();
> HttpHeader[] headers = null;
> if (responseHeaders != null) {
> headers = new
> HttpHeader[responseHeaders.length];
> for (int i=0; i<responseHeaders.length;
> i++) {
> headers[i] = new HttpHeader();
> headers[i].setName(responseHeaders[i].getName());
> headers[i].setValue(responseHeaders[i].getValue());
> }
> }
>
> response =
> handler.process(method.getResponseBody(),statusCode, headers);
> } catch (HttpException e) {
> throw new RuntimeException(e.getMessage());
> } catch (IOException e) {
> throw new RuntimeException(e.getMessage());
> } finally {
> method.releaseConnection();
> }
> return response;
> }
> private HttpResponseHandler handler;
> public void setHttpResponseHandler(HttpResponseHandler handler)
{
> this.handler = handler;
> }
> public static void debug(Object obj) {
>
System.out.println(HttpClientEngine.class.getName()+"::"+obj.toString());
> }
>
> Zhaohua Meng
> WebSphere Consultant
> This message is being sent from a law firm and may contain confidential
or privileged information. If you are not the intended recipient, please
advise the sender immediately by reply e-mail and delete this message and
any attachments without retaining a copy.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
This message is being sent from a law firm and may contain confidential or privileged information. If you are not the intended recipient, please advise the sender immediately by reply e-mail and delete this message and any attachments without retaining a copy.
Re: HttpClient: can it cause "too many open files" ?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, Nov 02, 2005 at 06:03:03PM -0500, Zhaohua Meng wrote:
> HttpClient gugus,
>
>
> I have a HttpClientEngine (see code listing at the bottom ) which wraps
> commons HttpClient. It is used as following in my code deployed on SuSe
> Linux.
>
> for (int i=0; i<requests.length; i++) }
> HttpClientEngine engine = new HttpClientEngine ();
> response = engine.doPost(requests[i]);
> //code to process response
> }
>
> In real situation t he request loop can be as big as 500+. After the code
> running for a while I got "too many open files" exception. I know in Linux
> a stream is considered a file. Please note that
> "method.releaseConnection()" is called before doPost() returns a response
> object. My questions are following:
>
> 1. Will the following code work? In other words, can HttpClient object be
> reused to post with different HttpBaseMethod?
>
> HttpClientEngine engine = new HttpClientEngine ();
> for (int i=0; i<requests.length; i++) }
> response = engine.doPost(requests[i]);
> //code to process response
> }
>
Actually it is strongly recommended to reuse the HttpClient instance.
For details see the HttpClient optimization guide:
http://jakarta.apache.org/commons/httpclient/performance.html
> 2. Will my code leave some streams open thus causes "too many open files"?
>
Yes it will. HttpMethod#releaseConnection() call does not close the
underlying socket if the connection can be kept alive. This method
merely returns connection back to the connection manager
>
> 3. Besides "method.releaseConnection()", what other methods should I call
> to close all streams?
>
MultiThreadedHttpConnectionManager#shutdown()
SimpleHttpConnectionManager#dhutdown()
> Thanks,
> Zhaohua
>
>
> public class HttpClientEngine {
> private HttpClient httpClient; //this is the commons HttpClient
> private HttpResponse response; //my proprietary class
> private HttpRequest request; //my proprietary class
> private URL url;
> private Credentials credentials;
> public HttpClientEngine() {
> httpClient = new HttpClient();
> }
> public HttpResponse doPost(HttpRequest request) {
> return
> doPost(request,request.getURL(),request.getCredentials());
> }
> public HttpResponse doPost(HttpRequest request, URL url) {
> return doPost(request,url,request.getCredentials());
> }
> private static Logger logger = Logger.getInstance();
> public HttpResponse doPost(HttpRequest request, URL url,
> Credentials credentials) {
> this.request = request;
> this.url = url;
> this.credentials = credentials;
> return service(request,new PostMethod(url.toString()));
> }
>
> public HttpResponse doGet(HttpRequest request) {
> return
> doGet(request,request.getURL(),request.getCredentials());
> }
> public HttpResponse doGet(HttpRequest request, URL url) {
> return doGet(request,url,request.getCredentials());
> }
> public HttpResponse doGet(HttpRequest request, URL url,
> Credentials credentials) {
> this.request = request;
> this.url = url;
> this.credentials = credentials;
> return service(request,new GetMethod(url.toString()));
> }
>
>
> private HttpResponse service(HttpRequest request, HttpMethodBase
> method) {
> HttpHeader[] requestHeaders = request.getHeaders();
> if ( requestHeaders != null) {
> for (int i=0; i<requestHeaders.length; i++) {
> //debug("header:
> ["+requestHeaders[i].getName()+"]["+requestHeaders[i].getValue()+"]");
>
> method.setRequestHeader(requestHeaders[i].getName(),requestHeaders[i].getValue());
> }
> }
> if (credentials != null) {
> UsernamePasswordCredentials login =
> new
> UsernamePasswordCredentials(credentials.getName(),credentials.getPassword());
>
> AuthScope auth= new AuthScope(url.getHost(),
> url.getPort(), null);
> httpClient.getState().setCredentials(auth, login);
> method.setDoAuthentication(true);
> }
> if (method instanceof PostMethod) {
> try {
> ((PostMethod)method).setRequestEntity(new
> ByteArrayRequestEntity(request.getBody()));
> } catch (Exception e) {
> method.releaseConnection();
> return null;
> }
> }
> try {
> int statusCode = httpClient.executeMethod(method);
> Header[] responseHeaders =
> method.getResponseHeaders();
> HttpHeader[] headers = null;
> if (responseHeaders != null) {
> headers = new
> HttpHeader[responseHeaders.length];
> for (int i=0; i<responseHeaders.length;
> i++) {
> headers[i] = new HttpHeader();
> headers[i].setName(responseHeaders[i].getName());
> headers[i].setValue(responseHeaders[i].getValue());
> }
> }
>
> response =
> handler.process(method.getResponseBody(),statusCode, headers);
> } catch (HttpException e) {
> throw new RuntimeException(e.getMessage());
> } catch (IOException e) {
> throw new RuntimeException(e.getMessage());
> } finally {
> method.releaseConnection();
> }
> return response;
> }
> private HttpResponseHandler handler;
> public void setHttpResponseHandler(HttpResponseHandler handler) {
> this.handler = handler;
> }
> public static void debug(Object obj) {
> System.out.println(HttpClientEngine.class.getName()+"::"+obj.toString());
> }
>
> Zhaohua Meng
> WebSphere Consultant
> This message is being sent from a law firm and may contain confidential or privileged information. If you are not the intended recipient, please advise the sender immediately by reply e-mail and delete this message and any attachments without retaining a copy.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: HttpClient: can it cause "too many open files" ?
Posted by robert burrell donkin <ro...@blueyonder.co.uk>.
On Wed, 2005-11-02 at 18:03 -0500, Zhaohua Meng wrote:
> HttpClient gugus,
>
>
> I have a HttpClientEngine (see code listing at the bottom ) which wraps
> commons HttpClient. It is used as following in my code deployed on SuSe
> Linux.
>
> for (int i=0; i<requests.length; i++) }
> HttpClientEngine engine = new HttpClientEngine ();
> response = engine.doPost(requests[i]);
> //code to process response
> }
>
> In real situation t he request loop can be as big as 500+. After the code
> running for a while I got "too many open files" exception. I know in Linux
> a stream is considered a file.
have you checked the number of file descriptors available?
(i'll not bore the list with the details: how to do this is easily found
on google)
many linux distributions now ship with default allocations which are far
too low for any serious server usage. so, this is usually my first port
of call when this issue crops up.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org