out out memory problem of Httpclient

[Antonio Xu <An...@Sun.COM>]

Hello All,

This is Antonio Xu, I'm writing a http application base on HttpClient 
2.02, but today, I found there is serious problem (out of memory) of 
httpclient, that make my http application keep on crash. In my code, I 
use httpclient's get method to download a file (more than 2MB) form my 
apache web server, but every time, when I try to get the stream of 
response, httpclient will throw a out of memory exception, that make my 
application crash. Please see my codes below:

       String res;
       GetMethod method = null;
       try {
           method = new GetMethod(path);
       } catch (IllegalArgumentException e) {
           throw new HttpException("Wrong fetch Url");
       }
       method.setDoAuthentication(true);
       try {
           executeMethod(method);

           InputStream is = method.getResponseBodyAsStream();  // if the 
download file more than 2MB, before the stream will be returned, the 
httpclient throw a out of memory exception.
           is.read(buf, 0, 20480-1);
           res = new String(buf);
           is.close();
       }catch(IOException e){
           return null;
       }
       finally {
           method.releaseConnection();
       }

I wonder who can help me resolve my problem, if this issue cannot be 
resolved, I have to give up to use HttpClient.

Thanks,
Anto



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org

Re: out out memory problem of Httpclient

[Oleg Kalnichevski <ol...@apache.org>]

Hi Anto,

Please take a look at the HttpClient optimization guide:

http://jakarta.apache.org/commons/httpclient/3.0/performance.html

This should give you an idea how to efficiently stream request/response
content. Please note this guide is based on newer version of HttpClient
3.0 but the same techniques can be applied to HttpClient 2.0.x

If you are still unable to solve the problem please take a look at the
troubleshooting guide
<http://jakarta.apache.org/commons/httpclient/3.0/troubleshooting.html>
and send us the wire/context log of the HTTP session.

Hope this helps

Oleg


On Thu, Feb 03, 2005 at 01:28:55PM +0800, Antonio Xu wrote:
> Hello All,
> 
> This is Antonio Xu, I'm writing a http application base on HttpClient 
> 2.02, but today, I found there is serious problem (out of memory) of 
> httpclient, that make my http application keep on crash. In my code, I 
> use httpclient's get method to download a file (more than 2MB) form my 
> apache web server, but every time, when I try to get the stream of 
> response, httpclient will throw a out of memory exception, that make my 
> application crash. Please see my codes below:
> 
>       String res;
>       GetMethod method = null;
>       try {
>           method = new GetMethod(path);
>       } catch (IllegalArgumentException e) {
>           throw new HttpException("Wrong fetch Url");
>       }
>       method.setDoAuthentication(true);
>       try {
>           executeMethod(method);
> 
>           InputStream is = method.getResponseBodyAsStream();  // if the 
> download file more than 2MB, before the stream will be returned, the 
> httpclient throw a out of memory exception.
>           is.read(buf, 0, 20480-1);
>           res = new String(buf);
>           is.close();
>       }catch(IOException e){
>           return null;
>       }
>       finally {
>           method.releaseConnection();
>       }
> 
> I wonder who can help me resolve my problem, if this issue cannot be 
> resolved, I have to give up to use HttpClient.
> 
> Thanks,
> Anto
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org

Re: out out memory problem of Httpclient

[Ortwin Glück <or...@nose.ch>]

Antonio,

This is not a problem with HttpClient but with the way you deal with the 
content. You are buffering the complete response in memory. Moreover the 
way you create the String, copying occurs, so in fact the data is stored 
twice in memory. I don't know what data you are dealing with. You should 
find a way to keep as little data in memory at a time. If the data is 
XML, you could feed the stream into a SAX parser for instance.

Ortwin Glück

Antonio Xu wrote:
>           InputStream is = method.getResponseBodyAsStream();  // if the 
> download file more than 2MB, before the stream will be returned, the 
> httpclient throw a out of memory exception.
>           is.read(buf, 0, 20480-1);
>           res = new String(buf);
>           is.close();

-- 
  _________________________________________________________________
  NOSE applied intelligence ag

  ortwin glück                      [www]      http://www.nose.ch
  software engineer
  hardturmstrasse 171               [pgp id]           0x81CF3416
  8005 zürich                       [office]      +41-1-277 57 35
  switzerland                       [fax]         +41-1-277 57 12

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org