You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Michael Taft <mi...@earthlink.net> on 2004/11/07 04:37:19 UTC
Problem Downloading Images
Hello -
I have a quick question. I am downloading images and writing them to
disk using code that looks like this:
-----------------------------------------------------------------------------------------------------
String nextPic = "img01.jpg";
for (int i=1; i<=24; i++)
{
String iString = Integer.toString(i);
if(i<10)iString = "0"+iString;
nextPic = "img" + iString + ".jpg";
// System.out.println("saved " + nextPic);
get.setURI(new URI("http://targetsite" + nextPic, false));
client.executeMethod( get );
InputStream is = get.getResponseBodyAsStream();
BufferedInputStream bis = new BufferedInputStream( is );
FileOutputStream fos = new FileOutputStream( nextPic );
byte[] bytes = new byte[ 200000 ];//room for a 200K file
int count = bis.read( bytes );
while( count != -1 && count <= 200000 ) {
System.out.print( "-" );
fos.write( bytes, 0, count );
count = bis.read( bytes );
}
System.out.println("saved " + nextPic);
if( count != -1 ) {
fos.write( bytes, 0, count );
}
fos.close();
bis.close();
}
System.out.println("saved" + nextPic);
-------------------------------------------------------------------------------------
In this example, the assumption is that there are 24 images, named in
numberical order. The code works fine, except that every once in a
while, I'm getting a "responseBodyConsumed" message like this one:
---------------------
Nov 6, 2004 7:10:53 PM org.apache.commons.httpclient.HttpMethodBase
responseBodyConsumed
INFO: Stream closed
---------------------
I've been trying to figure out what causes this problem, and how to fix
it. Any help?
Thanks,
M.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem Downloading Images
Posted by Oleg Kalnichevski <ol...@apache.org>.
Michael,
Try this and see it makes any difference.
package test;
import java.io.IOException;
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpConnection;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpState;
import org.apache.commons.httpclient.methods.GetMethod;
public class BrokenLenMethod extends GetMethod {
public BrokenLenMethod() {
super();
}
public BrokenLenMethod(final String uri) {
super(uri);
}
protected void readResponseHeaders(
HttpState state, HttpConnection conn) throws IOException, HttpException {
// Remove broken content length header(s)
Header[] headers =
getResponseHeaderGroup().getHeaders("Content-Length");
for (int i = 0; i < headers.length; i++) {
getResponseHeaderGroup().removeHeader(headers[i]);
}
// Force close connection
getResponseHeaderGroup().addHeader(
new Header("Connection", "close"));
super.readResponseHeaders(state, conn);
}
}
Hope this helps
Oleg
On Mon, Nov 08, 2004 at 01:37:00AM -0800, Michael Taft wrote:
> Oleg -
> Seems like the server is messing them up now and again. Is there a
> workaround so that the image gets downloaded anyway?
> M.
>
> Oleg Kalnichevski wrote:
>
> >Michael,
> >
> >In the responseBodyConsumed method HttpClient tries to determine if the
> >connection is still good to be reused. It checks whether there's some
> >extraneous content past the declared response body. Apparently something
> >goes wrong there. Can you check if the Content-Length /
> >Transfer-Encoding headers are kosher? Possibly the server miscalculates
> >the size of the images
> >
> >Oleg
> >
> >On Sun, 2004-11-07 at 23:24, Michael Taft wrote:
> >
> >>Oleg -
> >>Interesting. Something is happening when I get this info message,
> >>however, because those images aren't downloaded properly (they are cut
> >>short) whereas all the other ones work fine. Any ideas?
> >>M.
> >>
> >>Oleg Kalnichevski wrote:
> >>
> >>
> >>>Michael,
> >>>
> >>>This is not a problem, but merely an info message. You can get rid of it
> >>>by reducing the log verbosity from INFO to WARN
> >>>
> >>>Oleg
> >>>
> >>>On Sun, 2004-11-07 at 04:37, Michael Taft wrote:
> >>>
> >>>
> >>>>Hello -
> >>>>I have a quick question. I am downloading images and writing them to
> >>>>disk using code that looks like this:
> >>>>
> >>>>
> >>>>-----------------------------------------------------------------------------------------------------
> >>>>
> >>>>
> >>>>String nextPic = "img01.jpg";
> >>>> for (int i=1; i<=24; i++)
> >>>> {
> >>>> String iString = Integer.toString(i);
> >>>> if(i<10)iString = "0"+iString;
> >>>> nextPic = "img" + iString + ".jpg";
> >>>> // System.out.println("saved " + nextPic);
> >>>> get.setURI(new URI("http://targetsite" + nextPic, false));
> >>>> client.executeMethod( get );
> >>>> InputStream is = get.getResponseBodyAsStream();
> >>>> BufferedInputStream bis = new BufferedInputStream( is );
> >>>> FileOutputStream fos = new FileOutputStream( nextPic );
> >>>> byte[] bytes = new byte[ 200000 ];//room for a 200K file
> >>>> int count = bis.read( bytes );
> >>>> while( count != -1 && count <= 200000 ) {
> >>>> System.out.print( "-" );
> >>>> fos.write( bytes, 0, count );
> >>>> count = bis.read( bytes );
> >>>> }
> >>>> System.out.println("saved " + nextPic);
> >>>> if( count != -1 ) {
> >>>> fos.write( bytes, 0, count );
> >>>> }
> >>>> fos.close();
> >>>> bis.close();
> >>>> }
> >>>> System.out.println("saved" + nextPic);
> >>>>
> >>>>-------------------------------------------------------------------------------------
> >>>>
> >>>>
> >>>>
> >>>>In this example, the assumption is that there are 24 images, named in
> >>>>numberical order. The code works fine, except that every once in a
> >>>>while, I'm getting a "responseBodyConsumed" message like this one:
> >>>>
> >>>>---------------------
> >>>>Nov 6, 2004 7:10:53 PM org.apache.commons.httpclient.HttpMethodBase
> >>>>responseBodyConsumed
> >>>>INFO: Stream closed
> >>>>---------------------
> >>>>
> >>>>I've been trying to figure out what causes this problem, and how to fix
> >>>>it. Any help?
> >>>>
> >>>>Thanks,
> >>>>M.
> >>>>
> >>>>---------------------------------------------------------------------
> >>>>To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> >>>>For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >>>>
> >>>
> >>>
> >>>
> >>>---------------------------------------------------------------------
> >>>To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> >>>For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >>>
> >>>
> >
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >
> >
>
> --
> Michael W. Taft
> Screenwriter/Producer
> 4614 Finley Avenue, #3
> Los Angeles, CA 90027
> (323)663-6042
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem Downloading Images
Posted by Michael Taft <mi...@earthlink.net>.
Oleg -
Seems like the server is messing them up now and again. Is there a
workaround so that the image gets downloaded anyway?
M.
Oleg Kalnichevski wrote:
> Michael,
>
> In the responseBodyConsumed method HttpClient tries to determine if the
> connection is still good to be reused. It checks whether there's some
> extraneous content past the declared response body. Apparently something
> goes wrong there. Can you check if the Content-Length /
> Transfer-Encoding headers are kosher? Possibly the server miscalculates
> the size of the images
>
> Oleg
>
> On Sun, 2004-11-07 at 23:24, Michael Taft wrote:
>
>>Oleg -
>>Interesting. Something is happening when I get this info message,
>>however, because those images aren't downloaded properly (they are cut
>>short) whereas all the other ones work fine. Any ideas?
>>M.
>>
>>Oleg Kalnichevski wrote:
>>
>>
>>>Michael,
>>>
>>>This is not a problem, but merely an info message. You can get rid of it
>>>by reducing the log verbosity from INFO to WARN
>>>
>>>Oleg
>>>
>>>On Sun, 2004-11-07 at 04:37, Michael Taft wrote:
>>>
>>>
>>>>Hello -
>>>>I have a quick question. I am downloading images and writing them to
>>>>disk using code that looks like this:
>>>>
>>>>
>>>>-----------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>>String nextPic = "img01.jpg";
>>>> for (int i=1; i<=24; i++)
>>>> {
>>>> String iString = Integer.toString(i);
>>>> if(i<10)iString = "0"+iString;
>>>> nextPic = "img" + iString + ".jpg";
>>>> // System.out.println("saved " + nextPic);
>>>> get.setURI(new URI("http://targetsite" + nextPic, false));
>>>> client.executeMethod( get );
>>>> InputStream is = get.getResponseBodyAsStream();
>>>> BufferedInputStream bis = new BufferedInputStream( is );
>>>> FileOutputStream fos = new FileOutputStream( nextPic );
>>>> byte[] bytes = new byte[ 200000 ];//room for a 200K file
>>>> int count = bis.read( bytes );
>>>> while( count != -1 && count <= 200000 ) {
>>>> System.out.print( "-" );
>>>> fos.write( bytes, 0, count );
>>>> count = bis.read( bytes );
>>>> }
>>>> System.out.println("saved " + nextPic);
>>>> if( count != -1 ) {
>>>> fos.write( bytes, 0, count );
>>>> }
>>>> fos.close();
>>>> bis.close();
>>>> }
>>>> System.out.println("saved" + nextPic);
>>>>
>>>>-------------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>>In this example, the assumption is that there are 24 images, named in
>>>>numberical order. The code works fine, except that every once in a
>>>>while, I'm getting a "responseBodyConsumed" message like this one:
>>>>
>>>>---------------------
>>>>Nov 6, 2004 7:10:53 PM org.apache.commons.httpclient.HttpMethodBase
>>>>responseBodyConsumed
>>>>INFO: Stream closed
>>>>---------------------
>>>>
>>>>I've been trying to figure out what causes this problem, and how to fix
>>>>it. Any help?
>>>>
>>>>Thanks,
>>>>M.
>>>>
>>>>---------------------------------------------------------------------
>>>>To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
>>>>For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>>>>
>>>
>>>
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>>>
>>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
--
Michael W. Taft
Screenwriter/Producer
4614 Finley Avenue, #3
Los Angeles, CA 90027
(323)663-6042
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem Downloading Images
Posted by Oleg Kalnichevski <ol...@apache.org>.
Michael,
In the responseBodyConsumed method HttpClient tries to determine if the
connection is still good to be reused. It checks whether there's some
extraneous content past the declared response body. Apparently something
goes wrong there. Can you check if the Content-Length /
Transfer-Encoding headers are kosher? Possibly the server miscalculates
the size of the images
Oleg
On Sun, 2004-11-07 at 23:24, Michael Taft wrote:
> Oleg -
> Interesting. Something is happening when I get this info message,
> however, because those images aren't downloaded properly (they are cut
> short) whereas all the other ones work fine. Any ideas?
> M.
>
> Oleg Kalnichevski wrote:
>
> > Michael,
> >
> > This is not a problem, but merely an info message. You can get rid of it
> > by reducing the log verbosity from INFO to WARN
> >
> > Oleg
> >
> > On Sun, 2004-11-07 at 04:37, Michael Taft wrote:
> >
> >>Hello -
> >>I have a quick question. I am downloading images and writing them to
> >>disk using code that looks like this:
> >>
> >>
> >>-----------------------------------------------------------------------------------------------------
> >>
> >>
> >>String nextPic = "img01.jpg";
> >> for (int i=1; i<=24; i++)
> >> {
> >> String iString = Integer.toString(i);
> >> if(i<10)iString = "0"+iString;
> >> nextPic = "img" + iString + ".jpg";
> >> // System.out.println("saved " + nextPic);
> >> get.setURI(new URI("http://targetsite" + nextPic, false));
> >> client.executeMethod( get );
> >> InputStream is = get.getResponseBodyAsStream();
> >> BufferedInputStream bis = new BufferedInputStream( is );
> >> FileOutputStream fos = new FileOutputStream( nextPic );
> >> byte[] bytes = new byte[ 200000 ];//room for a 200K file
> >> int count = bis.read( bytes );
> >> while( count != -1 && count <= 200000 ) {
> >> System.out.print( "-" );
> >> fos.write( bytes, 0, count );
> >> count = bis.read( bytes );
> >> }
> >> System.out.println("saved " + nextPic);
> >> if( count != -1 ) {
> >> fos.write( bytes, 0, count );
> >> }
> >> fos.close();
> >> bis.close();
> >> }
> >> System.out.println("saved" + nextPic);
> >>
> >>-------------------------------------------------------------------------------------
> >>
> >>
> >>
> >>In this example, the assumption is that there are 24 images, named in
> >>numberical order. The code works fine, except that every once in a
> >>while, I'm getting a "responseBodyConsumed" message like this one:
> >>
> >>---------------------
> >>Nov 6, 2004 7:10:53 PM org.apache.commons.httpclient.HttpMethodBase
> >>responseBodyConsumed
> >>INFO: Stream closed
> >>---------------------
> >>
> >>I've been trying to figure out what causes this problem, and how to fix
> >>it. Any help?
> >>
> >>Thanks,
> >>M.
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> >>For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem Downloading Images
Posted by Michael Taft <mi...@earthlink.net>.
Oleg -
Interesting. Something is happening when I get this info message,
however, because those images aren't downloaded properly (they are cut
short) whereas all the other ones work fine. Any ideas?
M.
Oleg Kalnichevski wrote:
> Michael,
>
> This is not a problem, but merely an info message. You can get rid of it
> by reducing the log verbosity from INFO to WARN
>
> Oleg
>
> On Sun, 2004-11-07 at 04:37, Michael Taft wrote:
>
>>Hello -
>>I have a quick question. I am downloading images and writing them to
>>disk using code that looks like this:
>>
>>
>>-----------------------------------------------------------------------------------------------------
>>
>>
>>String nextPic = "img01.jpg";
>> for (int i=1; i<=24; i++)
>> {
>> String iString = Integer.toString(i);
>> if(i<10)iString = "0"+iString;
>> nextPic = "img" + iString + ".jpg";
>> // System.out.println("saved " + nextPic);
>> get.setURI(new URI("http://targetsite" + nextPic, false));
>> client.executeMethod( get );
>> InputStream is = get.getResponseBodyAsStream();
>> BufferedInputStream bis = new BufferedInputStream( is );
>> FileOutputStream fos = new FileOutputStream( nextPic );
>> byte[] bytes = new byte[ 200000 ];//room for a 200K file
>> int count = bis.read( bytes );
>> while( count != -1 && count <= 200000 ) {
>> System.out.print( "-" );
>> fos.write( bytes, 0, count );
>> count = bis.read( bytes );
>> }
>> System.out.println("saved " + nextPic);
>> if( count != -1 ) {
>> fos.write( bytes, 0, count );
>> }
>> fos.close();
>> bis.close();
>> }
>> System.out.println("saved" + nextPic);
>>
>>-------------------------------------------------------------------------------------
>>
>>
>>
>>In this example, the assumption is that there are 24 images, named in
>>numberical order. The code works fine, except that every once in a
>>while, I'm getting a "responseBodyConsumed" message like this one:
>>
>>---------------------
>>Nov 6, 2004 7:10:53 PM org.apache.commons.httpclient.HttpMethodBase
>>responseBodyConsumed
>>INFO: Stream closed
>>---------------------
>>
>>I've been trying to figure out what causes this problem, and how to fix
>>it. Any help?
>>
>>Thanks,
>>M.
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
--
Michael W. Taft
Screenwriter/Producer
4614 Finley Avenue, #3
Los Angeles, CA 90027
(323)663-6042
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem Downloading Images
Posted by Oleg Kalnichevski <ol...@apache.org>.
Michael,
This is not a problem, but merely an info message. You can get rid of it
by reducing the log verbosity from INFO to WARN
Oleg
On Sun, 2004-11-07 at 04:37, Michael Taft wrote:
> Hello -
> I have a quick question. I am downloading images and writing them to
> disk using code that looks like this:
>
>
> -----------------------------------------------------------------------------------------------------
>
>
> String nextPic = "img01.jpg";
> for (int i=1; i<=24; i++)
> {
> String iString = Integer.toString(i);
> if(i<10)iString = "0"+iString;
> nextPic = "img" + iString + ".jpg";
> // System.out.println("saved " + nextPic);
> get.setURI(new URI("http://targetsite" + nextPic, false));
> client.executeMethod( get );
> InputStream is = get.getResponseBodyAsStream();
> BufferedInputStream bis = new BufferedInputStream( is );
> FileOutputStream fos = new FileOutputStream( nextPic );
> byte[] bytes = new byte[ 200000 ];//room for a 200K file
> int count = bis.read( bytes );
> while( count != -1 && count <= 200000 ) {
> System.out.print( "-" );
> fos.write( bytes, 0, count );
> count = bis.read( bytes );
> }
> System.out.println("saved " + nextPic);
> if( count != -1 ) {
> fos.write( bytes, 0, count );
> }
> fos.close();
> bis.close();
> }
> System.out.println("saved" + nextPic);
>
> -------------------------------------------------------------------------------------
>
>
>
> In this example, the assumption is that there are 24 images, named in
> numberical order. The code works fine, except that every once in a
> while, I'm getting a "responseBodyConsumed" message like this one:
>
> ---------------------
> Nov 6, 2004 7:10:53 PM org.apache.commons.httpclient.HttpMethodBase
> responseBodyConsumed
> INFO: Stream closed
> ---------------------
>
> I've been trying to figure out what causes this problem, and how to fix
> it. Any help?
>
> Thanks,
> M.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org