You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by "S.L" <si...@gmail.com> on 2013/04/02 05:42:04 UTC
Error trying to download a Gzip file.
Hi All,
I use the following code and run the method multiple times, a few times I
get a response in gzip which is what I expect and a few time I get a
response that is completely different(non Gzip and html format) .However if
I download the same URL multiple times using Mozilla or IE I consistently
get the same GZIP response , Is this an error with the server I am trying
to reach to , or do I need to set parameters to get a consistent response ?
Thanks.
The URL I am trying to download is *
http://www.walmart.com/navigation6.xml.gz* , can you please let me know ?
Thanks
public static byte[] dowloadURL(URL urlToDownload) {
InputStream iStream = null;
byte[] urlBytes = null;
try {
//HttpClient httpClient = new HttpClient();
org.apache.http.client.
HttpClient httpClient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(urlToDownload.toString());
HttpResponse response = httpClient.execute(httpget);
iStream = response.getEntity().getContent();
urlBytes = IOUtils.toByteArray(iStream);
String responseString = new String(urlBytes);
System.out.println(" >>> The response string for "
+urlToDownload.toString()+ " is " +responseString);
} catch (IOException e) {
System.err.printf("Failed while reading bytes from %s: %s",
urlToDownload.toExternalForm(), e.getMessage());
e.printStackTrace();
// Perform any other exception handling that's appropriate.
} finally {
if (iStream != null) {
try {
iStream.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
return urlBytes;
}
Re: Error trying to download a Gzip file.
Posted by sebb <se...@gmail.com>.
On 2 April 2013 22:28, S.L <si...@gmail.com> wrote:
> Thanks Stephen .I completely agree with you , I am just accessing publicly
> available data , one thing that I would like to mention though is that I
> get the following warning every time this happens .
>
> *Invalid cookie header: "Set-Cookie: ". Cookie name may not be empty*.
>
> Even though I have set a lenient cookie -policy as illustrated here in
> section 3.5 .
>
>
> http://hc.apache.org/httpcomponents-client-ga/tutorial/html/statemgmt.html#d4e777
>
>
>
Even a lenient cookie policy cannot allow a cookie with no name.
Sounds like the server is having problems processing your request and is
returning invalid HTTP headers as a result.
I suggest you capture the headers and check.
>
>
> On Tue, Apr 2, 2013 at 4:19 PM, Stephen J. Butler
> <st...@gmail.com>wrote:
>
> > This really doesn't sound like a problem with HttpClient. Rather, an
> issue
> > with Walmart's servers. Maybe you're querying the server too often and
> > they're rate limiting you. Or if the same pattern always works in a
> browser
> > then they're sniffing your UserAgent and doing something different.
> >
> > In any event, if this is an approved API for Walmart.com, you need to
> > contact them. Or just cache the response yourself in the code and use a
> > cached value when you get a 404. There's nothing wrong with HttpClient.
> > It's doing exactly what the server told it to do.
> >
> >
> >
> > On Tue, Apr 2, 2013 at 3:12 PM, S.L <si...@gmail.com> wrote:
> >
> > > Stephen,
> > >
> > > Thanks, I now checked the reason phrase and it returns "Not Found"
> when I
> > > get a non gzip page not found error and an OK when I successfully get a
> > > gzip file download. As I said earlier in multiple iterations , the
> first
> > > one is successful almost all the time and any subsequent request after
> > > that can fail. Is there any way I can "stick" to the fist successful
> > server
> > > instance using the HttpClient , is there any way around ? Thanks for
> > your
> > > help.
> > >
> > >
> > > On Mon, Apr 1, 2013 at 11:48 PM, Stephen J. Butler <
> > > stephen.butler@gmail.com
> > > > wrote:
> > >
> > > > I notice you aren't checking your status code. Error response have a
> > > > body/response entity too. Take a look at
> > > > response.getStatusLine().getStatusCode() and
> getStatusReasonPhrase(). I
> > > bet
> > > > the server is limiting you in the instances where you're seeing
> > > non-gziped
> > > > content.
> > > >
> > > >
> > > > On Mon, Apr 1, 2013 at 10:42 PM, S.L <si...@gmail.com>
> > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I use the following code and run the method multiple times, a few
> > > times I
> > > > > get a response in gzip which is what I expect and a few time I get
> a
> > > > > response that is completely different(non Gzip and html format)
> > > .However
> > > > if
> > > > > I download the same URL multiple times using Mozilla or IE I
> > > consistently
> > > > > get the same GZIP response , Is this an error with the server I am
> > > trying
> > > > > to reach to , or do I need to set parameters to get a consistent
> > > > response ?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > The URL I am trying to download is *
> > > > > http://www.walmart.com/navigation6.xml.gz* , can you please let me
> > > know
> > > > ?
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > public static byte[] dowloadURL(URL urlToDownload) {
> > > > >
> > > > > InputStream iStream = null;
> > > > > byte[] urlBytes = null;
> > > > >
> > > > > try {
> > > > >
> > > > > //HttpClient httpClient = new HttpClient();
> > > > > org.apache.http.client.
> > > > > HttpClient httpClient = new DefaultHttpClient();
> > > > >
> > > > >
> > > > > HttpGet httpget = new
> HttpGet(urlToDownload.toString());
> > > > >
> > > > > HttpResponse response = httpClient.execute(httpget);
> > > > >
> > > > > iStream = response.getEntity().getContent();
> > > > >
> > > > > urlBytes = IOUtils.toByteArray(iStream);
> > > > > String responseString = new String(urlBytes);
> > > > > System.out.println(" >>> The response string for "
> > > > > +urlToDownload.toString()+ " is " +responseString);
> > > > >
> > > > > } catch (IOException e) {
> > > > > System.err.printf("Failed while reading bytes from %s:
> > %s",
> > > > > urlToDownload.toExternalForm(),
> e.getMessage());
> > > > > e.printStackTrace();
> > > > > // Perform any other exception handling that's
> > appropriate.
> > > > > } finally {
> > > > > if (iStream != null) {
> > > > > try {
> > > > > iStream.close();
> > > > > } catch (IOException e) {
> > > > > // TODO Auto-generated catch block
> > > > > e.printStackTrace();
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > return urlBytes;
> > > > > }
> > > > >
> > > >
> > >
> >
>
Re: Error trying to download a Gzip file.
Posted by "S.L" <si...@gmail.com>.
Thanks Stephen .I completely agree with you , I am just accessing publicly
available data , one thing that I would like to mention though is that I
get the following warning every time this happens .
*Invalid cookie header: "Set-Cookie: ". Cookie name may not be empty*.
Even though I have set a lenient cookie -policy as illustrated here in
section 3.5 .
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/statemgmt.html#d4e777
On Tue, Apr 2, 2013 at 4:19 PM, Stephen J. Butler
<st...@gmail.com>wrote:
> This really doesn't sound like a problem with HttpClient. Rather, an issue
> with Walmart's servers. Maybe you're querying the server too often and
> they're rate limiting you. Or if the same pattern always works in a browser
> then they're sniffing your UserAgent and doing something different.
>
> In any event, if this is an approved API for Walmart.com, you need to
> contact them. Or just cache the response yourself in the code and use a
> cached value when you get a 404. There's nothing wrong with HttpClient.
> It's doing exactly what the server told it to do.
>
>
>
> On Tue, Apr 2, 2013 at 3:12 PM, S.L <si...@gmail.com> wrote:
>
> > Stephen,
> >
> > Thanks, I now checked the reason phrase and it returns "Not Found" when I
> > get a non gzip page not found error and an OK when I successfully get a
> > gzip file download. As I said earlier in multiple iterations , the first
> > one is successful almost all the time and any subsequent request after
> > that can fail. Is there any way I can "stick" to the fist successful
> server
> > instance using the HttpClient , is there any way around ? Thanks for
> your
> > help.
> >
> >
> > On Mon, Apr 1, 2013 at 11:48 PM, Stephen J. Butler <
> > stephen.butler@gmail.com
> > > wrote:
> >
> > > I notice you aren't checking your status code. Error response have a
> > > body/response entity too. Take a look at
> > > response.getStatusLine().getStatusCode() and getStatusReasonPhrase(). I
> > bet
> > > the server is limiting you in the instances where you're seeing
> > non-gziped
> > > content.
> > >
> > >
> > > On Mon, Apr 1, 2013 at 10:42 PM, S.L <si...@gmail.com>
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > I use the following code and run the method multiple times, a few
> > times I
> > > > get a response in gzip which is what I expect and a few time I get a
> > > > response that is completely different(non Gzip and html format)
> > .However
> > > if
> > > > I download the same URL multiple times using Mozilla or IE I
> > consistently
> > > > get the same GZIP response , Is this an error with the server I am
> > trying
> > > > to reach to , or do I need to set parameters to get a consistent
> > > response ?
> > > >
> > > > Thanks.
> > > >
> > > > The URL I am trying to download is *
> > > > http://www.walmart.com/navigation6.xml.gz* , can you please let me
> > know
> > > ?
> > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > > > public static byte[] dowloadURL(URL urlToDownload) {
> > > >
> > > > InputStream iStream = null;
> > > > byte[] urlBytes = null;
> > > >
> > > > try {
> > > >
> > > > //HttpClient httpClient = new HttpClient();
> > > > org.apache.http.client.
> > > > HttpClient httpClient = new DefaultHttpClient();
> > > >
> > > >
> > > > HttpGet httpget = new HttpGet(urlToDownload.toString());
> > > >
> > > > HttpResponse response = httpClient.execute(httpget);
> > > >
> > > > iStream = response.getEntity().getContent();
> > > >
> > > > urlBytes = IOUtils.toByteArray(iStream);
> > > > String responseString = new String(urlBytes);
> > > > System.out.println(" >>> The response string for "
> > > > +urlToDownload.toString()+ " is " +responseString);
> > > >
> > > > } catch (IOException e) {
> > > > System.err.printf("Failed while reading bytes from %s:
> %s",
> > > > urlToDownload.toExternalForm(), e.getMessage());
> > > > e.printStackTrace();
> > > > // Perform any other exception handling that's
> appropriate.
> > > > } finally {
> > > > if (iStream != null) {
> > > > try {
> > > > iStream.close();
> > > > } catch (IOException e) {
> > > > // TODO Auto-generated catch block
> > > > e.printStackTrace();
> > > > }
> > > > }
> > > > }
> > > >
> > > > return urlBytes;
> > > > }
> > > >
> > >
> >
>
Re: Error trying to download a Gzip file.
Posted by "Stephen J. Butler" <st...@gmail.com>.
This really doesn't sound like a problem with HttpClient. Rather, an issue
with Walmart's servers. Maybe you're querying the server too often and
they're rate limiting you. Or if the same pattern always works in a browser
then they're sniffing your UserAgent and doing something different.
In any event, if this is an approved API for Walmart.com, you need to
contact them. Or just cache the response yourself in the code and use a
cached value when you get a 404. There's nothing wrong with HttpClient.
It's doing exactly what the server told it to do.
On Tue, Apr 2, 2013 at 3:12 PM, S.L <si...@gmail.com> wrote:
> Stephen,
>
> Thanks, I now checked the reason phrase and it returns "Not Found" when I
> get a non gzip page not found error and an OK when I successfully get a
> gzip file download. As I said earlier in multiple iterations , the first
> one is successful almost all the time and any subsequent request after
> that can fail. Is there any way I can "stick" to the fist successful server
> instance using the HttpClient , is there any way around ? Thanks for your
> help.
>
>
> On Mon, Apr 1, 2013 at 11:48 PM, Stephen J. Butler <
> stephen.butler@gmail.com
> > wrote:
>
> > I notice you aren't checking your status code. Error response have a
> > body/response entity too. Take a look at
> > response.getStatusLine().getStatusCode() and getStatusReasonPhrase(). I
> bet
> > the server is limiting you in the instances where you're seeing
> non-gziped
> > content.
> >
> >
> > On Mon, Apr 1, 2013 at 10:42 PM, S.L <si...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > I use the following code and run the method multiple times, a few
> times I
> > > get a response in gzip which is what I expect and a few time I get a
> > > response that is completely different(non Gzip and html format)
> .However
> > if
> > > I download the same URL multiple times using Mozilla or IE I
> consistently
> > > get the same GZIP response , Is this an error with the server I am
> trying
> > > to reach to , or do I need to set parameters to get a consistent
> > response ?
> > >
> > > Thanks.
> > >
> > > The URL I am trying to download is *
> > > http://www.walmart.com/navigation6.xml.gz* , can you please let me
> know
> > ?
> > > Thanks
> > >
> > >
> > >
> > >
> > > public static byte[] dowloadURL(URL urlToDownload) {
> > >
> > > InputStream iStream = null;
> > > byte[] urlBytes = null;
> > >
> > > try {
> > >
> > > //HttpClient httpClient = new HttpClient();
> > > org.apache.http.client.
> > > HttpClient httpClient = new DefaultHttpClient();
> > >
> > >
> > > HttpGet httpget = new HttpGet(urlToDownload.toString());
> > >
> > > HttpResponse response = httpClient.execute(httpget);
> > >
> > > iStream = response.getEntity().getContent();
> > >
> > > urlBytes = IOUtils.toByteArray(iStream);
> > > String responseString = new String(urlBytes);
> > > System.out.println(" >>> The response string for "
> > > +urlToDownload.toString()+ " is " +responseString);
> > >
> > > } catch (IOException e) {
> > > System.err.printf("Failed while reading bytes from %s: %s",
> > > urlToDownload.toExternalForm(), e.getMessage());
> > > e.printStackTrace();
> > > // Perform any other exception handling that's appropriate.
> > > } finally {
> > > if (iStream != null) {
> > > try {
> > > iStream.close();
> > > } catch (IOException e) {
> > > // TODO Auto-generated catch block
> > > e.printStackTrace();
> > > }
> > > }
> > > }
> > >
> > > return urlBytes;
> > > }
> > >
> >
>
Re: Error trying to download a Gzip file.
Posted by "S.L" <si...@gmail.com>.
Stephen,
Thanks, I now checked the reason phrase and it returns "Not Found" when I
get a non gzip page not found error and an OK when I successfully get a
gzip file download. As I said earlier in multiple iterations , the first
one is successful almost all the time and any subsequent request after
that can fail. Is there any way I can "stick" to the fist successful server
instance using the HttpClient , is there any way around ? Thanks for your
help.
On Mon, Apr 1, 2013 at 11:48 PM, Stephen J. Butler <stephen.butler@gmail.com
> wrote:
> I notice you aren't checking your status code. Error response have a
> body/response entity too. Take a look at
> response.getStatusLine().getStatusCode() and getStatusReasonPhrase(). I bet
> the server is limiting you in the instances where you're seeing non-gziped
> content.
>
>
> On Mon, Apr 1, 2013 at 10:42 PM, S.L <si...@gmail.com> wrote:
>
> > Hi All,
> >
> > I use the following code and run the method multiple times, a few times I
> > get a response in gzip which is what I expect and a few time I get a
> > response that is completely different(non Gzip and html format) .However
> if
> > I download the same URL multiple times using Mozilla or IE I consistently
> > get the same GZIP response , Is this an error with the server I am trying
> > to reach to , or do I need to set parameters to get a consistent
> response ?
> >
> > Thanks.
> >
> > The URL I am trying to download is *
> > http://www.walmart.com/navigation6.xml.gz* , can you please let me know
> ?
> > Thanks
> >
> >
> >
> >
> > public static byte[] dowloadURL(URL urlToDownload) {
> >
> > InputStream iStream = null;
> > byte[] urlBytes = null;
> >
> > try {
> >
> > //HttpClient httpClient = new HttpClient();
> > org.apache.http.client.
> > HttpClient httpClient = new DefaultHttpClient();
> >
> >
> > HttpGet httpget = new HttpGet(urlToDownload.toString());
> >
> > HttpResponse response = httpClient.execute(httpget);
> >
> > iStream = response.getEntity().getContent();
> >
> > urlBytes = IOUtils.toByteArray(iStream);
> > String responseString = new String(urlBytes);
> > System.out.println(" >>> The response string for "
> > +urlToDownload.toString()+ " is " +responseString);
> >
> > } catch (IOException e) {
> > System.err.printf("Failed while reading bytes from %s: %s",
> > urlToDownload.toExternalForm(), e.getMessage());
> > e.printStackTrace();
> > // Perform any other exception handling that's appropriate.
> > } finally {
> > if (iStream != null) {
> > try {
> > iStream.close();
> > } catch (IOException e) {
> > // TODO Auto-generated catch block
> > e.printStackTrace();
> > }
> > }
> > }
> >
> > return urlBytes;
> > }
> >
>
Re: Error trying to download a Gzip file.
Posted by "Stephen J. Butler" <st...@gmail.com>.
I notice you aren't checking your status code. Error response have a
body/response entity too. Take a look at
response.getStatusLine().getStatusCode() and getStatusReasonPhrase(). I bet
the server is limiting you in the instances where you're seeing non-gziped
content.
On Mon, Apr 1, 2013 at 10:42 PM, S.L <si...@gmail.com> wrote:
> Hi All,
>
> I use the following code and run the method multiple times, a few times I
> get a response in gzip which is what I expect and a few time I get a
> response that is completely different(non Gzip and html format) .However if
> I download the same URL multiple times using Mozilla or IE I consistently
> get the same GZIP response , Is this an error with the server I am trying
> to reach to , or do I need to set parameters to get a consistent response ?
>
> Thanks.
>
> The URL I am trying to download is *
> http://www.walmart.com/navigation6.xml.gz* , can you please let me know ?
> Thanks
>
>
>
>
> public static byte[] dowloadURL(URL urlToDownload) {
>
> InputStream iStream = null;
> byte[] urlBytes = null;
>
> try {
>
> //HttpClient httpClient = new HttpClient();
> org.apache.http.client.
> HttpClient httpClient = new DefaultHttpClient();
>
>
> HttpGet httpget = new HttpGet(urlToDownload.toString());
>
> HttpResponse response = httpClient.execute(httpget);
>
> iStream = response.getEntity().getContent();
>
> urlBytes = IOUtils.toByteArray(iStream);
> String responseString = new String(urlBytes);
> System.out.println(" >>> The response string for "
> +urlToDownload.toString()+ " is " +responseString);
>
> } catch (IOException e) {
> System.err.printf("Failed while reading bytes from %s: %s",
> urlToDownload.toExternalForm(), e.getMessage());
> e.printStackTrace();
> // Perform any other exception handling that's appropriate.
> } finally {
> if (iStream != null) {
> try {
> iStream.close();
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> }
> }
>
> return urlBytes;
> }
>