You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Chad La Joie <la...@itumi.biz> on 2011/04/01 15:06:54 UTC
Bytes Missing from HTTP Response
I'm experiencing an odd problem with HttpClient 4.1.1. I perform a GET
on a document and then use the following code to get the bytes of the
response entity (assuming a 200 status code):
byte[] responseEntity = EntityUtils.toByteArray(response.getEntity());
The problem I'm having is that this returns 16 fewer bytes than are
actually in the document. So far I've checked:
- that downloading the file via wget gives me the expected byte account
- that the downloaded content is not compressed
The document itself has a digital signature over it and this is failing
to validate with the content as downloaded by HttpClient, but not with
the document downloaded by wget so there is some material difference in
the canonical form of the document (i.e., it's not just a lack of a new
line at the start/end of the document).
Any thoughts? Is EntityUtils.toByteArray not the right method to use to
get the complete byte[] of the response entity?
Thanks.
--
Chad La Joie
http://itumi.biz
trusted identities, delivered
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Ken Krugler <kk...@transpac.com>.
On Apr 2, 2011, at 3:29am, Oleg Kalnichevski wrote:
> On Sat, 2011-04-02 at 06:10 -0400, Chad La Joie wrote:
>> Okay, that makes sense.
>>
>> To test this, is there a way I can force the content type on the client
>> side, prior to requesting the response entity, via the response object?
>>
>
> You can try adding Accept and / or Accept-Charset header to the request
> message and see if the origin server responds appropriately.
>
> However, generally you might be better off using some sort of a content
> detection algorithm such that provided by Apache Tika toolkit. I suspect
> wget does exactly that.
Tika tries to follow the recommendations of RFC 3023:
If an application/xml entity is received where the charset
parameter is omitted, no information is being provided about the
charset by the MIME Content-Type header. Conforming XML
processors MUST follow the requirements in section 4.3.3 of [XML]
that directly address this contingency.
Which means it will look for a byte-order-mark and encoding declaration inside of the XML content.
-- Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Chad La Joie <la...@itumi.biz>.
Okay, I'll look in to the server side of things.
As an aside, converting the stream to characters is generally a really
bad idea when it comes to XML parsing. Normally you want just give the
parser the stream and let it figure out all the details.
Thanks for the help.
On 4/4/11 8:22 AM, Oleg Kalnichevski wrote:
> On Mon, 2011-04-04 at 08:16 -0400, Chad La Joie wrote:
>> Then I guess I misunderstood what you were saying before. Are you
>> suggesting then that the server is transcoding the file when it serves
>> it up? And that the missing bytes actually go missing before HttpClient
>> gets the response?
>
>
> That is one possibility. Besides, I suspect that your application also
> needs to convert the response content to a stream of characters in order
> to be able to parse the XML message. This is another possibility for
> things to go screwy.
>
> Hope this helps
>
> Oleg
>
>> On 4/4/11 8:08 AM, Oleg Kalnichevski wrote:
>>> On Mon, 2011-04-04 at 07:34 -0400, Chad La Joie wrote:
>>>> Yeah, unfortunately that didn't work.
>>>>
>>>> Is there any way to get the old v3 behavior that gives you access to the
>>>> raw bytes of the entity before any sort of character decoding is done?
>>>>
>>>> I strongly suspect that very few web servers out there are properly
>>>> configured to return the correct character encoding so this could
>>>> definitely be an ongoing problem.
>>>>
>>>
>>> EntityUtils.toByteArray returns raw response content without attempting
>>> to decode it.
>>>
>>> http://hc.apache.org/httpcomponents-core-ga/httpcore/xref/org/apache/http/util/EntityUtils.html#81
>>>
>>> Oleg
>>>
>>>
>>>> On 4/2/11 6:29 AM, Oleg Kalnichevski wrote:
>>>>> On Sat, 2011-04-02 at 06:10 -0400, Chad La Joie wrote:
>>>>>> Okay, that makes sense.
>>>>>>
>>>>>> To test this, is there a way I can force the content type on the client
>>>>>> side, prior to requesting the response entity, via the response object?
>>>>>>
>>>>>
>>>>> You can try adding Accept and / or Accept-Charset header to the request
>>>>> message and see if the origin server responds appropriately.
>>>>>
>>>>> However, generally you might be better off using some sort of a content
>>>>> detection algorithm such that provided by Apache Tika toolkit. I suspect
>>>>> wget does exactly that.
>>>>>
>>>>> http://tika.apache.org/0.9/detection.html
>>>>> http://tika.apache.org/
>>>>>
>>>>> Oleg
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Chad La Joie
http://itumi.biz
trusted identities, delivered
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2011-04-04 at 08:16 -0400, Chad La Joie wrote:
> Then I guess I misunderstood what you were saying before. Are you
> suggesting then that the server is transcoding the file when it serves
> it up? And that the missing bytes actually go missing before HttpClient
> gets the response?
That is one possibility. Besides, I suspect that your application also
needs to convert the response content to a stream of characters in order
to be able to parse the XML message. This is another possibility for
things to go screwy.
Hope this helps
Oleg
> On 4/4/11 8:08 AM, Oleg Kalnichevski wrote:
> > On Mon, 2011-04-04 at 07:34 -0400, Chad La Joie wrote:
> >> Yeah, unfortunately that didn't work.
> >>
> >> Is there any way to get the old v3 behavior that gives you access to the
> >> raw bytes of the entity before any sort of character decoding is done?
> >>
> >> I strongly suspect that very few web servers out there are properly
> >> configured to return the correct character encoding so this could
> >> definitely be an ongoing problem.
> >>
> >
> > EntityUtils.toByteArray returns raw response content without attempting
> > to decode it.
> >
> > http://hc.apache.org/httpcomponents-core-ga/httpcore/xref/org/apache/http/util/EntityUtils.html#81
> >
> > Oleg
> >
> >
> >> On 4/2/11 6:29 AM, Oleg Kalnichevski wrote:
> >>> On Sat, 2011-04-02 at 06:10 -0400, Chad La Joie wrote:
> >>>> Okay, that makes sense.
> >>>>
> >>>> To test this, is there a way I can force the content type on the client
> >>>> side, prior to requesting the response entity, via the response object?
> >>>>
> >>>
> >>> You can try adding Accept and / or Accept-Charset header to the request
> >>> message and see if the origin server responds appropriately.
> >>>
> >>> However, generally you might be better off using some sort of a content
> >>> detection algorithm such that provided by Apache Tika toolkit. I suspect
> >>> wget does exactly that.
> >>>
> >>> http://tika.apache.org/0.9/detection.html
> >>> http://tika.apache.org/
> >>>
> >>> Oleg
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> >>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >>>
> >>>
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Chad La Joie <la...@itumi.biz>.
Then I guess I misunderstood what you were saying before. Are you
suggesting then that the server is transcoding the file when it serves
it up? And that the missing bytes actually go missing before HttpClient
gets the response?
On 4/4/11 8:08 AM, Oleg Kalnichevski wrote:
> On Mon, 2011-04-04 at 07:34 -0400, Chad La Joie wrote:
>> Yeah, unfortunately that didn't work.
>>
>> Is there any way to get the old v3 behavior that gives you access to the
>> raw bytes of the entity before any sort of character decoding is done?
>>
>> I strongly suspect that very few web servers out there are properly
>> configured to return the correct character encoding so this could
>> definitely be an ongoing problem.
>>
>
> EntityUtils.toByteArray returns raw response content without attempting
> to decode it.
>
> http://hc.apache.org/httpcomponents-core-ga/httpcore/xref/org/apache/http/util/EntityUtils.html#81
>
> Oleg
>
>
>> On 4/2/11 6:29 AM, Oleg Kalnichevski wrote:
>>> On Sat, 2011-04-02 at 06:10 -0400, Chad La Joie wrote:
>>>> Okay, that makes sense.
>>>>
>>>> To test this, is there a way I can force the content type on the client
>>>> side, prior to requesting the response entity, via the response object?
>>>>
>>>
>>> You can try adding Accept and / or Accept-Charset header to the request
>>> message and see if the origin server responds appropriately.
>>>
>>> However, generally you might be better off using some sort of a content
>>> detection algorithm such that provided by Apache Tika toolkit. I suspect
>>> wget does exactly that.
>>>
>>> http://tika.apache.org/0.9/detection.html
>>> http://tika.apache.org/
>>>
>>> Oleg
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Chad La Joie
http://itumi.biz
trusted identities, delivered
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2011-04-04 at 07:34 -0400, Chad La Joie wrote:
> Yeah, unfortunately that didn't work.
>
> Is there any way to get the old v3 behavior that gives you access to the
> raw bytes of the entity before any sort of character decoding is done?
>
> I strongly suspect that very few web servers out there are properly
> configured to return the correct character encoding so this could
> definitely be an ongoing problem.
>
EntityUtils.toByteArray returns raw response content without attempting
to decode it.
http://hc.apache.org/httpcomponents-core-ga/httpcore/xref/org/apache/http/util/EntityUtils.html#81
Oleg
> On 4/2/11 6:29 AM, Oleg Kalnichevski wrote:
> > On Sat, 2011-04-02 at 06:10 -0400, Chad La Joie wrote:
> >> Okay, that makes sense.
> >>
> >> To test this, is there a way I can force the content type on the client
> >> side, prior to requesting the response entity, via the response object?
> >>
> >
> > You can try adding Accept and / or Accept-Charset header to the request
> > message and see if the origin server responds appropriately.
> >
> > However, generally you might be better off using some sort of a content
> > detection algorithm such that provided by Apache Tika toolkit. I suspect
> > wget does exactly that.
> >
> > http://tika.apache.org/0.9/detection.html
> > http://tika.apache.org/
> >
> > Oleg
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Chad La Joie <la...@itumi.biz>.
Yeah, unfortunately that didn't work.
Is there any way to get the old v3 behavior that gives you access to the
raw bytes of the entity before any sort of character decoding is done?
I strongly suspect that very few web servers out there are properly
configured to return the correct character encoding so this could
definitely be an ongoing problem.
On 4/2/11 6:29 AM, Oleg Kalnichevski wrote:
> On Sat, 2011-04-02 at 06:10 -0400, Chad La Joie wrote:
>> Okay, that makes sense.
>>
>> To test this, is there a way I can force the content type on the client
>> side, prior to requesting the response entity, via the response object?
>>
>
> You can try adding Accept and / or Accept-Charset header to the request
> message and see if the origin server responds appropriately.
>
> However, generally you might be better off using some sort of a content
> detection algorithm such that provided by Apache Tika toolkit. I suspect
> wget does exactly that.
>
> http://tika.apache.org/0.9/detection.html
> http://tika.apache.org/
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Chad La Joie
http://itumi.biz
trusted identities, delivered
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2011-04-02 at 06:10 -0400, Chad La Joie wrote:
> Okay, that makes sense.
>
> To test this, is there a way I can force the content type on the client
> side, prior to requesting the response entity, via the response object?
>
You can try adding Accept and / or Accept-Charset header to the request
message and see if the origin server responds appropriately.
However, generally you might be better off using some sort of a content
detection algorithm such that provided by Apache Tika toolkit. I suspect
wget does exactly that.
http://tika.apache.org/0.9/detection.html
http://tika.apache.org/
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Chad La Joie <la...@itumi.biz>.
Okay, that makes sense.
To test this, is there a way I can force the content type on the client
side, prior to requesting the response entity, via the response object?
On 4/2/11 6:05 AM, Oleg Kalnichevski wrote:
> This is a problem with content decoding.
>
> << "HTTP/1.1 200 OK[\r][\n]"
> << "Date: Fri, 01 Apr 2011 14:14:28 GMT[\r][\n]"
> << "Server: Apache/2.2.3 (Unix) mod_ssl/2.2.3 OpenSSL/0.9.7d[\r][\n]"
> << "Last-Modified: Thu, 31 Mar 2011 17:00:05 GMT[\r][\n]"
> << "ETag: "4d0e-39932f40"[\r][\n]"
> << "Accept-Ranges: bytes[\r][\n]"
> << "Content-Length: 19726[\r][\n]"
> << "Keep-Alive: timeout=5, max=99[\r][\n]"
> << "Connection: Keep-Alive[\r][\n]"
> << "Content-Type: application/xml[\r][\n]"
> << "[\r][\n]"
> << "<?xml version="1.0" encoding="UTF-8"?><!--[\n]"
>
> The response content is clearly UTF-8 encoded. However the Content-Type
> header does not specify a charset. Per HTTP specification if content
> charset is not explicitly set in the Content-Type content charset is
> assumed to be ISO-8859-1
--
Chad La Joie
http://itumi.biz
trusted identities, delivered
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2011-04-01 at 18:14 -0400, Chad La Joie wrote:
> Here you go:
> http://shibboleth.net/dumps.tgz
>
> I found a much smaller document than the one I was initially testing
> with. It's off by one byte.
>
> On 4/1/11 9:38 AM, Oleg Kalnichevski wrote:
> > On Fri, 2011-04-01 at 09:06 -0400, Chad La Joie wrote:
> >> I'm experiencing an odd problem with HttpClient 4.1.1. I perform a GET
> >> on a document and then use the following code to get the bytes of the
> >> response entity (assuming a 200 status code):
> >>
> >> byte[] responseEntity = EntityUtils.toByteArray(response.getEntity());
> >>
> >> The problem I'm having is that this returns 16 fewer bytes than are
> >> actually in the document. So far I've checked:
> >> - that downloading the file via wget gives me the expected byte account
> >> - that the downloaded content is not compressed
> >>
> >> The document itself has a digital signature over it and this is failing
> >> to validate with the content as downloaded by HttpClient, but not with
> >> the document downloaded by wget so there is some material difference in
> >> the canonical form of the document (i.e., it's not just a lack of a new
> >> line at the start/end of the document).
> >>
> >> Any thoughts? Is EntityUtils.toByteArray not the right method to use to
> >> get the complete byte[] of the response entity?
> >>
> >> Thanks.
> >
This is a problem with content decoding.
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Date: Fri, 01 Apr 2011 14:14:28 GMT[\r][\n]"
<< "Server: Apache/2.2.3 (Unix) mod_ssl/2.2.3 OpenSSL/0.9.7d[\r][\n]"
<< "Last-Modified: Thu, 31 Mar 2011 17:00:05 GMT[\r][\n]"
<< "ETag: "4d0e-39932f40"[\r][\n]"
<< "Accept-Ranges: bytes[\r][\n]"
<< "Content-Length: 19726[\r][\n]"
<< "Keep-Alive: timeout=5, max=99[\r][\n]"
<< "Connection: Keep-Alive[\r][\n]"
<< "Content-Type: application/xml[\r][\n]"
<< "[\r][\n]"
<< "<?xml version="1.0" encoding="UTF-8"?><!--[\n]"
The response content is clearly UTF-8 encoded. However the Content-Type
header does not specify a charset. Per HTTP specification if content
charset is not explicitly set in the Content-Type content charset is
assumed to be ISO-8859-1
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Chad La Joie <la...@itumi.biz>.
Here you go:
http://shibboleth.net/dumps.tgz
I found a much smaller document than the one I was initially testing
with. It's off by one byte.
On 4/1/11 9:38 AM, Oleg Kalnichevski wrote:
> On Fri, 2011-04-01 at 09:06 -0400, Chad La Joie wrote:
>> I'm experiencing an odd problem with HttpClient 4.1.1. I perform a GET
>> on a document and then use the following code to get the bytes of the
>> response entity (assuming a 200 status code):
>>
>> byte[] responseEntity = EntityUtils.toByteArray(response.getEntity());
>>
>> The problem I'm having is that this returns 16 fewer bytes than are
>> actually in the document. So far I've checked:
>> - that downloading the file via wget gives me the expected byte account
>> - that the downloaded content is not compressed
>>
>> The document itself has a digital signature over it and this is failing
>> to validate with the content as downloaded by HttpClient, but not with
>> the document downloaded by wget so there is some material difference in
>> the canonical form of the document (i.e., it's not just a lack of a new
>> line at the start/end of the document).
>>
>> Any thoughts? Is EntityUtils.toByteArray not the right method to use to
>> get the complete byte[] of the response entity?
>>
>> Thanks.
>
> Post a complete wire / context log of the HTTP session generated with
> HttpClient [1] and define what the expected result should be like (dump
> of HTTP packets received with wget would be probably the best)
>
> Oleg
>
> [1] http://hc.apache.org/httpcomponents-client-ga/logging.html
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Chad La Joie
http://itumi.biz
trusted identities, delivered
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bytes Missing from HTTP Response
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2011-04-01 at 09:06 -0400, Chad La Joie wrote:
> I'm experiencing an odd problem with HttpClient 4.1.1. I perform a GET
> on a document and then use the following code to get the bytes of the
> response entity (assuming a 200 status code):
>
> byte[] responseEntity = EntityUtils.toByteArray(response.getEntity());
>
> The problem I'm having is that this returns 16 fewer bytes than are
> actually in the document. So far I've checked:
> - that downloading the file via wget gives me the expected byte account
> - that the downloaded content is not compressed
>
> The document itself has a digital signature over it and this is failing
> to validate with the content as downloaded by HttpClient, but not with
> the document downloaded by wget so there is some material difference in
> the canonical form of the document (i.e., it's not just a lack of a new
> line at the start/end of the document).
>
> Any thoughts? Is EntityUtils.toByteArray not the right method to use to
> get the complete byte[] of the response entity?
>
> Thanks.
Post a complete wire / context log of the HTTP session generated with
HttpClient [1] and define what the expected result should be like (dump
of HTTP packets received with wget would be probably the best)
Oleg
[1] http://hc.apache.org/httpcomponents-client-ga/logging.html
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org