You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Alessandro Manzoni <ma...@gmail.com> on 2015/03/02 11:45:17 UTC
Re: 400 bad request POSTing to Tomcat 7.0.42
Using InputStreamEntity and forcing UFF-8 as enconding in HttpPost the
problem was solved. InputStreamEntity only, without UFF-8, was not
enough. That's sond definitively as a Tomcat issue.
Il 26.02.2015 13.35, Oleg Kalnichevski ha scritto:
> On Thu, 2015-02-26 at 13:28 +0100, Alessandro Manzoni wrote:
>> Thank you Oleg,
>> Il 26.02.2015 09.52, Oleg Kalnichevski ha scritto:
>>> On Thu, 2015-02-26 at 19:04 +1100, Brett Ryan wrote:
>>>>> Since I produce the xml in memory, that's the way Marshal.marshal method works, I could use the ByteArrayEntity using the byte[] from the ByteArrayOutputStream supplied to marshal.
>>>> Could you not do something like
>>>>
>>>> PipedOutputStream out = new PipedOutputStream();
>>>> InputStream instr = new PipedInputStream(out);
>>>> marshaller.marshal(object, out);
>>>> HttpPost post = new HttpPost();
>>>> post.setEntity(new InputStreamEntity(instr));
>>>>
>>> A custom HttpEntity implementation that internally makes use of JAXB
>>> marshaling would be massively more efficient.
>>>
>>> Oleg
>> I agree with you, but efficiency is not an issue.
>> The client is manageg by an application that gather data to send as an
>> xml to the target Tomcat.
>> Data are validated by marshalling before sending by HTTPCLient.
>> If validation didn't pass, a notification of invalid data is sent to
>> another service. When validation is succesfull, validated data is sent
>> by http client.
>> So putting validation into a custom entity would be a duplicated
>> operation, then the responsability of validation is not a transport
>> agent one. Don't you agree?
>>
> HttpEntity is essentially a data producer meant to encapsulate whatever
> data generation mechanism there already is, including validation and
> what not. No code duplication should ever be necessary.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Alexey Panchenko <al...@gmail.com>.
And that's kind of correct.
application/xml does not care of charset. but text/xml does.
On Tue, Mar 3, 2015 at 5:06 PM, Alessandro Manzoni <
manzoni.alessandro4@gmail.com> wrote:
> Il 03.03.2015 09.47, Alexey Panchenko ha scritto:
>
>> I am curious about how the client code looks now, before we continue
>> complaining on the other end.
>> Just trying to be fair.
>>
> 1 HttpClient httpclient = new DefaultHttpClient();
> 2 HttpPost httppost = new HttpPost(uri);
> 3 httppost.addHeader("Content-Type", "text/xml;charset=utf-8");
> 4 ByteArrayEntity entity = new ByteArrayEntity(output.
> toByteArray());
> 5 // StringEntity entity = new StringEntity(new
> String(output.toByteArray()), ContentType.TEXT_XML);
> 6 httppost.setEntity(entity);
> 7 HttpResponse response = httpclient.execute(httppost);
>
> That's it. Added line 3 and line 4 replaced line 5.
>
>
>
>> On Tue, Mar 3, 2015 at 2:36 PM, Brett Ryan <br...@gmail.com> wrote:
>>
>> What confused me, is that I thought that tomcat should honor the
>>>>
>>> encoding set inside xml, while it just use content-type encoding or its
>>> default one instead.
>>>
>>> Thats the correct behaviour. Your servlet container may choose to inspect
>>> the POST data but there is no requirement for it to, thats up to you to
>>> implement a filter as mentioned earlier. What if your XML was malformed,
>>> what do you think it should do? The content could be anything for all it
>>> cares. What would your processor do if the header of the XML said the
>>> encoding was 8859-1, but the actual encoding was UCS-2? You wouldn't even
>>> be able to read the header reliably, you would have to perform a series
>>> of
>>> rereads to get the right encoding.
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Brett Ryan <br...@gmail.com>.
You didn't optimise as we suggested in the end, you're still building the content up in memory. As suggested by a few, write to a stream so nothing needs to consume your clients memory and will scale better
Sent from my iPhone
> On 3 Mar 2015, at 22:06, Alessandro Manzoni <ma...@gmail.com> wrote:
>
> Il 03.03.2015 09.47, Alexey Panchenko ha scritto:
>> I am curious about how the client code looks now, before we continue
>> complaining on the other end.
>> Just trying to be fair.
> 1 HttpClient httpclient = new DefaultHttpClient();
> 2 HttpPost httppost = new HttpPost(uri);
> 3 httppost.addHeader("Content-Type", "text/xml;charset=utf-8");
> 4 ByteArrayEntity entity = new ByteArrayEntity(output.toByteArray());
> 5 // StringEntity entity = new StringEntity(new String(output.toByteArray()), ContentType.TEXT_XML);
> 6 httppost.setEntity(entity);
> 7 HttpResponse response = httpclient.execute(httppost);
>
> That's it. Added line 3 and line 4 replaced line 5.
>
>>
>> On Tue, Mar 3, 2015 at 2:36 PM, Brett Ryan <br...@gmail.com> wrote:
>>
>>>> What confused me, is that I thought that tomcat should honor the
>>> encoding set inside xml, while it just use content-type encoding or its
>>> default one instead.
>>>
>>> Thats the correct behaviour. Your servlet container may choose to inspect
>>> the POST data but there is no requirement for it to, thats up to you to
>>> implement a filter as mentioned earlier. What if your XML was malformed,
>>> what do you think it should do? The content could be anything for all it
>>> cares. What would your processor do if the header of the XML said the
>>> encoding was 8859-1, but the actual encoding was UCS-2? You wouldn't even
>>> be able to read the header reliably, you would have to perform a series of
>>> rereads to get the right encoding.
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Alessandro Manzoni <ma...@gmail.com>.
Il 03.03.2015 09.47, Alexey Panchenko ha scritto:
> I am curious about how the client code looks now, before we continue
> complaining on the other end.
> Just trying to be fair.
1 HttpClient httpclient = new DefaultHttpClient();
2 HttpPost httppost = new HttpPost(uri);
3 httppost.addHeader("Content-Type", "text/xml;charset=utf-8");
4 ByteArrayEntity entity = new
ByteArrayEntity(output.toByteArray());
5 // StringEntity entity = new StringEntity(new
String(output.toByteArray()), ContentType.TEXT_XML);
6 httppost.setEntity(entity);
7 HttpResponse response = httpclient.execute(httppost);
That's it. Added line 3 and line 4 replaced line 5.
>
> On Tue, Mar 3, 2015 at 2:36 PM, Brett Ryan <br...@gmail.com> wrote:
>
>>> What confused me, is that I thought that tomcat should honor the
>> encoding set inside xml, while it just use content-type encoding or its
>> default one instead.
>>
>> Thats the correct behaviour. Your servlet container may choose to inspect
>> the POST data but there is no requirement for it to, thats up to you to
>> implement a filter as mentioned earlier. What if your XML was malformed,
>> what do you think it should do? The content could be anything for all it
>> cares. What would your processor do if the header of the XML said the
>> encoding was 8859-1, but the actual encoding was UCS-2? You wouldn't even
>> be able to read the header reliably, you would have to perform a series of
>> rereads to get the right encoding.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Alexey Panchenko <al...@gmail.com>.
I am curious about how the client code looks now, before we continue
complaining on the other end.
Just trying to be fair.
On Tue, Mar 3, 2015 at 2:36 PM, Brett Ryan <br...@gmail.com> wrote:
>
> > What confused me, is that I thought that tomcat should honor the
> encoding set inside xml, while it just use content-type encoding or its
> default one instead.
>
> Thats the correct behaviour. Your servlet container may choose to inspect
> the POST data but there is no requirement for it to, thats up to you to
> implement a filter as mentioned earlier. What if your XML was malformed,
> what do you think it should do? The content could be anything for all it
> cares. What would your processor do if the header of the XML said the
> encoding was 8859-1, but the actual encoding was UCS-2? You wouldn't even
> be able to read the header reliably, you would have to perform a series of
> rereads to get the right encoding.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Brett Ryan <br...@gmail.com>.
> What confused me, is that I thought that tomcat should honor the encoding set inside xml, while it just use content-type encoding or its default one instead.
Thats the correct behaviour. Your servlet container may choose to inspect the POST data but there is no requirement for it to, thats up to you to implement a filter as mentioned earlier. What if your XML was malformed, what do you think it should do? The content could be anything for all it cares. What would your processor do if the header of the XML said the encoding was 8859-1, but the actual encoding was UCS-2? You wouldn't even be able to read the header reliably, you would have to perform a series of rereads to get the right encoding.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Alessandro Manzoni <ma...@gmail.com>.
Il 02.03.2015 21.45, Brett Ryan ha scritto:
> Hah. If it wasn't for that greek letter we wouldn't have been able to help you optimise your code, and the bug could have come up at a far more dire time in production.
Yes indeed. Further more, as at test time the bug didn't come up (due to
a limited set of data supplied by customer), it was already in production!!
What confused me, is that I thought that tomcat should honor the
encoding set inside xml, while it just use content-type encoding or its
default one instead.
Usually I use axis with soap services, so I just don't care about
encoding, this time the server is implementing a sort of post only
miscontrolled REST service.
>
> Sent from my iPhone
>
>> On 3 Mar 2015, at 03:27, Alessandro Manzoni <ma...@gmail.com> wrote:
>>
>> Il 02.03.2015 17.18, Brett Ryan ha scritto:
>>> Actually its not tomcats issue, when sending an encoding other than the default which is ISO-8859-1 according to RFC-2616 (HTTP/1.1) you should send the encoding type (i.e. text/xml;charset=utf-8). By default the JAXB marshaller will use UTF-8 encoding when marshalling.
>>>
>>> You can switch Tomcat to use UTF-8 encoding everywhere by default by setting the URI encoding, adding an encoding filter such as [1](Springs CharacterEncodingFilter) implementation etc... follow details at [2](the tomcat wiki).
>>>
>>> What I'm guessing was happening was you were sending a character like a copyright symbol but instead of sending 0xA9 you were sending 0xC2 0xA9
>>>
>>> [1]: http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/filter/CharacterEncodingFilter.html
>>> [2]: http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q8
>> More or less you guessed well.
>> In fact inside the xml there was a special character (a fi greek letter), but what I found in wire log, this character had been converted in a string like "[0xd8]" as is (including brackets). So from my point of view, in that string there weren't spacial characters. Unless wire log make some kind of formatting to render the content.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Brett Ryan <br...@gmail.com>.
Hah. If it wasn't for that greek letter we wouldn't have been able to help you optimise your code, and the bug could have come up at a far more dire time in production.
Sent from my iPhone
> On 3 Mar 2015, at 03:27, Alessandro Manzoni <ma...@gmail.com> wrote:
>
> Il 02.03.2015 17.18, Brett Ryan ha scritto:
>> Actually its not tomcats issue, when sending an encoding other than the default which is ISO-8859-1 according to RFC-2616 (HTTP/1.1) you should send the encoding type (i.e. text/xml;charset=utf-8). By default the JAXB marshaller will use UTF-8 encoding when marshalling.
>>
>> You can switch Tomcat to use UTF-8 encoding everywhere by default by setting the URI encoding, adding an encoding filter such as [1](Springs CharacterEncodingFilter) implementation etc... follow details at [2](the tomcat wiki).
>>
>> What I'm guessing was happening was you were sending a character like a copyright symbol but instead of sending 0xA9 you were sending 0xC2 0xA9
>>
>> [1]: http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/filter/CharacterEncodingFilter.html
>> [2]: http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q8
> More or less you guessed well.
> In fact inside the xml there was a special character (a fi greek letter), but what I found in wire log, this character had been converted in a string like "[0xd8]" as is (including brackets). So from my point of view, in that string there weren't spacial characters. Unless wire log make some kind of formatting to render the content.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Alessandro Manzoni <ma...@gmail.com>.
Il 02.03.2015 17.18, Brett Ryan ha scritto:
> Actually its not tomcats issue, when sending an encoding other than the default which is ISO-8859-1 according to RFC-2616 (HTTP/1.1) you should send the encoding type (i.e. text/xml;charset=utf-8). By default the JAXB marshaller will use UTF-8 encoding when marshalling.
>
> You can switch Tomcat to use UTF-8 encoding everywhere by default by setting the URI encoding, adding an encoding filter such as [1](Springs CharacterEncodingFilter) implementation etc... follow details at [2](the tomcat wiki).
>
> What I'm guessing was happening was you were sending a character like a copyright symbol but instead of sending 0xA9 you were sending 0xC2 0xA9
>
> [1]: http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/filter/CharacterEncodingFilter.html
> [2]: http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q8
>
More or less you guessed well.
In fact inside the xml there was a special character (a fi greek
letter), but what I found in wire log, this character had been converted
in a string like "[0xd8]" as is (including brackets). So from my point
of view, in that string there weren't spacial characters. Unless wire
log make some kind of formatting to render the content.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Brett Ryan <br...@gmail.com>.
Actually its not tomcats issue, when sending an encoding other than the default which is ISO-8859-1 according to RFC-2616 (HTTP/1.1) you should send the encoding type (i.e. text/xml;charset=utf-8). By default the JAXB marshaller will use UTF-8 encoding when marshalling.
You can switch Tomcat to use UTF-8 encoding everywhere by default by setting the URI encoding, adding an encoding filter such as [1](Springs CharacterEncodingFilter) implementation etc... follow details at [2](the tomcat wiki).
What I'm guessing was happening was you were sending a character like a copyright symbol but instead of sending 0xA9 you were sending 0xC2 0xA9
[1]: http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/filter/CharacterEncodingFilter.html
[2]: http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q8
On 2 Mar 2015, at 21:45, Alessandro Manzoni <manzoni.alessandro4@gmail.com <ma...@gmail.com>> wrote:
> Using InputStreamEntity and forcing UFF-8 as enconding in HttpPost the problem was solved. InputStreamEntity only, without UFF-8, was not enough. That's sond definitively as a Tomcat issue.
>
> Il 26.02.2015 13.35, Oleg Kalnichevski ha scritto:
>> On Thu, 2015-02-26 at 13:28 +0100, Alessandro Manzoni wrote:
>>> Thank you Oleg,
>>> Il 26.02.2015 09.52, Oleg Kalnichevski ha scritto:
>>>> On Thu, 2015-02-26 at 19:04 +1100, Brett Ryan wrote:
>>>>>> Since I produce the xml in memory, that's the way Marshal.marshal method works, I could use the ByteArrayEntity using the byte[] from the ByteArrayOutputStream supplied to marshal.
>>>>> Could you not do something like
>>>>>
>>>>> PipedOutputStream out = new PipedOutputStream();
>>>>> InputStream instr = new PipedInputStream(out);
>>>>> marshaller.marshal(object, out);
>>>>> HttpPost post = new HttpPost();
>>>>> post.setEntity(new InputStreamEntity(instr));
>>>>>
>>>> A custom HttpEntity implementation that internally makes use of JAXB
>>>> marshaling would be massively more efficient.
>>>>
>>>> Oleg
>>> I agree with you, but efficiency is not an issue.
>>> The client is manageg by an application that gather data to send as an
>>> xml to the target Tomcat.
>>> Data are validated by marshalling before sending by HTTPCLient.
>>> If validation didn't pass, a notification of invalid data is sent to
>>> another service. When validation is succesfull, validated data is sent
>>> by http client.
>>> So putting validation into a custom entity would be a duplicated
>>> operation, then the responsability of validation is not a transport
>>> agent one. Don't you agree?
>>>
>> HttpEntity is essentially a data producer meant to encapsulate whatever
>> data generation mechanism there already is, including validation and
>> what not. No code duplication should ever be necessary.
>>
>> Oleg
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org <ma...@hc.apache.org>
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org <ma...@hc.apache.org>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org <ma...@hc.apache.org>
> For additional commands, e-mail: httpclient-users-help@hc.apache.org <ma...@hc.apache.org>
>
Re: 400 bad request POSTing to Tomcat 7.0.42
Posted by Stefan Magnus Landrø <st...@gmail.com>.
what's the encoding used by the tomcat jvm? If it's let's say ISO 8859-1 that would explain everything.
Sendt fra min iPhone
> Den 2. mar. 2015 kl. 11.45 skrev Alessandro Manzoni <ma...@gmail.com>:
>
> Using InputStreamEntity and forcing UFF-8 as enconding in HttpPost the problem was solved. InputStreamEntity only, without UFF-8, was not enough. That's sond definitively as a Tomcat issue.
>
> Il 26.02.2015 13.35, Oleg Kalnichevski ha scritto:
>>> On Thu, 2015-02-26 at 13:28 +0100, Alessandro Manzoni wrote:
>>> Thank you Oleg,
>>> Il 26.02.2015 09.52, Oleg Kalnichevski ha scritto:
>>>> On Thu, 2015-02-26 at 19:04 +1100, Brett Ryan wrote:
>>>>>> Since I produce the xml in memory, that's the way Marshal.marshal method works, I could use the ByteArrayEntity using the byte[] from the ByteArrayOutputStream supplied to marshal.
>>>>> Could you not do something like
>>>>>
>>>>> PipedOutputStream out = new PipedOutputStream();
>>>>> InputStream instr = new PipedInputStream(out);
>>>>> marshaller.marshal(object, out);
>>>>> HttpPost post = new HttpPost();
>>>>> post.setEntity(new InputStreamEntity(instr));
>>>> A custom HttpEntity implementation that internally makes use of JAXB
>>>> marshaling would be massively more efficient.
>>>>
>>>> Oleg
>>> I agree with you, but efficiency is not an issue.
>>> The client is manageg by an application that gather data to send as an
>>> xml to the target Tomcat.
>>> Data are validated by marshalling before sending by HTTPCLient.
>>> If validation didn't pass, a notification of invalid data is sent to
>>> another service. When validation is succesfull, validated data is sent
>>> by http client.
>>> So putting validation into a custom entity would be a duplicated
>>> operation, then the responsability of validation is not a transport
>>> agent one. Don't you agree?
>> HttpEntity is essentially a data producer meant to encapsulate whatever
>> data generation mechanism there already is, including validation and
>> what not. No code duplication should ever be necessary.
>>
>> Oleg
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org