You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Magnus Heino <ma...@filur.org> on 2008/03/13 21:32:48 UTC
MarshalProcessor utf-8 bug?
A xml-message encoded in utf-8 containing åäö routed through this route is
trashed. åäö is not åäö after this route. If I remove the marshal call, the
problem goes away.
this.from("direct:request").marshal(soapDataFormat).convertBodyTo(
String.class).to(requestUri);
To be sure that it wasn't my dataformat that trashed it, I wrote this
processor:
final Processor wrapSoap = new Processor() {
public void process(Exchange exchange) throws Exception {
ByteArrayOutputStream outputStream = new
ByteArrayOutputStream();
soapDataFormat.marshal(exchange, new
StringSource((String) exchange.getIn().getBody()), outputStream);
exchange.getOut(true).setBody(outputStream.toString
("UTF-8"));
}
};
and applied this route:
this.from("direct:request").process(this.wrapSoap).convertBodyTo(
String.class).to(requestUri);
And now things are working...
Looking at org.apache.camel.processor.MarshalProcessor.java that is used in
the marshal()-call, is this really ok to do with utf-8 data?
dataFormat.marshal(exchange, body, buffer);
byte[] data = buffer.toByteArray();
out.setBody(data);
--
/Magnus Heino
Re: MarshalProcessor utf-8 bug?
Posted by Magnus Heino <ma...@filur.org>.
>
> James, or any of the core comitters what is your view of this? Is it
> somehow
> doable to improve the type conversion framework to support meta parameters
> such as encoding.
>
> It is quite common in integrations to use byte[] and String objects when
> passing data around and thus the encoding is important.
>
Well, the current lack of support for encodings makes you not trust what
Camel does with your data. The type conversion framework is great, but since
it's not respecting the encoding of the data, in practice, its like
supporting ASCII only.
--
/Magnus Heino
Re: MarshalProcessor utf-8 bug?
Posted by davsclaus <ci...@yahoo.dk>.
Hi Magnus
I have recently been working on some patches for camel-mina for encodings.
During this work I do think the camel codebase currently does not consider
encoding when it does convertions to String.
I think camel need to be improved in its core to support encoding parameters
for its type conversion framework. Currently the type converters does not
support meta parameters such as encoding etc.
In camel-mina I did manage to get by this as I could handle the type
conversion in camel-mina and use the provided encoding parameter. However
the mock endpoint used for mock testing could no do this and thus a few
tests has to be done without the mocking.
James, or any of the core comitters what is your view of this? Is it somehow
doable to improve the type conversion framework to support meta parameters
such as encoding.
It is quite common in integrations to use byte[] and String objects when
passing data around and thus the encoding is important.
/Claus
Magnus Heino wrote:
>
> A xml-message encoded in utf-8 containing åäö routed through this route is
> trashed. åäö is not åäö after this route. If I remove the marshal call,
> the
> problem goes away.
>
> this.from("direct:request").marshal(soapDataFormat).convertBodyTo(
> String.class).to(requestUri);
>
> To be sure that it wasn't my dataformat that trashed it, I wrote this
> processor:
>
> final Processor wrapSoap = new Processor() {
>
> public void process(Exchange exchange) throws Exception {
> ByteArrayOutputStream outputStream = new
> ByteArrayOutputStream();
> soapDataFormat.marshal(exchange, new
> StringSource((String) exchange.getIn().getBody()), outputStream);
> exchange.getOut(true).setBody(outputStream.toString
> ("UTF-8"));
>
> }
> };
>
> and applied this route:
>
> this.from("direct:request").process(this.wrapSoap).convertBodyTo(
> String.class).to(requestUri);
>
> And now things are working...
>
> Looking at org.apache.camel.processor.MarshalProcessor.java that is used
> in
> the marshal()-call, is this really ok to do with utf-8 data?
>
> dataFormat.marshal(exchange, body, buffer);
> byte[] data = buffer.toByteArray();
> out.setBody(data);
>
> --
>
> /Magnus Heino
>
>
--
View this message in context: http://www.nabble.com/MarshalProcessor-utf-8-bug--tp16037986s22882p16046369.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MarshalProcessor utf-8 bug?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Can you create a JIRA ticket and attach a small example with one of
your XML file that has this invalid character.
Then we can use that as a base for a fix and unit test.
On Wed, Sep 23, 2009 at 1:02 AM, SoaMattH <ma...@netpacket.com.au> wrote:
>
> I have a problem where in my in coming XML messages I have ° symbols these
> are marking
> degrees on lat and long values. I am using version camel 2.0.0
>
> I have specified my data format as follows
>
> <camel:dataFormats>
> <camel:jaxb encoding="uft-8"
> id="incidentJaxb"
> prettyPrint="true"
> contextPath="au.bla.bla.bla.bla.bla.bla.incident" />
> </camel:dataFormats>
> and call it in my route:
> <camel:unmarshal ref="incidentJaxb" />
>
> Thousands of messages process with out failure except the ones with the
> invalid ° character they end up in to my dead letter queue
>
> I suppose my question is, is encoding still a problem in camel as per this
> thread
> from earlier in the year?
>
> Thanks Matt
>
>
>
>
>
> Magnus Heino wrote:
>>
>> A xml-message encoded in utf-8 containing åäö routed through this route is
>> trashed. åäö is not åäö after this route. If I remove the marshal call,
>> the
>> problem goes away.
>>
>> this.from("direct:request").marshal(soapDataFormat).convertBodyTo(
>> String.class).to(requestUri);
>>
>> To be sure that it wasn't my dataformat that trashed it, I wrote this
>> processor:
>>
>> final Processor wrapSoap = new Processor() {
>>
>> public void process(Exchange exchange) throws Exception {
>> ByteArrayOutputStream outputStream = new
>> ByteArrayOutputStream();
>> soapDataFormat.marshal(exchange, new
>> StringSource((String) exchange.getIn().getBody()), outputStream);
>> exchange.getOut(true).setBody(outputStream.toString
>> ("UTF-8"));
>>
>> }
>> };
>>
>> and applied this route:
>>
>> this.from("direct:request").process(this.wrapSoap).convertBodyTo(
>> String.class).to(requestUri);
>>
>> And now things are working...
>>
>> Looking at org.apache.camel.processor.MarshalProcessor.java that is used
>> in
>> the marshal()-call, is this really ok to do with utf-8 data?
>>
>> dataFormat.marshal(exchange, body, buffer);
>> byte[] data = buffer.toByteArray();
>> out.setBody(data);
>>
>> --
>>
>> /Magnus Heino
>>
>>
>
> --
> View this message in context: http://www.nabble.com/MarshalProcessor-utf-8-bug--tp16037986p25530930.html
> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: MarshalProcessor utf-8 bug?
Posted by SoaMattH <ma...@netpacket.com.au>.
I have a problem where in my in coming XML messages I have ° symbols these
are marking
degrees on lat and long values. I am using version camel 2.0.0
I have specified my data format as follows
<camel:dataFormats>
<camel:jaxb encoding="uft-8"
id="incidentJaxb"
prettyPrint="true"
contextPath="au.bla.bla.bla.bla.bla.bla.incident" />
</camel:dataFormats>
and call it in my route:
<camel:unmarshal ref="incidentJaxb" />
Thousands of messages process with out failure except the ones with the
invalid ° character they end up in to my dead letter queue
I suppose my question is, is encoding still a problem in camel as per this
thread
from earlier in the year?
Thanks Matt
Magnus Heino wrote:
>
> A xml-message encoded in utf-8 containing åäö routed through this route is
> trashed. åäö is not åäö after this route. If I remove the marshal call,
> the
> problem goes away.
>
> this.from("direct:request").marshal(soapDataFormat).convertBodyTo(
> String.class).to(requestUri);
>
> To be sure that it wasn't my dataformat that trashed it, I wrote this
> processor:
>
> final Processor wrapSoap = new Processor() {
>
> public void process(Exchange exchange) throws Exception {
> ByteArrayOutputStream outputStream = new
> ByteArrayOutputStream();
> soapDataFormat.marshal(exchange, new
> StringSource((String) exchange.getIn().getBody()), outputStream);
> exchange.getOut(true).setBody(outputStream.toString
> ("UTF-8"));
>
> }
> };
>
> and applied this route:
>
> this.from("direct:request").process(this.wrapSoap).convertBodyTo(
> String.class).to(requestUri);
>
> And now things are working...
>
> Looking at org.apache.camel.processor.MarshalProcessor.java that is used
> in
> the marshal()-call, is this really ok to do with utf-8 data?
>
> dataFormat.marshal(exchange, body, buffer);
> byte[] data = buffer.toByteArray();
> out.setBody(data);
>
> --
>
> /Magnus Heino
>
>
--
View this message in context: http://www.nabble.com/MarshalProcessor-utf-8-bug--tp16037986p25530930.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.