You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Mikael Fernandus S <mi...@gmail.com> on 2012/11/22 08:04:18 UTC
[camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Hi,
While testing Camel Mina 2 component, we noticed that big message
payload received via UDP endpoint was always truncated to 2048 bytes
at Mina2Consumer. We did some investigation and found out that this
problem was caused by the default UDP codec. Comparing default UDP
codec in Mina 2 with its counterpart in Mina 2, the different behavior
might be caused by the replacement of ByteBuffer with IoBuffer and
setting auto expand value to true (even though this may seem counter
intuitive).
Camel - Mina (MinaUdpProtocolCodecFactory):
...
private ByteBuffer toByteBuffer(Object message, CharsetEncoder encoder)
throws CharacterCodingException, NoTypeConversionAvailableException {
String value =
context.getTypeConverter().convertTo(String.class, message);
if (value != null) {
ByteBuffer answer =
ByteBuffer.allocate(value.length()).setAutoExpand(false);
answer.putString(value, encoder);
return answer;
}
// failback to use a byte buffer converter
return
context.getTypeConverter().mandatoryConvertTo(ByteBuffer.class,
message);
}
Camel - Mina 2 (Mina2UdpProtocolCodecFactory):
...
private IoBuffer toIoBuffer(Object message, CharsetEncoder encoder)
throws CharacterCodingException, NoTypeConversionAvailableException {
String value =
context.getTypeConverter().convertTo(String.class, message);
if (value != null) {
IoBuffer answer =
IoBuffer.allocate(value.length()).setAutoExpand(true);
answer.putString(value, encoder);
return answer;
}
// failback to use a byte buffer converter
return
context.getTypeConverter().mandatoryConvertTo(IoBuffer.class,
message);
}
We would like to know if this issue is also reproducible by others.
Camel version we use for this testing is 2.10.2.
Regards,
Mike
Re: [camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Posted by Mikael Fernandus Simalango <mi...@gmail.com>.
Here I provide a sample test case to reproduce the surprising behavior.
public class Mina2UdpTest extends CamelTestSupport {
private static volatile int port;
private static volatile String longMessage = "A very long UTF-8
string with length more than 2048 bytes";
@BeforeClass
public static void chooseFreePort() throws Exception {
port = AvailablePortFinder.getNextAvailable();
}
protected int getPort() {
return port;
}
protected RouteBuilder createRouteBuilder() {
return new RouteBuilder() {
@Override
public void configure() throws Exception {
from(String.format("mina2:udp://localhost:%1$s?sync=false&textline=false&allowDefaultCodec=false",getPort()))
.to("mock:result");
}
};
}
@Test
public void testDefaultMina2UdpBehavior() throws Exception{
template().sendBody(String.format("mina2:udp://localhost:%1$s?sync=false&textline=false&allowDefaultCodec=false",
getPort()), longMessage);
MockEndpoint mockEndpoint = getMockEndpoint("mock:result");
mockEndpoint.setExpectedMessageCount(1);
mockEndpoint.assertIsSatisfied();
//Equality test
Exchange e = mockEndpoint.getReceivedExchanges().get(0);
Object o = e.getIn().getBody();
String result = context().getTypeConverter().convertTo(String.class,o);
assertEquals(longMessage,result);
}
}
Debug message snippet from the log:
...
2012-11-23 10:18:54,874 [main ] DEBUG Mina2Producer
- Writing body: A very long UTF-8 string with length more than
2048 bytes
...
2012-11-23 10:18:54,906 [agramAcceptor-1] DEBUG Mina2Consumer
- Received body: A very long UTF-8 string with.. (truncated to
2048 bytes)
As another note, even though allowDefaultCodec property is set to
false, it is noticeable that the default codec
Mina2UdpProtocolCodecFactory is still called.
I'll be glad hearing comments and feedback from others regarding this finding.
Regards,
Mike
Re: [camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Posted by Mikael Fernandus Simalango <mi...@gmail.com>.
On Mon, Dec 3, 2012 at 6:58 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Mon, Dec 3, 2012 at 10:44 AM, Christian Ohr <ch...@gmail.com> wrote:
>> The HL7 Codec works on TCP messages, not UDP. On the other hand I
>> don't think that the codec really depends on that.
>> Note that HL7 messages have specific begin and end bytes that make it
>> rather easy to cumulate the parts.
>>
>
> Ah yeah good point about the markers.
>
We decided to fail back to Camel Mina as the base component. The udp
endpoint of our component behaves as expected when running on top of
Camel Mina. We, however, keep the issue open and will revisit later
when we can afford more time to investigate session management and
packet reassembling in Camel Mina 2. I'm expecting that I/O
performance will be better in Mina 2 and Camel Mina 2. So, switching
to Camel Mina 2 is still in the trajectory.
Thanks to Claus and Christian for the inputs and comments.
Regards,
Mike
Re: [camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Dec 3, 2012 at 10:44 AM, Christian Ohr <ch...@gmail.com> wrote:
> The HL7 Codec works on TCP messages, not UDP. On the other hand I
> don't think that the codec really depends on that.
> Note that HL7 messages have specific begin and end bytes that make it
> rather easy to cumulate the parts.
>
Ah yeah good point about the markers.
> Christian
>
> 2012/12/3 Claus Ibsen <cl...@gmail.com>:
>> Hi
>>
>> The camel-hl7 component has a mina codec that works with hl7 message formats.
>> And I think there is some logic in there to "assemble" udp messages
>> that is larger than 2048.
>>
>> I remembered some patches we got years ago about something related to this.
>> Maybe take a peek in the camel-hl7 source code.
>>
>>
>>
>> On Tue, Nov 27, 2012 at 6:50 AM, Mikael Fernandus Simalango
>> <mi...@gmail.com> wrote:
>>> On Sat, Nov 24, 2012 at 5:29 PM, Claus Ibsen <cl...@gmail.com> wrote:
>>>>
>>>> So have you tried with setting auto expand = false in camel-mina2, and
>>>> see if that fixes the issue for you?
>>>>
>>>>
>>>
>>> Hi Claus,
>>>
>>> Thanks for your comment. I have conducted more tests to trace the root
>>> cause. Unfortunately, setting auto expand to false did not fix the
>>> problem because such flag is needed when dealing with unicode
>>> characters. I once tried to set it to false to only observe
>>> BufferOverflowException.
>>>
>>> When debugging, I found out that Mina2Consumer received the message
>>> n-times depending on the fragmentation (doDecode is called n-times) if
>>> a class extending CumulativeProtocolDecoder is used as the decoder in
>>> a custom codec. Yet, the content in each call is exactly the same,
>>> which is the first 2048 bytes of the message sent. As a remark,
>>> session.getTransportMetadata().hasFragmentation() is false, which was
>>> different with my expectation. Additionally, the call is only once
>>> when the received message is decoded using the default codec, i.e.
>>> Mina2UdpProtocolCodecFactory. In that call, only the first 2048 bytes
>>> are received.
>>>
>>> I come into a proposition that the problem can be caused by different
>>> session management mechanism in Camel Mina 2 compared with that of
>>> Camel Mina. Yet, I haven't been able to refine and isolate the exact
>>> part of the code causing such behavior. I'm not that familiar with the
>>> codebase and I'm not quite sure if it's a feature in the new
>>> component. I'd be glad for further clarification from others with more
>>> competency in this matter.
>>>
>>> Regards,
>>> Mike
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> FuseSource is now part of Red Hat
>> Email: cibsen@redhat.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: [camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Posted by Christian Ohr <ch...@gmail.com>.
The HL7 Codec works on TCP messages, not UDP. On the other hand I
don't think that the codec really depends on that.
Note that HL7 messages have specific begin and end bytes that make it
rather easy to cumulate the parts.
Christian
2012/12/3 Claus Ibsen <cl...@gmail.com>:
> Hi
>
> The camel-hl7 component has a mina codec that works with hl7 message formats.
> And I think there is some logic in there to "assemble" udp messages
> that is larger than 2048.
>
> I remembered some patches we got years ago about something related to this.
> Maybe take a peek in the camel-hl7 source code.
>
>
>
> On Tue, Nov 27, 2012 at 6:50 AM, Mikael Fernandus Simalango
> <mi...@gmail.com> wrote:
>> On Sat, Nov 24, 2012 at 5:29 PM, Claus Ibsen <cl...@gmail.com> wrote:
>>>
>>> So have you tried with setting auto expand = false in camel-mina2, and
>>> see if that fixes the issue for you?
>>>
>>>
>>
>> Hi Claus,
>>
>> Thanks for your comment. I have conducted more tests to trace the root
>> cause. Unfortunately, setting auto expand to false did not fix the
>> problem because such flag is needed when dealing with unicode
>> characters. I once tried to set it to false to only observe
>> BufferOverflowException.
>>
>> When debugging, I found out that Mina2Consumer received the message
>> n-times depending on the fragmentation (doDecode is called n-times) if
>> a class extending CumulativeProtocolDecoder is used as the decoder in
>> a custom codec. Yet, the content in each call is exactly the same,
>> which is the first 2048 bytes of the message sent. As a remark,
>> session.getTransportMetadata().hasFragmentation() is false, which was
>> different with my expectation. Additionally, the call is only once
>> when the received message is decoded using the default codec, i.e.
>> Mina2UdpProtocolCodecFactory. In that call, only the first 2048 bytes
>> are received.
>>
>> I come into a proposition that the problem can be caused by different
>> session management mechanism in Camel Mina 2 compared with that of
>> Camel Mina. Yet, I haven't been able to refine and isolate the exact
>> part of the code causing such behavior. I'm not that familiar with the
>> codebase and I'm not quite sure if it's a feature in the new
>> component. I'd be glad for further clarification from others with more
>> competency in this matter.
>>
>> Regards,
>> Mike
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
Re: [camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
The camel-hl7 component has a mina codec that works with hl7 message formats.
And I think there is some logic in there to "assemble" udp messages
that is larger than 2048.
I remembered some patches we got years ago about something related to this.
Maybe take a peek in the camel-hl7 source code.
On Tue, Nov 27, 2012 at 6:50 AM, Mikael Fernandus Simalango
<mi...@gmail.com> wrote:
> On Sat, Nov 24, 2012 at 5:29 PM, Claus Ibsen <cl...@gmail.com> wrote:
>>
>> So have you tried with setting auto expand = false in camel-mina2, and
>> see if that fixes the issue for you?
>>
>>
>
> Hi Claus,
>
> Thanks for your comment. I have conducted more tests to trace the root
> cause. Unfortunately, setting auto expand to false did not fix the
> problem because such flag is needed when dealing with unicode
> characters. I once tried to set it to false to only observe
> BufferOverflowException.
>
> When debugging, I found out that Mina2Consumer received the message
> n-times depending on the fragmentation (doDecode is called n-times) if
> a class extending CumulativeProtocolDecoder is used as the decoder in
> a custom codec. Yet, the content in each call is exactly the same,
> which is the first 2048 bytes of the message sent. As a remark,
> session.getTransportMetadata().hasFragmentation() is false, which was
> different with my expectation. Additionally, the call is only once
> when the received message is decoded using the default codec, i.e.
> Mina2UdpProtocolCodecFactory. In that call, only the first 2048 bytes
> are received.
>
> I come into a proposition that the problem can be caused by different
> session management mechanism in Camel Mina 2 compared with that of
> Camel Mina. Yet, I haven't been able to refine and isolate the exact
> part of the code causing such behavior. I'm not that familiar with the
> codebase and I'm not quite sure if it's a feature in the new
> component. I'd be glad for further clarification from others with more
> competency in this matter.
>
> Regards,
> Mike
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: [camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Posted by Mikael Fernandus Simalango <mi...@gmail.com>.
On Sat, Nov 24, 2012 at 5:29 PM, Claus Ibsen <cl...@gmail.com> wrote:
>
> So have you tried with setting auto expand = false in camel-mina2, and
> see if that fixes the issue for you?
>
>
Hi Claus,
Thanks for your comment. I have conducted more tests to trace the root
cause. Unfortunately, setting auto expand to false did not fix the
problem because such flag is needed when dealing with unicode
characters. I once tried to set it to false to only observe
BufferOverflowException.
When debugging, I found out that Mina2Consumer received the message
n-times depending on the fragmentation (doDecode is called n-times) if
a class extending CumulativeProtocolDecoder is used as the decoder in
a custom codec. Yet, the content in each call is exactly the same,
which is the first 2048 bytes of the message sent. As a remark,
session.getTransportMetadata().hasFragmentation() is false, which was
different with my expectation. Additionally, the call is only once
when the received message is decoded using the default codec, i.e.
Mina2UdpProtocolCodecFactory. In that call, only the first 2048 bytes
are received.
I come into a proposition that the problem can be caused by different
session management mechanism in Camel Mina 2 compared with that of
Camel Mina. Yet, I haven't been able to refine and isolate the exact
part of the code causing such behavior. I'm not that familiar with the
codebase and I'm not quite sure if it's a feature in the new
component. I'd be glad for further clarification from others with more
competency in this matter.
Regards,
Mike
Re: [camel-mina2] Bug in default UDP codec (Mina2UdpProtocolCodecFactory)?
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Nov 22, 2012 at 8:04 AM, Mikael Fernandus S <mi...@gmail.com> wrote:
> Hi,
>
> While testing Camel Mina 2 component, we noticed that big message
> payload received via UDP endpoint was always truncated to 2048 bytes
> at Mina2Consumer. We did some investigation and found out that this
> problem was caused by the default UDP codec. Comparing default UDP
> codec in Mina 2 with its counterpart in Mina 2, the different behavior
> might be caused by the replacement of ByteBuffer with IoBuffer and
> setting auto expand value to true (even though this may seem counter
> intuitive).
>
> Camel - Mina (MinaUdpProtocolCodecFactory):
> ...
> private ByteBuffer toByteBuffer(Object message, CharsetEncoder encoder)
> throws CharacterCodingException, NoTypeConversionAvailableException {
> String value =
> context.getTypeConverter().convertTo(String.class, message);
> if (value != null) {
> ByteBuffer answer =
> ByteBuffer.allocate(value.length()).setAutoExpand(false);
> answer.putString(value, encoder);
> return answer;
> }
>
> // failback to use a byte buffer converter
> return
> context.getTypeConverter().mandatoryConvertTo(ByteBuffer.class,
> message);
> }
>
> Camel - Mina 2 (Mina2UdpProtocolCodecFactory):
> ...
> private IoBuffer toIoBuffer(Object message, CharsetEncoder encoder)
> throws CharacterCodingException, NoTypeConversionAvailableException {
> String value =
> context.getTypeConverter().convertTo(String.class, message);
> if (value != null) {
> IoBuffer answer =
> IoBuffer.allocate(value.length()).setAutoExpand(true);
> answer.putString(value, encoder);
> return answer;
> }
>
> // failback to use a byte buffer converter
> return
> context.getTypeConverter().mandatoryConvertTo(IoBuffer.class,
> message);
> }
>
>
> We would like to know if this issue is also reproducible by others.
> Camel version we use for this testing is 2.10.2.
>
So have you tried with setting auto expand = false in camel-mina2, and
see if that fixes the issue for you?
> Regards,
> Mike
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen