You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Lahiru Gunathilake <gl...@gmail.com> on 2010/11/26 13:14:55 UTC
How to handle large messages in Qpid Java client
Hi Devs,
When I consume a big message with 1MB file size and pick it from Qpid Queue,
I am getting an error like this
avax.jms.JMSException: Could not decode string data:
java.nio.charset.MalformedInputException: Input length = 1<br> at
org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
After debugging the code, I feel like this happen, due to a malformed input
stream coming for the client.
Can I increase the frame-size of the client and get this message in to a
single frame ? Is there any other solution to this error.
Regards
Lahiru
Re: How to handle large messages in Qpid Java client
Posted by Marnie McCormack <ma...@googlemail.com>.
Hi Lahiru,
I've had a quick look at trunk - the exception actually ripples up from the
MINA ByteBuffer.getString() call.
Are you sure that your message content is UTF-8 ? Most of the reports of
this issue with MINA appear to be caused by binary/non-UTF-8 content in text
messages.
Regards,
Marnie
On Fri, Nov 26, 2010 at 1:08 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Marnie,
>
> Thanks Marnie for your response.
>
> No, i do not think this is a timing issue, even when I debug I am getting
> this error. It's a good news that this is a known issue.
>
> Are you capable of reading messages larger than 1 MB ? If you are using any
> configuration can you please post here ? That will be very useful !
>
> Lahiru
>
> On Fri, Nov 26, 2010 at 6:23 PM, Marnie McCormack <
> marnie.mccormack@googlemail.com> wrote:
>
> > Hi Lahiru,
> >
> > There's a JIRA about this exception/issue - QPID-2902.
> >
> > Does it fail for you reliably ? I wonder if there's a timing issue here
> > since it didn't occur all the time on the CI instance.
> >
> > Marnie
> >
> > On Fri, Nov 26, 2010 at 12:14 PM, Lahiru Gunathilake <glahiru@gmail.com
> > >wrote:
> >
> > > Hi Devs,
> > >
> > > When I consume a big message with 1MB file size and pick it from Qpid
> > > Queue,
> > > I am getting an error like this
> > >
> > > avax.jms.JMSException: Could not decode string data:
> > > java.nio.charset.MalformedInputException: Input length = 1<br> at
> > >
> > >
> >
> org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
> > >
> > > After debugging the code, I feel like this happen, due to a malformed
> > input
> > > stream coming for the client.
> > >
> > > Can I increase the frame-size of the client and get this message in to
> a
> > > single frame ? Is there any other solution to this error.
> > >
> > > Regards
> > > Lahiru
> > >
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Marnie,
Thanks Marnie for your response.
No, i do not think this is a timing issue, even when I debug I am getting
this error. It's a good news that this is a known issue.
Are you capable of reading messages larger than 1 MB ? If you are using any
configuration can you please post here ? That will be very useful !
Lahiru
On Fri, Nov 26, 2010 at 6:23 PM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> Hi Lahiru,
>
> There's a JIRA about this exception/issue - QPID-2902.
>
> Does it fail for you reliably ? I wonder if there's a timing issue here
> since it didn't occur all the time on the CI instance.
>
> Marnie
>
> On Fri, Nov 26, 2010 at 12:14 PM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi Devs,
> >
> > When I consume a big message with 1MB file size and pick it from Qpid
> > Queue,
> > I am getting an error like this
> >
> > avax.jms.JMSException: Could not decode string data:
> > java.nio.charset.MalformedInputException: Input length = 1<br> at
> >
> >
> org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
> >
> > After debugging the code, I feel like this happen, due to a malformed
> input
> > stream coming for the client.
> >
> > Can I increase the frame-size of the client and get this message in to a
> > single frame ? Is there any other solution to this error.
> >
> > Regards
> > Lahiru
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Andrew Kennedy <an...@gmail.com>.
Note: I was going to send this as a reply to Lahiru on qpid-users,
but better judgement prevailed, as I think it would probably not have
the desired effect. Can anyone knowledgeable in the Java broker
confirm this is the way things really work?
Ok,
That 'IoTransport' class is *never* used, unless the system property
'UseTransportIo' has a value other than 'false' and that code path is
not tested (See AMQConnectionDelegate_8_0 lines 91-99). I would *not*
recommend modifying that property, or even trying this, as it is not
included in our test suite and is experimental code. Also, it will be
disappearing in a future release anyway .
In fact, the code will use a 'SocketTransportConnection' object,
which sets up TCP socket receive and send buffer sizes based on
either the system properties 'amqj.sendBufferSize' and
'amqj.receiveBufferSize' or the default, which is 32 kB for both. If
you want to change these, the system properties are the things to use
(on the client side, of course) instead.
Confusingly, on the broker side of things, this is handled by the
'MinaNetworkDriver' class, which uses a default of 262144 bytes for
the send and receive transport buffers. The send buffer size is
changed by setting the '//connector/protectio/writeBufferLimitSize'
element in config.xml, and the receive buffer likewise by changing
the '//connector/protectio/readBufferLimitSize' element.
I am quite surprised that changing constants in IoTransport had any
effect whatsoever, to be honest. Also, note that this has nothing to
do with *Frame* size, which is an AMQP thing. In fact, the broker
itself informs each connection of its maximum framse size. This
changes, depending on whether any SASL authentication mechanisms are
defined, obviously. If there are none, it is hard coded as 65535
bytes, as can be seen in this snippet from ServerDelegate, around
line 70 or so (Note that 'ConnectionBinding' serves only to provide
this constant, no other code is used from this class):
conn.connectionTune(Integer.MAX_VALUE,
org.apache.qpid.transport.network.ConnectionBinding.MAX_FRAME_SIZE,
0, Integer.MAX_VALUE);
If, on the other hand, the connection is secured using SASL, the
*default* maximum frame size is 65536 bytes (note the one byte
difference...) however this can, again, be changed using a config.xml
element, namely the '//advanced/framesize' element.
I am not sure if the above has made things any clearer, sadly, but
that is what the code appears to do... If you feel that there is a
bug somewhere here, please raise a JIRA so that we can track it
properly and fix the behavior.
Andrew.
--
-- andrew d kennedy ? do not fold, bend, spindle, or mutilate ;
-- http://grkvlt.blogspot.com/ ? edinburgh : +44 7582 293 255 ;
On 29 Nov 2010, at 02:54, Lahiru Gunathilake wrote:
> Hi Marnie,
>
> In IoSender and IoReceiver classes we are using a ByteBuffer to
> read the
> write messages, to increase the size of this buffer, I simply
> changed a
> constant defined in IoTransport.java file, if you look in to those
> three
> classes, you will get it.
>
> In IoTransport.java I changed the following line of code,
>
> private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
>
> but I saw there are some other constants defined with the name of
> MAX_FRAME_SIZE but after going through the code i saw they are not
> affecting.
>
> I feel there's an issue when we send a large message in to a Queue and
> something get currupted, when do framing and send. because when I
> keep the
> same small frame size for read buffer and increase the size for write
> buffer, it worked for 500K messages, but couldn't get it work for
> more than
> 1MB large messages.
>
> Lahiru
>
> Lahiru
>
> On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
> marnie.mccormack@googlemail.com> wrote:
>
>> Hi Lahiru,
>>
>> What exactly have you done/set for the frame size change ?
>>
>> Thanks,
>> Marnie
>>
>> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake
>> <glahiru@gmail.com
>>> wrote:
>>
>>> Hi all,
>>>
>>> I did a mistake when increase the framesize without increasing to
>> required
>>> size, when I increase the size I was able to read some of the
>>> messages
>>> which
>>> are in the size of 700K, but sometimes it fails.
>>>
>>>
>>> I kept on increasing the frame-size since in my setup there are
>>> messages
>>> with the size of 1.5M but I couldn't read a any of them though i
>>> increase
>>> the frame-size more.
>>>
>>> other than changing in IoTransport.java, do I have to change any
>>> other
>>> constant values to sync the maximum-frame-size ?
>>>
>>> Regards
>>> Lahiru
>>>
>>> On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake
>>> <glahiru@gmail.com
>>>> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> Thanks Andrew for the response.
>>>> I have doubled the default frame-size which was (64 * 1024) in
>>>> IoTransport.java, but didn't worked for me. FYI, activeMQ works
>>>> fine in
>>> this
>>>> scenario.
>>>>
>>>> On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
>>>> andrewinternational@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The out of order frames only occur in 0-10 and with transports
>>>>> that
>> have
>>>>> no
>>>>> concept of sequencing, i.e. in-vm or udp, not tcp. Having said
>>>>> that, a
>>>>> frame
>>>>> body being interpreted as a header would give those symptoms.
>>>>
>>>> I will debug and see.
>>>>
>>>> Regards
>>>> Lahiru
>>>>
>>>>> Test code that
>>>>> exercises the issue would be illuminating.
>>>>>
>>>>> Unsure if this helps?
>>>>>
>>>>> Andrew.
>>>>>
>>>>> On 26 Nov 2010 16:56, "Marnie McCormack" <
>>> marnie.mccormack@googlemail.com
>>>>>>
>>>>> wrote:
>>>>>
>>>>> Hi Lahiru,
>>>>>
>>>>> Robbie mentioned earlier when I asked him about this that he
>>>>> *thinks*
>>> it's
>>>>> a
>>>>> bug Andrew might have come across and possibly fixed to do with
>>>>> OoO
>>>>> frames.
>>>>> Certainly the exception showing that the buffer length is 1 might
>>> support
>>>>> that.
>>>>>
>>>>> I haven't had a chance to look about for this - Andrew is not
>>>>> about
>>> today.
>>>>>
>>>>> I'll check with him when he's back if you don't get a better reply
>> here
>>>>> first.
>>>>>
>>>>> Rgds,
>>>>> Marnie
>>>>>
>>>>> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
>> glahiru@gmail.com
>>>>>> wrote:
>>>>>
>>>>>
>>>>>> Hi Marnie,
>>>>>>
>>>>>> Thanks for your prompt response. yes i tried to wait before
>> consuming,
>>>>> and
>>>>>> it wo...
>>>>>
>>>>
>>>>
>>>
>>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Forwarding to dev list !
On Thu, Dec 2, 2010 at 3:28 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> FYI, I try to monitor all the reads from the socket InputStream from the
> client and I can see that Client is reading a curupted message, so
> I guarantee that there's no issue in Client side IoSendere and IoReceiver
> classes.
>
> I think the issue is with the broker IoReceiver or Sending, we found that
> this is directly an issue with Java broker because things are working
> perfect with Qpid C++ broker 0.8 RC2.
>
> Thanks
> Lahiru
>
>
> On Mon, Nov 29, 2010 at 3:21 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>> FYI, during receiving a message, I can see following
>> malformed characters in my JMSTextMessage object,
>>
>> <OUTMOD>4</OUTMOD>^D^C��^@^A^@^@^@^@^@^@
>>
>> This cause the exception i have sent at the very first email, after
>> increasing the frame-size sometimes it works sometimes it doesn't.
>>
>> Lahiru
>>
>>
>> On Mon, Nov 29, 2010 at 8:24 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>>
>>> Hi Marnie,
>>>
>>> In IoSender and IoReceiver classes we are using a ByteBuffer to read the
>>> write messages, to increase the size of this buffer, I simply changed a
>>> constant defined in IoTransport.java file, if you look in to those three
>>> classes, you will get it.
>>>
>>> In IoTransport.java I changed the following line of code,
>>>
>>> private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
>>>
>>> but I saw there are some other constants defined with the name of
>>> MAX_FRAME_SIZE but after going through the code i saw they are not
>>> affecting.
>>>
>>> I feel there's an issue when we send a large message in to a Queue and
>>> something get currupted, when do framing and send. because when I keep the
>>> same small frame size for read buffer and increase the size for write
>>> buffer, it worked for 500K messages, but couldn't get it work for more than
>>> 1MB large messages.
>>>
>>> Lahiru
>>>
>>> Lahiru
>>>
>>>
>>> On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
>>> marnie.mccormack@googlemail.com> wrote:
>>>
>>>> Hi Lahiru,
>>>>
>>>> What exactly have you done/set for the frame size change ?
>>>>
>>>> Thanks,
>>>> Marnie
>>>>
>>>> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <glahiru@gmail.com
>>>> >wrote:
>>>>
>>>> > Hi all,
>>>> >
>>>> > I did a mistake when increase the framesize without increasing to
>>>> required
>>>> > size, when I increase the size I was able to read some of the messages
>>>> > which
>>>> > are in the size of 700K, but sometimes it fails.
>>>> >
>>>> >
>>>> > I kept on increasing the frame-size since in my setup there are
>>>> messages
>>>> > with the size of 1.5M but I couldn't read a any of them though i
>>>> increase
>>>> > the frame-size more.
>>>> >
>>>> > other than changing in IoTransport.java, do I have to change any other
>>>> > constant values to sync the maximum-frame-size ?
>>>> >
>>>> > Regards
>>>> > Lahiru
>>>> >
>>>> > On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <
>>>> glahiru@gmail.com
>>>> > >wrote:
>>>> >
>>>> > > Hi Andrew,
>>>> > >
>>>> > > Thanks Andrew for the response.
>>>> > > I have doubled the default frame-size which was (64 * 1024) in
>>>> > > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine
>>>> in
>>>> > this
>>>> > > scenario.
>>>> > >
>>>> > > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
>>>> > > andrewinternational@gmail.com> wrote:
>>>> > >
>>>> > >> Hi,
>>>> > >>
>>>> > >> The out of order frames only occur in 0-10 and with transports that
>>>> have
>>>> > >> no
>>>> > >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said
>>>> that, a
>>>> > >> frame
>>>> > >> body being interpreted as a header would give those symptoms.
>>>> > >
>>>> > > I will debug and see.
>>>> > >
>>>> > > Regards
>>>> > > Lahiru
>>>> > >
>>>> > >> Test code that
>>>> > >> exercises the issue would be illuminating.
>>>> > >>
>>>> > >> Unsure if this helps?
>>>> > >>
>>>> > >> Andrew.
>>>> > >>
>>>> > >> On 26 Nov 2010 16:56, "Marnie McCormack" <
>>>> > marnie.mccormack@googlemail.com
>>>> > >> >
>>>> > >> wrote:
>>>> > >>
>>>> > >> Hi Lahiru,
>>>> > >>
>>>> > >> Robbie mentioned earlier when I asked him about this that he
>>>> *thinks*
>>>> > it's
>>>> > >> a
>>>> > >> bug Andrew might have come across and possibly fixed to do with OoO
>>>> > >> frames.
>>>> > >> Certainly the exception showing that the buffer length is 1 might
>>>> > support
>>>> > >> that.
>>>> > >>
>>>> > >> I haven't had a chance to look about for this - Andrew is not about
>>>> > today.
>>>> > >>
>>>> > >> I'll check with him when he's back if you don't get a better reply
>>>> here
>>>> > >> first.
>>>> > >>
>>>> > >> Rgds,
>>>> > >> Marnie
>>>> > >>
>>>> > >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
>>>> glahiru@gmail.com
>>>> > >> >wrote:
>>>> > >>
>>>> > >>
>>>> > >> > Hi Marnie,
>>>> > >> >
>>>> > >> > Thanks for your prompt response. yes i tried to wait before
>>>> consuming,
>>>> > >> and
>>>> > >> > it wo...
>>>> > >>
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Forwarding to dev list !
On Thu, Dec 2, 2010 at 3:28 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> FYI, I try to monitor all the reads from the socket InputStream from the
> client and I can see that Client is reading a curupted message, so
> I guarantee that there's no issue in Client side IoSendere and IoReceiver
> classes.
>
> I think the issue is with the broker IoReceiver or Sending, we found that
> this is directly an issue with Java broker because things are working
> perfect with Qpid C++ broker 0.8 RC2.
>
> Thanks
> Lahiru
>
>
> On Mon, Nov 29, 2010 at 3:21 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>> FYI, during receiving a message, I can see following
>> malformed characters in my JMSTextMessage object,
>>
>> <OUTMOD>4</OUTMOD>^D^C��^@^A^@^@^@^@^@^@
>>
>> This cause the exception i have sent at the very first email, after
>> increasing the frame-size sometimes it works sometimes it doesn't.
>>
>> Lahiru
>>
>>
>> On Mon, Nov 29, 2010 at 8:24 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>>
>>> Hi Marnie,
>>>
>>> In IoSender and IoReceiver classes we are using a ByteBuffer to read the
>>> write messages, to increase the size of this buffer, I simply changed a
>>> constant defined in IoTransport.java file, if you look in to those three
>>> classes, you will get it.
>>>
>>> In IoTransport.java I changed the following line of code,
>>>
>>> private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
>>>
>>> but I saw there are some other constants defined with the name of
>>> MAX_FRAME_SIZE but after going through the code i saw they are not
>>> affecting.
>>>
>>> I feel there's an issue when we send a large message in to a Queue and
>>> something get currupted, when do framing and send. because when I keep the
>>> same small frame size for read buffer and increase the size for write
>>> buffer, it worked for 500K messages, but couldn't get it work for more than
>>> 1MB large messages.
>>>
>>> Lahiru
>>>
>>> Lahiru
>>>
>>>
>>> On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
>>> marnie.mccormack@googlemail.com> wrote:
>>>
>>>> Hi Lahiru,
>>>>
>>>> What exactly have you done/set for the frame size change ?
>>>>
>>>> Thanks,
>>>> Marnie
>>>>
>>>> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <glahiru@gmail.com
>>>> >wrote:
>>>>
>>>> > Hi all,
>>>> >
>>>> > I did a mistake when increase the framesize without increasing to
>>>> required
>>>> > size, when I increase the size I was able to read some of the messages
>>>> > which
>>>> > are in the size of 700K, but sometimes it fails.
>>>> >
>>>> >
>>>> > I kept on increasing the frame-size since in my setup there are
>>>> messages
>>>> > with the size of 1.5M but I couldn't read a any of them though i
>>>> increase
>>>> > the frame-size more.
>>>> >
>>>> > other than changing in IoTransport.java, do I have to change any other
>>>> > constant values to sync the maximum-frame-size ?
>>>> >
>>>> > Regards
>>>> > Lahiru
>>>> >
>>>> > On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <
>>>> glahiru@gmail.com
>>>> > >wrote:
>>>> >
>>>> > > Hi Andrew,
>>>> > >
>>>> > > Thanks Andrew for the response.
>>>> > > I have doubled the default frame-size which was (64 * 1024) in
>>>> > > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine
>>>> in
>>>> > this
>>>> > > scenario.
>>>> > >
>>>> > > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
>>>> > > andrewinternational@gmail.com> wrote:
>>>> > >
>>>> > >> Hi,
>>>> > >>
>>>> > >> The out of order frames only occur in 0-10 and with transports that
>>>> have
>>>> > >> no
>>>> > >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said
>>>> that, a
>>>> > >> frame
>>>> > >> body being interpreted as a header would give those symptoms.
>>>> > >
>>>> > > I will debug and see.
>>>> > >
>>>> > > Regards
>>>> > > Lahiru
>>>> > >
>>>> > >> Test code that
>>>> > >> exercises the issue would be illuminating.
>>>> > >>
>>>> > >> Unsure if this helps?
>>>> > >>
>>>> > >> Andrew.
>>>> > >>
>>>> > >> On 26 Nov 2010 16:56, "Marnie McCormack" <
>>>> > marnie.mccormack@googlemail.com
>>>> > >> >
>>>> > >> wrote:
>>>> > >>
>>>> > >> Hi Lahiru,
>>>> > >>
>>>> > >> Robbie mentioned earlier when I asked him about this that he
>>>> *thinks*
>>>> > it's
>>>> > >> a
>>>> > >> bug Andrew might have come across and possibly fixed to do with OoO
>>>> > >> frames.
>>>> > >> Certainly the exception showing that the buffer length is 1 might
>>>> > support
>>>> > >> that.
>>>> > >>
>>>> > >> I haven't had a chance to look about for this - Andrew is not about
>>>> > today.
>>>> > >>
>>>> > >> I'll check with him when he's back if you don't get a better reply
>>>> here
>>>> > >> first.
>>>> > >>
>>>> > >> Rgds,
>>>> > >> Marnie
>>>> > >>
>>>> > >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
>>>> glahiru@gmail.com
>>>> > >> >wrote:
>>>> > >>
>>>> > >>
>>>> > >> > Hi Marnie,
>>>> > >> >
>>>> > >> > Thanks for your prompt response. yes i tried to wait before
>>>> consuming,
>>>> > >> and
>>>> > >> > it wo...
>>>> > >>
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
FYI, I try to monitor all the reads from the socket InputStream from the
client and I can see that Client is reading a curupted message, so
I guarantee that there's no issue in Client side IoSendere and IoReceiver
classes.
I think the issue is with the broker IoReceiver or Sending, we found that
this is directly an issue with Java broker because things are working
perfect with Qpid C++ broker 0.8 RC2.
Thanks
Lahiru
On Mon, Nov 29, 2010 at 3:21 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> FYI, during receiving a message, I can see following
> malformed characters in my JMSTextMessage object,
>
> <OUTMOD>4</OUTMOD>^D^C��^@^A^@^@^@^@^@^@
>
> This cause the exception i have sent at the very first email, after
> increasing the frame-size sometimes it works sometimes it doesn't.
>
> Lahiru
>
>
> On Mon, Nov 29, 2010 at 8:24 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>> Hi Marnie,
>>
>> In IoSender and IoReceiver classes we are using a ByteBuffer to read the
>> write messages, to increase the size of this buffer, I simply changed a
>> constant defined in IoTransport.java file, if you look in to those three
>> classes, you will get it.
>>
>> In IoTransport.java I changed the following line of code,
>>
>> private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
>>
>> but I saw there are some other constants defined with the name of
>> MAX_FRAME_SIZE but after going through the code i saw they are not
>> affecting.
>>
>> I feel there's an issue when we send a large message in to a Queue and
>> something get currupted, when do framing and send. because when I keep the
>> same small frame size for read buffer and increase the size for write
>> buffer, it worked for 500K messages, but couldn't get it work for more than
>> 1MB large messages.
>>
>> Lahiru
>>
>> Lahiru
>>
>>
>> On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
>> marnie.mccormack@googlemail.com> wrote:
>>
>>> Hi Lahiru,
>>>
>>> What exactly have you done/set for the frame size change ?
>>>
>>> Thanks,
>>> Marnie
>>>
>>> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <glahiru@gmail.com
>>> >wrote:
>>>
>>> > Hi all,
>>> >
>>> > I did a mistake when increase the framesize without increasing to
>>> required
>>> > size, when I increase the size I was able to read some of the messages
>>> > which
>>> > are in the size of 700K, but sometimes it fails.
>>> >
>>> >
>>> > I kept on increasing the frame-size since in my setup there are
>>> messages
>>> > with the size of 1.5M but I couldn't read a any of them though i
>>> increase
>>> > the frame-size more.
>>> >
>>> > other than changing in IoTransport.java, do I have to change any other
>>> > constant values to sync the maximum-frame-size ?
>>> >
>>> > Regards
>>> > Lahiru
>>> >
>>> > On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <
>>> glahiru@gmail.com
>>> > >wrote:
>>> >
>>> > > Hi Andrew,
>>> > >
>>> > > Thanks Andrew for the response.
>>> > > I have doubled the default frame-size which was (64 * 1024) in
>>> > > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine
>>> in
>>> > this
>>> > > scenario.
>>> > >
>>> > > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
>>> > > andrewinternational@gmail.com> wrote:
>>> > >
>>> > >> Hi,
>>> > >>
>>> > >> The out of order frames only occur in 0-10 and with transports that
>>> have
>>> > >> no
>>> > >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said that,
>>> a
>>> > >> frame
>>> > >> body being interpreted as a header would give those symptoms.
>>> > >
>>> > > I will debug and see.
>>> > >
>>> > > Regards
>>> > > Lahiru
>>> > >
>>> > >> Test code that
>>> > >> exercises the issue would be illuminating.
>>> > >>
>>> > >> Unsure if this helps?
>>> > >>
>>> > >> Andrew.
>>> > >>
>>> > >> On 26 Nov 2010 16:56, "Marnie McCormack" <
>>> > marnie.mccormack@googlemail.com
>>> > >> >
>>> > >> wrote:
>>> > >>
>>> > >> Hi Lahiru,
>>> > >>
>>> > >> Robbie mentioned earlier when I asked him about this that he
>>> *thinks*
>>> > it's
>>> > >> a
>>> > >> bug Andrew might have come across and possibly fixed to do with OoO
>>> > >> frames.
>>> > >> Certainly the exception showing that the buffer length is 1 might
>>> > support
>>> > >> that.
>>> > >>
>>> > >> I haven't had a chance to look about for this - Andrew is not about
>>> > today.
>>> > >>
>>> > >> I'll check with him when he's back if you don't get a better reply
>>> here
>>> > >> first.
>>> > >>
>>> > >> Rgds,
>>> > >> Marnie
>>> > >>
>>> > >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
>>> glahiru@gmail.com
>>> > >> >wrote:
>>> > >>
>>> > >>
>>> > >> > Hi Marnie,
>>> > >> >
>>> > >> > Thanks for your prompt response. yes i tried to wait before
>>> consuming,
>>> > >> and
>>> > >> > it wo...
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
FYI, during receiving a message, I can see following malformed characters in
my JMSTextMessage object,
<OUTMOD>4</OUTMOD>^D^C��^@^A^@^@^@^@^@^@
This cause the exception i have sent at the very first email, after
increasing the frame-size sometimes it works sometimes it doesn't.
Lahiru
On Mon, Nov 29, 2010 at 8:24 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Marnie,
>
> In IoSender and IoReceiver classes we are using a ByteBuffer to read the
> write messages, to increase the size of this buffer, I simply changed a
> constant defined in IoTransport.java file, if you look in to those three
> classes, you will get it.
>
> In IoTransport.java I changed the following line of code,
>
> private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
>
> but I saw there are some other constants defined with the name of
> MAX_FRAME_SIZE but after going through the code i saw they are not
> affecting.
>
> I feel there's an issue when we send a large message in to a Queue and
> something get currupted, when do framing and send. because when I keep the
> same small frame size for read buffer and increase the size for write
> buffer, it worked for 500K messages, but couldn't get it work for more than
> 1MB large messages.
>
> Lahiru
>
> Lahiru
>
>
> On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
> marnie.mccormack@googlemail.com> wrote:
>
>> Hi Lahiru,
>>
>> What exactly have you done/set for the frame size change ?
>>
>> Thanks,
>> Marnie
>>
>> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <glahiru@gmail.com
>> >wrote:
>>
>> > Hi all,
>> >
>> > I did a mistake when increase the framesize without increasing to
>> required
>> > size, when I increase the size I was able to read some of the messages
>> > which
>> > are in the size of 700K, but sometimes it fails.
>> >
>> >
>> > I kept on increasing the frame-size since in my setup there are messages
>> > with the size of 1.5M but I couldn't read a any of them though i
>> increase
>> > the frame-size more.
>> >
>> > other than changing in IoTransport.java, do I have to change any other
>> > constant values to sync the maximum-frame-size ?
>> >
>> > Regards
>> > Lahiru
>> >
>> > On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <glahiru@gmail.com
>> > >wrote:
>> >
>> > > Hi Andrew,
>> > >
>> > > Thanks Andrew for the response.
>> > > I have doubled the default frame-size which was (64 * 1024) in
>> > > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine
>> in
>> > this
>> > > scenario.
>> > >
>> > > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
>> > > andrewinternational@gmail.com> wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> The out of order frames only occur in 0-10 and with transports that
>> have
>> > >> no
>> > >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said that,
>> a
>> > >> frame
>> > >> body being interpreted as a header would give those symptoms.
>> > >
>> > > I will debug and see.
>> > >
>> > > Regards
>> > > Lahiru
>> > >
>> > >> Test code that
>> > >> exercises the issue would be illuminating.
>> > >>
>> > >> Unsure if this helps?
>> > >>
>> > >> Andrew.
>> > >>
>> > >> On 26 Nov 2010 16:56, "Marnie McCormack" <
>> > marnie.mccormack@googlemail.com
>> > >> >
>> > >> wrote:
>> > >>
>> > >> Hi Lahiru,
>> > >>
>> > >> Robbie mentioned earlier when I asked him about this that he *thinks*
>> > it's
>> > >> a
>> > >> bug Andrew might have come across and possibly fixed to do with OoO
>> > >> frames.
>> > >> Certainly the exception showing that the buffer length is 1 might
>> > support
>> > >> that.
>> > >>
>> > >> I haven't had a chance to look about for this - Andrew is not about
>> > today.
>> > >>
>> > >> I'll check with him when he's back if you don't get a better reply
>> here
>> > >> first.
>> > >>
>> > >> Rgds,
>> > >> Marnie
>> > >>
>> > >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
>> glahiru@gmail.com
>> > >> >wrote:
>> > >>
>> > >>
>> > >> > Hi Marnie,
>> > >> >
>> > >> > Thanks for your prompt response. yes i tried to wait before
>> consuming,
>> > >> and
>> > >> > it wo...
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Marnie,
In IoSender and IoReceiver classes we are using a ByteBuffer to read the
write messages, to increase the size of this buffer, I simply changed a
constant defined in IoTransport.java file, if you look in to those three
classes, you will get it.
In IoTransport.java I changed the following line of code,
private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
but I saw there are some other constants defined with the name of
MAX_FRAME_SIZE but after going through the code i saw they are not
affecting.
I feel there's an issue when we send a large message in to a Queue and
something get currupted, when do framing and send. because when I keep the
same small frame size for read buffer and increase the size for write
buffer, it worked for 500K messages, but couldn't get it work for more than
1MB large messages.
Lahiru
Lahiru
On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> Hi Lahiru,
>
> What exactly have you done/set for the frame size change ?
>
> Thanks,
> Marnie
>
> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi all,
> >
> > I did a mistake when increase the framesize without increasing to
> required
> > size, when I increase the size I was able to read some of the messages
> > which
> > are in the size of 700K, but sometimes it fails.
> >
> >
> > I kept on increasing the frame-size since in my setup there are messages
> > with the size of 1.5M but I couldn't read a any of them though i increase
> > the frame-size more.
> >
> > other than changing in IoTransport.java, do I have to change any other
> > constant values to sync the maximum-frame-size ?
> >
> > Regards
> > Lahiru
> >
> > On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <glahiru@gmail.com
> > >wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks Andrew for the response.
> > > I have doubled the default frame-size which was (64 * 1024) in
> > > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine in
> > this
> > > scenario.
> > >
> > > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
> > > andrewinternational@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> The out of order frames only occur in 0-10 and with transports that
> have
> > >> no
> > >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said that, a
> > >> frame
> > >> body being interpreted as a header would give those symptoms.
> > >
> > > I will debug and see.
> > >
> > > Regards
> > > Lahiru
> > >
> > >> Test code that
> > >> exercises the issue would be illuminating.
> > >>
> > >> Unsure if this helps?
> > >>
> > >> Andrew.
> > >>
> > >> On 26 Nov 2010 16:56, "Marnie McCormack" <
> > marnie.mccormack@googlemail.com
> > >> >
> > >> wrote:
> > >>
> > >> Hi Lahiru,
> > >>
> > >> Robbie mentioned earlier when I asked him about this that he *thinks*
> > it's
> > >> a
> > >> bug Andrew might have come across and possibly fixed to do with OoO
> > >> frames.
> > >> Certainly the exception showing that the buffer length is 1 might
> > support
> > >> that.
> > >>
> > >> I haven't had a chance to look about for this - Andrew is not about
> > today.
> > >>
> > >> I'll check with him when he's back if you don't get a better reply
> here
> > >> first.
> > >>
> > >> Rgds,
> > >> Marnie
> > >>
> > >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
> glahiru@gmail.com
> > >> >wrote:
> > >>
> > >>
> > >> > Hi Marnie,
> > >> >
> > >> > Thanks for your prompt response. yes i tried to wait before
> consuming,
> > >> and
> > >> > it wo...
> > >>
> > >
> > >
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Marnie McCormack <ma...@googlemail.com>.
Hi Lahiru,
What exactly have you done/set for the frame size change ?
Thanks,
Marnie
On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi all,
>
> I did a mistake when increase the framesize without increasing to required
> size, when I increase the size I was able to read some of the messages
> which
> are in the size of 700K, but sometimes it fails.
>
>
> I kept on increasing the frame-size since in my setup there are messages
> with the size of 1.5M but I couldn't read a any of them though i increase
> the frame-size more.
>
> other than changing in IoTransport.java, do I have to change any other
> constant values to sync the maximum-frame-size ?
>
> Regards
> Lahiru
>
> On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi Andrew,
> >
> > Thanks Andrew for the response.
> > I have doubled the default frame-size which was (64 * 1024) in
> > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine in
> this
> > scenario.
> >
> > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
> > andrewinternational@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> The out of order frames only occur in 0-10 and with transports that have
> >> no
> >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said that, a
> >> frame
> >> body being interpreted as a header would give those symptoms.
> >
> > I will debug and see.
> >
> > Regards
> > Lahiru
> >
> >> Test code that
> >> exercises the issue would be illuminating.
> >>
> >> Unsure if this helps?
> >>
> >> Andrew.
> >>
> >> On 26 Nov 2010 16:56, "Marnie McCormack" <
> marnie.mccormack@googlemail.com
> >> >
> >> wrote:
> >>
> >> Hi Lahiru,
> >>
> >> Robbie mentioned earlier when I asked him about this that he *thinks*
> it's
> >> a
> >> bug Andrew might have come across and possibly fixed to do with OoO
> >> frames.
> >> Certainly the exception showing that the buffer length is 1 might
> support
> >> that.
> >>
> >> I haven't had a chance to look about for this - Andrew is not about
> today.
> >>
> >> I'll check with him when he's back if you don't get a better reply here
> >> first.
> >>
> >> Rgds,
> >> Marnie
> >>
> >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <glahiru@gmail.com
> >> >wrote:
> >>
> >>
> >> > Hi Marnie,
> >> >
> >> > Thanks for your prompt response. yes i tried to wait before consuming,
> >> and
> >> > it wo...
> >>
> >
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi all,
I did a mistake when increase the framesize without increasing to required
size, when I increase the size I was able to read some of the messages which
are in the size of 700K, but sometimes it fails.
I kept on increasing the frame-size since in my setup there are messages
with the size of 1.5M but I couldn't read a any of them though i increase
the frame-size more.
other than changing in IoTransport.java, do I have to change any other
constant values to sync the maximum-frame-size ?
Regards
Lahiru
On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Andrew,
>
> Thanks Andrew for the response.
> I have doubled the default frame-size which was (64 * 1024) in
> IoTransport.java, but didn't worked for me. FYI, activeMQ works fine in this
> scenario.
>
> On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
> andrewinternational@gmail.com> wrote:
>
>> Hi,
>>
>> The out of order frames only occur in 0-10 and with transports that have
>> no
>> concept of sequencing, i.e. in-vm or udp, not tcp. Having said that, a
>> frame
>> body being interpreted as a header would give those symptoms.
>
> I will debug and see.
>
> Regards
> Lahiru
>
>> Test code that
>> exercises the issue would be illuminating.
>>
>> Unsure if this helps?
>>
>> Andrew.
>>
>> On 26 Nov 2010 16:56, "Marnie McCormack" <marnie.mccormack@googlemail.com
>> >
>> wrote:
>>
>> Hi Lahiru,
>>
>> Robbie mentioned earlier when I asked him about this that he *thinks* it's
>> a
>> bug Andrew might have come across and possibly fixed to do with OoO
>> frames.
>> Certainly the exception showing that the buffer length is 1 might support
>> that.
>>
>> I haven't had a chance to look about for this - Andrew is not about today.
>>
>> I'll check with him when he's back if you don't get a better reply here
>> first.
>>
>> Rgds,
>> Marnie
>>
>> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <glahiru@gmail.com
>> >wrote:
>>
>>
>> > Hi Marnie,
>> >
>> > Thanks for your prompt response. yes i tried to wait before consuming,
>> and
>> > it wo...
>>
>
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Andrew,
Thanks Andrew for the response.
I have doubled the default frame-size which was (64 * 1024) in
IoTransport.java, but didn't worked for me. FYI, activeMQ works fine in this
scenario.
On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
andrewinternational@gmail.com> wrote:
> Hi,
>
> The out of order frames only occur in 0-10 and with transports that have no
> concept of sequencing, i.e. in-vm or udp, not tcp. Having said that, a
> frame
> body being interpreted as a header would give those symptoms.
I will debug and see.
Regards
Lahiru
> Test code that
> exercises the issue would be illuminating.
>
> Unsure if this helps?
>
> Andrew.
>
> On 26 Nov 2010 16:56, "Marnie McCormack" <ma...@googlemail.com>
> wrote:
>
> Hi Lahiru,
>
> Robbie mentioned earlier when I asked him about this that he *thinks* it's
> a
> bug Andrew might have come across and possibly fixed to do with OoO frames.
> Certainly the exception showing that the buffer length is 1 might support
> that.
>
> I haven't had a chance to look about for this - Andrew is not about today.
>
> I'll check with him when he's back if you don't get a better reply here
> first.
>
> Rgds,
> Marnie
>
> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
>
> > Hi Marnie,
> >
> > Thanks for your prompt response. yes i tried to wait before consuming,
> and
> > it wo...
>
Re: How to handle large messages in Qpid Java client
Posted by Andrew Kennedy <an...@gmail.com>.
Hi,
The out of order frames only occur in 0-10 and with transports that have no
concept of sequencing, i.e. in-vm or udp, not tcp. Having said that, a frame
body being interpreted as a header would give those symptoms. Test code that
exercises the issue would be illuminating.
Unsure if this helps?
Andrew.
On 26 Nov 2010 16:56, "Marnie McCormack" <ma...@googlemail.com>
wrote:
Hi Lahiru,
Robbie mentioned earlier when I asked him about this that he *thinks* it's a
bug Andrew might have come across and possibly fixed to do with OoO frames.
Certainly the exception showing that the buffer length is 1 might support
that.
I haven't had a chance to look about for this - Andrew is not about today.
I'll check with him when he's back if you don't get a better reply here
first.
Rgds,
Marnie
On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <glahiru@gmail.com
>wrote:
> Hi Marnie,
>
> Thanks for your prompt response. yes i tried to wait before consuming, and
> it wo...
Re: How to handle large messages in Qpid Java client
Posted by Marnie McCormack <ma...@googlemail.com>.
Hi Lahiru,
Robbie mentioned earlier when I asked him about this that he *thinks* it's a
bug Andrew might have come across and possibly fixed to do with OoO frames.
Certainly the exception showing that the buffer length is 1 might support
that.
I haven't had a chance to look about for this - Andrew is not about today.
I'll check with him when he's back if you don't get a better reply here
first.
Rgds,
Marnie
On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Marnie,
>
> Thanks for your prompt response. yes i tried to wait before consuming, and
> it worked for me. But the practical issue is we are sending and consuming
> concurrently in two different threads, I think if the consumer assume
> there's a message in the queue before it successfully stored in to Qpid,
> that is not correct.
>
> I believe there's an issue in Qpid Java client consumer implementation. I
> will look in to transaction work and get back to the thread.
>
> WDYT ?
>
> Regards
> Lahiru
>
> On Fri, Nov 26, 2010 at 8:27 PM, Marnie McCormack <
> marnie.mccormack@googlemail.com> wrote:
>
> > If the whole message hasn't been buffered on the broker before you try to
> > consume it then this might account for this exception. Try sleeping
> before
> > consuming and see if the issue goes away, that'd prove it - or use
> > transactions on the send() and wait for commit to return before consuming
> ?
> >
> > Marnie
> >
> > On Fri, Nov 26, 2010 at 2:50 PM, Lahiru Gunathilake <glahiru@gmail.com
> > >wrote:
> >
> > > Hi Marnie,
> > >
> > > On Fri, Nov 26, 2010 at 6:23 PM, Marnie McCormack <
> > > marnie.mccormack@googlemail.com> wrote:
> > >
> > > > Hi Lahiru,
> > > >
> > > > There's a JIRA about this exception/issue - QPID-2902.
> > > >
> > > > Does it fail for you reliably ? I wonder if there's a timing issue
> here
> > > > since it didn't occur all the time on the CI instance.
> > > >
> > > +1, I noticed some of the messages were picked successfully. I think
> this
> > > is
> > > a timing issue comes due to large sizes of the messages.
> > >
> > > Lahiru
> > >
> > > >
> > > > Marnie
> > > >
> > > > On Fri, Nov 26, 2010 at 12:14 PM, Lahiru Gunathilake <
> > glahiru@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi Devs,
> > > > >
> > > > > When I consume a big message with 1MB file size and pick it from
> Qpid
> > > > > Queue,
> > > > > I am getting an error like this
> > > > >
> > > > > avax.jms.JMSException: Could not decode string data:
> > > > > java.nio.charset.MalformedInputException: Input length = 1<br> at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
> > > > >
> > > > > After debugging the code, I feel like this happen, due to a
> malformed
> > > > input
> > > > > stream coming for the client.
> > > > >
> > > > > Can I increase the frame-size of the client and get this message in
> > to
> > > a
> > > > > single frame ? Is there any other solution to this error.
> > > > >
> > > > > Regards
> > > > > Lahiru
> > > > >
> > > >
> > >
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Marnie,
Thanks for your prompt response. yes i tried to wait before consuming, and
it worked for me. But the practical issue is we are sending and consuming
concurrently in two different threads, I think if the consumer assume
there's a message in the queue before it successfully stored in to Qpid,
that is not correct.
I believe there's an issue in Qpid Java client consumer implementation. I
will look in to transaction work and get back to the thread.
WDYT ?
Regards
Lahiru
On Fri, Nov 26, 2010 at 8:27 PM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> If the whole message hasn't been buffered on the broker before you try to
> consume it then this might account for this exception. Try sleeping before
> consuming and see if the issue goes away, that'd prove it - or use
> transactions on the send() and wait for commit to return before consuming ?
>
> Marnie
>
> On Fri, Nov 26, 2010 at 2:50 PM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi Marnie,
> >
> > On Fri, Nov 26, 2010 at 6:23 PM, Marnie McCormack <
> > marnie.mccormack@googlemail.com> wrote:
> >
> > > Hi Lahiru,
> > >
> > > There's a JIRA about this exception/issue - QPID-2902.
> > >
> > > Does it fail for you reliably ? I wonder if there's a timing issue here
> > > since it didn't occur all the time on the CI instance.
> > >
> > +1, I noticed some of the messages were picked successfully. I think this
> > is
> > a timing issue comes due to large sizes of the messages.
> >
> > Lahiru
> >
> > >
> > > Marnie
> > >
> > > On Fri, Nov 26, 2010 at 12:14 PM, Lahiru Gunathilake <
> glahiru@gmail.com
> > > >wrote:
> > >
> > > > Hi Devs,
> > > >
> > > > When I consume a big message with 1MB file size and pick it from Qpid
> > > > Queue,
> > > > I am getting an error like this
> > > >
> > > > avax.jms.JMSException: Could not decode string data:
> > > > java.nio.charset.MalformedInputException: Input length = 1<br> at
> > > >
> > > >
> > >
> >
> org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
> > > >
> > > > After debugging the code, I feel like this happen, due to a malformed
> > > input
> > > > stream coming for the client.
> > > >
> > > > Can I increase the frame-size of the client and get this message in
> to
> > a
> > > > single frame ? Is there any other solution to this error.
> > > >
> > > > Regards
> > > > Lahiru
> > > >
> > >
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Marnie McCormack <ma...@googlemail.com>.
If the whole message hasn't been buffered on the broker before you try to
consume it then this might account for this exception. Try sleeping before
consuming and see if the issue goes away, that'd prove it - or use
transactions on the send() and wait for commit to return before consuming ?
Marnie
On Fri, Nov 26, 2010 at 2:50 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Marnie,
>
> On Fri, Nov 26, 2010 at 6:23 PM, Marnie McCormack <
> marnie.mccormack@googlemail.com> wrote:
>
> > Hi Lahiru,
> >
> > There's a JIRA about this exception/issue - QPID-2902.
> >
> > Does it fail for you reliably ? I wonder if there's a timing issue here
> > since it didn't occur all the time on the CI instance.
> >
> +1, I noticed some of the messages were picked successfully. I think this
> is
> a timing issue comes due to large sizes of the messages.
>
> Lahiru
>
> >
> > Marnie
> >
> > On Fri, Nov 26, 2010 at 12:14 PM, Lahiru Gunathilake <glahiru@gmail.com
> > >wrote:
> >
> > > Hi Devs,
> > >
> > > When I consume a big message with 1MB file size and pick it from Qpid
> > > Queue,
> > > I am getting an error like this
> > >
> > > avax.jms.JMSException: Could not decode string data:
> > > java.nio.charset.MalformedInputException: Input length = 1<br> at
> > >
> > >
> >
> org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
> > >
> > > After debugging the code, I feel like this happen, due to a malformed
> > input
> > > stream coming for the client.
> > >
> > > Can I increase the frame-size of the client and get this message in to
> a
> > > single frame ? Is there any other solution to this error.
> > >
> > > Regards
> > > Lahiru
> > >
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Marnie,
On Fri, Nov 26, 2010 at 6:23 PM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> Hi Lahiru,
>
> There's a JIRA about this exception/issue - QPID-2902.
>
> Does it fail for you reliably ? I wonder if there's a timing issue here
> since it didn't occur all the time on the CI instance.
>
+1, I noticed some of the messages were picked successfully. I think this is
a timing issue comes due to large sizes of the messages.
Lahiru
>
> Marnie
>
> On Fri, Nov 26, 2010 at 12:14 PM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi Devs,
> >
> > When I consume a big message with 1MB file size and pick it from Qpid
> > Queue,
> > I am getting an error like this
> >
> > avax.jms.JMSException: Could not decode string data:
> > java.nio.charset.MalformedInputException: Input length = 1<br> at
> >
> >
> org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
> >
> > After debugging the code, I feel like this happen, due to a malformed
> input
> > stream coming for the client.
> >
> > Can I increase the frame-size of the client and get this message in to a
> > single frame ? Is there any other solution to this error.
> >
> > Regards
> > Lahiru
> >
>
Re: How to handle large messages in Qpid Java client
Posted by Marnie McCormack <ma...@googlemail.com>.
Hi Lahiru,
There's a JIRA about this exception/issue - QPID-2902.
Does it fail for you reliably ? I wonder if there's a timing issue here
since it didn't occur all the time on the CI instance.
Marnie
On Fri, Nov 26, 2010 at 12:14 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Devs,
>
> When I consume a big message with 1MB file size and pick it from Qpid
> Queue,
> I am getting an error like this
>
> avax.jms.JMSException: Could not decode string data:
> java.nio.charset.MalformedInputException: Input length = 1<br> at
>
> org.apache.qpid.client.message.JMSTextMessage.getText(JMSTextMessage.java:150)
>
> After debugging the code, I feel like this happen, due to a malformed input
> stream coming for the client.
>
> Can I increase the frame-size of the client and get this message in to a
> single frame ? Is there any other solution to this error.
>
> Regards
> Lahiru
>