Re: How to handle large messages in Qpid Java client

[Lahiru Gunathilake <gl...@gmail.com>]

Forwarding to dev list !

On Thu, Dec 2, 2010 at 3:28 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:

> FYI, I try to monitor all the reads from the socket InputStream from the
> client and I can see that Client is reading a curupted message, so
> I guarantee that there's no issue in Client side IoSendere and IoReceiver
> classes.
>
> I think the issue is with the broker IoReceiver or Sending, we found that
> this is directly an issue with Java broker because things are working
> perfect with Qpid C++ broker 0.8 RC2.
>
> Thanks
> Lahiru
>
>
> On Mon, Nov 29, 2010 at 3:21 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>> FYI, during receiving a message, I can see following
>> malformed characters in my JMSTextMessage object,
>>
>> <OUTMOD>4</OUTMOD>^D^C��^@^A^@^@^@^@^@^@
>>
>> This cause the exception i have sent at the very first email, after
>> increasing the frame-size sometimes it works sometimes it doesn't.
>>
>> Lahiru
>>
>>
>> On Mon, Nov 29, 2010 at 8:24 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>>
>>> Hi Marnie,
>>>
>>> In IoSender and IoReceiver classes we are using a ByteBuffer to read the
>>> write messages, to increase the size of this buffer, I simply changed a
>>> constant defined in IoTransport.java file, if you look in to those three
>>> classes, you will get it.
>>>
>>> In IoTransport.java I changed the following line of code,
>>>
>>> private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
>>>
>>> but I saw there are some other constants defined with the name of
>>> MAX_FRAME_SIZE but after going through the code i saw they are not
>>> affecting.
>>>
>>> I feel there's an issue when we send a large message in to a Queue and
>>> something get currupted, when do framing and send. because when I keep the
>>> same small frame size for read buffer and increase the size for write
>>> buffer, it worked for 500K messages, but couldn't get it work for more than
>>> 1MB large messages.
>>>
>>> Lahiru
>>>
>>> Lahiru
>>>
>>>
>>> On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
>>> marnie.mccormack@googlemail.com> wrote:
>>>
>>>> Hi Lahiru,
>>>>
>>>> What exactly have you done/set for the frame size change ?
>>>>
>>>> Thanks,
>>>> Marnie
>>>>
>>>> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <glahiru@gmail.com
>>>> >wrote:
>>>>
>>>> > Hi all,
>>>> >
>>>> > I did a mistake when increase the framesize without increasing to
>>>> required
>>>> > size, when I increase the size I was able to read some of the messages
>>>> > which
>>>> > are in the size of 700K, but sometimes it fails.
>>>> >
>>>> >
>>>> > I kept on increasing the frame-size since in my setup there are
>>>> messages
>>>> > with the size of 1.5M but I couldn't read a any of them though i
>>>> increase
>>>> > the frame-size more.
>>>> >
>>>> > other than changing in IoTransport.java, do I have to change any other
>>>> > constant values to sync the maximum-frame-size ?
>>>> >
>>>> > Regards
>>>> > Lahiru
>>>> >
>>>> > On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <
>>>> glahiru@gmail.com
>>>> > >wrote:
>>>> >
>>>> > > Hi Andrew,
>>>> > >
>>>> > > Thanks Andrew for the response.
>>>> > > I have doubled  the default frame-size which was (64 * 1024) in
>>>> > > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine
>>>> in
>>>> > this
>>>> > > scenario.
>>>> > >
>>>> > > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
>>>> > > andrewinternational@gmail.com> wrote:
>>>> > >
>>>> > >> Hi,
>>>> > >>
>>>> > >> The out of order frames only occur in 0-10 and with transports that
>>>> have
>>>> > >> no
>>>> > >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said
>>>> that, a
>>>> > >> frame
>>>> > >> body being interpreted as a header would give those symptoms.
>>>> > >
>>>> > > I will debug and see.
>>>> > >
>>>> > > Regards
>>>> > > Lahiru
>>>> > >
>>>> > >> Test code that
>>>> > >> exercises the issue would be illuminating.
>>>> > >>
>>>> > >> Unsure if this helps?
>>>> > >>
>>>> > >> Andrew.
>>>> > >>
>>>> > >> On 26 Nov 2010 16:56, "Marnie McCormack" <
>>>> > marnie.mccormack@googlemail.com
>>>> > >> >
>>>> > >> wrote:
>>>> > >>
>>>> > >> Hi Lahiru,
>>>> > >>
>>>> > >> Robbie mentioned earlier when I asked him about this that he
>>>> *thinks*
>>>> > it's
>>>> > >> a
>>>> > >> bug Andrew might have come across and possibly fixed to do with OoO
>>>> > >> frames.
>>>> > >> Certainly the exception showing that the buffer length is 1 might
>>>> > support
>>>> > >> that.
>>>> > >>
>>>> > >> I haven't had a chance to look about for this - Andrew is not about
>>>> > today.
>>>> > >>
>>>> > >> I'll check with him when he's back if you don't get a better reply
>>>> here
>>>> > >> first.
>>>> > >>
>>>> > >> Rgds,
>>>> > >> Marnie
>>>> > >>
>>>> > >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
>>>> glahiru@gmail.com
>>>> > >> >wrote:
>>>> > >>
>>>> > >>
>>>> > >> > Hi Marnie,
>>>> > >> >
>>>> > >> > Thanks for your prompt response. yes i tried to wait before
>>>> consuming,
>>>> > >> and
>>>> > >> > it wo...
>>>> > >>
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>