[DISCUSS] AMQP Large Messages

[Clebert Suconic <cl...@gmail.com>]

I am currently implementing Large Message support for AMQP Messages.

There's currently no support for Large Messages in AMQP, although I
have recently implemented a dum copy and convert, but if you send a
large message, say with 1 G, the whole body would be read into memory
before a conversion to AMQP.. what won't fit in most cases of course.
(even so I can always make a case where it wouldn't fit).

LargeMessages are always converted as Core currently. With this change
they will be kept as AMQP all the way through, adding full support for
large messages, which is nice!


Anyway... the design I got so far is the following:


- LargeServerMessageImpl will delegate its implementation to another
class called LargeMessageBody

- LargeMessageBody would then be used by AMQPMessage when incomplete
messages are sent through the wire.

- When sending messages on the server, I will use sender.send multiple
times until the whole buffer is read. I'm not 100% sure qpid.Proton
will work fine with that approach. I will start with that, and deal
with further bugs if they come along.


- When receiving messages on the server. Messages are marked as
incomplete. So, I will create the file using the LargeMessageBody. on
that case I will just use the current recv method we already have
available on Proton multiple times. This seems to work fine already.

Re: [DISCUSS] AMQP Large Messages

[Clebert Suconic <cl...@gmail.com>]

This is a bit more complex.


LargeServerMessafe is too tight with core.

Will be a bit challenging changing it. I will make large Message a property
of the message so it can be resumed in AMQP.

It will still be compatible change.  Won’t need to change it to 3.0.

I will update the design here once I do some more work.

On Thu, Nov 16, 2017 at 9:46 AM Clebert Suconic <cl...@gmail.com>
wrote:

> I will try to make a conversion. I would rather treat than make another
> parameter.
>
> On Thu, Nov 16, 2017 at 9:31 AM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> There is no distinction.
>>
>> It will stream if the packet is more then min Large message size on
>> core.  Or max frame size in AMQP.
>>
>> I don’t know how distinguish really large ones from the ones that are
>> just above the limit.
>>
>> On Thu, Nov 16, 2017 at 9:11 AM Martyn Taylor <mt...@redhat.com> wrote:
>>
>>> On Thu, Nov 16, 2017 at 2:01 PM, Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> wrote:
>>>
>>> > On Thu, Nov 16, 2017 at 6:10 AM, Martyn Taylor <mt...@redhat.com>
>>> wrote:
>>> > > Clebert,
>>> > >
>>> > > We need to distinguish between streamed messages and large
>>> messages.  For
>>> > > the basic large message case, i.e. a single large message sent to the
>>> > > broker.  I agree with what you have here.
>>> > >
>>> > > Streamed large messages (i.e. messages that are received in chunks)
>>> > allows
>>> > > us to store the message without having to keep the whole thing in
>>> broker
>>> > > memory.  This get's complicated with cross protocol as:
>>> > >
>>> > > 1. Not all protocols support streamed messages.
>>> > Exactly.. right now in Stomp.. we read the whole body in memory before
>>> > sending. Just like what is in master now for AMQP.
>>> >
>>> > > 2. Cross protocol may require message conversions
>>> > Just like in Stomp... we read the whole message in memory
>>> >
>>> > >
>>> > > Both 1. and 2. would likely mean that we'd likely need to read the
>>> whole
>>> > > message into memory and I'm not sure we really want to do this?  It
>>> > defeats
>>> > > one of the main purposes of streamed messages.  (The others being
>>> saving
>>> > > client memory and reducing the amount of data to resend on connection
>>> > drop.)
>>> >
>>> > Most times the user will have no control over this. Say you sent a
>>> > 100K + 1 byte over the configured limit, String messages in Core.  the
>>> > client will stream it as it being large
>>>
>>> This is why I think we need to distinguish between broker large messages
>>> and streamed messages.  This configuration option is really for streaming
>>> messages.  In our docs we talk about the ability to send messages that
>>> are
>>> larger than the broker memory using streaming.  What happens if I do this
>>> and try to consume it via STOMP or another protocol that doesn't support
>>> streaming?
>>>
>>> > .
>>> >
>>> >
>>> > >
>>> > > For 1. I wonder whether or not we want to support this at all, or if
>>> we
>>> > do,
>>> > > make it configurable.
>>> > > For 2. Is it possible to transform the message via a stream?  If not
>>> then
>>> > > perhaps we treat this as a raw bytes message?
>>> >
>>> > It is possible as I said.. I think it's a separate task. I may try to
>>> > do it within this scope.
>>> >
>>> OK sounds good.
>>>
>> --
>> Clebert Suconic
>>
> --
> Clebert Suconic
>
-- 
Clebert Suconic

Re: [DISCUSS] AMQP Large Messages

[Clebert Suconic <cl...@gmail.com>]

I will try to make a conversion. I would rather treat than make another
parameter.

On Thu, Nov 16, 2017 at 9:31 AM Clebert Suconic <cl...@gmail.com>
wrote:

> There is no distinction.
>
> It will stream if the packet is more then min Large message size on core.
> Or max frame size in AMQP.
>
> I don’t know how distinguish really large ones from the ones that are just
> above the limit.
>
> On Thu, Nov 16, 2017 at 9:11 AM Martyn Taylor <mt...@redhat.com> wrote:
>
>> On Thu, Nov 16, 2017 at 2:01 PM, Clebert Suconic <
>> clebert.suconic@gmail.com>
>> wrote:
>>
>> > On Thu, Nov 16, 2017 at 6:10 AM, Martyn Taylor <mt...@redhat.com>
>> wrote:
>> > > Clebert,
>> > >
>> > > We need to distinguish between streamed messages and large messages.
>> For
>> > > the basic large message case, i.e. a single large message sent to the
>> > > broker.  I agree with what you have here.
>> > >
>> > > Streamed large messages (i.e. messages that are received in chunks)
>> > allows
>> > > us to store the message without having to keep the whole thing in
>> broker
>> > > memory.  This get's complicated with cross protocol as:
>> > >
>> > > 1. Not all protocols support streamed messages.
>> > Exactly.. right now in Stomp.. we read the whole body in memory before
>> > sending. Just like what is in master now for AMQP.
>> >
>> > > 2. Cross protocol may require message conversions
>> > Just like in Stomp... we read the whole message in memory
>> >
>> > >
>> > > Both 1. and 2. would likely mean that we'd likely need to read the
>> whole
>> > > message into memory and I'm not sure we really want to do this?  It
>> > defeats
>> > > one of the main purposes of streamed messages.  (The others being
>> saving
>> > > client memory and reducing the amount of data to resend on connection
>> > drop.)
>> >
>> > Most times the user will have no control over this. Say you sent a
>> > 100K + 1 byte over the configured limit, String messages in Core.  the
>> > client will stream it as it being large
>>
>> This is why I think we need to distinguish between broker large messages
>> and streamed messages.  This configuration option is really for streaming
>> messages.  In our docs we talk about the ability to send messages that are
>> larger than the broker memory using streaming.  What happens if I do this
>> and try to consume it via STOMP or another protocol that doesn't support
>> streaming?
>>
>> > .
>> >
>> >
>> > >
>> > > For 1. I wonder whether or not we want to support this at all, or if
>> we
>> > do,
>> > > make it configurable.
>> > > For 2. Is it possible to transform the message via a stream?  If not
>> then
>> > > perhaps we treat this as a raw bytes message?
>> >
>> > It is possible as I said.. I think it's a separate task. I may try to
>> > do it within this scope.
>> >
>> OK sounds good.
>>
> --
> Clebert Suconic
>
-- 
Clebert Suconic

Re: [DISCUSS] AMQP Large Messages

[Clebert Suconic <cl...@gmail.com>]

There is no distinction.

It will stream if the packet is more then min Large message size on core.
Or max frame size in AMQP.

I don’t know how distinguish really large ones from the ones that are just
above the limit.

On Thu, Nov 16, 2017 at 9:11 AM Martyn Taylor <mt...@redhat.com> wrote:

> On Thu, Nov 16, 2017 at 2:01 PM, Clebert Suconic <
> clebert.suconic@gmail.com>
> wrote:
>
> > On Thu, Nov 16, 2017 at 6:10 AM, Martyn Taylor <mt...@redhat.com>
> wrote:
> > > Clebert,
> > >
> > > We need to distinguish between streamed messages and large messages.
> For
> > > the basic large message case, i.e. a single large message sent to the
> > > broker.  I agree with what you have here.
> > >
> > > Streamed large messages (i.e. messages that are received in chunks)
> > allows
> > > us to store the message without having to keep the whole thing in
> broker
> > > memory.  This get's complicated with cross protocol as:
> > >
> > > 1. Not all protocols support streamed messages.
> > Exactly.. right now in Stomp.. we read the whole body in memory before
> > sending. Just like what is in master now for AMQP.
> >
> > > 2. Cross protocol may require message conversions
> > Just like in Stomp... we read the whole message in memory
> >
> > >
> > > Both 1. and 2. would likely mean that we'd likely need to read the
> whole
> > > message into memory and I'm not sure we really want to do this?  It
> > defeats
> > > one of the main purposes of streamed messages.  (The others being
> saving
> > > client memory and reducing the amount of data to resend on connection
> > drop.)
> >
> > Most times the user will have no control over this. Say you sent a
> > 100K + 1 byte over the configured limit, String messages in Core.  the
> > client will stream it as it being large
>
> This is why I think we need to distinguish between broker large messages
> and streamed messages.  This configuration option is really for streaming
> messages.  In our docs we talk about the ability to send messages that are
> larger than the broker memory using streaming.  What happens if I do this
> and try to consume it via STOMP or another protocol that doesn't support
> streaming?
>
> > .
> >
> >
> > >
> > > For 1. I wonder whether or not we want to support this at all, or if we
> > do,
> > > make it configurable.
> > > For 2. Is it possible to transform the message via a stream?  If not
> then
> > > perhaps we treat this as a raw bytes message?
> >
> > It is possible as I said.. I think it's a separate task. I may try to
> > do it within this scope.
> >
> OK sounds good.
>
-- 
Clebert Suconic

Re: [DISCUSS] AMQP Large Messages

[Martyn Taylor <mt...@redhat.com>]

On Thu, Nov 16, 2017 at 2:01 PM, Clebert Suconic <cl...@gmail.com>
wrote:

> On Thu, Nov 16, 2017 at 6:10 AM, Martyn Taylor <mt...@redhat.com> wrote:
> > Clebert,
> >
> > We need to distinguish between streamed messages and large messages.  For
> > the basic large message case, i.e. a single large message sent to the
> > broker.  I agree with what you have here.
> >
> > Streamed large messages (i.e. messages that are received in chunks)
> allows
> > us to store the message without having to keep the whole thing in broker
> > memory.  This get's complicated with cross protocol as:
> >
> > 1. Not all protocols support streamed messages.
> Exactly.. right now in Stomp.. we read the whole body in memory before
> sending. Just like what is in master now for AMQP.
>
> > 2. Cross protocol may require message conversions
> Just like in Stomp... we read the whole message in memory
>
> >
> > Both 1. and 2. would likely mean that we'd likely need to read the whole
> > message into memory and I'm not sure we really want to do this?  It
> defeats
> > one of the main purposes of streamed messages.  (The others being saving
> > client memory and reducing the amount of data to resend on connection
> drop.)
>
> Most times the user will have no control over this. Say you sent a
> 100K + 1 byte over the configured limit, String messages in Core.  the
> client will stream it as it being large

This is why I think we need to distinguish between broker large messages
and streamed messages.  This configuration option is really for streaming
messages.  In our docs we talk about the ability to send messages that are
larger than the broker memory using streaming.  What happens if I do this
and try to consume it via STOMP or another protocol that doesn't support
streaming?

> .
>
>
> >
> > For 1. I wonder whether or not we want to support this at all, or if we
> do,
> > make it configurable.
> > For 2. Is it possible to transform the message via a stream?  If not then
> > perhaps we treat this as a raw bytes message?
>
> It is possible as I said.. I think it's a separate task. I may try to
> do it within this scope.
>
OK sounds good.

Re: [DISCUSS] AMQP Large Messages

[Clebert Suconic <cl...@gmail.com>]

On Thu, Nov 16, 2017 at 6:10 AM, Martyn Taylor <mt...@redhat.com> wrote:
> Clebert,
>
> We need to distinguish between streamed messages and large messages.  For
> the basic large message case, i.e. a single large message sent to the
> broker.  I agree with what you have here.
>
> Streamed large messages (i.e. messages that are received in chunks) allows
> us to store the message without having to keep the whole thing in broker
> memory.  This get's complicated with cross protocol as:
>
> 1. Not all protocols support streamed messages.
Exactly.. right now in Stomp.. we read the whole body in memory before
sending. Just like what is in master now for AMQP.

> 2. Cross protocol may require message conversions
Just like in Stomp... we read the whole message in memory

>
> Both 1. and 2. would likely mean that we'd likely need to read the whole
> message into memory and I'm not sure we really want to do this?  It defeats
> one of the main purposes of streamed messages.  (The others being saving
> client memory and reducing the amount of data to resend on connection drop.)

Most times the user will have no control over this. Say you sent a
100K + 1 byte over the configured limit, String messages in Core.  the
client will stream it as it being large.


>
> For 1. I wonder whether or not we want to support this at all, or if we do,
> make it configurable.
> For 2. Is it possible to transform the message via a stream?  If not then
> perhaps we treat this as a raw bytes message?

It is possible as I said.. I think it's a separate task. I may try to
do it within this scope.

Re: [DISCUSS] AMQP Large Messages

[Martyn Taylor <mt...@redhat.com>]

Clebert,

We need to distinguish between streamed messages and large messages.  For
the basic large message case, i.e. a single large message sent to the
broker.  I agree with what you have here.

Streamed large messages (i.e. messages that are received in chunks) allows
us to store the message without having to keep the whole thing in broker
memory.  This get's complicated with cross protocol as:

1. Not all protocols support streamed messages.
2. Cross protocol may require message conversions

Both 1. and 2. would likely mean that we'd likely need to read the whole
message into memory and I'm not sure we really want to do this?  It defeats
one of the main purposes of streamed messages.  (The others being saving
client memory and reducing the amount of data to resend on connection drop.)

For 1. I wonder whether or not we want to support this at all, or if we do,
make it configurable.
For 2. Is it possible to transform the message via a stream?  If not then
perhaps we treat this as a raw bytes message?

Cheers

On Wed, Nov 15, 2017 at 11:33 PM, Clebert Suconic <clebert.suconic@gmail.com
> wrote:

> It would be possible... look at
> https://github.com/apache/activemq-artemis/pull/1655
>
> I will keep the same current semantic. That is.. you would read the
> whole body.. and convert in memory.
> That's because I wouldn't know how to convert the body without having
> it into memory.. for things like string reading.
>
> Possible to do it without using memory.. but not very practical at
> this point... we could make this a separate task.. to convert large
> messages without requiring full memory read.
>
>
>
> I would add a note to docs saying that large messages will be read
> into memory before converting protocol barriers.
>
> On Wed, Nov 15, 2017 at 6:24 PM, Michael André Pearce
> <mi...@me.com> wrote:
> > Looks good for AMQP to AMQP client.
> >
> > One question is in a poly protocol supported setup. Eg senders and
> receivers one uses Core and AMQP.
> >
> > It’s not clear if this should now be possible with the new
> LargeMessageBody abstraction you discuss.
> >
> > Cheers
> > Mike
> >
> > Sent from my iPhone
> >
> >> On 16 Nov 2017, at 02:10, Clebert Suconic <cl...@gmail.com>
> wrote:
> >>
> >> I am currently implementing Large Message support for AMQP Messages.
> >>
> >> There's currently no support for Large Messages in AMQP, although I
> >> have recently implemented a dum copy and convert, but if you send a
> >> large message, say with 1 G, the whole body would be read into memory
> >> before a conversion to AMQP.. what won't fit in most cases of course.
> >> (even so I can always make a case where it wouldn't fit).
> >>
> >> LargeMessages are always converted as Core currently. With this change
> >> they will be kept as AMQP all the way through, adding full support for
> >> large messages, which is nice!
> >>
> >>
> >> Anyway... the design I got so far is the following:
> >>
> >>
> >> - LargeServerMessageImpl will delegate its implementation to another
> >> class called LargeMessageBody
> >>
> >> - LargeMessageBody would then be used by AMQPMessage when incomplete
> >> messages are sent through the wire.
> >>
> >> - When sending messages on the server, I will use sender.send multiple
> >> times until the whole buffer is read. I'm not 100% sure qpid.Proton
> >> will work fine with that approach. I will start with that, and deal
> >> with further bugs if they come along.
> >>
> >>
> >> - When receiving messages on the server. Messages are marked as
> >> incomplete. So, I will create the file using the LargeMessageBody. on
> >> that case I will just use the current recv method we already have
> >> available on Proton multiple times. This seems to work fine already.
>
>
>
> --
> Clebert Suconic
>

Re: [DISCUSS] AMQP Large Messages

[Clebert Suconic <cl...@gmail.com>]

It would be possible... look at
https://github.com/apache/activemq-artemis/pull/1655

I will keep the same current semantic. That is.. you would read the
whole body.. and convert in memory.
That's because I wouldn't know how to convert the body without having
it into memory.. for things like string reading.

Possible to do it without using memory.. but not very practical at
this point... we could make this a separate task.. to convert large
messages without requiring full memory read.



I would add a note to docs saying that large messages will be read
into memory before converting protocol barriers.

On Wed, Nov 15, 2017 at 6:24 PM, Michael André Pearce
<mi...@me.com> wrote:
> Looks good for AMQP to AMQP client.
>
> One question is in a poly protocol supported setup. Eg senders and receivers one uses Core and AMQP.
>
> It’s not clear if this should now be possible with the new LargeMessageBody abstraction you discuss.
>
> Cheers
> Mike
>
> Sent from my iPhone
>
>> On 16 Nov 2017, at 02:10, Clebert Suconic <cl...@gmail.com> wrote:
>>
>> I am currently implementing Large Message support for AMQP Messages.
>>
>> There's currently no support for Large Messages in AMQP, although I
>> have recently implemented a dum copy and convert, but if you send a
>> large message, say with 1 G, the whole body would be read into memory
>> before a conversion to AMQP.. what won't fit in most cases of course.
>> (even so I can always make a case where it wouldn't fit).
>>
>> LargeMessages are always converted as Core currently. With this change
>> they will be kept as AMQP all the way through, adding full support for
>> large messages, which is nice!
>>
>>
>> Anyway... the design I got so far is the following:
>>
>>
>> - LargeServerMessageImpl will delegate its implementation to another
>> class called LargeMessageBody
>>
>> - LargeMessageBody would then be used by AMQPMessage when incomplete
>> messages are sent through the wire.
>>
>> - When sending messages on the server, I will use sender.send multiple
>> times until the whole buffer is read. I'm not 100% sure qpid.Proton
>> will work fine with that approach. I will start with that, and deal
>> with further bugs if they come along.
>>
>>
>> - When receiving messages on the server. Messages are marked as
>> incomplete. So, I will create the file using the LargeMessageBody. on
>> that case I will just use the current recv method we already have
>> available on Proton multiple times. This seems to work fine already.



-- 
Clebert Suconic

Re: [DISCUSS] AMQP Large Messages

[Michael André Pearce <mi...@me.com>]

Looks good for AMQP to AMQP client.

One question is in a poly protocol supported setup. Eg senders and receivers one uses Core and AMQP. 

It’s not clear if this should now be possible with the new LargeMessageBody abstraction you discuss. 

Cheers
Mike 

Sent from my iPhone

> On 16 Nov 2017, at 02:10, Clebert Suconic <cl...@gmail.com> wrote:
> 
> I am currently implementing Large Message support for AMQP Messages.
> 
> There's currently no support for Large Messages in AMQP, although I
> have recently implemented a dum copy and convert, but if you send a
> large message, say with 1 G, the whole body would be read into memory
> before a conversion to AMQP.. what won't fit in most cases of course.
> (even so I can always make a case where it wouldn't fit).
> 
> LargeMessages are always converted as Core currently. With this change
> they will be kept as AMQP all the way through, adding full support for
> large messages, which is nice!
> 
> 
> Anyway... the design I got so far is the following:
> 
> 
> - LargeServerMessageImpl will delegate its implementation to another
> class called LargeMessageBody
> 
> - LargeMessageBody would then be used by AMQPMessage when incomplete
> messages are sent through the wire.
> 
> - When sending messages on the server, I will use sender.send multiple
> times until the whole buffer is read. I'm not 100% sure qpid.Proton
> will work fine with that approach. I will start with that, and deal
> with further bugs if they come along.
> 
> 
> - When receiving messages on the server. Messages are marked as
> incomplete. So, I will create the file using the LargeMessageBody. on
> that case I will just use the current recv method we already have
> available on Proton multiple times. This seems to work fine already.