You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Andy Yar <an...@gmail.com> on 2022/07/14 14:19:40 UTC

Artemis - Large Message - Core => AMQP

Hello,
Sending a message as a Large Message via Core protocol and receiving
it via AMQP on Artemis 2.17.0 ends with receiving the following string
as message body:

"Conversion to AMQP error: Error reading in simpleString, length=y is
greater than readableBytes=x"

Broker lists the message as Type: Default and Large: True.

The receiver is basically Qpid's:
https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py

The doc https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
states both Core and AMQP support Large Message. Is this approach
valid?

Thanks

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
Another possibility would be Stomp... although I have not seen anyone
using large messages with stomp... they will for sure read into memory
before sent to the clients.

On Tue, Jul 26, 2022 at 1:38 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> Either that of have your Java producer to switch to AMQP.
>
> And that is true that the whole message will be read into memory
> before it's converted.
>
> On Tue, Jul 26, 2022 at 12:46 AM Jan Šmucr <Ja...@aimtecglobal.com> wrote:
> >
> > Hello. I'm sorry for the late reply but somehow the message got caught by my spam filter.
> > I believe we'd like to keep the broker from becoming overwhelmed by loading the entire message to memory just for conversion purposes, so if we can avoid it by sticking with the core protocol, I think it's going to be the way to go.
> >
> > Thank you though. 🙂
> > Jan
> >
> > Dne 23. 7. 2022 21:23 napsal uživatel Clebert Suconic <cl...@gmail.com>:
> > If you want help with your core producer to generate a compatible message
> > it’s a different thing. So far the issue was reported as a bug.
> >
> > I can certainly help next week. (From Monday )
> >
> > I will make sure I connect on slack on Monday perhaps we could talk there.
> >
> > On Sat, Jul 23, 2022 at 2:49 PM Clebert Suconic <cl...@gmail.com>
> > wrote:
> >
> > > We could look at enhancing the converters.  But the best would be to
> > > change your producer to have the exact format the converter would have.
> > >
> > > On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> > > wrote:
> > >
> > >> Change your message in a way is compatible ?
> > >>
> > >> On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> > >> wrote:
> > >>
> > >>> The reason for this is that there's a whole infrastructure built using
> > >>> the core protocol, and now we need to connect a Python-based Lambda capable
> > >>> of receiving large messages. Is there any other, core-compatible method?
> > >>>
> > >>> Jan
> > >>>
> > >>> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
> > >>> clebert.suconic@gmail.com>:
> > >>> If you expect conversions to happen, you have to send messages from
> > >>> qpid-ms or core-jms client.
> > >>>
> > >>>
> > >>> On the case:
> > >>>
> > >>>
> > >>> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
> > >>> Connection conn = factory.createConnection(...);
> > >>> Session session = conn.createSession(...)
> > >>> Producer producer = session.createProducer(....);
> > >>> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
> > >>>
> > >>>
> > >>>
> > >>> Then the conversion from Core to AMQP should happen fine when you
> > >>> receive the message on the other side at your Python client.
> > >>>
> > >>>
> > >>>
> > >>> If you really must use the Core-API, then you will need to hack your
> > >>> way in and find what the JMS client would do.. but at that point it's
> > >>> just a non standard way of doing this... You would need to have a very
> > >>> good reason to not do the right thing on this case.
> > >>>
> > >>> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
> > >>> <cl...@gmail.com> wrote:
> > >>> >
> > >>> > If you’re not using qpid JMS it will not work.
> > >>> >
> > >>> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
> > >>> >>
> > >>> >> There is another attempt employing AMQP test structures. It produces
> > >>> the same incorrect message body.
> > >>> >>
> > >>> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com>
> > >>> wrote:
> > >>> >>>
> > >>> >>> Hello,
> > >>> >>> Using the LargeServerMessageImpl was just an attempt providing a
> > >>> seemingly easy way to introduce a kind of Large Message into the test. I'm
> > >>> not familiar with the code base at all - so I simply tried to somehow
> > >>> provide the requested test replicating my "sending via Core -> receiving
> > >>> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> > >>> encountered error simply pointed on it. Is there an intended "test way" to
> > >>> model the message transfer among different clients/protocols?
> > >>> >>>
> > >>> >>> Anyway, I've changed the test to employ
> > >>> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> > >>> LargeServerMessageImpl with a very similar result. Does that implementation
> > >>> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> > >>> better or is the explicit AMQP conversion misused?
> > >>> >>>
> > >>> >>> Please, see the attached patch.
> > >>> >>>
> > >>> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> > >>> clebert.suconic@gmail.com> wrote:
> > >>> >>>>
> > >>> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
> > >>> clients
> > >>> >>>> to generate your message. There are certain caveats that both
> > >>> clients
> > >>> >>>> will add to the messages.
> > >>> >>>>
> > >>> >>>>
> > >>> >>>> Usually I have seen more users doing weird conversions in AMQP when
> > >>> >>>> they use a C++ or non Java Client.. but on your case you used a Core
> > >>> >>>> Client, using a non standard way to send a message (ServerMessage
> > >>> from
> > >>> >>>> the client)... and it's not clear the message is correctly set for a
> > >>> >>>> conversion to work.
> > >>> >>>>
> > >>> >>>>
> > >>> >>>> I need you to provide some explanation on what you're doing.. if
> > >>> this
> > >>> >>>> was  a way to hack a bug your saw or if this is just the way you're
> > >>> >>>> using it.
> > >>> >>>>
> > >>> >>>>
> > >>> >>>> if this is how you're actually using, I would suggest you either use
> > >>> >>>> the proper APIs or if you're doing some hack for performance
> > >>> >>>> consideration you have to set the message in a better way the
> > >>> >>>> converters would understand...
> > >>> >>>>
> > >>> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> > >>> >>>> <cl...@gmail.com> wrote:
> > >>> >>>> >
> > >>> >>>> > Are you using the LargeServerMessageImpl on your client, or that
> > >>> was
> > >>> >>>> > just a "hack" to reproduce your issue.
> > >>> >>>> >
> > >>> >>>> >
> > >>> >>>> > LargeServerMessageImpl was not meant to be used on the client.
> > >>> But if
> > >>> >>>> > you're doing that just to show something you faced in production
> > >>> it's
> > >>> >>>> > ok.. but using the LargeServerMessageImpl to send a Client
> > >>> Message is
> > >>> >>>> > a big not for me.
> > >>> >>>> >
> > >>> >>>> > The only reason we do that is for server to server transfer.
> > >>> >>>> >
> > >>> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
> > >>> wrote:
> > >>> >>>> > >
> > >>> >>>> > > Hello,
> > >>> >>>> > > Yes, the 2.23.1 test instance was freshly created.
> > >>> >>>> > >
> > >>> >>>> > > I've attached a test as a part of ARTEMIS-3897
> > >>> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope
> > >>> it helps.
> > >>> >>>> > >
> > >>> >>>> > >
> > >>> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> > >>> clebert.suconic@gmail.com>
> > >>> >>>> > > wrote:
> > >>> >>>> > >
> > >>> >>>> > > > Did you start with fresh data on 2.23.1.
> > >>> >>>> > > >
> > >>> >>>> > > > If you did please provide a self enclosing test reproducing
> > >>> your issue.
> > >>> >>>> > > >
> > >>> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> > >>> wrote:
> > >>> >>>> > > >
> > >>> >>>> > > > > A quick test using 2.23.1 results in the same error payload
> > >>> being
> > >>> >>>> > > > received:
> > >>> >>>> > > > >
> > >>> >>>> > > > > "Message(address='test', durable=True, priority=4,
> > >>> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
> > >>> byte(0),
> > >>> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> > >>> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
> > >>> 67},
> > >>> >>>> > > > > body='Conversion to AMQP error: Error reading in
> > >>> simpleString,
> > >>> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
> > >>> >>>> > > > >
> > >>> >>>> > > > > Best regards
> > >>> >>>> > > > >
> > >>> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> > >>> >>>> > > > > <cl...@gmail.com> wrote:
> > >>> >>>> > > > > >
> > >>> >>>> > > > > > There's been a few fixes in AMQP Large message.
> > >>> >>>> > > > > >
> > >>> >>>> > > > > >
> > >>> >>>> > > > > > More prominently a fix with the JDBC implementation
> > >>> between 2.17 and
> > >>> >>>> > > > > HEAD.
> > >>> >>>> > > > > >
> > >>> >>>> > > > > >
> > >>> >>>> > > > > > I would definitely recommend you to upgrade.
> > >>> >>>> > > > > >
> > >>> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> > >>> jbertram@apache.org>
> > >>> >>>> > > > > wrote:
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > > I would expect this to work. Can you try this on the
> > >>> latest release
> > >>> >>>> > > > > (i.e.
> > >>> >>>> > > > > > > 2.23.1) [1]?
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > > If it still doesn't work please open a Jira with all
> > >>> the details
> > >>> >>>> > > > > necessary
> > >>> >>>> > > > > > > to reproduce the problem. An actual test-case would be
> > >>> ideal.
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > > Thanks!
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > > Justin
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > > [1]
> > >>> https://activemq.apache.org/components/artemis/download/
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> > >>> andyyar66@gmail.com>
> > >>> >>>> > > > wrote:
> > >>> >>>> > > > > > >
> > >>> >>>> > > > > > > > Hello,
> > >>> >>>> > > > > > > > Sending a message as a Large Message via Core
> > >>> protocol and
> > >>> >>>> > > > receiving
> > >>> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> > >>> following
> > >>> >>>> > > > > string
> > >>> >>>> > > > > > > > as message body:
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
> > >>> simpleString, length=y
> > >>> >>>> > > > is
> > >>> >>>> > > > > > > > greater than readableBytes=x"
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
> > >>> True.
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > > > > The receiver is basically Qpid's:
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > >
> > >>> >>>> > > >
> > >>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > > > > The doc
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > >
> > >>> >>>> > > >
> > >>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > >>> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
> > >>> this approach
> > >>> >>>> > > > > > > > valid?
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > > > > Thanks
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > > > >
> > >>> >>>> > > > > >
> > >>> >>>> > > > > >
> > >>> >>>> > > > > >
> > >>> >>>> > > > > > --
> > >>> >>>> > > > > > Clebert Suconic
> > >>> >>>> > > > >
> > >>> >>>> > > > --
> > >>> >>>> > > > Clebert Suconic
> > >>> >>>> > > >
> > >>> >>>> >
> > >>> >>>> >
> > >>> >>>> >
> > >>> >>>> > --
> > >>> >>>> > Clebert Suconic
> > >>> >>>>
> > >>> >>>>
> > >>> >>>>
> > >>> >>>> --
> > >>> >>>> Clebert Suconic
> > >>> >
> > >>> > --
> > >>> > Clebert Suconic
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Clebert Suconic
> > >>>
> > >> --
> > >> Clebert Suconic
> > >>
> > > --
> > > Clebert Suconic
> > >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
Either that of have your Java producer to switch to AMQP.

And that is true that the whole message will be read into memory
before it's converted.

On Tue, Jul 26, 2022 at 12:46 AM Jan Šmucr <Ja...@aimtecglobal.com> wrote:
>
> Hello. I'm sorry for the late reply but somehow the message got caught by my spam filter.
> I believe we'd like to keep the broker from becoming overwhelmed by loading the entire message to memory just for conversion purposes, so if we can avoid it by sticking with the core protocol, I think it's going to be the way to go.
>
> Thank you though. 🙂
> Jan
>
> Dne 23. 7. 2022 21:23 napsal uživatel Clebert Suconic <cl...@gmail.com>:
> If you want help with your core producer to generate a compatible message
> it’s a different thing. So far the issue was reported as a bug.
>
> I can certainly help next week. (From Monday )
>
> I will make sure I connect on slack on Monday perhaps we could talk there.
>
> On Sat, Jul 23, 2022 at 2:49 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > We could look at enhancing the converters.  But the best would be to
> > change your producer to have the exact format the converter would have.
> >
> > On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> > wrote:
> >
> >> Change your message in a way is compatible ?
> >>
> >> On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> >> wrote:
> >>
> >>> The reason for this is that there's a whole infrastructure built using
> >>> the core protocol, and now we need to connect a Python-based Lambda capable
> >>> of receiving large messages. Is there any other, core-compatible method?
> >>>
> >>> Jan
> >>>
> >>> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
> >>> clebert.suconic@gmail.com>:
> >>> If you expect conversions to happen, you have to send messages from
> >>> qpid-ms or core-jms client.
> >>>
> >>>
> >>> On the case:
> >>>
> >>>
> >>> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
> >>> Connection conn = factory.createConnection(...);
> >>> Session session = conn.createSession(...)
> >>> Producer producer = session.createProducer(....);
> >>> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
> >>>
> >>>
> >>>
> >>> Then the conversion from Core to AMQP should happen fine when you
> >>> receive the message on the other side at your Python client.
> >>>
> >>>
> >>>
> >>> If you really must use the Core-API, then you will need to hack your
> >>> way in and find what the JMS client would do.. but at that point it's
> >>> just a non standard way of doing this... You would need to have a very
> >>> good reason to not do the right thing on this case.
> >>>
> >>> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
> >>> <cl...@gmail.com> wrote:
> >>> >
> >>> > If you’re not using qpid JMS it will not work.
> >>> >
> >>> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
> >>> >>
> >>> >> There is another attempt employing AMQP test structures. It produces
> >>> the same incorrect message body.
> >>> >>
> >>> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com>
> >>> wrote:
> >>> >>>
> >>> >>> Hello,
> >>> >>> Using the LargeServerMessageImpl was just an attempt providing a
> >>> seemingly easy way to introduce a kind of Large Message into the test. I'm
> >>> not familiar with the code base at all - so I simply tried to somehow
> >>> provide the requested test replicating my "sending via Core -> receiving
> >>> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> >>> encountered error simply pointed on it. Is there an intended "test way" to
> >>> model the message transfer among different clients/protocols?
> >>> >>>
> >>> >>> Anyway, I've changed the test to employ
> >>> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> >>> LargeServerMessageImpl with a very similar result. Does that implementation
> >>> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> >>> better or is the explicit AMQP conversion misused?
> >>> >>>
> >>> >>> Please, see the attached patch.
> >>> >>>
> >>> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> >>> clebert.suconic@gmail.com> wrote:
> >>> >>>>
> >>> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
> >>> clients
> >>> >>>> to generate your message. There are certain caveats that both
> >>> clients
> >>> >>>> will add to the messages.
> >>> >>>>
> >>> >>>>
> >>> >>>> Usually I have seen more users doing weird conversions in AMQP when
> >>> >>>> they use a C++ or non Java Client.. but on your case you used a Core
> >>> >>>> Client, using a non standard way to send a message (ServerMessage
> >>> from
> >>> >>>> the client)... and it's not clear the message is correctly set for a
> >>> >>>> conversion to work.
> >>> >>>>
> >>> >>>>
> >>> >>>> I need you to provide some explanation on what you're doing.. if
> >>> this
> >>> >>>> was  a way to hack a bug your saw or if this is just the way you're
> >>> >>>> using it.
> >>> >>>>
> >>> >>>>
> >>> >>>> if this is how you're actually using, I would suggest you either use
> >>> >>>> the proper APIs or if you're doing some hack for performance
> >>> >>>> consideration you have to set the message in a better way the
> >>> >>>> converters would understand...
> >>> >>>>
> >>> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> >>> >>>> <cl...@gmail.com> wrote:
> >>> >>>> >
> >>> >>>> > Are you using the LargeServerMessageImpl on your client, or that
> >>> was
> >>> >>>> > just a "hack" to reproduce your issue.
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > LargeServerMessageImpl was not meant to be used on the client.
> >>> But if
> >>> >>>> > you're doing that just to show something you faced in production
> >>> it's
> >>> >>>> > ok.. but using the LargeServerMessageImpl to send a Client
> >>> Message is
> >>> >>>> > a big not for me.
> >>> >>>> >
> >>> >>>> > The only reason we do that is for server to server transfer.
> >>> >>>> >
> >>> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
> >>> wrote:
> >>> >>>> > >
> >>> >>>> > > Hello,
> >>> >>>> > > Yes, the 2.23.1 test instance was freshly created.
> >>> >>>> > >
> >>> >>>> > > I've attached a test as a part of ARTEMIS-3897
> >>> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope
> >>> it helps.
> >>> >>>> > >
> >>> >>>> > >
> >>> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> >>> clebert.suconic@gmail.com>
> >>> >>>> > > wrote:
> >>> >>>> > >
> >>> >>>> > > > Did you start with fresh data on 2.23.1.
> >>> >>>> > > >
> >>> >>>> > > > If you did please provide a self enclosing test reproducing
> >>> your issue.
> >>> >>>> > > >
> >>> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> >>> wrote:
> >>> >>>> > > >
> >>> >>>> > > > > A quick test using 2.23.1 results in the same error payload
> >>> being
> >>> >>>> > > > received:
> >>> >>>> > > > >
> >>> >>>> > > > > "Message(address='test', durable=True, priority=4,
> >>> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
> >>> byte(0),
> >>> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> >>> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
> >>> 67},
> >>> >>>> > > > > body='Conversion to AMQP error: Error reading in
> >>> simpleString,
> >>> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
> >>> >>>> > > > >
> >>> >>>> > > > > Best regards
> >>> >>>> > > > >
> >>> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> >>> >>>> > > > > <cl...@gmail.com> wrote:
> >>> >>>> > > > > >
> >>> >>>> > > > > > There's been a few fixes in AMQP Large message.
> >>> >>>> > > > > >
> >>> >>>> > > > > >
> >>> >>>> > > > > > More prominently a fix with the JDBC implementation
> >>> between 2.17 and
> >>> >>>> > > > > HEAD.
> >>> >>>> > > > > >
> >>> >>>> > > > > >
> >>> >>>> > > > > > I would definitely recommend you to upgrade.
> >>> >>>> > > > > >
> >>> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> >>> jbertram@apache.org>
> >>> >>>> > > > > wrote:
> >>> >>>> > > > > > >
> >>> >>>> > > > > > > I would expect this to work. Can you try this on the
> >>> latest release
> >>> >>>> > > > > (i.e.
> >>> >>>> > > > > > > 2.23.1) [1]?
> >>> >>>> > > > > > >
> >>> >>>> > > > > > > If it still doesn't work please open a Jira with all
> >>> the details
> >>> >>>> > > > > necessary
> >>> >>>> > > > > > > to reproduce the problem. An actual test-case would be
> >>> ideal.
> >>> >>>> > > > > > >
> >>> >>>> > > > > > > Thanks!
> >>> >>>> > > > > > >
> >>> >>>> > > > > > >
> >>> >>>> > > > > > > Justin
> >>> >>>> > > > > > >
> >>> >>>> > > > > > > [1]
> >>> https://activemq.apache.org/components/artemis/download/
> >>> >>>> > > > > > >
> >>> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> >>> andyyar66@gmail.com>
> >>> >>>> > > > wrote:
> >>> >>>> > > > > > >
> >>> >>>> > > > > > > > Hello,
> >>> >>>> > > > > > > > Sending a message as a Large Message via Core
> >>> protocol and
> >>> >>>> > > > receiving
> >>> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> >>> following
> >>> >>>> > > > > string
> >>> >>>> > > > > > > > as message body:
> >>> >>>> > > > > > > >
> >>> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
> >>> simpleString, length=y
> >>> >>>> > > > is
> >>> >>>> > > > > > > > greater than readableBytes=x"
> >>> >>>> > > > > > > >
> >>> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
> >>> True.
> >>> >>>> > > > > > > >
> >>> >>>> > > > > > > > The receiver is basically Qpid's:
> >>> >>>> > > > > > > >
> >>> >>>> > > > > > > >
> >>> >>>> > > > >
> >>> >>>> > > >
> >>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> >>> >>>> > > > > > > >
> >>> >>>> > > > > > > > The doc
> >>> >>>> > > > > > > >
> >>> >>>> > > > >
> >>> >>>> > > >
> >>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> >>> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
> >>> this approach
> >>> >>>> > > > > > > > valid?
> >>> >>>> > > > > > > >
> >>> >>>> > > > > > > > Thanks
> >>> >>>> > > > > > > >
> >>> >>>> > > > > > > >
> >>> >>>> > > > > >
> >>> >>>> > > > > >
> >>> >>>> > > > > >
> >>> >>>> > > > > > --
> >>> >>>> > > > > > Clebert Suconic
> >>> >>>> > > > >
> >>> >>>> > > > --
> >>> >>>> > > > Clebert Suconic
> >>> >>>> > > >
> >>> >>>> >
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > --
> >>> >>>> > Clebert Suconic
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> --
> >>> >>>> Clebert Suconic
> >>> >
> >>> > --
> >>> > Clebert Suconic
> >>>
> >>>
> >>>
> >>> --
> >>> Clebert Suconic
> >>>
> >> --
> >> Clebert Suconic
> >>
> > --
> > Clebert Suconic
> >
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Jan Šmucr <Ja...@aimtecglobal.com>.
Hello. I'm sorry for the late reply but somehow the message got caught by my spam filter.
I believe we'd like to keep the broker from becoming overwhelmed by loading the entire message to memory just for conversion purposes, so if we can avoid it by sticking with the core protocol, I think it's going to be the way to go.

Thank you though. 🙂
Jan

Dne 23. 7. 2022 21:23 napsal uživatel Clebert Suconic <cl...@gmail.com>:
If you want help with your core producer to generate a compatible message
it’s a different thing. So far the issue was reported as a bug.

I can certainly help next week. (From Monday )

I will make sure I connect on slack on Monday perhaps we could talk there.

On Sat, Jul 23, 2022 at 2:49 PM Clebert Suconic <cl...@gmail.com>
wrote:

> We could look at enhancing the converters.  But the best would be to
> change your producer to have the exact format the converter would have.
>
> On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> Change your message in a way is compatible ?
>>
>> On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
>> wrote:
>>
>>> The reason for this is that there's a whole infrastructure built using
>>> the core protocol, and now we need to connect a Python-based Lambda capable
>>> of receiving large messages. Is there any other, core-compatible method?
>>>
>>> Jan
>>>
>>> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
>>> clebert.suconic@gmail.com>:
>>> If you expect conversions to happen, you have to send messages from
>>> qpid-ms or core-jms client.
>>>
>>>
>>> On the case:
>>>
>>>
>>> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
>>> Connection conn = factory.createConnection(...);
>>> Session session = conn.createSession(...)
>>> Producer producer = session.createProducer(....);
>>> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
>>>
>>>
>>>
>>> Then the conversion from Core to AMQP should happen fine when you
>>> receive the message on the other side at your Python client.
>>>
>>>
>>>
>>> If you really must use the Core-API, then you will need to hack your
>>> way in and find what the JMS client would do.. but at that point it's
>>> just a non standard way of doing this... You would need to have a very
>>> good reason to not do the right thing on this case.
>>>
>>> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
>>> <cl...@gmail.com> wrote:
>>> >
>>> > If you’re not using qpid JMS it will not work.
>>> >
>>> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
>>> >>
>>> >> There is another attempt employing AMQP test structures. It produces
>>> the same incorrect message body.
>>> >>
>>> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> Hello,
>>> >>> Using the LargeServerMessageImpl was just an attempt providing a
>>> seemingly easy way to introduce a kind of Large Message into the test. I'm
>>> not familiar with the code base at all - so I simply tried to somehow
>>> provide the requested test replicating my "sending via Core -> receiving
>>> via Qpid" issue. The same applies to the explicit AMQP conversion - the
>>> encountered error simply pointed on it. Is there an intended "test way" to
>>> model the message transfer among different clients/protocols?
>>> >>>
>>> >>> Anyway, I've changed the test to employ
>>> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
>>> LargeServerMessageImpl with a very similar result. Does that implementation
>>> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
>>> better or is the explicit AMQP conversion misused?
>>> >>>
>>> >>> Please, see the attached patch.
>>> >>>
>>> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
>>> clebert.suconic@gmail.com> wrote:
>>> >>>>
>>> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
>>> clients
>>> >>>> to generate your message. There are certain caveats that both
>>> clients
>>> >>>> will add to the messages.
>>> >>>>
>>> >>>>
>>> >>>> Usually I have seen more users doing weird conversions in AMQP when
>>> >>>> they use a C++ or non Java Client.. but on your case you used a Core
>>> >>>> Client, using a non standard way to send a message (ServerMessage
>>> from
>>> >>>> the client)... and it's not clear the message is correctly set for a
>>> >>>> conversion to work.
>>> >>>>
>>> >>>>
>>> >>>> I need you to provide some explanation on what you're doing.. if
>>> this
>>> >>>> was  a way to hack a bug your saw or if this is just the way you're
>>> >>>> using it.
>>> >>>>
>>> >>>>
>>> >>>> if this is how you're actually using, I would suggest you either use
>>> >>>> the proper APIs or if you're doing some hack for performance
>>> >>>> consideration you have to set the message in a better way the
>>> >>>> converters would understand...
>>> >>>>
>>> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>>> >>>> <cl...@gmail.com> wrote:
>>> >>>> >
>>> >>>> > Are you using the LargeServerMessageImpl on your client, or that
>>> was
>>> >>>> > just a "hack" to reproduce your issue.
>>> >>>> >
>>> >>>> >
>>> >>>> > LargeServerMessageImpl was not meant to be used on the client.
>>> But if
>>> >>>> > you're doing that just to show something you faced in production
>>> it's
>>> >>>> > ok.. but using the LargeServerMessageImpl to send a Client
>>> Message is
>>> >>>> > a big not for me.
>>> >>>> >
>>> >>>> > The only reason we do that is for server to server transfer.
>>> >>>> >
>>> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
>>> wrote:
>>> >>>> > >
>>> >>>> > > Hello,
>>> >>>> > > Yes, the 2.23.1 test instance was freshly created.
>>> >>>> > >
>>> >>>> > > I've attached a test as a part of ARTEMIS-3897
>>> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope
>>> it helps.
>>> >>>> > >
>>> >>>> > >
>>> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> >>>> > > wrote:
>>> >>>> > >
>>> >>>> > > > Did you start with fresh data on 2.23.1.
>>> >>>> > > >
>>> >>>> > > > If you did please provide a self enclosing test reproducing
>>> your issue.
>>> >>>> > > >
>>> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
>>> wrote:
>>> >>>> > > >
>>> >>>> > > > > A quick test using 2.23.1 results in the same error payload
>>> being
>>> >>>> > > > received:
>>> >>>> > > > >
>>> >>>> > > > > "Message(address='test', durable=True, priority=4,
>>> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
>>> byte(0),
>>> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>>> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
>>> 67},
>>> >>>> > > > > body='Conversion to AMQP error: Error reading in
>>> simpleString,
>>> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
>>> >>>> > > > >
>>> >>>> > > > > Best regards
>>> >>>> > > > >
>>> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>>> >>>> > > > > <cl...@gmail.com> wrote:
>>> >>>> > > > > >
>>> >>>> > > > > > There's been a few fixes in AMQP Large message.
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > > More prominently a fix with the JDBC implementation
>>> between 2.17 and
>>> >>>> > > > > HEAD.
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > > I would definitely recommend you to upgrade.
>>> >>>> > > > > >
>>> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
>>> jbertram@apache.org>
>>> >>>> > > > > wrote:
>>> >>>> > > > > > >
>>> >>>> > > > > > > I would expect this to work. Can you try this on the
>>> latest release
>>> >>>> > > > > (i.e.
>>> >>>> > > > > > > 2.23.1) [1]?
>>> >>>> > > > > > >
>>> >>>> > > > > > > If it still doesn't work please open a Jira with all
>>> the details
>>> >>>> > > > > necessary
>>> >>>> > > > > > > to reproduce the problem. An actual test-case would be
>>> ideal.
>>> >>>> > > > > > >
>>> >>>> > > > > > > Thanks!
>>> >>>> > > > > > >
>>> >>>> > > > > > >
>>> >>>> > > > > > > Justin
>>> >>>> > > > > > >
>>> >>>> > > > > > > [1]
>>> https://activemq.apache.org/components/artemis/download/
>>> >>>> > > > > > >
>>> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
>>> andyyar66@gmail.com>
>>> >>>> > > > wrote:
>>> >>>> > > > > > >
>>> >>>> > > > > > > > Hello,
>>> >>>> > > > > > > > Sending a message as a Large Message via Core
>>> protocol and
>>> >>>> > > > receiving
>>> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
>>> following
>>> >>>> > > > > string
>>> >>>> > > > > > > > as message body:
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
>>> simpleString, length=y
>>> >>>> > > > is
>>> >>>> > > > > > > > greater than readableBytes=x"
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
>>> True.
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > The receiver is basically Qpid's:
>>> >>>> > > > > > > >
>>> >>>> > > > > > > >
>>> >>>> > > > >
>>> >>>> > > >
>>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > The doc
>>> >>>> > > > > > > >
>>> >>>> > > > >
>>> >>>> > > >
>>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>>> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
>>> this approach
>>> >>>> > > > > > > > valid?
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > Thanks
>>> >>>> > > > > > > >
>>> >>>> > > > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > > --
>>> >>>> > > > > > Clebert Suconic
>>> >>>> > > > >
>>> >>>> > > > --
>>> >>>> > > > Clebert Suconic
>>> >>>> > > >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > --
>>> >>>> > Clebert Suconic
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Clebert Suconic
>>> >
>>> > --
>>> > Clebert Suconic
>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>>
>> --
>> Clebert Suconic
>>
> --
> Clebert Suconic
>
--
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
If you want help with your core producer to generate a compatible message
it’s a different thing. So far the issue was reported as a bug.

I can certainly help next week. (From Monday )

I will make sure I connect on slack on Monday perhaps we could talk there.

On Sat, Jul 23, 2022 at 2:49 PM Clebert Suconic <cl...@gmail.com>
wrote:

> We could look at enhancing the converters.  But the best would be to
> change your producer to have the exact format the converter would have.
>
> On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> Change your message in a way is compatible ?
>>
>> On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
>> wrote:
>>
>>> The reason for this is that there's a whole infrastructure built using
>>> the core protocol, and now we need to connect a Python-based Lambda capable
>>> of receiving large messages. Is there any other, core-compatible method?
>>>
>>> Jan
>>>
>>> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
>>> clebert.suconic@gmail.com>:
>>> If you expect conversions to happen, you have to send messages from
>>> qpid-ms or core-jms client.
>>>
>>>
>>> On the case:
>>>
>>>
>>> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
>>> Connection conn = factory.createConnection(...);
>>> Session session = conn.createSession(...)
>>> Producer producer = session.createProducer(....);
>>> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
>>>
>>>
>>>
>>> Then the conversion from Core to AMQP should happen fine when you
>>> receive the message on the other side at your Python client.
>>>
>>>
>>>
>>> If you really must use the Core-API, then you will need to hack your
>>> way in and find what the JMS client would do.. but at that point it's
>>> just a non standard way of doing this... You would need to have a very
>>> good reason to not do the right thing on this case.
>>>
>>> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
>>> <cl...@gmail.com> wrote:
>>> >
>>> > If you’re not using qpid JMS it will not work.
>>> >
>>> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
>>> >>
>>> >> There is another attempt employing AMQP test structures. It produces
>>> the same incorrect message body.
>>> >>
>>> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> Hello,
>>> >>> Using the LargeServerMessageImpl was just an attempt providing a
>>> seemingly easy way to introduce a kind of Large Message into the test. I'm
>>> not familiar with the code base at all - so I simply tried to somehow
>>> provide the requested test replicating my "sending via Core -> receiving
>>> via Qpid" issue. The same applies to the explicit AMQP conversion - the
>>> encountered error simply pointed on it. Is there an intended "test way" to
>>> model the message transfer among different clients/protocols?
>>> >>>
>>> >>> Anyway, I've changed the test to employ
>>> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
>>> LargeServerMessageImpl with a very similar result. Does that implementation
>>> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
>>> better or is the explicit AMQP conversion misused?
>>> >>>
>>> >>> Please, see the attached patch.
>>> >>>
>>> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
>>> clebert.suconic@gmail.com> wrote:
>>> >>>>
>>> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
>>> clients
>>> >>>> to generate your message. There are certain caveats that both
>>> clients
>>> >>>> will add to the messages.
>>> >>>>
>>> >>>>
>>> >>>> Usually I have seen more users doing weird conversions in AMQP when
>>> >>>> they use a C++ or non Java Client.. but on your case you used a Core
>>> >>>> Client, using a non standard way to send a message (ServerMessage
>>> from
>>> >>>> the client)... and it's not clear the message is correctly set for a
>>> >>>> conversion to work.
>>> >>>>
>>> >>>>
>>> >>>> I need you to provide some explanation on what you're doing.. if
>>> this
>>> >>>> was  a way to hack a bug your saw or if this is just the way you're
>>> >>>> using it.
>>> >>>>
>>> >>>>
>>> >>>> if this is how you're actually using, I would suggest you either use
>>> >>>> the proper APIs or if you're doing some hack for performance
>>> >>>> consideration you have to set the message in a better way the
>>> >>>> converters would understand...
>>> >>>>
>>> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>>> >>>> <cl...@gmail.com> wrote:
>>> >>>> >
>>> >>>> > Are you using the LargeServerMessageImpl on your client, or that
>>> was
>>> >>>> > just a "hack" to reproduce your issue.
>>> >>>> >
>>> >>>> >
>>> >>>> > LargeServerMessageImpl was not meant to be used on the client.
>>> But if
>>> >>>> > you're doing that just to show something you faced in production
>>> it's
>>> >>>> > ok.. but using the LargeServerMessageImpl to send a Client
>>> Message is
>>> >>>> > a big not for me.
>>> >>>> >
>>> >>>> > The only reason we do that is for server to server transfer.
>>> >>>> >
>>> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
>>> wrote:
>>> >>>> > >
>>> >>>> > > Hello,
>>> >>>> > > Yes, the 2.23.1 test instance was freshly created.
>>> >>>> > >
>>> >>>> > > I've attached a test as a part of ARTEMIS-3897
>>> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope
>>> it helps.
>>> >>>> > >
>>> >>>> > >
>>> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> >>>> > > wrote:
>>> >>>> > >
>>> >>>> > > > Did you start with fresh data on 2.23.1.
>>> >>>> > > >
>>> >>>> > > > If you did please provide a self enclosing test reproducing
>>> your issue.
>>> >>>> > > >
>>> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
>>> wrote:
>>> >>>> > > >
>>> >>>> > > > > A quick test using 2.23.1 results in the same error payload
>>> being
>>> >>>> > > > received:
>>> >>>> > > > >
>>> >>>> > > > > "Message(address='test', durable=True, priority=4,
>>> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
>>> byte(0),
>>> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>>> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
>>> 67},
>>> >>>> > > > > body='Conversion to AMQP error: Error reading in
>>> simpleString,
>>> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
>>> >>>> > > > >
>>> >>>> > > > > Best regards
>>> >>>> > > > >
>>> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>>> >>>> > > > > <cl...@gmail.com> wrote:
>>> >>>> > > > > >
>>> >>>> > > > > > There's been a few fixes in AMQP Large message.
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > > More prominently a fix with the JDBC implementation
>>> between 2.17 and
>>> >>>> > > > > HEAD.
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > > I would definitely recommend you to upgrade.
>>> >>>> > > > > >
>>> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
>>> jbertram@apache.org>
>>> >>>> > > > > wrote:
>>> >>>> > > > > > >
>>> >>>> > > > > > > I would expect this to work. Can you try this on the
>>> latest release
>>> >>>> > > > > (i.e.
>>> >>>> > > > > > > 2.23.1) [1]?
>>> >>>> > > > > > >
>>> >>>> > > > > > > If it still doesn't work please open a Jira with all
>>> the details
>>> >>>> > > > > necessary
>>> >>>> > > > > > > to reproduce the problem. An actual test-case would be
>>> ideal.
>>> >>>> > > > > > >
>>> >>>> > > > > > > Thanks!
>>> >>>> > > > > > >
>>> >>>> > > > > > >
>>> >>>> > > > > > > Justin
>>> >>>> > > > > > >
>>> >>>> > > > > > > [1]
>>> https://activemq.apache.org/components/artemis/download/
>>> >>>> > > > > > >
>>> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
>>> andyyar66@gmail.com>
>>> >>>> > > > wrote:
>>> >>>> > > > > > >
>>> >>>> > > > > > > > Hello,
>>> >>>> > > > > > > > Sending a message as a Large Message via Core
>>> protocol and
>>> >>>> > > > receiving
>>> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
>>> following
>>> >>>> > > > > string
>>> >>>> > > > > > > > as message body:
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
>>> simpleString, length=y
>>> >>>> > > > is
>>> >>>> > > > > > > > greater than readableBytes=x"
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
>>> True.
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > The receiver is basically Qpid's:
>>> >>>> > > > > > > >
>>> >>>> > > > > > > >
>>> >>>> > > > >
>>> >>>> > > >
>>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > The doc
>>> >>>> > > > > > > >
>>> >>>> > > > >
>>> >>>> > > >
>>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>>> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
>>> this approach
>>> >>>> > > > > > > > valid?
>>> >>>> > > > > > > >
>>> >>>> > > > > > > > Thanks
>>> >>>> > > > > > > >
>>> >>>> > > > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > >
>>> >>>> > > > > > --
>>> >>>> > > > > > Clebert Suconic
>>> >>>> > > > >
>>> >>>> > > > --
>>> >>>> > > > Clebert Suconic
>>> >>>> > > >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > --
>>> >>>> > Clebert Suconic
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Clebert Suconic
>>> >
>>> > --
>>> > Clebert Suconic
>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>>
>> --
>> Clebert Suconic
>>
> --
> Clebert Suconic
>
-- 
Clebert Suconic

RE: Artemis - Large Message - Core => AMQP

Posted by Jan Šmucr <Ja...@aimtecglobal.com>.
Yes, pretty much.

Jan

From: Robbie Gemmell<ma...@gmail.com>
Sent: pondělí 25. července 2022 14:04
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Artemis - Large Message - Core => AMQP

By "in Java" I assume you more specifically mean 'using the same Core
client as the existing senders/receivers' ?

On Mon, 25 Jul 2022 at 11:10, Jan Šmucr <Ja...@aimtecglobal.com> wrote:
>
> > For the broker, it would ordinarily treat the large message as a
> > chunked stream if doing e.g core->core, or amqp->amqp...but in this
> > case having to do core->amqp means the broker will necessarily have to
> > convert the message and in doing so it will load the entire message.
>
> I think this is exactly what we needed to know to drop Python support and create the Lambda in Java. Thank you! 😊
>
> Jan
>
> From: Robbie Gemmell<ma...@gmail.com>
> Sent: pondělí 25. července 2022 12:05
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Artemis - Large Message - Core => AMQP
>
> Are you using StreamMessage because you think it will enable
> 'steaming'? If so I wouldnt bother, the Stream name isnt related, its
> just a JMS message type name meaning there is a list/sequence of
> things in it. Previously you just seemed to want to send text/bytes.
>
> For the Core client side you can see
> https://activemq.apache.org/components/artemis/documentation/latest/large-messages.html
>
> For the broker, it would ordinarily treat the large message as a
> chunked stream if doing e.g core->core, or amqp->amqp...but in this
> case having to do core->amqp means the broker will necessarily have to
> convert the message and in doing so it will load the entire message.
>
> (It seems like its that last step currently not handling your sent
> Core messages, because you are sending raw Core payloads that dont
> match the JMS style Core message structure the brokers core->AMQP
> converter is geared around, meaning it falls back to one that tries to
> convert it as a string, which presumably fails due to starting to read
> your bare payload as a size indicator)
>
> On Mon, 25 Jul 2022 at 06:40, Jan Šmucr <Ja...@aimtecglobal.com> wrote:
> >
> > Thank you for the feedback.
> > So what would this be the proper way of streaming files (if the code below is correct)?
> >
> >       final String body = "foo";
> >
> > try (Connection connection = createConnection()) {
> >          Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
> >          MessageProducer producer = session.createProducer(session.createQueue(getQueueName()));
> >          StreamMessage message = session.createStreamMessage();
> >          message.writeBytes(body.getBytes());
> >          producer.send(message);
> >       }
> >
> >       AmqpClient client = createAmqpClient();
> >       AmqpConnection amqpConnection = addConnection(client.connect());
> >       AmqpSession amqpSession = amqpConnection.createSession();
> >       AmqpReceiver receiver = amqpSession.createReceiver(getQueueName());
> >       receiver.flow(100);
> >       AmqpMessage amqpMessage = receiver.receive(1, TimeUnit.SECONDS);
> >       Assert.assertNotNull(amqpMessage);
> >       Assert.assertEquals(body, new String(((Binary) ((AmqpSequence) amqpMessage.getWrappedMessage().getBody()).getValue().get(0)).getArray()));
> >
> > We want to stream the payloads due to the fact that these can take up to hundreds of megabytes.
> >
> > Thank you.
> > Jan
> >
> > From: Clebert Suconic<ma...@gmail.com>
> > Sent: sobota 23. července 2022 20:50
> > To: users@activemq.apache.org<ma...@activemq.apache.org>
> > Subject: Re: Artemis - Large Message - Core => AMQP
> >
> > We could look at enhancing the converters.  But the best would be to change
> > your producer to have the exact format the converter would have.
> >
> > On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> > wrote:
> >
> > > Change your message in a way is compatible ?
> > >
> > > On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> > > wrote:
> > >
> > >> The reason for this is that there's a whole infrastructure built using
> > >> the core protocol, and now we need to connect a Python-based Lambda capable
> > >> of receiving large messages. Is there any other, core-compatible method?
> > >>
> > >> Jan
> > >>
> > >> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
> > >> clebert.suconic@gmail.com>:
> > >> If you expect conversions to happen, you have to send messages from
> > >> qpid-ms or core-jms client.
> > >>
> > >>
> > >> On the case:
> > >>
> > >>
> > >> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
> > >> Connection conn = factory.createConnection(...);
> > >> Session session = conn.createSession(...)
> > >> Producer producer = session.createProducer(....);
> > >> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
> > >>
> > >>
> > >>
> > >> Then the conversion from Core to AMQP should happen fine when you
> > >> receive the message on the other side at your Python client.
> > >>
> > >>
> > >>
> > >> If you really must use the Core-API, then you will need to hack your
> > >> way in and find what the JMS client would do.. but at that point it's
> > >> just a non standard way of doing this... You would need to have a very
> > >> good reason to not do the right thing on this case.
> > >>
> > >> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
> > >> <cl...@gmail.com> wrote:
> > >> >
> > >> > If you’re not using qpid JMS it will not work.
> > >> >
> > >> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
> > >> >>
> > >> >> There is another attempt employing AMQP test structures. It produces
> > >> the same incorrect message body.
> > >> >>
> > >> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
> > >> >>>
> > >> >>> Hello,
> > >> >>> Using the LargeServerMessageImpl was just an attempt providing a
> > >> seemingly easy way to introduce a kind of Large Message into the test. I'm
> > >> not familiar with the code base at all - so I simply tried to somehow
> > >> provide the requested test replicating my "sending via Core -> receiving
> > >> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> > >> encountered error simply pointed on it. Is there an intended "test way" to
> > >> model the message transfer among different clients/protocols?
> > >> >>>
> > >> >>> Anyway, I've changed the test to employ
> > >> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> > >> LargeServerMessageImpl with a very similar result. Does that implementation
> > >> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> > >> better or is the explicit AMQP conversion misused?
> > >> >>>
> > >> >>> Please, see the attached patch.
> > >> >>>
> > >> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> > >> clebert.suconic@gmail.com> wrote:
> > >> >>>>
> > >> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
> > >> clients
> > >> >>>> to generate your message. There are certain caveats that both clients
> > >> >>>> will add to the messages.
> > >> >>>>
> > >> >>>>
> > >> >>>> Usually I have seen more users doing weird conversions in AMQP when
> > >> >>>> they use a C++ or non Java Client.. but on your case you used a Core
> > >> >>>> Client, using a non standard way to send a message (ServerMessage
> > >> from
> > >> >>>> the client)... and it's not clear the message is correctly set for a
> > >> >>>> conversion to work.
> > >> >>>>
> > >> >>>>
> > >> >>>> I need you to provide some explanation on what you're doing.. if this
> > >> >>>> was  a way to hack a bug your saw or if this is just the way you're
> > >> >>>> using it.
> > >> >>>>
> > >> >>>>
> > >> >>>> if this is how you're actually using, I would suggest you either use
> > >> >>>> the proper APIs or if you're doing some hack for performance
> > >> >>>> consideration you have to set the message in a better way the
> > >> >>>> converters would understand...
> > >> >>>>
> > >> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> > >> >>>> <cl...@gmail.com> wrote:
> > >> >>>> >
> > >> >>>> > Are you using the LargeServerMessageImpl on your client, or that
> > >> was
> > >> >>>> > just a "hack" to reproduce your issue.
> > >> >>>> >
> > >> >>>> >
> > >> >>>> > LargeServerMessageImpl was not meant to be used on the client. But
> > >> if
> > >> >>>> > you're doing that just to show something you faced in production
> > >> it's
> > >> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message
> > >> is
> > >> >>>> > a big not for me.
> > >> >>>> >
> > >> >>>> > The only reason we do that is for server to server transfer.
> > >> >>>> >
> > >> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
> > >> wrote:
> > >> >>>> > >
> > >> >>>> > > Hello,
> > >> >>>> > > Yes, the 2.23.1 test instance was freshly created.
> > >> >>>> > >
> > >> >>>> > > I've attached a test as a part of ARTEMIS-3897
> > >> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
> > >> helps.
> > >> >>>> > >
> > >> >>>> > >
> > >> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> > >> clebert.suconic@gmail.com>
> > >> >>>> > > wrote:
> > >> >>>> > >
> > >> >>>> > > > Did you start with fresh data on 2.23.1.
> > >> >>>> > > >
> > >> >>>> > > > If you did please provide a self enclosing test reproducing
> > >> your issue.
> > >> >>>> > > >
> > >> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> > >> wrote:
> > >> >>>> > > >
> > >> >>>> > > > > A quick test using 2.23.1 results in the same error payload
> > >> being
> > >> >>>> > > > received:
> > >> >>>> > > > >
> > >> >>>> > > > > "Message(address='test', durable=True, priority=4,
> > >> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
> > >> byte(0),
> > >> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> > >> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
> > >> 67},
> > >> >>>> > > > > body='Conversion to AMQP error: Error reading in
> > >> simpleString,
> > >> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
> > >> >>>> > > > >
> > >> >>>> > > > > Best regards
> > >> >>>> > > > >
> > >> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> > >> >>>> > > > > <cl...@gmail.com> wrote:
> > >> >>>> > > > > >
> > >> >>>> > > > > > There's been a few fixes in AMQP Large message.
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > > More prominently a fix with the JDBC implementation
> > >> between 2.17 and
> > >> >>>> > > > > HEAD.
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > > I would definitely recommend you to upgrade.
> > >> >>>> > > > > >
> > >> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> > >> jbertram@apache.org>
> > >> >>>> > > > > wrote:
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > I would expect this to work. Can you try this on the
> > >> latest release
> > >> >>>> > > > > (i.e.
> > >> >>>> > > > > > > 2.23.1) [1]?
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > If it still doesn't work please open a Jira with all the
> > >> details
> > >> >>>> > > > > necessary
> > >> >>>> > > > > > > to reproduce the problem. An actual test-case would be
> > >> ideal.
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > Thanks!
> > >> >>>> > > > > > >
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > Justin
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > [1]
> > >> https://activemq.apache.org/components/artemis/download/
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> > >> andyyar66@gmail.com>
> > >> >>>> > > > wrote:
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > > Hello,
> > >> >>>> > > > > > > > Sending a message as a Large Message via Core protocol
> > >> and
> > >> >>>> > > > receiving
> > >> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> > >> following
> > >> >>>> > > > > string
> > >> >>>> > > > > > > > as message body:
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
> > >> simpleString, length=y
> > >> >>>> > > > is
> > >> >>>> > > > > > > > greater than readableBytes=x"
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
> > >> True.
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > The receiver is basically Qpid's:
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > >
> > >> >>>> > > > >
> > >> >>>> > > >
> > >> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > The doc
> > >> >>>> > > > > > > >
> > >> >>>> > > > >
> > >> >>>> > > >
> > >> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > >> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
> > >> this approach
> > >> >>>> > > > > > > > valid?
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > Thanks
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > > --
> > >> >>>> > > > > > Clebert Suconic
> > >> >>>> > > > >
> > >> >>>> > > > --
> > >> >>>> > > > Clebert Suconic
> > >> >>>> > > >
> > >> >>>> >
> > >> >>>> >
> > >> >>>> >
> > >> >>>> > --
> > >> >>>> > Clebert Suconic
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>> --
> > >> >>>> Clebert Suconic
> > >> >
> > >> > --
> > >> > Clebert Suconic
> > >>
> > >>
> > >>
> > >> --
> > >> Clebert Suconic
> > >>
> > > --
> > > Clebert Suconic
> > >
> > --
> > Clebert Suconic
> >
>

Re: Artemis - Large Message - Core => AMQP

Posted by Robbie Gemmell <ro...@gmail.com>.
By "in Java" I assume you more specifically mean 'using the same Core
client as the existing senders/receivers' ?

On Mon, 25 Jul 2022 at 11:10, Jan Šmucr <Ja...@aimtecglobal.com> wrote:
>
> > For the broker, it would ordinarily treat the large message as a
> > chunked stream if doing e.g core->core, or amqp->amqp...but in this
> > case having to do core->amqp means the broker will necessarily have to
> > convert the message and in doing so it will load the entire message.
>
> I think this is exactly what we needed to know to drop Python support and create the Lambda in Java. Thank you! 😊
>
> Jan
>
> From: Robbie Gemmell<ma...@gmail.com>
> Sent: pondělí 25. července 2022 12:05
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Artemis - Large Message - Core => AMQP
>
> Are you using StreamMessage because you think it will enable
> 'steaming'? If so I wouldnt bother, the Stream name isnt related, its
> just a JMS message type name meaning there is a list/sequence of
> things in it. Previously you just seemed to want to send text/bytes.
>
> For the Core client side you can see
> https://activemq.apache.org/components/artemis/documentation/latest/large-messages.html
>
> For the broker, it would ordinarily treat the large message as a
> chunked stream if doing e.g core->core, or amqp->amqp...but in this
> case having to do core->amqp means the broker will necessarily have to
> convert the message and in doing so it will load the entire message.
>
> (It seems like its that last step currently not handling your sent
> Core messages, because you are sending raw Core payloads that dont
> match the JMS style Core message structure the brokers core->AMQP
> converter is geared around, meaning it falls back to one that tries to
> convert it as a string, which presumably fails due to starting to read
> your bare payload as a size indicator)
>
> On Mon, 25 Jul 2022 at 06:40, Jan Šmucr <Ja...@aimtecglobal.com> wrote:
> >
> > Thank you for the feedback.
> > So what would this be the proper way of streaming files (if the code below is correct)?
> >
> >       final String body = "foo";
> >
> > try (Connection connection = createConnection()) {
> >          Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
> >          MessageProducer producer = session.createProducer(session.createQueue(getQueueName()));
> >          StreamMessage message = session.createStreamMessage();
> >          message.writeBytes(body.getBytes());
> >          producer.send(message);
> >       }
> >
> >       AmqpClient client = createAmqpClient();
> >       AmqpConnection amqpConnection = addConnection(client.connect());
> >       AmqpSession amqpSession = amqpConnection.createSession();
> >       AmqpReceiver receiver = amqpSession.createReceiver(getQueueName());
> >       receiver.flow(100);
> >       AmqpMessage amqpMessage = receiver.receive(1, TimeUnit.SECONDS);
> >       Assert.assertNotNull(amqpMessage);
> >       Assert.assertEquals(body, new String(((Binary) ((AmqpSequence) amqpMessage.getWrappedMessage().getBody()).getValue().get(0)).getArray()));
> >
> > We want to stream the payloads due to the fact that these can take up to hundreds of megabytes.
> >
> > Thank you.
> > Jan
> >
> > From: Clebert Suconic<ma...@gmail.com>
> > Sent: sobota 23. července 2022 20:50
> > To: users@activemq.apache.org<ma...@activemq.apache.org>
> > Subject: Re: Artemis - Large Message - Core => AMQP
> >
> > We could look at enhancing the converters.  But the best would be to change
> > your producer to have the exact format the converter would have.
> >
> > On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> > wrote:
> >
> > > Change your message in a way is compatible ?
> > >
> > > On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> > > wrote:
> > >
> > >> The reason for this is that there's a whole infrastructure built using
> > >> the core protocol, and now we need to connect a Python-based Lambda capable
> > >> of receiving large messages. Is there any other, core-compatible method?
> > >>
> > >> Jan
> > >>
> > >> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
> > >> clebert.suconic@gmail.com>:
> > >> If you expect conversions to happen, you have to send messages from
> > >> qpid-ms or core-jms client.
> > >>
> > >>
> > >> On the case:
> > >>
> > >>
> > >> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
> > >> Connection conn = factory.createConnection(...);
> > >> Session session = conn.createSession(...)
> > >> Producer producer = session.createProducer(....);
> > >> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
> > >>
> > >>
> > >>
> > >> Then the conversion from Core to AMQP should happen fine when you
> > >> receive the message on the other side at your Python client.
> > >>
> > >>
> > >>
> > >> If you really must use the Core-API, then you will need to hack your
> > >> way in and find what the JMS client would do.. but at that point it's
> > >> just a non standard way of doing this... You would need to have a very
> > >> good reason to not do the right thing on this case.
> > >>
> > >> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
> > >> <cl...@gmail.com> wrote:
> > >> >
> > >> > If you’re not using qpid JMS it will not work.
> > >> >
> > >> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
> > >> >>
> > >> >> There is another attempt employing AMQP test structures. It produces
> > >> the same incorrect message body.
> > >> >>
> > >> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
> > >> >>>
> > >> >>> Hello,
> > >> >>> Using the LargeServerMessageImpl was just an attempt providing a
> > >> seemingly easy way to introduce a kind of Large Message into the test. I'm
> > >> not familiar with the code base at all - so I simply tried to somehow
> > >> provide the requested test replicating my "sending via Core -> receiving
> > >> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> > >> encountered error simply pointed on it. Is there an intended "test way" to
> > >> model the message transfer among different clients/protocols?
> > >> >>>
> > >> >>> Anyway, I've changed the test to employ
> > >> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> > >> LargeServerMessageImpl with a very similar result. Does that implementation
> > >> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> > >> better or is the explicit AMQP conversion misused?
> > >> >>>
> > >> >>> Please, see the attached patch.
> > >> >>>
> > >> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> > >> clebert.suconic@gmail.com> wrote:
> > >> >>>>
> > >> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
> > >> clients
> > >> >>>> to generate your message. There are certain caveats that both clients
> > >> >>>> will add to the messages.
> > >> >>>>
> > >> >>>>
> > >> >>>> Usually I have seen more users doing weird conversions in AMQP when
> > >> >>>> they use a C++ or non Java Client.. but on your case you used a Core
> > >> >>>> Client, using a non standard way to send a message (ServerMessage
> > >> from
> > >> >>>> the client)... and it's not clear the message is correctly set for a
> > >> >>>> conversion to work.
> > >> >>>>
> > >> >>>>
> > >> >>>> I need you to provide some explanation on what you're doing.. if this
> > >> >>>> was  a way to hack a bug your saw or if this is just the way you're
> > >> >>>> using it.
> > >> >>>>
> > >> >>>>
> > >> >>>> if this is how you're actually using, I would suggest you either use
> > >> >>>> the proper APIs or if you're doing some hack for performance
> > >> >>>> consideration you have to set the message in a better way the
> > >> >>>> converters would understand...
> > >> >>>>
> > >> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> > >> >>>> <cl...@gmail.com> wrote:
> > >> >>>> >
> > >> >>>> > Are you using the LargeServerMessageImpl on your client, or that
> > >> was
> > >> >>>> > just a "hack" to reproduce your issue.
> > >> >>>> >
> > >> >>>> >
> > >> >>>> > LargeServerMessageImpl was not meant to be used on the client. But
> > >> if
> > >> >>>> > you're doing that just to show something you faced in production
> > >> it's
> > >> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message
> > >> is
> > >> >>>> > a big not for me.
> > >> >>>> >
> > >> >>>> > The only reason we do that is for server to server transfer.
> > >> >>>> >
> > >> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
> > >> wrote:
> > >> >>>> > >
> > >> >>>> > > Hello,
> > >> >>>> > > Yes, the 2.23.1 test instance was freshly created.
> > >> >>>> > >
> > >> >>>> > > I've attached a test as a part of ARTEMIS-3897
> > >> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
> > >> helps.
> > >> >>>> > >
> > >> >>>> > >
> > >> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> > >> clebert.suconic@gmail.com>
> > >> >>>> > > wrote:
> > >> >>>> > >
> > >> >>>> > > > Did you start with fresh data on 2.23.1.
> > >> >>>> > > >
> > >> >>>> > > > If you did please provide a self enclosing test reproducing
> > >> your issue.
> > >> >>>> > > >
> > >> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> > >> wrote:
> > >> >>>> > > >
> > >> >>>> > > > > A quick test using 2.23.1 results in the same error payload
> > >> being
> > >> >>>> > > > received:
> > >> >>>> > > > >
> > >> >>>> > > > > "Message(address='test', durable=True, priority=4,
> > >> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
> > >> byte(0),
> > >> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> > >> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
> > >> 67},
> > >> >>>> > > > > body='Conversion to AMQP error: Error reading in
> > >> simpleString,
> > >> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
> > >> >>>> > > > >
> > >> >>>> > > > > Best regards
> > >> >>>> > > > >
> > >> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> > >> >>>> > > > > <cl...@gmail.com> wrote:
> > >> >>>> > > > > >
> > >> >>>> > > > > > There's been a few fixes in AMQP Large message.
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > > More prominently a fix with the JDBC implementation
> > >> between 2.17 and
> > >> >>>> > > > > HEAD.
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > > I would definitely recommend you to upgrade.
> > >> >>>> > > > > >
> > >> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> > >> jbertram@apache.org>
> > >> >>>> > > > > wrote:
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > I would expect this to work. Can you try this on the
> > >> latest release
> > >> >>>> > > > > (i.e.
> > >> >>>> > > > > > > 2.23.1) [1]?
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > If it still doesn't work please open a Jira with all the
> > >> details
> > >> >>>> > > > > necessary
> > >> >>>> > > > > > > to reproduce the problem. An actual test-case would be
> > >> ideal.
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > Thanks!
> > >> >>>> > > > > > >
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > Justin
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > [1]
> > >> https://activemq.apache.org/components/artemis/download/
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> > >> andyyar66@gmail.com>
> > >> >>>> > > > wrote:
> > >> >>>> > > > > > >
> > >> >>>> > > > > > > > Hello,
> > >> >>>> > > > > > > > Sending a message as a Large Message via Core protocol
> > >> and
> > >> >>>> > > > receiving
> > >> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> > >> following
> > >> >>>> > > > > string
> > >> >>>> > > > > > > > as message body:
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
> > >> simpleString, length=y
> > >> >>>> > > > is
> > >> >>>> > > > > > > > greater than readableBytes=x"
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
> > >> True.
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > The receiver is basically Qpid's:
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > >
> > >> >>>> > > > >
> > >> >>>> > > >
> > >> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > The doc
> > >> >>>> > > > > > > >
> > >> >>>> > > > >
> > >> >>>> > > >
> > >> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > >> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
> > >> this approach
> > >> >>>> > > > > > > > valid?
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > > Thanks
> > >> >>>> > > > > > > >
> > >> >>>> > > > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > > > --
> > >> >>>> > > > > > Clebert Suconic
> > >> >>>> > > > >
> > >> >>>> > > > --
> > >> >>>> > > > Clebert Suconic
> > >> >>>> > > >
> > >> >>>> >
> > >> >>>> >
> > >> >>>> >
> > >> >>>> > --
> > >> >>>> > Clebert Suconic
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>> --
> > >> >>>> Clebert Suconic
> > >> >
> > >> > --
> > >> > Clebert Suconic
> > >>
> > >>
> > >>
> > >> --
> > >> Clebert Suconic
> > >>
> > > --
> > > Clebert Suconic
> > >
> > --
> > Clebert Suconic
> >
>

RE: Artemis - Large Message - Core => AMQP

Posted by Jan Šmucr <Ja...@aimtecglobal.com>.
> For the broker, it would ordinarily treat the large message as a
> chunked stream if doing e.g core->core, or amqp->amqp...but in this
> case having to do core->amqp means the broker will necessarily have to
> convert the message and in doing so it will load the entire message.

I think this is exactly what we needed to know to drop Python support and create the Lambda in Java. Thank you! 😊

Jan

From: Robbie Gemmell<ma...@gmail.com>
Sent: pondělí 25. července 2022 12:05
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Artemis - Large Message - Core => AMQP

Are you using StreamMessage because you think it will enable
'steaming'? If so I wouldnt bother, the Stream name isnt related, its
just a JMS message type name meaning there is a list/sequence of
things in it. Previously you just seemed to want to send text/bytes.

For the Core client side you can see
https://activemq.apache.org/components/artemis/documentation/latest/large-messages.html

For the broker, it would ordinarily treat the large message as a
chunked stream if doing e.g core->core, or amqp->amqp...but in this
case having to do core->amqp means the broker will necessarily have to
convert the message and in doing so it will load the entire message.

(It seems like its that last step currently not handling your sent
Core messages, because you are sending raw Core payloads that dont
match the JMS style Core message structure the brokers core->AMQP
converter is geared around, meaning it falls back to one that tries to
convert it as a string, which presumably fails due to starting to read
your bare payload as a size indicator)

On Mon, 25 Jul 2022 at 06:40, Jan Šmucr <Ja...@aimtecglobal.com> wrote:
>
> Thank you for the feedback.
> So what would this be the proper way of streaming files (if the code below is correct)?
>
>       final String body = "foo";
>
> try (Connection connection = createConnection()) {
>          Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
>          MessageProducer producer = session.createProducer(session.createQueue(getQueueName()));
>          StreamMessage message = session.createStreamMessage();
>          message.writeBytes(body.getBytes());
>          producer.send(message);
>       }
>
>       AmqpClient client = createAmqpClient();
>       AmqpConnection amqpConnection = addConnection(client.connect());
>       AmqpSession amqpSession = amqpConnection.createSession();
>       AmqpReceiver receiver = amqpSession.createReceiver(getQueueName());
>       receiver.flow(100);
>       AmqpMessage amqpMessage = receiver.receive(1, TimeUnit.SECONDS);
>       Assert.assertNotNull(amqpMessage);
>       Assert.assertEquals(body, new String(((Binary) ((AmqpSequence) amqpMessage.getWrappedMessage().getBody()).getValue().get(0)).getArray()));
>
> We want to stream the payloads due to the fact that these can take up to hundreds of megabytes.
>
> Thank you.
> Jan
>
> From: Clebert Suconic<ma...@gmail.com>
> Sent: sobota 23. července 2022 20:50
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Artemis - Large Message - Core => AMQP
>
> We could look at enhancing the converters.  But the best would be to change
> your producer to have the exact format the converter would have.
>
> On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > Change your message in a way is compatible ?
> >
> > On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> > wrote:
> >
> >> The reason for this is that there's a whole infrastructure built using
> >> the core protocol, and now we need to connect a Python-based Lambda capable
> >> of receiving large messages. Is there any other, core-compatible method?
> >>
> >> Jan
> >>
> >> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
> >> clebert.suconic@gmail.com>:
> >> If you expect conversions to happen, you have to send messages from
> >> qpid-ms or core-jms client.
> >>
> >>
> >> On the case:
> >>
> >>
> >> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
> >> Connection conn = factory.createConnection(...);
> >> Session session = conn.createSession(...)
> >> Producer producer = session.createProducer(....);
> >> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
> >>
> >>
> >>
> >> Then the conversion from Core to AMQP should happen fine when you
> >> receive the message on the other side at your Python client.
> >>
> >>
> >>
> >> If you really must use the Core-API, then you will need to hack your
> >> way in and find what the JMS client would do.. but at that point it's
> >> just a non standard way of doing this... You would need to have a very
> >> good reason to not do the right thing on this case.
> >>
> >> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
> >> <cl...@gmail.com> wrote:
> >> >
> >> > If you’re not using qpid JMS it will not work.
> >> >
> >> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
> >> >>
> >> >> There is another attempt employing AMQP test structures. It produces
> >> the same incorrect message body.
> >> >>
> >> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
> >> >>>
> >> >>> Hello,
> >> >>> Using the LargeServerMessageImpl was just an attempt providing a
> >> seemingly easy way to introduce a kind of Large Message into the test. I'm
> >> not familiar with the code base at all - so I simply tried to somehow
> >> provide the requested test replicating my "sending via Core -> receiving
> >> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> >> encountered error simply pointed on it. Is there an intended "test way" to
> >> model the message transfer among different clients/protocols?
> >> >>>
> >> >>> Anyway, I've changed the test to employ
> >> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> >> LargeServerMessageImpl with a very similar result. Does that implementation
> >> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> >> better or is the explicit AMQP conversion misused?
> >> >>>
> >> >>> Please, see the attached patch.
> >> >>>
> >> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> >> clebert.suconic@gmail.com> wrote:
> >> >>>>
> >> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
> >> clients
> >> >>>> to generate your message. There are certain caveats that both clients
> >> >>>> will add to the messages.
> >> >>>>
> >> >>>>
> >> >>>> Usually I have seen more users doing weird conversions in AMQP when
> >> >>>> they use a C++ or non Java Client.. but on your case you used a Core
> >> >>>> Client, using a non standard way to send a message (ServerMessage
> >> from
> >> >>>> the client)... and it's not clear the message is correctly set for a
> >> >>>> conversion to work.
> >> >>>>
> >> >>>>
> >> >>>> I need you to provide some explanation on what you're doing.. if this
> >> >>>> was  a way to hack a bug your saw or if this is just the way you're
> >> >>>> using it.
> >> >>>>
> >> >>>>
> >> >>>> if this is how you're actually using, I would suggest you either use
> >> >>>> the proper APIs or if you're doing some hack for performance
> >> >>>> consideration you have to set the message in a better way the
> >> >>>> converters would understand...
> >> >>>>
> >> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> >> >>>> <cl...@gmail.com> wrote:
> >> >>>> >
> >> >>>> > Are you using the LargeServerMessageImpl on your client, or that
> >> was
> >> >>>> > just a "hack" to reproduce your issue.
> >> >>>> >
> >> >>>> >
> >> >>>> > LargeServerMessageImpl was not meant to be used on the client. But
> >> if
> >> >>>> > you're doing that just to show something you faced in production
> >> it's
> >> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message
> >> is
> >> >>>> > a big not for me.
> >> >>>> >
> >> >>>> > The only reason we do that is for server to server transfer.
> >> >>>> >
> >> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
> >> wrote:
> >> >>>> > >
> >> >>>> > > Hello,
> >> >>>> > > Yes, the 2.23.1 test instance was freshly created.
> >> >>>> > >
> >> >>>> > > I've attached a test as a part of ARTEMIS-3897
> >> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
> >> helps.
> >> >>>> > >
> >> >>>> > >
> >> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> >> clebert.suconic@gmail.com>
> >> >>>> > > wrote:
> >> >>>> > >
> >> >>>> > > > Did you start with fresh data on 2.23.1.
> >> >>>> > > >
> >> >>>> > > > If you did please provide a self enclosing test reproducing
> >> your issue.
> >> >>>> > > >
> >> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> >> wrote:
> >> >>>> > > >
> >> >>>> > > > > A quick test using 2.23.1 results in the same error payload
> >> being
> >> >>>> > > > received:
> >> >>>> > > > >
> >> >>>> > > > > "Message(address='test', durable=True, priority=4,
> >> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
> >> byte(0),
> >> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> >> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
> >> 67},
> >> >>>> > > > > body='Conversion to AMQP error: Error reading in
> >> simpleString,
> >> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
> >> >>>> > > > >
> >> >>>> > > > > Best regards
> >> >>>> > > > >
> >> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> >> >>>> > > > > <cl...@gmail.com> wrote:
> >> >>>> > > > > >
> >> >>>> > > > > > There's been a few fixes in AMQP Large message.
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > > More prominently a fix with the JDBC implementation
> >> between 2.17 and
> >> >>>> > > > > HEAD.
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > > I would definitely recommend you to upgrade.
> >> >>>> > > > > >
> >> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> >> jbertram@apache.org>
> >> >>>> > > > > wrote:
> >> >>>> > > > > > >
> >> >>>> > > > > > > I would expect this to work. Can you try this on the
> >> latest release
> >> >>>> > > > > (i.e.
> >> >>>> > > > > > > 2.23.1) [1]?
> >> >>>> > > > > > >
> >> >>>> > > > > > > If it still doesn't work please open a Jira with all the
> >> details
> >> >>>> > > > > necessary
> >> >>>> > > > > > > to reproduce the problem. An actual test-case would be
> >> ideal.
> >> >>>> > > > > > >
> >> >>>> > > > > > > Thanks!
> >> >>>> > > > > > >
> >> >>>> > > > > > >
> >> >>>> > > > > > > Justin
> >> >>>> > > > > > >
> >> >>>> > > > > > > [1]
> >> https://activemq.apache.org/components/artemis/download/
> >> >>>> > > > > > >
> >> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> >> andyyar66@gmail.com>
> >> >>>> > > > wrote:
> >> >>>> > > > > > >
> >> >>>> > > > > > > > Hello,
> >> >>>> > > > > > > > Sending a message as a Large Message via Core protocol
> >> and
> >> >>>> > > > receiving
> >> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> >> following
> >> >>>> > > > > string
> >> >>>> > > > > > > > as message body:
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
> >> simpleString, length=y
> >> >>>> > > > is
> >> >>>> > > > > > > > greater than readableBytes=x"
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
> >> True.
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > The receiver is basically Qpid's:
> >> >>>> > > > > > > >
> >> >>>> > > > > > > >
> >> >>>> > > > >
> >> >>>> > > >
> >> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > The doc
> >> >>>> > > > > > > >
> >> >>>> > > > >
> >> >>>> > > >
> >> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> >> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
> >> this approach
> >> >>>> > > > > > > > valid?
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > Thanks
> >> >>>> > > > > > > >
> >> >>>> > > > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > > --
> >> >>>> > > > > > Clebert Suconic
> >> >>>> > > > >
> >> >>>> > > > --
> >> >>>> > > > Clebert Suconic
> >> >>>> > > >
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > --
> >> >>>> > Clebert Suconic
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Clebert Suconic
> >> >
> >> > --
> >> > Clebert Suconic
> >>
> >>
> >>
> >> --
> >> Clebert Suconic
> >>
> > --
> > Clebert Suconic
> >
> --
> Clebert Suconic
>

Re: Artemis - Large Message - Core => AMQP

Posted by Robbie Gemmell <ro...@gmail.com>.
Are you using StreamMessage because you think it will enable
'steaming'? If so I wouldnt bother, the Stream name isnt related, its
just a JMS message type name meaning there is a list/sequence of
things in it. Previously you just seemed to want to send text/bytes.

For the Core client side you can see
https://activemq.apache.org/components/artemis/documentation/latest/large-messages.html

For the broker, it would ordinarily treat the large message as a
chunked stream if doing e.g core->core, or amqp->amqp...but in this
case having to do core->amqp means the broker will necessarily have to
convert the message and in doing so it will load the entire message.

(It seems like its that last step currently not handling your sent
Core messages, because you are sending raw Core payloads that dont
match the JMS style Core message structure the brokers core->AMQP
converter is geared around, meaning it falls back to one that tries to
convert it as a string, which presumably fails due to starting to read
your bare payload as a size indicator)

On Mon, 25 Jul 2022 at 06:40, Jan Šmucr <Ja...@aimtecglobal.com> wrote:
>
> Thank you for the feedback.
> So what would this be the proper way of streaming files (if the code below is correct)?
>
>       final String body = "foo";
>
> try (Connection connection = createConnection()) {
>          Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
>          MessageProducer producer = session.createProducer(session.createQueue(getQueueName()));
>          StreamMessage message = session.createStreamMessage();
>          message.writeBytes(body.getBytes());
>          producer.send(message);
>       }
>
>       AmqpClient client = createAmqpClient();
>       AmqpConnection amqpConnection = addConnection(client.connect());
>       AmqpSession amqpSession = amqpConnection.createSession();
>       AmqpReceiver receiver = amqpSession.createReceiver(getQueueName());
>       receiver.flow(100);
>       AmqpMessage amqpMessage = receiver.receive(1, TimeUnit.SECONDS);
>       Assert.assertNotNull(amqpMessage);
>       Assert.assertEquals(body, new String(((Binary) ((AmqpSequence) amqpMessage.getWrappedMessage().getBody()).getValue().get(0)).getArray()));
>
> We want to stream the payloads due to the fact that these can take up to hundreds of megabytes.
>
> Thank you.
> Jan
>
> From: Clebert Suconic<ma...@gmail.com>
> Sent: sobota 23. července 2022 20:50
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Artemis - Large Message - Core => AMQP
>
> We could look at enhancing the converters.  But the best would be to change
> your producer to have the exact format the converter would have.
>
> On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > Change your message in a way is compatible ?
> >
> > On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> > wrote:
> >
> >> The reason for this is that there's a whole infrastructure built using
> >> the core protocol, and now we need to connect a Python-based Lambda capable
> >> of receiving large messages. Is there any other, core-compatible method?
> >>
> >> Jan
> >>
> >> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
> >> clebert.suconic@gmail.com>:
> >> If you expect conversions to happen, you have to send messages from
> >> qpid-ms or core-jms client.
> >>
> >>
> >> On the case:
> >>
> >>
> >> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
> >> Connection conn = factory.createConnection(...);
> >> Session session = conn.createSession(...)
> >> Producer producer = session.createProducer(....);
> >> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
> >>
> >>
> >>
> >> Then the conversion from Core to AMQP should happen fine when you
> >> receive the message on the other side at your Python client.
> >>
> >>
> >>
> >> If you really must use the Core-API, then you will need to hack your
> >> way in and find what the JMS client would do.. but at that point it's
> >> just a non standard way of doing this... You would need to have a very
> >> good reason to not do the right thing on this case.
> >>
> >> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
> >> <cl...@gmail.com> wrote:
> >> >
> >> > If you’re not using qpid JMS it will not work.
> >> >
> >> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
> >> >>
> >> >> There is another attempt employing AMQP test structures. It produces
> >> the same incorrect message body.
> >> >>
> >> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
> >> >>>
> >> >>> Hello,
> >> >>> Using the LargeServerMessageImpl was just an attempt providing a
> >> seemingly easy way to introduce a kind of Large Message into the test. I'm
> >> not familiar with the code base at all - so I simply tried to somehow
> >> provide the requested test replicating my "sending via Core -> receiving
> >> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> >> encountered error simply pointed on it. Is there an intended "test way" to
> >> model the message transfer among different clients/protocols?
> >> >>>
> >> >>> Anyway, I've changed the test to employ
> >> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> >> LargeServerMessageImpl with a very similar result. Does that implementation
> >> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> >> better or is the explicit AMQP conversion misused?
> >> >>>
> >> >>> Please, see the attached patch.
> >> >>>
> >> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> >> clebert.suconic@gmail.com> wrote:
> >> >>>>
> >> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
> >> clients
> >> >>>> to generate your message. There are certain caveats that both clients
> >> >>>> will add to the messages.
> >> >>>>
> >> >>>>
> >> >>>> Usually I have seen more users doing weird conversions in AMQP when
> >> >>>> they use a C++ or non Java Client.. but on your case you used a Core
> >> >>>> Client, using a non standard way to send a message (ServerMessage
> >> from
> >> >>>> the client)... and it's not clear the message is correctly set for a
> >> >>>> conversion to work.
> >> >>>>
> >> >>>>
> >> >>>> I need you to provide some explanation on what you're doing.. if this
> >> >>>> was  a way to hack a bug your saw or if this is just the way you're
> >> >>>> using it.
> >> >>>>
> >> >>>>
> >> >>>> if this is how you're actually using, I would suggest you either use
> >> >>>> the proper APIs or if you're doing some hack for performance
> >> >>>> consideration you have to set the message in a better way the
> >> >>>> converters would understand...
> >> >>>>
> >> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> >> >>>> <cl...@gmail.com> wrote:
> >> >>>> >
> >> >>>> > Are you using the LargeServerMessageImpl on your client, or that
> >> was
> >> >>>> > just a "hack" to reproduce your issue.
> >> >>>> >
> >> >>>> >
> >> >>>> > LargeServerMessageImpl was not meant to be used on the client. But
> >> if
> >> >>>> > you're doing that just to show something you faced in production
> >> it's
> >> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message
> >> is
> >> >>>> > a big not for me.
> >> >>>> >
> >> >>>> > The only reason we do that is for server to server transfer.
> >> >>>> >
> >> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
> >> wrote:
> >> >>>> > >
> >> >>>> > > Hello,
> >> >>>> > > Yes, the 2.23.1 test instance was freshly created.
> >> >>>> > >
> >> >>>> > > I've attached a test as a part of ARTEMIS-3897
> >> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
> >> helps.
> >> >>>> > >
> >> >>>> > >
> >> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> >> clebert.suconic@gmail.com>
> >> >>>> > > wrote:
> >> >>>> > >
> >> >>>> > > > Did you start with fresh data on 2.23.1.
> >> >>>> > > >
> >> >>>> > > > If you did please provide a self enclosing test reproducing
> >> your issue.
> >> >>>> > > >
> >> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> >> wrote:
> >> >>>> > > >
> >> >>>> > > > > A quick test using 2.23.1 results in the same error payload
> >> being
> >> >>>> > > > received:
> >> >>>> > > > >
> >> >>>> > > > > "Message(address='test', durable=True, priority=4,
> >> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
> >> byte(0),
> >> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> >> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
> >> 67},
> >> >>>> > > > > body='Conversion to AMQP error: Error reading in
> >> simpleString,
> >> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
> >> >>>> > > > >
> >> >>>> > > > > Best regards
> >> >>>> > > > >
> >> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> >> >>>> > > > > <cl...@gmail.com> wrote:
> >> >>>> > > > > >
> >> >>>> > > > > > There's been a few fixes in AMQP Large message.
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > > More prominently a fix with the JDBC implementation
> >> between 2.17 and
> >> >>>> > > > > HEAD.
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > > I would definitely recommend you to upgrade.
> >> >>>> > > > > >
> >> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> >> jbertram@apache.org>
> >> >>>> > > > > wrote:
> >> >>>> > > > > > >
> >> >>>> > > > > > > I would expect this to work. Can you try this on the
> >> latest release
> >> >>>> > > > > (i.e.
> >> >>>> > > > > > > 2.23.1) [1]?
> >> >>>> > > > > > >
> >> >>>> > > > > > > If it still doesn't work please open a Jira with all the
> >> details
> >> >>>> > > > > necessary
> >> >>>> > > > > > > to reproduce the problem. An actual test-case would be
> >> ideal.
> >> >>>> > > > > > >
> >> >>>> > > > > > > Thanks!
> >> >>>> > > > > > >
> >> >>>> > > > > > >
> >> >>>> > > > > > > Justin
> >> >>>> > > > > > >
> >> >>>> > > > > > > [1]
> >> https://activemq.apache.org/components/artemis/download/
> >> >>>> > > > > > >
> >> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> >> andyyar66@gmail.com>
> >> >>>> > > > wrote:
> >> >>>> > > > > > >
> >> >>>> > > > > > > > Hello,
> >> >>>> > > > > > > > Sending a message as a Large Message via Core protocol
> >> and
> >> >>>> > > > receiving
> >> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> >> following
> >> >>>> > > > > string
> >> >>>> > > > > > > > as message body:
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
> >> simpleString, length=y
> >> >>>> > > > is
> >> >>>> > > > > > > > greater than readableBytes=x"
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
> >> True.
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > The receiver is basically Qpid's:
> >> >>>> > > > > > > >
> >> >>>> > > > > > > >
> >> >>>> > > > >
> >> >>>> > > >
> >> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > The doc
> >> >>>> > > > > > > >
> >> >>>> > > > >
> >> >>>> > > >
> >> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> >> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
> >> this approach
> >> >>>> > > > > > > > valid?
> >> >>>> > > > > > > >
> >> >>>> > > > > > > > Thanks
> >> >>>> > > > > > > >
> >> >>>> > > > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > > > --
> >> >>>> > > > > > Clebert Suconic
> >> >>>> > > > >
> >> >>>> > > > --
> >> >>>> > > > Clebert Suconic
> >> >>>> > > >
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > --
> >> >>>> > Clebert Suconic
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Clebert Suconic
> >> >
> >> > --
> >> > Clebert Suconic
> >>
> >>
> >>
> >> --
> >> Clebert Suconic
> >>
> > --
> > Clebert Suconic
> >
> --
> Clebert Suconic
>

RE: Artemis - Large Message - Core => AMQP

Posted by Jan Šmucr <Ja...@aimtecglobal.com>.
Thank you for the feedback.
So what would this be the proper way of streaming files (if the code below is correct)?

      final String body = "foo";

try (Connection connection = createConnection()) {
         Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
         MessageProducer producer = session.createProducer(session.createQueue(getQueueName()));
         StreamMessage message = session.createStreamMessage();
         message.writeBytes(body.getBytes());
         producer.send(message);
      }

      AmqpClient client = createAmqpClient();
      AmqpConnection amqpConnection = addConnection(client.connect());
      AmqpSession amqpSession = amqpConnection.createSession();
      AmqpReceiver receiver = amqpSession.createReceiver(getQueueName());
      receiver.flow(100);
      AmqpMessage amqpMessage = receiver.receive(1, TimeUnit.SECONDS);
      Assert.assertNotNull(amqpMessage);
      Assert.assertEquals(body, new String(((Binary) ((AmqpSequence) amqpMessage.getWrappedMessage().getBody()).getValue().get(0)).getArray()));

We want to stream the payloads due to the fact that these can take up to hundreds of megabytes.

Thank you.
Jan

From: Clebert Suconic<ma...@gmail.com>
Sent: sobota 23. července 2022 20:50
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Artemis - Large Message - Core => AMQP

We could look at enhancing the converters.  But the best would be to change
your producer to have the exact format the converter would have.

On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
wrote:

> Change your message in a way is compatible ?
>
> On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> wrote:
>
>> The reason for this is that there's a whole infrastructure built using
>> the core protocol, and now we need to connect a Python-based Lambda capable
>> of receiving large messages. Is there any other, core-compatible method?
>>
>> Jan
>>
>> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
>> clebert.suconic@gmail.com>:
>> If you expect conversions to happen, you have to send messages from
>> qpid-ms or core-jms client.
>>
>>
>> On the case:
>>
>>
>> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
>> Connection conn = factory.createConnection(...);
>> Session session = conn.createSession(...)
>> Producer producer = session.createProducer(....);
>> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
>>
>>
>>
>> Then the conversion from Core to AMQP should happen fine when you
>> receive the message on the other side at your Python client.
>>
>>
>>
>> If you really must use the Core-API, then you will need to hack your
>> way in and find what the JMS client would do.. but at that point it's
>> just a non standard way of doing this... You would need to have a very
>> good reason to not do the right thing on this case.
>>
>> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
>> <cl...@gmail.com> wrote:
>> >
>> > If you’re not using qpid JMS it will not work.
>> >
>> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
>> >>
>> >> There is another attempt employing AMQP test structures. It produces
>> the same incorrect message body.
>> >>
>> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
>> >>>
>> >>> Hello,
>> >>> Using the LargeServerMessageImpl was just an attempt providing a
>> seemingly easy way to introduce a kind of Large Message into the test. I'm
>> not familiar with the code base at all - so I simply tried to somehow
>> provide the requested test replicating my "sending via Core -> receiving
>> via Qpid" issue. The same applies to the explicit AMQP conversion - the
>> encountered error simply pointed on it. Is there an intended "test way" to
>> model the message transfer among different clients/protocols?
>> >>>
>> >>> Anyway, I've changed the test to employ
>> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
>> LargeServerMessageImpl with a very similar result. Does that implementation
>> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
>> better or is the explicit AMQP conversion misused?
>> >>>
>> >>> Please, see the attached patch.
>> >>>
>> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
>> clebert.suconic@gmail.com> wrote:
>> >>>>
>> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
>> clients
>> >>>> to generate your message. There are certain caveats that both clients
>> >>>> will add to the messages.
>> >>>>
>> >>>>
>> >>>> Usually I have seen more users doing weird conversions in AMQP when
>> >>>> they use a C++ or non Java Client.. but on your case you used a Core
>> >>>> Client, using a non standard way to send a message (ServerMessage
>> from
>> >>>> the client)... and it's not clear the message is correctly set for a
>> >>>> conversion to work.
>> >>>>
>> >>>>
>> >>>> I need you to provide some explanation on what you're doing.. if this
>> >>>> was  a way to hack a bug your saw or if this is just the way you're
>> >>>> using it.
>> >>>>
>> >>>>
>> >>>> if this is how you're actually using, I would suggest you either use
>> >>>> the proper APIs or if you're doing some hack for performance
>> >>>> consideration you have to set the message in a better way the
>> >>>> converters would understand...
>> >>>>
>> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>> >>>> <cl...@gmail.com> wrote:
>> >>>> >
>> >>>> > Are you using the LargeServerMessageImpl on your client, or that
>> was
>> >>>> > just a "hack" to reproduce your issue.
>> >>>> >
>> >>>> >
>> >>>> > LargeServerMessageImpl was not meant to be used on the client. But
>> if
>> >>>> > you're doing that just to show something you faced in production
>> it's
>> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message
>> is
>> >>>> > a big not for me.
>> >>>> >
>> >>>> > The only reason we do that is for server to server transfer.
>> >>>> >
>> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
>> wrote:
>> >>>> > >
>> >>>> > > Hello,
>> >>>> > > Yes, the 2.23.1 test instance was freshly created.
>> >>>> > >
>> >>>> > > I've attached a test as a part of ARTEMIS-3897
>> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
>> helps.
>> >>>> > >
>> >>>> > >
>> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> >>>> > > wrote:
>> >>>> > >
>> >>>> > > > Did you start with fresh data on 2.23.1.
>> >>>> > > >
>> >>>> > > > If you did please provide a self enclosing test reproducing
>> your issue.
>> >>>> > > >
>> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
>> wrote:
>> >>>> > > >
>> >>>> > > > > A quick test using 2.23.1 results in the same error payload
>> being
>> >>>> > > > received:
>> >>>> > > > >
>> >>>> > > > > "Message(address='test', durable=True, priority=4,
>> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
>> byte(0),
>> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
>> 67},
>> >>>> > > > > body='Conversion to AMQP error: Error reading in
>> simpleString,
>> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
>> >>>> > > > >
>> >>>> > > > > Best regards
>> >>>> > > > >
>> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>> >>>> > > > > <cl...@gmail.com> wrote:
>> >>>> > > > > >
>> >>>> > > > > > There's been a few fixes in AMQP Large message.
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > > More prominently a fix with the JDBC implementation
>> between 2.17 and
>> >>>> > > > > HEAD.
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > > I would definitely recommend you to upgrade.
>> >>>> > > > > >
>> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
>> jbertram@apache.org>
>> >>>> > > > > wrote:
>> >>>> > > > > > >
>> >>>> > > > > > > I would expect this to work. Can you try this on the
>> latest release
>> >>>> > > > > (i.e.
>> >>>> > > > > > > 2.23.1) [1]?
>> >>>> > > > > > >
>> >>>> > > > > > > If it still doesn't work please open a Jira with all the
>> details
>> >>>> > > > > necessary
>> >>>> > > > > > > to reproduce the problem. An actual test-case would be
>> ideal.
>> >>>> > > > > > >
>> >>>> > > > > > > Thanks!
>> >>>> > > > > > >
>> >>>> > > > > > >
>> >>>> > > > > > > Justin
>> >>>> > > > > > >
>> >>>> > > > > > > [1]
>> https://activemq.apache.org/components/artemis/download/
>> >>>> > > > > > >
>> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
>> andyyar66@gmail.com>
>> >>>> > > > wrote:
>> >>>> > > > > > >
>> >>>> > > > > > > > Hello,
>> >>>> > > > > > > > Sending a message as a Large Message via Core protocol
>> and
>> >>>> > > > receiving
>> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
>> following
>> >>>> > > > > string
>> >>>> > > > > > > > as message body:
>> >>>> > > > > > > >
>> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
>> simpleString, length=y
>> >>>> > > > is
>> >>>> > > > > > > > greater than readableBytes=x"
>> >>>> > > > > > > >
>> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
>> True.
>> >>>> > > > > > > >
>> >>>> > > > > > > > The receiver is basically Qpid's:
>> >>>> > > > > > > >
>> >>>> > > > > > > >
>> >>>> > > > >
>> >>>> > > >
>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>> >>>> > > > > > > >
>> >>>> > > > > > > > The doc
>> >>>> > > > > > > >
>> >>>> > > > >
>> >>>> > > >
>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
>> this approach
>> >>>> > > > > > > > valid?
>> >>>> > > > > > > >
>> >>>> > > > > > > > Thanks
>> >>>> > > > > > > >
>> >>>> > > > > > > >
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > > --
>> >>>> > > > > > Clebert Suconic
>> >>>> > > > >
>> >>>> > > > --
>> >>>> > > > Clebert Suconic
>> >>>> > > >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Clebert Suconic
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Clebert Suconic
>> >
>> > --
>> > Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>>
> --
> Clebert Suconic
>
--
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
We could look at enhancing the converters.  But the best would be to change
your producer to have the exact format the converter would have.

On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <cl...@gmail.com>
wrote:

> Change your message in a way is compatible ?
>
> On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
> wrote:
>
>> The reason for this is that there's a whole infrastructure built using
>> the core protocol, and now we need to connect a Python-based Lambda capable
>> of receiving large messages. Is there any other, core-compatible method?
>>
>> Jan
>>
>> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
>> clebert.suconic@gmail.com>:
>> If you expect conversions to happen, you have to send messages from
>> qpid-ms or core-jms client.
>>
>>
>> On the case:
>>
>>
>> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
>> Connection conn = factory.createConnection(...);
>> Session session = conn.createSession(...)
>> Producer producer = session.createProducer(....);
>> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
>>
>>
>>
>> Then the conversion from Core to AMQP should happen fine when you
>> receive the message on the other side at your Python client.
>>
>>
>>
>> If you really must use the Core-API, then you will need to hack your
>> way in and find what the JMS client would do.. but at that point it's
>> just a non standard way of doing this... You would need to have a very
>> good reason to not do the right thing on this case.
>>
>> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
>> <cl...@gmail.com> wrote:
>> >
>> > If you’re not using qpid JMS it will not work.
>> >
>> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
>> >>
>> >> There is another attempt employing AMQP test structures. It produces
>> the same incorrect message body.
>> >>
>> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
>> >>>
>> >>> Hello,
>> >>> Using the LargeServerMessageImpl was just an attempt providing a
>> seemingly easy way to introduce a kind of Large Message into the test. I'm
>> not familiar with the code base at all - so I simply tried to somehow
>> provide the requested test replicating my "sending via Core -> receiving
>> via Qpid" issue. The same applies to the explicit AMQP conversion - the
>> encountered error simply pointed on it. Is there an intended "test way" to
>> model the message transfer among different clients/protocols?
>> >>>
>> >>> Anyway, I've changed the test to employ
>> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
>> LargeServerMessageImpl with a very similar result. Does that implementation
>> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
>> better or is the explicit AMQP conversion misused?
>> >>>
>> >>> Please, see the attached patch.
>> >>>
>> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
>> clebert.suconic@gmail.com> wrote:
>> >>>>
>> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS
>> clients
>> >>>> to generate your message. There are certain caveats that both clients
>> >>>> will add to the messages.
>> >>>>
>> >>>>
>> >>>> Usually I have seen more users doing weird conversions in AMQP when
>> >>>> they use a C++ or non Java Client.. but on your case you used a Core
>> >>>> Client, using a non standard way to send a message (ServerMessage
>> from
>> >>>> the client)... and it's not clear the message is correctly set for a
>> >>>> conversion to work.
>> >>>>
>> >>>>
>> >>>> I need you to provide some explanation on what you're doing.. if this
>> >>>> was  a way to hack a bug your saw or if this is just the way you're
>> >>>> using it.
>> >>>>
>> >>>>
>> >>>> if this is how you're actually using, I would suggest you either use
>> >>>> the proper APIs or if you're doing some hack for performance
>> >>>> consideration you have to set the message in a better way the
>> >>>> converters would understand...
>> >>>>
>> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>> >>>> <cl...@gmail.com> wrote:
>> >>>> >
>> >>>> > Are you using the LargeServerMessageImpl on your client, or that
>> was
>> >>>> > just a "hack" to reproduce your issue.
>> >>>> >
>> >>>> >
>> >>>> > LargeServerMessageImpl was not meant to be used on the client. But
>> if
>> >>>> > you're doing that just to show something you faced in production
>> it's
>> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message
>> is
>> >>>> > a big not for me.
>> >>>> >
>> >>>> > The only reason we do that is for server to server transfer.
>> >>>> >
>> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
>> wrote:
>> >>>> > >
>> >>>> > > Hello,
>> >>>> > > Yes, the 2.23.1 test instance was freshly created.
>> >>>> > >
>> >>>> > > I've attached a test as a part of ARTEMIS-3897
>> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
>> helps.
>> >>>> > >
>> >>>> > >
>> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> >>>> > > wrote:
>> >>>> > >
>> >>>> > > > Did you start with fresh data on 2.23.1.
>> >>>> > > >
>> >>>> > > > If you did please provide a self enclosing test reproducing
>> your issue.
>> >>>> > > >
>> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
>> wrote:
>> >>>> > > >
>> >>>> > > > > A quick test using 2.23.1 results in the same error payload
>> being
>> >>>> > > > received:
>> >>>> > > > >
>> >>>> > > > > "Message(address='test', durable=True, priority=4,
>> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'):
>> byte(0),
>> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE':
>> 67},
>> >>>> > > > > body='Conversion to AMQP error: Error reading in
>> simpleString,
>> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
>> >>>> > > > >
>> >>>> > > > > Best regards
>> >>>> > > > >
>> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>> >>>> > > > > <cl...@gmail.com> wrote:
>> >>>> > > > > >
>> >>>> > > > > > There's been a few fixes in AMQP Large message.
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > > More prominently a fix with the JDBC implementation
>> between 2.17 and
>> >>>> > > > > HEAD.
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > > I would definitely recommend you to upgrade.
>> >>>> > > > > >
>> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
>> jbertram@apache.org>
>> >>>> > > > > wrote:
>> >>>> > > > > > >
>> >>>> > > > > > > I would expect this to work. Can you try this on the
>> latest release
>> >>>> > > > > (i.e.
>> >>>> > > > > > > 2.23.1) [1]?
>> >>>> > > > > > >
>> >>>> > > > > > > If it still doesn't work please open a Jira with all the
>> details
>> >>>> > > > > necessary
>> >>>> > > > > > > to reproduce the problem. An actual test-case would be
>> ideal.
>> >>>> > > > > > >
>> >>>> > > > > > > Thanks!
>> >>>> > > > > > >
>> >>>> > > > > > >
>> >>>> > > > > > > Justin
>> >>>> > > > > > >
>> >>>> > > > > > > [1]
>> https://activemq.apache.org/components/artemis/download/
>> >>>> > > > > > >
>> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
>> andyyar66@gmail.com>
>> >>>> > > > wrote:
>> >>>> > > > > > >
>> >>>> > > > > > > > Hello,
>> >>>> > > > > > > > Sending a message as a Large Message via Core protocol
>> and
>> >>>> > > > receiving
>> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
>> following
>> >>>> > > > > string
>> >>>> > > > > > > > as message body:
>> >>>> > > > > > > >
>> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
>> simpleString, length=y
>> >>>> > > > is
>> >>>> > > > > > > > greater than readableBytes=x"
>> >>>> > > > > > > >
>> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
>> True.
>> >>>> > > > > > > >
>> >>>> > > > > > > > The receiver is basically Qpid's:
>> >>>> > > > > > > >
>> >>>> > > > > > > >
>> >>>> > > > >
>> >>>> > > >
>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>> >>>> > > > > > > >
>> >>>> > > > > > > > The doc
>> >>>> > > > > > > >
>> >>>> > > > >
>> >>>> > > >
>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
>> this approach
>> >>>> > > > > > > > valid?
>> >>>> > > > > > > >
>> >>>> > > > > > > > Thanks
>> >>>> > > > > > > >
>> >>>> > > > > > > >
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > > > --
>> >>>> > > > > > Clebert Suconic
>> >>>> > > > >
>> >>>> > > > --
>> >>>> > > > Clebert Suconic
>> >>>> > > >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Clebert Suconic
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Clebert Suconic
>> >
>> > --
>> > Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>>
> --
> Clebert Suconic
>
-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
Change your message in a way is compatible ?

On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <Ja...@aimtecglobal.com>
wrote:

> The reason for this is that there's a whole infrastructure built using the
> core protocol, and now we need to connect a Python-based Lambda capable of
> receiving large messages. Is there any other, core-compatible method?
>
> Jan
>
> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <
> clebert.suconic@gmail.com>:
> If you expect conversions to happen, you have to send messages from
> qpid-ms or core-jms client.
>
>
> On the case:
>
>
> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
> Connection conn = factory.createConnection(...);
> Session session = conn.createSession(...)
> Producer producer = session.createProducer(....);
> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);
>
>
>
> Then the conversion from Core to AMQP should happen fine when you
> receive the message on the other side at your Python client.
>
>
>
> If you really must use the Core-API, then you will need to hack your
> way in and find what the JMS client would do.. but at that point it's
> just a non standard way of doing this... You would need to have a very
> good reason to not do the right thing on this case.
>
> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > If you’re not using qpid JMS it will not work.
> >
> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
> >>
> >> There is another attempt employing AMQP test structures. It produces
> the same incorrect message body.
> >>
> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
> >>>
> >>> Hello,
> >>> Using the LargeServerMessageImpl was just an attempt providing a
> seemingly easy way to introduce a kind of Large Message into the test. I'm
> not familiar with the code base at all - so I simply tried to somehow
> provide the requested test replicating my "sending via Core -> receiving
> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> encountered error simply pointed on it. Is there an intended "test way" to
> model the message transfer among different clients/protocols?
> >>>
> >>> Anyway, I've changed the test to employ
> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> LargeServerMessageImpl with a very similar result. Does that implementation
> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> better or is the explicit AMQP conversion misused?
> >>>
> >>> Please, see the attached patch.
> >>>
> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> clebert.suconic@gmail.com> wrote:
> >>>>
> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS clients
> >>>> to generate your message. There are certain caveats that both clients
> >>>> will add to the messages.
> >>>>
> >>>>
> >>>> Usually I have seen more users doing weird conversions in AMQP when
> >>>> they use a C++ or non Java Client.. but on your case you used a Core
> >>>> Client, using a non standard way to send a message (ServerMessage from
> >>>> the client)... and it's not clear the message is correctly set for a
> >>>> conversion to work.
> >>>>
> >>>>
> >>>> I need you to provide some explanation on what you're doing.. if this
> >>>> was  a way to hack a bug your saw or if this is just the way you're
> >>>> using it.
> >>>>
> >>>>
> >>>> if this is how you're actually using, I would suggest you either use
> >>>> the proper APIs or if you're doing some hack for performance
> >>>> consideration you have to set the message in a better way the
> >>>> converters would understand...
> >>>>
> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> >>>> <cl...@gmail.com> wrote:
> >>>> >
> >>>> > Are you using the LargeServerMessageImpl on your client, or that was
> >>>> > just a "hack" to reproduce your issue.
> >>>> >
> >>>> >
> >>>> > LargeServerMessageImpl was not meant to be used on the client. But
> if
> >>>> > you're doing that just to show something you faced in production
> it's
> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message
> is
> >>>> > a big not for me.
> >>>> >
> >>>> > The only reason we do that is for server to server transfer.
> >>>> >
> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com>
> wrote:
> >>>> > >
> >>>> > > Hello,
> >>>> > > Yes, the 2.23.1 test instance was freshly created.
> >>>> > >
> >>>> > > I've attached a test as a part of ARTEMIS-3897
> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
> helps.
> >>>> > >
> >>>> > >
> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> clebert.suconic@gmail.com>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Did you start with fresh data on 2.23.1.
> >>>> > > >
> >>>> > > > If you did please provide a self enclosing test reproducing
> your issue.
> >>>> > > >
> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> wrote:
> >>>> > > >
> >>>> > > > > A quick test using 2.23.1 results in the same error payload
> being
> >>>> > > > received:
> >>>> > > > >
> >>>> > > > > "Message(address='test', durable=True, priority=4,
> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
> >>>> > > > > body='Conversion to AMQP error: Error reading in simpleString,
> >>>> > > > > length=1330464032 is greater than readableBytes=62')"
> >>>> > > > >
> >>>> > > > > Best regards
> >>>> > > > >
> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> >>>> > > > > <cl...@gmail.com> wrote:
> >>>> > > > > >
> >>>> > > > > > There's been a few fixes in AMQP Large message.
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > > More prominently a fix with the JDBC implementation between
> 2.17 and
> >>>> > > > > HEAD.
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > > I would definitely recommend you to upgrade.
> >>>> > > > > >
> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> jbertram@apache.org>
> >>>> > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > I would expect this to work. Can you try this on the
> latest release
> >>>> > > > > (i.e.
> >>>> > > > > > > 2.23.1) [1]?
> >>>> > > > > > >
> >>>> > > > > > > If it still doesn't work please open a Jira with all the
> details
> >>>> > > > > necessary
> >>>> > > > > > > to reproduce the problem. An actual test-case would be
> ideal.
> >>>> > > > > > >
> >>>> > > > > > > Thanks!
> >>>> > > > > > >
> >>>> > > > > > >
> >>>> > > > > > > Justin
> >>>> > > > > > >
> >>>> > > > > > > [1]
> https://activemq.apache.org/components/artemis/download/
> >>>> > > > > > >
> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> andyyar66@gmail.com>
> >>>> > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > > Hello,
> >>>> > > > > > > > Sending a message as a Large Message via Core protocol
> and
> >>>> > > > receiving
> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> following
> >>>> > > > > string
> >>>> > > > > > > > as message body:
> >>>> > > > > > > >
> >>>> > > > > > > > "Conversion to AMQP error: Error reading in
> simpleString, length=y
> >>>> > > > is
> >>>> > > > > > > > greater than readableBytes=x"
> >>>> > > > > > > >
> >>>> > > > > > > > Broker lists the message as Type: Default and Large:
> True.
> >>>> > > > > > > >
> >>>> > > > > > > > The receiver is basically Qpid's:
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > >
> >>>> > > >
> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> >>>> > > > > > > >
> >>>> > > > > > > > The doc
> >>>> > > > > > > >
> >>>> > > > >
> >>>> > > >
> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> >>>> > > > > > > > states both Core and AMQP support Large Message. Is
> this approach
> >>>> > > > > > > > valid?
> >>>> > > > > > > >
> >>>> > > > > > > > Thanks
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > > --
> >>>> > > > > > Clebert Suconic
> >>>> > > > >
> >>>> > > > --
> >>>> > > > Clebert Suconic
> >>>> > > >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > Clebert Suconic
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Clebert Suconic
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Jan Šmucr <Ja...@aimtecglobal.com>.
The reason for this is that there's a whole infrastructure built using the core protocol, and now we need to connect a Python-based Lambda capable of receiving large messages. Is there any other, core-compatible method?

Jan

Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic <cl...@gmail.com>:
If you expect conversions to happen, you have to send messages from
qpid-ms or core-jms client.


On the case:


ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
Connection conn = factory.createConnection(...);
Session session = conn.createSession(...)
Producer producer = session.createProducer(....);
producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);



Then the conversion from Core to AMQP should happen fine when you
receive the message on the other side at your Python client.



If you really must use the Core-API, then you will need to hack your
way in and find what the JMS client would do.. but at that point it's
just a non standard way of doing this... You would need to have a very
good reason to not do the right thing on this case.

On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> If you’re not using qpid JMS it will not work.
>
> On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
>>
>> There is another attempt employing AMQP test structures. It produces the same incorrect message body.
>>
>> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
>>>
>>> Hello,
>>> Using the LargeServerMessageImpl was just an attempt providing a seemingly easy way to introduce a kind of Large Message into the test. I'm not familiar with the code base at all - so I simply tried to somehow provide the requested test replicating my "sending via Core -> receiving via Qpid" issue. The same applies to the explicit AMQP conversion - the encountered error simply pointed on it. Is there an intended "test way" to model the message transfer among different clients/protocols?
>>>
>>> Anyway, I've changed the test to employ ClientLargeMessageImpl/ClientMessageImpl instead of the server-related LargeServerMessageImpl with a very similar result. Does that implementation fit the simulated use case (sending via Core -> receiving via Qpid AMQP) better or is the explicit AMQP conversion misused?
>>>
>>> Please, see the attached patch.
>>>
>>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <cl...@gmail.com> wrote:
>>>>
>>>> AMQP to Core and Core to AMQP conversions expect you using JMS clients
>>>> to generate your message. There are certain caveats that both clients
>>>> will add to the messages.
>>>>
>>>>
>>>> Usually I have seen more users doing weird conversions in AMQP when
>>>> they use a C++ or non Java Client.. but on your case you used a Core
>>>> Client, using a non standard way to send a message (ServerMessage from
>>>> the client)... and it's not clear the message is correctly set for a
>>>> conversion to work.
>>>>
>>>>
>>>> I need you to provide some explanation on what you're doing.. if this
>>>> was  a way to hack a bug your saw or if this is just the way you're
>>>> using it.
>>>>
>>>>
>>>> if this is how you're actually using, I would suggest you either use
>>>> the proper APIs or if you're doing some hack for performance
>>>> consideration you have to set the message in a better way the
>>>> converters would understand...
>>>>
>>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>>>> <cl...@gmail.com> wrote:
>>>> >
>>>> > Are you using the LargeServerMessageImpl on your client, or that was
>>>> > just a "hack" to reproduce your issue.
>>>> >
>>>> >
>>>> > LargeServerMessageImpl was not meant to be used on the client. But if
>>>> > you're doing that just to show something you faced in production it's
>>>> > ok.. but using the LargeServerMessageImpl to send a Client Message is
>>>> > a big not for me.
>>>> >
>>>> > The only reason we do that is for server to server transfer.
>>>> >
>>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
>>>> > >
>>>> > > Hello,
>>>> > > Yes, the 2.23.1 test instance was freshly created.
>>>> > >
>>>> > > I've attached a test as a part of ARTEMIS-3897
>>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it helps.
>>>> > >
>>>> > >
>>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <cl...@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Did you start with fresh data on 2.23.1.
>>>> > > >
>>>> > > > If you did please provide a self enclosing test reproducing your issue.
>>>> > > >
>>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com> wrote:
>>>> > > >
>>>> > > > > A quick test using 2.23.1 results in the same error payload being
>>>> > > > received:
>>>> > > > >
>>>> > > > > "Message(address='test', durable=True, priority=4,
>>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
>>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
>>>> > > > > body='Conversion to AMQP error: Error reading in simpleString,
>>>> > > > > length=1330464032 is greater than readableBytes=62')"
>>>> > > > >
>>>> > > > > Best regards
>>>> > > > >
>>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>>>> > > > > <cl...@gmail.com> wrote:
>>>> > > > > >
>>>> > > > > > There's been a few fixes in AMQP Large message.
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > More prominently a fix with the JDBC implementation between 2.17 and
>>>> > > > > HEAD.
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > I would definitely recommend you to upgrade.
>>>> > > > > >
>>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org>
>>>> > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > I would expect this to work. Can you try this on the latest release
>>>> > > > > (i.e.
>>>> > > > > > > 2.23.1) [1]?
>>>> > > > > > >
>>>> > > > > > > If it still doesn't work please open a Jira with all the details
>>>> > > > > necessary
>>>> > > > > > > to reproduce the problem. An actual test-case would be ideal.
>>>> > > > > > >
>>>> > > > > > > Thanks!
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > Justin
>>>> > > > > > >
>>>> > > > > > > [1] https://activemq.apache.org/components/artemis/download/
>>>> > > > > > >
>>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com>
>>>> > > > wrote:
>>>> > > > > > >
>>>> > > > > > > > Hello,
>>>> > > > > > > > Sending a message as a Large Message via Core protocol and
>>>> > > > receiving
>>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the following
>>>> > > > > string
>>>> > > > > > > > as message body:
>>>> > > > > > > >
>>>> > > > > > > > "Conversion to AMQP error: Error reading in simpleString, length=y
>>>> > > > is
>>>> > > > > > > > greater than readableBytes=x"
>>>> > > > > > > >
>>>> > > > > > > > Broker lists the message as Type: Default and Large: True.
>>>> > > > > > > >
>>>> > > > > > > > The receiver is basically Qpid's:
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > >
>>>> > > > https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>>>> > > > > > > >
>>>> > > > > > > > The doc
>>>> > > > > > > >
>>>> > > > >
>>>> > > > https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>>>> > > > > > > > states both Core and AMQP support Large Message. Is this approach
>>>> > > > > > > > valid?
>>>> > > > > > > >
>>>> > > > > > > > Thanks
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > --
>>>> > > > > > Clebert Suconic
>>>> > > > >
>>>> > > > --
>>>> > > > Clebert Suconic
>>>> > > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Clebert Suconic
>>>>
>>>>
>>>>
>>>> --
>>>> Clebert Suconic
>
> --
> Clebert Suconic



--
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
If you expect conversions to happen, you have to send messages from
qpid-ms or core-jms client.


On the case:


ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
Connection conn = factory.createConnection(...);
Session session = conn.createSession(...)
Producer producer = session.createProducer(....);
producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE);



Then the conversion from Core to AMQP should happen fine when you
receive the message on the other side at your Python client.



If you really must use the Core-API, then you will need to hack your
way in and find what the JMS client would do.. but at that point it's
just a non standard way of doing this... You would need to have a very
good reason to not do the right thing on this case.

On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> If you’re not using qpid JMS it will not work.
>
> On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
>>
>> There is another attempt employing AMQP test structures. It produces the same incorrect message body.
>>
>> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
>>>
>>> Hello,
>>> Using the LargeServerMessageImpl was just an attempt providing a seemingly easy way to introduce a kind of Large Message into the test. I'm not familiar with the code base at all - so I simply tried to somehow provide the requested test replicating my "sending via Core -> receiving via Qpid" issue. The same applies to the explicit AMQP conversion - the encountered error simply pointed on it. Is there an intended "test way" to model the message transfer among different clients/protocols?
>>>
>>> Anyway, I've changed the test to employ ClientLargeMessageImpl/ClientMessageImpl instead of the server-related LargeServerMessageImpl with a very similar result. Does that implementation fit the simulated use case (sending via Core -> receiving via Qpid AMQP) better or is the explicit AMQP conversion misused?
>>>
>>> Please, see the attached patch.
>>>
>>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <cl...@gmail.com> wrote:
>>>>
>>>> AMQP to Core and Core to AMQP conversions expect you using JMS clients
>>>> to generate your message. There are certain caveats that both clients
>>>> will add to the messages.
>>>>
>>>>
>>>> Usually I have seen more users doing weird conversions in AMQP when
>>>> they use a C++ or non Java Client.. but on your case you used a Core
>>>> Client, using a non standard way to send a message (ServerMessage from
>>>> the client)... and it's not clear the message is correctly set for a
>>>> conversion to work.
>>>>
>>>>
>>>> I need you to provide some explanation on what you're doing.. if this
>>>> was  a way to hack a bug your saw or if this is just the way you're
>>>> using it.
>>>>
>>>>
>>>> if this is how you're actually using, I would suggest you either use
>>>> the proper APIs or if you're doing some hack for performance
>>>> consideration you have to set the message in a better way the
>>>> converters would understand...
>>>>
>>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>>>> <cl...@gmail.com> wrote:
>>>> >
>>>> > Are you using the LargeServerMessageImpl on your client, or that was
>>>> > just a "hack" to reproduce your issue.
>>>> >
>>>> >
>>>> > LargeServerMessageImpl was not meant to be used on the client. But if
>>>> > you're doing that just to show something you faced in production it's
>>>> > ok.. but using the LargeServerMessageImpl to send a Client Message is
>>>> > a big not for me.
>>>> >
>>>> > The only reason we do that is for server to server transfer.
>>>> >
>>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
>>>> > >
>>>> > > Hello,
>>>> > > Yes, the 2.23.1 test instance was freshly created.
>>>> > >
>>>> > > I've attached a test as a part of ARTEMIS-3897
>>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it helps.
>>>> > >
>>>> > >
>>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <cl...@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Did you start with fresh data on 2.23.1.
>>>> > > >
>>>> > > > If you did please provide a self enclosing test reproducing your issue.
>>>> > > >
>>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com> wrote:
>>>> > > >
>>>> > > > > A quick test using 2.23.1 results in the same error payload being
>>>> > > > received:
>>>> > > > >
>>>> > > > > "Message(address='test', durable=True, priority=4,
>>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
>>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
>>>> > > > > body='Conversion to AMQP error: Error reading in simpleString,
>>>> > > > > length=1330464032 is greater than readableBytes=62')"
>>>> > > > >
>>>> > > > > Best regards
>>>> > > > >
>>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>>>> > > > > <cl...@gmail.com> wrote:
>>>> > > > > >
>>>> > > > > > There's been a few fixes in AMQP Large message.
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > More prominently a fix with the JDBC implementation between 2.17 and
>>>> > > > > HEAD.
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > I would definitely recommend you to upgrade.
>>>> > > > > >
>>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org>
>>>> > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > I would expect this to work. Can you try this on the latest release
>>>> > > > > (i.e.
>>>> > > > > > > 2.23.1) [1]?
>>>> > > > > > >
>>>> > > > > > > If it still doesn't work please open a Jira with all the details
>>>> > > > > necessary
>>>> > > > > > > to reproduce the problem. An actual test-case would be ideal.
>>>> > > > > > >
>>>> > > > > > > Thanks!
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > Justin
>>>> > > > > > >
>>>> > > > > > > [1] https://activemq.apache.org/components/artemis/download/
>>>> > > > > > >
>>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com>
>>>> > > > wrote:
>>>> > > > > > >
>>>> > > > > > > > Hello,
>>>> > > > > > > > Sending a message as a Large Message via Core protocol and
>>>> > > > receiving
>>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the following
>>>> > > > > string
>>>> > > > > > > > as message body:
>>>> > > > > > > >
>>>> > > > > > > > "Conversion to AMQP error: Error reading in simpleString, length=y
>>>> > > > is
>>>> > > > > > > > greater than readableBytes=x"
>>>> > > > > > > >
>>>> > > > > > > > Broker lists the message as Type: Default and Large: True.
>>>> > > > > > > >
>>>> > > > > > > > The receiver is basically Qpid's:
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > >
>>>> > > > https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>>>> > > > > > > >
>>>> > > > > > > > The doc
>>>> > > > > > > >
>>>> > > > >
>>>> > > > https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>>>> > > > > > > > states both Core and AMQP support Large Message. Is this approach
>>>> > > > > > > > valid?
>>>> > > > > > > >
>>>> > > > > > > > Thanks
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > --
>>>> > > > > > Clebert Suconic
>>>> > > > >
>>>> > > > --
>>>> > > > Clebert Suconic
>>>> > > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Clebert Suconic
>>>>
>>>>
>>>>
>>>> --
>>>> Clebert Suconic
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Robbie Gemmell <ro...@gmail.com>.
The snagging described seems to be in the Core sending+conversion legs
of things, so the receiving AMQP client wouldnt seem likely to make
much difference currently. Once its converted any AMQP 1.0 client
should be able to receive it really.

On Fri, 22 Jul 2022 at 20:01, Clebert Suconic <cl...@gmail.com> wrote:
>
> If you’re not using qpid JMS it will not work.
>
> On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:
>
> > There is another attempt employing AMQP test structures. It produces the
> > same incorrect message body.
> >
> > On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
> >
> >> Hello,
> >> Using the LargeServerMessageImpl was just an attempt providing a
> >> seemingly easy way to introduce a kind of Large Message into the test. I'm
> >> not familiar with the code base at all - so I simply tried to somehow
> >> provide the requested test replicating my "sending via Core -> receiving
> >> via Qpid" issue. The same applies to the explicit AMQP conversion - the
> >> encountered error simply pointed on it. Is there an intended "test way" to
> >> model the message transfer among different clients/protocols?
> >>
> >> Anyway, I've changed the test to employ
> >> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> >> LargeServerMessageImpl with a very similar result. Does that implementation
> >> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> >> better or is the explicit AMQP conversion misused?
> >>
> >> Please, see the attached patch.
> >>
> >> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
> >> clebert.suconic@gmail.com> wrote:
> >>
> >>> AMQP to Core and Core to AMQP conversions expect you using JMS clients
> >>> to generate your message. There are certain caveats that both clients
> >>> will add to the messages.
> >>>
> >>>
> >>> Usually I have seen more users doing weird conversions in AMQP when
> >>> they use a C++ or non Java Client.. but on your case you used a Core
> >>> Client, using a non standard way to send a message (ServerMessage from
> >>> the client)... and it's not clear the message is correctly set for a
> >>> conversion to work.
> >>>
> >>>
> >>> I need you to provide some explanation on what you're doing.. if this
> >>> was  a way to hack a bug your saw or if this is just the way you're
> >>> using it.
> >>>
> >>>
> >>> if this is how you're actually using, I would suggest you either use
> >>> the proper APIs or if you're doing some hack for performance
> >>> consideration you have to set the message in a better way the
> >>> converters would understand...
> >>>
> >>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> >>> <cl...@gmail.com> wrote:
> >>> >
> >>> > Are you using the LargeServerMessageImpl on your client, or that was
> >>> > just a "hack" to reproduce your issue.
> >>> >
> >>> >
> >>> > LargeServerMessageImpl was not meant to be used on the client. But if
> >>> > you're doing that just to show something you faced in production it's
> >>> > ok.. but using the LargeServerMessageImpl to send a Client Message is
> >>> > a big not for me.
> >>> >
> >>> > The only reason we do that is for server to server transfer.
> >>> >
> >>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
> >>> > >
> >>> > > Hello,
> >>> > > Yes, the 2.23.1 test instance was freshly created.
> >>> > >
> >>> > > I've attached a test as a part of ARTEMIS-3897
> >>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
> >>> helps.
> >>> > >
> >>> > >
> >>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> >>> clebert.suconic@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Did you start with fresh data on 2.23.1.
> >>> > > >
> >>> > > > If you did please provide a self enclosing test reproducing your
> >>> issue.
> >>> > > >
> >>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> >>> wrote:
> >>> > > >
> >>> > > > > A quick test using 2.23.1 results in the same error payload being
> >>> > > > received:
> >>> > > > >
> >>> > > > > "Message(address='test', durable=True, priority=4,
> >>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
> >>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> >>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
> >>> > > > > body='Conversion to AMQP error: Error reading in simpleString,
> >>> > > > > length=1330464032 is greater than readableBytes=62')"
> >>> > > > >
> >>> > > > > Best regards
> >>> > > > >
> >>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> >>> > > > > <cl...@gmail.com> wrote:
> >>> > > > > >
> >>> > > > > > There's been a few fixes in AMQP Large message.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > More prominently a fix with the JDBC implementation between
> >>> 2.17 and
> >>> > > > > HEAD.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > I would definitely recommend you to upgrade.
> >>> > > > > >
> >>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> >>> jbertram@apache.org>
> >>> > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > I would expect this to work. Can you try this on the latest
> >>> release
> >>> > > > > (i.e.
> >>> > > > > > > 2.23.1) [1]?
> >>> > > > > > >
> >>> > > > > > > If it still doesn't work please open a Jira with all the
> >>> details
> >>> > > > > necessary
> >>> > > > > > > to reproduce the problem. An actual test-case would be ideal.
> >>> > > > > > >
> >>> > > > > > > Thanks!
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > Justin
> >>> > > > > > >
> >>> > > > > > > [1] https://activemq.apache.org/components/artemis/download/
> >>> > > > > > >
> >>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
> >>> andyyar66@gmail.com>
> >>> > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hello,
> >>> > > > > > > > Sending a message as a Large Message via Core protocol and
> >>> > > > receiving
> >>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> >>> following
> >>> > > > > string
> >>> > > > > > > > as message body:
> >>> > > > > > > >
> >>> > > > > > > > "Conversion to AMQP error: Error reading in simpleString,
> >>> length=y
> >>> > > > is
> >>> > > > > > > > greater than readableBytes=x"
> >>> > > > > > > >
> >>> > > > > > > > Broker lists the message as Type: Default and Large: True.
> >>> > > > > > > >
> >>> > > > > > > > The receiver is basically Qpid's:
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > >
> >>> > > >
> >>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> >>> > > > > > > >
> >>> > > > > > > > The doc
> >>> > > > > > > >
> >>> > > > >
> >>> > > >
> >>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> >>> > > > > > > > states both Core and AMQP support Large Message. Is this
> >>> approach
> >>> > > > > > > > valid?
> >>> > > > > > > >
> >>> > > > > > > > Thanks
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > --
> >>> > > > > > Clebert Suconic
> >>> > > > >
> >>> > > > --
> >>> > > > Clebert Suconic
> >>> > > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Clebert Suconic
> >>>
> >>>
> >>>
> >>> --
> >>> Clebert Suconic
> >>>
> >> --
> Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
If you’re not using qpid JMS it will not work.

On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <an...@gmail.com> wrote:

> There is another attempt employing AMQP test structures. It produces the
> same incorrect message body.
>
> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:
>
>> Hello,
>> Using the LargeServerMessageImpl was just an attempt providing a
>> seemingly easy way to introduce a kind of Large Message into the test. I'm
>> not familiar with the code base at all - so I simply tried to somehow
>> provide the requested test replicating my "sending via Core -> receiving
>> via Qpid" issue. The same applies to the explicit AMQP conversion - the
>> encountered error simply pointed on it. Is there an intended "test way" to
>> model the message transfer among different clients/protocols?
>>
>> Anyway, I've changed the test to employ
>> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
>> LargeServerMessageImpl with a very similar result. Does that implementation
>> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
>> better or is the explicit AMQP conversion misused?
>>
>> Please, see the attached patch.
>>
>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <
>> clebert.suconic@gmail.com> wrote:
>>
>>> AMQP to Core and Core to AMQP conversions expect you using JMS clients
>>> to generate your message. There are certain caveats that both clients
>>> will add to the messages.
>>>
>>>
>>> Usually I have seen more users doing weird conversions in AMQP when
>>> they use a C++ or non Java Client.. but on your case you used a Core
>>> Client, using a non standard way to send a message (ServerMessage from
>>> the client)... and it's not clear the message is correctly set for a
>>> conversion to work.
>>>
>>>
>>> I need you to provide some explanation on what you're doing.. if this
>>> was  a way to hack a bug your saw or if this is just the way you're
>>> using it.
>>>
>>>
>>> if this is how you're actually using, I would suggest you either use
>>> the proper APIs or if you're doing some hack for performance
>>> consideration you have to set the message in a better way the
>>> converters would understand...
>>>
>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>>> <cl...@gmail.com> wrote:
>>> >
>>> > Are you using the LargeServerMessageImpl on your client, or that was
>>> > just a "hack" to reproduce your issue.
>>> >
>>> >
>>> > LargeServerMessageImpl was not meant to be used on the client. But if
>>> > you're doing that just to show something you faced in production it's
>>> > ok.. but using the LargeServerMessageImpl to send a Client Message is
>>> > a big not for me.
>>> >
>>> > The only reason we do that is for server to server transfer.
>>> >
>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
>>> > >
>>> > > Hello,
>>> > > Yes, the 2.23.1 test instance was freshly created.
>>> > >
>>> > > I've attached a test as a part of ARTEMIS-3897
>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
>>> helps.
>>> > >
>>> > >
>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Did you start with fresh data on 2.23.1.
>>> > > >
>>> > > > If you did please provide a self enclosing test reproducing your
>>> issue.
>>> > > >
>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > > A quick test using 2.23.1 results in the same error payload being
>>> > > > received:
>>> > > > >
>>> > > > > "Message(address='test', durable=True, priority=4,
>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
>>> > > > > body='Conversion to AMQP error: Error reading in simpleString,
>>> > > > > length=1330464032 is greater than readableBytes=62')"
>>> > > > >
>>> > > > > Best regards
>>> > > > >
>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>>> > > > > <cl...@gmail.com> wrote:
>>> > > > > >
>>> > > > > > There's been a few fixes in AMQP Large message.
>>> > > > > >
>>> > > > > >
>>> > > > > > More prominently a fix with the JDBC implementation between
>>> 2.17 and
>>> > > > > HEAD.
>>> > > > > >
>>> > > > > >
>>> > > > > > I would definitely recommend you to upgrade.
>>> > > > > >
>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
>>> jbertram@apache.org>
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > > I would expect this to work. Can you try this on the latest
>>> release
>>> > > > > (i.e.
>>> > > > > > > 2.23.1) [1]?
>>> > > > > > >
>>> > > > > > > If it still doesn't work please open a Jira with all the
>>> details
>>> > > > > necessary
>>> > > > > > > to reproduce the problem. An actual test-case would be ideal.
>>> > > > > > >
>>> > > > > > > Thanks!
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > Justin
>>> > > > > > >
>>> > > > > > > [1] https://activemq.apache.org/components/artemis/download/
>>> > > > > > >
>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <
>>> andyyar66@gmail.com>
>>> > > > wrote:
>>> > > > > > >
>>> > > > > > > > Hello,
>>> > > > > > > > Sending a message as a Large Message via Core protocol and
>>> > > > receiving
>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
>>> following
>>> > > > > string
>>> > > > > > > > as message body:
>>> > > > > > > >
>>> > > > > > > > "Conversion to AMQP error: Error reading in simpleString,
>>> length=y
>>> > > > is
>>> > > > > > > > greater than readableBytes=x"
>>> > > > > > > >
>>> > > > > > > > Broker lists the message as Type: Default and Large: True.
>>> > > > > > > >
>>> > > > > > > > The receiver is basically Qpid's:
>>> > > > > > > >
>>> > > > > > > >
>>> > > > >
>>> > > >
>>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>>> > > > > > > >
>>> > > > > > > > The doc
>>> > > > > > > >
>>> > > > >
>>> > > >
>>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>>> > > > > > > > states both Core and AMQP support Large Message. Is this
>>> approach
>>> > > > > > > > valid?
>>> > > > > > > >
>>> > > > > > > > Thanks
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Clebert Suconic
>>> > > > >
>>> > > > --
>>> > > > Clebert Suconic
>>> > > >
>>> >
>>> >
>>> >
>>> > --
>>> > Clebert Suconic
>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>>
>> --
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Andy Yar <an...@gmail.com>.
There is another attempt employing AMQP test structures. It produces the
same incorrect message body.

On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <an...@gmail.com> wrote:

> Hello,
> Using the LargeServerMessageImpl was just an attempt providing a seemingly
> easy way to introduce a kind of Large Message into the test. I'm not
> familiar with the code base at all - so I simply tried to somehow provide
> the requested test replicating my "sending via Core -> receiving via Qpid"
> issue. The same applies to the explicit AMQP conversion - the encountered
> error simply pointed on it. Is there an intended "test way" to model the
> message transfer among different clients/protocols?
>
> Anyway, I've changed the test to employ
> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
> LargeServerMessageImpl with a very similar result. Does that implementation
> fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
> better or is the explicit AMQP conversion misused?
>
> Please, see the attached patch.
>
> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> AMQP to Core and Core to AMQP conversions expect you using JMS clients
>> to generate your message. There are certain caveats that both clients
>> will add to the messages.
>>
>>
>> Usually I have seen more users doing weird conversions in AMQP when
>> they use a C++ or non Java Client.. but on your case you used a Core
>> Client, using a non standard way to send a message (ServerMessage from
>> the client)... and it's not clear the message is correctly set for a
>> conversion to work.
>>
>>
>> I need you to provide some explanation on what you're doing.. if this
>> was  a way to hack a bug your saw or if this is just the way you're
>> using it.
>>
>>
>> if this is how you're actually using, I would suggest you either use
>> the proper APIs or if you're doing some hack for performance
>> consideration you have to set the message in a better way the
>> converters would understand...
>>
>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>> <cl...@gmail.com> wrote:
>> >
>> > Are you using the LargeServerMessageImpl on your client, or that was
>> > just a "hack" to reproduce your issue.
>> >
>> >
>> > LargeServerMessageImpl was not meant to be used on the client. But if
>> > you're doing that just to show something you faced in production it's
>> > ok.. but using the LargeServerMessageImpl to send a Client Message is
>> > a big not for me.
>> >
>> > The only reason we do that is for server to server transfer.
>> >
>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
>> > >
>> > > Hello,
>> > > Yes, the 2.23.1 test instance was freshly created.
>> > >
>> > > I've attached a test as a part of ARTEMIS-3897
>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it
>> helps.
>> > >
>> > >
>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> > > wrote:
>> > >
>> > > > Did you start with fresh data on 2.23.1.
>> > > >
>> > > > If you did please provide a self enclosing test reproducing your
>> issue.
>> > > >
>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
>> wrote:
>> > > >
>> > > > > A quick test using 2.23.1 results in the same error payload being
>> > > > received:
>> > > > >
>> > > > > "Message(address='test', durable=True, priority=4,
>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
>> > > > > body='Conversion to AMQP error: Error reading in simpleString,
>> > > > > length=1330464032 is greater than readableBytes=62')"
>> > > > >
>> > > > > Best regards
>> > > > >
>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>> > > > > <cl...@gmail.com> wrote:
>> > > > > >
>> > > > > > There's been a few fixes in AMQP Large message.
>> > > > > >
>> > > > > >
>> > > > > > More prominently a fix with the JDBC implementation between
>> 2.17 and
>> > > > > HEAD.
>> > > > > >
>> > > > > >
>> > > > > > I would definitely recommend you to upgrade.
>> > > > > >
>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
>> jbertram@apache.org>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > I would expect this to work. Can you try this on the latest
>> release
>> > > > > (i.e.
>> > > > > > > 2.23.1) [1]?
>> > > > > > >
>> > > > > > > If it still doesn't work please open a Jira with all the
>> details
>> > > > > necessary
>> > > > > > > to reproduce the problem. An actual test-case would be ideal.
>> > > > > > >
>> > > > > > > Thanks!
>> > > > > > >
>> > > > > > >
>> > > > > > > Justin
>> > > > > > >
>> > > > > > > [1] https://activemq.apache.org/components/artemis/download/
>> > > > > > >
>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <andyyar66@gmail.com
>> >
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Hello,
>> > > > > > > > Sending a message as a Large Message via Core protocol and
>> > > > receiving
>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
>> following
>> > > > > string
>> > > > > > > > as message body:
>> > > > > > > >
>> > > > > > > > "Conversion to AMQP error: Error reading in simpleString,
>> length=y
>> > > > is
>> > > > > > > > greater than readableBytes=x"
>> > > > > > > >
>> > > > > > > > Broker lists the message as Type: Default and Large: True.
>> > > > > > > >
>> > > > > > > > The receiver is basically Qpid's:
>> > > > > > > >
>> > > > > > > >
>> > > > >
>> > > >
>> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>> > > > > > > >
>> > > > > > > > The doc
>> > > > > > > >
>> > > > >
>> > > >
>> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>> > > > > > > > states both Core and AMQP support Large Message. Is this
>> approach
>> > > > > > > > valid?
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Clebert Suconic
>> > > > >
>> > > > --
>> > > > Clebert Suconic
>> > > >
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>>
>

Re: Artemis - Large Message - Core => AMQP

Posted by Robbie Gemmell <ro...@gmail.com>.
The mailing lists strip basically all attachments so noone saw what
you sent. You could attach it to the JIRA where the original was.

On Fri, 22 Jul 2022 at 11:22, Andy Yar <an...@gmail.com> wrote:
>
> Hello,
> Using the LargeServerMessageImpl was just an attempt providing a seemingly easy way to introduce a kind of Large Message into the test. I'm not familiar with the code base at all - so I simply tried to somehow provide the requested test replicating my "sending via Core -> receiving via Qpid" issue. The same applies to the explicit AMQP conversion - the encountered error simply pointed on it. Is there an intended "test way" to model the message transfer among different clients/protocols?
>
> Anyway, I've changed the test to employ ClientLargeMessageImpl/ClientMessageImpl instead of the server-related LargeServerMessageImpl with a very similar result. Does that implementation fit the simulated use case (sending via Core -> receiving via Qpid AMQP) better or is the explicit AMQP conversion misused?
>
> Please, see the attached patch.
>
> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <cl...@gmail.com> wrote:
>>
>> AMQP to Core and Core to AMQP conversions expect you using JMS clients
>> to generate your message. There are certain caveats that both clients
>> will add to the messages.
>>
>>
>> Usually I have seen more users doing weird conversions in AMQP when
>> they use a C++ or non Java Client.. but on your case you used a Core
>> Client, using a non standard way to send a message (ServerMessage from
>> the client)... and it's not clear the message is correctly set for a
>> conversion to work.
>>
>>
>> I need you to provide some explanation on what you're doing.. if this
>> was  a way to hack a bug your saw or if this is just the way you're
>> using it.
>>
>>
>> if this is how you're actually using, I would suggest you either use
>> the proper APIs or if you're doing some hack for performance
>> consideration you have to set the message in a better way the
>> converters would understand...
>>
>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
>> <cl...@gmail.com> wrote:
>> >
>> > Are you using the LargeServerMessageImpl on your client, or that was
>> > just a "hack" to reproduce your issue.
>> >
>> >
>> > LargeServerMessageImpl was not meant to be used on the client. But if
>> > you're doing that just to show something you faced in production it's
>> > ok.. but using the LargeServerMessageImpl to send a Client Message is
>> > a big not for me.
>> >
>> > The only reason we do that is for server to server transfer.
>> >
>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
>> > >
>> > > Hello,
>> > > Yes, the 2.23.1 test instance was freshly created.
>> > >
>> > > I've attached a test as a part of ARTEMIS-3897
>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it helps.
>> > >
>> > >
>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <cl...@gmail.com>
>> > > wrote:
>> > >
>> > > > Did you start with fresh data on 2.23.1.
>> > > >
>> > > > If you did please provide a self enclosing test reproducing your issue.
>> > > >
>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com> wrote:
>> > > >
>> > > > > A quick test using 2.23.1 results in the same error payload being
>> > > > received:
>> > > > >
>> > > > > "Message(address='test', durable=True, priority=4,
>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
>> > > > > body='Conversion to AMQP error: Error reading in simpleString,
>> > > > > length=1330464032 is greater than readableBytes=62')"
>> > > > >
>> > > > > Best regards
>> > > > >
>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
>> > > > > <cl...@gmail.com> wrote:
>> > > > > >
>> > > > > > There's been a few fixes in AMQP Large message.
>> > > > > >
>> > > > > >
>> > > > > > More prominently a fix with the JDBC implementation between 2.17 and
>> > > > > HEAD.
>> > > > > >
>> > > > > >
>> > > > > > I would definitely recommend you to upgrade.
>> > > > > >
>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > I would expect this to work. Can you try this on the latest release
>> > > > > (i.e.
>> > > > > > > 2.23.1) [1]?
>> > > > > > >
>> > > > > > > If it still doesn't work please open a Jira with all the details
>> > > > > necessary
>> > > > > > > to reproduce the problem. An actual test-case would be ideal.
>> > > > > > >
>> > > > > > > Thanks!
>> > > > > > >
>> > > > > > >
>> > > > > > > Justin
>> > > > > > >
>> > > > > > > [1] https://activemq.apache.org/components/artemis/download/
>> > > > > > >
>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Hello,
>> > > > > > > > Sending a message as a Large Message via Core protocol and
>> > > > receiving
>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the following
>> > > > > string
>> > > > > > > > as message body:
>> > > > > > > >
>> > > > > > > > "Conversion to AMQP error: Error reading in simpleString, length=y
>> > > > is
>> > > > > > > > greater than readableBytes=x"
>> > > > > > > >
>> > > > > > > > Broker lists the message as Type: Default and Large: True.
>> > > > > > > >
>> > > > > > > > The receiver is basically Qpid's:
>> > > > > > > >
>> > > > > > > >
>> > > > >
>> > > > https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>> > > > > > > >
>> > > > > > > > The doc
>> > > > > > > >
>> > > > >
>> > > > https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
>> > > > > > > > states both Core and AMQP support Large Message. Is this approach
>> > > > > > > > valid?
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Clebert Suconic
>> > > > >
>> > > > --
>> > > > Clebert Suconic
>> > > >
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Andy Yar <an...@gmail.com>.
Hello,
Using the LargeServerMessageImpl was just an attempt providing a seemingly
easy way to introduce a kind of Large Message into the test. I'm not
familiar with the code base at all - so I simply tried to somehow provide
the requested test replicating my "sending via Core -> receiving via Qpid"
issue. The same applies to the explicit AMQP conversion - the encountered
error simply pointed on it. Is there an intended "test way" to model the
message transfer among different clients/protocols?

Anyway, I've changed the test to employ
ClientLargeMessageImpl/ClientMessageImpl instead of the server-related
LargeServerMessageImpl with a very similar result. Does that implementation
fit the simulated use case (sending via Core -> receiving via Qpid AMQP)
better or is the explicit AMQP conversion misused?

Please, see the attached patch.

On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic <cl...@gmail.com>
wrote:

> AMQP to Core and Core to AMQP conversions expect you using JMS clients
> to generate your message. There are certain caveats that both clients
> will add to the messages.
>
>
> Usually I have seen more users doing weird conversions in AMQP when
> they use a C++ or non Java Client.. but on your case you used a Core
> Client, using a non standard way to send a message (ServerMessage from
> the client)... and it's not clear the message is correctly set for a
> conversion to work.
>
>
> I need you to provide some explanation on what you're doing.. if this
> was  a way to hack a bug your saw or if this is just the way you're
> using it.
>
>
> if this is how you're actually using, I would suggest you either use
> the proper APIs or if you're doing some hack for performance
> consideration you have to set the message in a better way the
> converters would understand...
>
> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > Are you using the LargeServerMessageImpl on your client, or that was
> > just a "hack" to reproduce your issue.
> >
> >
> > LargeServerMessageImpl was not meant to be used on the client. But if
> > you're doing that just to show something you faced in production it's
> > ok.. but using the LargeServerMessageImpl to send a Client Message is
> > a big not for me.
> >
> > The only reason we do that is for server to server transfer.
> >
> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
> > >
> > > Hello,
> > > Yes, the 2.23.1 test instance was freshly created.
> > >
> > > I've attached a test as a part of ARTEMIS-3897
> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it helps.
> > >
> > >
> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <
> clebert.suconic@gmail.com>
> > > wrote:
> > >
> > > > Did you start with fresh data on 2.23.1.
> > > >
> > > > If you did please provide a self enclosing test reproducing your
> issue.
> > > >
> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com>
> wrote:
> > > >
> > > > > A quick test using 2.23.1 results in the same error payload being
> > > > received:
> > > > >
> > > > > "Message(address='test', durable=True, priority=4,
> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
> > > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
> > > > > body='Conversion to AMQP error: Error reading in simpleString,
> > > > > length=1330464032 is greater than readableBytes=62')"
> > > > >
> > > > > Best regards
> > > > >
> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> > > > > <cl...@gmail.com> wrote:
> > > > > >
> > > > > > There's been a few fixes in AMQP Large message.
> > > > > >
> > > > > >
> > > > > > More prominently a fix with the JDBC implementation between 2.17
> and
> > > > > HEAD.
> > > > > >
> > > > > >
> > > > > > I would definitely recommend you to upgrade.
> > > > > >
> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <
> jbertram@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > I would expect this to work. Can you try this on the latest
> release
> > > > > (i.e.
> > > > > > > 2.23.1) [1]?
> > > > > > >
> > > > > > > If it still doesn't work please open a Jira with all the
> details
> > > > > necessary
> > > > > > > to reproduce the problem. An actual test-case would be ideal.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > >
> > > > > > > Justin
> > > > > > >
> > > > > > > [1] https://activemq.apache.org/components/artemis/download/
> > > > > > >
> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > > Sending a message as a Large Message via Core protocol and
> > > > receiving
> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the
> following
> > > > > string
> > > > > > > > as message body:
> > > > > > > >
> > > > > > > > "Conversion to AMQP error: Error reading in simpleString,
> length=y
> > > > is
> > > > > > > > greater than readableBytes=x"
> > > > > > > >
> > > > > > > > Broker lists the message as Type: Default and Large: True.
> > > > > > > >
> > > > > > > > The receiver is basically Qpid's:
> > > > > > > >
> > > > > > > >
> > > > >
> > > >
> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > > > > > > >
> > > > > > > > The doc
> > > > > > > >
> > > > >
> > > >
> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > > > > > > > states both Core and AMQP support Large Message. Is this
> approach
> > > > > > > > valid?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > >
> > > > --
> > > > Clebert Suconic
> > > >
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
AMQP to Core and Core to AMQP conversions expect you using JMS clients
to generate your message. There are certain caveats that both clients
will add to the messages.


Usually I have seen more users doing weird conversions in AMQP when
they use a C++ or non Java Client.. but on your case you used a Core
Client, using a non standard way to send a message (ServerMessage from
the client)... and it's not clear the message is correctly set for a
conversion to work.


I need you to provide some explanation on what you're doing.. if this
was  a way to hack a bug your saw or if this is just the way you're
using it.


if this is how you're actually using, I would suggest you either use
the proper APIs or if you're doing some hack for performance
consideration you have to set the message in a better way the
converters would understand...

On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic
<cl...@gmail.com> wrote:
>
> Are you using the LargeServerMessageImpl on your client, or that was
> just a "hack" to reproduce your issue.
>
>
> LargeServerMessageImpl was not meant to be used on the client. But if
> you're doing that just to show something you faced in production it's
> ok.. but using the LargeServerMessageImpl to send a Client Message is
> a big not for me.
>
> The only reason we do that is for server to server transfer.
>
> On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
> >
> > Hello,
> > Yes, the 2.23.1 test instance was freshly created.
> >
> > I've attached a test as a part of ARTEMIS-3897
> > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it helps.
> >
> >
> > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <cl...@gmail.com>
> > wrote:
> >
> > > Did you start with fresh data on 2.23.1.
> > >
> > > If you did please provide a self enclosing test reproducing your issue.
> > >
> > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com> wrote:
> > >
> > > > A quick test using 2.23.1 results in the same error payload being
> > > received:
> > > >
> > > > "Message(address='test', durable=True, priority=4,
> > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
> > > > symbol('x-opt-jms-msg-type'): byte(0)}),
> > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
> > > > body='Conversion to AMQP error: Error reading in simpleString,
> > > > length=1330464032 is greater than readableBytes=62')"
> > > >
> > > > Best regards
> > > >
> > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> > > > <cl...@gmail.com> wrote:
> > > > >
> > > > > There's been a few fixes in AMQP Large message.
> > > > >
> > > > >
> > > > > More prominently a fix with the JDBC implementation between 2.17 and
> > > > HEAD.
> > > > >
> > > > >
> > > > > I would definitely recommend you to upgrade.
> > > > >
> > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > I would expect this to work. Can you try this on the latest release
> > > > (i.e.
> > > > > > 2.23.1) [1]?
> > > > > >
> > > > > > If it still doesn't work please open a Jira with all the details
> > > > necessary
> > > > > > to reproduce the problem. An actual test-case would be ideal.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > >
> > > > > > Justin
> > > > > >
> > > > > > [1] https://activemq.apache.org/components/artemis/download/
> > > > > >
> > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > > Sending a message as a Large Message via Core protocol and
> > > receiving
> > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the following
> > > > string
> > > > > > > as message body:
> > > > > > >
> > > > > > > "Conversion to AMQP error: Error reading in simpleString, length=y
> > > is
> > > > > > > greater than readableBytes=x"
> > > > > > >
> > > > > > > Broker lists the message as Type: Default and Large: True.
> > > > > > >
> > > > > > > The receiver is basically Qpid's:
> > > > > > >
> > > > > > >
> > > >
> > > https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > > > > > >
> > > > > > > The doc
> > > > > > >
> > > >
> > > https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > > > > > > states both Core and AMQP support Large Message. Is this approach
> > > > > > > valid?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > >
> > > --
> > > Clebert Suconic
> > >
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
Are you using the LargeServerMessageImpl on your client, or that was
just a "hack" to reproduce your issue.


LargeServerMessageImpl was not meant to be used on the client. But if
you're doing that just to show something you faced in production it's
ok.. but using the LargeServerMessageImpl to send a Client Message is
a big not for me.

The only reason we do that is for server to server transfer.

On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <an...@gmail.com> wrote:
>
> Hello,
> Yes, the 2.23.1 test instance was freshly created.
>
> I've attached a test as a part of ARTEMIS-3897
> <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it helps.
>
>
> On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > Did you start with fresh data on 2.23.1.
> >
> > If you did please provide a self enclosing test reproducing your issue.
> >
> > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com> wrote:
> >
> > > A quick test using 2.23.1 results in the same error payload being
> > received:
> > >
> > > "Message(address='test', durable=True, priority=4,
> > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
> > > symbol('x-opt-jms-msg-type'): byte(0)}),
> > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
> > > body='Conversion to AMQP error: Error reading in simpleString,
> > > length=1330464032 is greater than readableBytes=62')"
> > >
> > > Best regards
> > >
> > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> > > <cl...@gmail.com> wrote:
> > > >
> > > > There's been a few fixes in AMQP Large message.
> > > >
> > > >
> > > > More prominently a fix with the JDBC implementation between 2.17 and
> > > HEAD.
> > > >
> > > >
> > > > I would definitely recommend you to upgrade.
> > > >
> > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org>
> > > wrote:
> > > > >
> > > > > I would expect this to work. Can you try this on the latest release
> > > (i.e.
> > > > > 2.23.1) [1]?
> > > > >
> > > > > If it still doesn't work please open a Jira with all the details
> > > necessary
> > > > > to reproduce the problem. An actual test-case would be ideal.
> > > > >
> > > > > Thanks!
> > > > >
> > > > >
> > > > > Justin
> > > > >
> > > > > [1] https://activemq.apache.org/components/artemis/download/
> > > > >
> > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hello,
> > > > > > Sending a message as a Large Message via Core protocol and
> > receiving
> > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the following
> > > string
> > > > > > as message body:
> > > > > >
> > > > > > "Conversion to AMQP error: Error reading in simpleString, length=y
> > is
> > > > > > greater than readableBytes=x"
> > > > > >
> > > > > > Broker lists the message as Type: Default and Large: True.
> > > > > >
> > > > > > The receiver is basically Qpid's:
> > > > > >
> > > > > >
> > >
> > https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > > > > >
> > > > > > The doc
> > > > > >
> > >
> > https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > > > > > states both Core and AMQP support Large Message. Is this approach
> > > > > > valid?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > >
> > --
> > Clebert Suconic
> >



-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Andy Yar <an...@gmail.com>.
Hello,
Yes, the 2.23.1 test instance was freshly created.

I've attached a test as a part of ARTEMIS-3897
<https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it helps.


On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic <cl...@gmail.com>
wrote:

> Did you start with fresh data on 2.23.1.
>
> If you did please provide a self enclosing test reproducing your issue.
>
> On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com> wrote:
>
> > A quick test using 2.23.1 results in the same error payload being
> received:
> >
> > "Message(address='test', durable=True, priority=4,
> > annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
> > symbol('x-opt-jms-msg-type'): byte(0)}),
> > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
> > body='Conversion to AMQP error: Error reading in simpleString,
> > length=1330464032 is greater than readableBytes=62')"
> >
> > Best regards
> >
> > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> > <cl...@gmail.com> wrote:
> > >
> > > There's been a few fixes in AMQP Large message.
> > >
> > >
> > > More prominently a fix with the JDBC implementation between 2.17 and
> > HEAD.
> > >
> > >
> > > I would definitely recommend you to upgrade.
> > >
> > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org>
> > wrote:
> > > >
> > > > I would expect this to work. Can you try this on the latest release
> > (i.e.
> > > > 2.23.1) [1]?
> > > >
> > > > If it still doesn't work please open a Jira with all the details
> > necessary
> > > > to reproduce the problem. An actual test-case would be ideal.
> > > >
> > > > Thanks!
> > > >
> > > >
> > > > Justin
> > > >
> > > > [1] https://activemq.apache.org/components/artemis/download/
> > > >
> > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com>
> wrote:
> > > >
> > > > > Hello,
> > > > > Sending a message as a Large Message via Core protocol and
> receiving
> > > > > it via AMQP on Artemis 2.17.0 ends with receiving the following
> > string
> > > > > as message body:
> > > > >
> > > > > "Conversion to AMQP error: Error reading in simpleString, length=y
> is
> > > > > greater than readableBytes=x"
> > > > >
> > > > > Broker lists the message as Type: Default and Large: True.
> > > > >
> > > > > The receiver is basically Qpid's:
> > > > >
> > > > >
> >
> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > > > >
> > > > > The doc
> > > > >
> >
> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > > > > states both Core and AMQP support Large Message. Is this approach
> > > > > valid?
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> >
> --
> Clebert Suconic
>

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
Did you start with fresh data on 2.23.1.

If you did please provide a self enclosing test reproducing your issue.

On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <an...@gmail.com> wrote:

> A quick test using 2.23.1 results in the same error payload being received:
>
> "Message(address='test', durable=True, priority=4,
> annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
> symbol('x-opt-jms-msg-type'): byte(0)}),
> properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
> body='Conversion to AMQP error: Error reading in simpleString,
> length=1330464032 is greater than readableBytes=62')"
>
> Best regards
>
> On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > There's been a few fixes in AMQP Large message.
> >
> >
> > More prominently a fix with the JDBC implementation between 2.17 and
> HEAD.
> >
> >
> > I would definitely recommend you to upgrade.
> >
> > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org>
> wrote:
> > >
> > > I would expect this to work. Can you try this on the latest release
> (i.e.
> > > 2.23.1) [1]?
> > >
> > > If it still doesn't work please open a Jira with all the details
> necessary
> > > to reproduce the problem. An actual test-case would be ideal.
> > >
> > > Thanks!
> > >
> > >
> > > Justin
> > >
> > > [1] https://activemq.apache.org/components/artemis/download/
> > >
> > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com> wrote:
> > >
> > > > Hello,
> > > > Sending a message as a Large Message via Core protocol and receiving
> > > > it via AMQP on Artemis 2.17.0 ends with receiving the following
> string
> > > > as message body:
> > > >
> > > > "Conversion to AMQP error: Error reading in simpleString, length=y is
> > > > greater than readableBytes=x"
> > > >
> > > > Broker lists the message as Type: Default and Large: True.
> > > >
> > > > The receiver is basically Qpid's:
> > > >
> > > >
> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > > >
> > > > The doc
> > > >
> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > > > states both Core and AMQP support Large Message. Is this approach
> > > > valid?
> > > >
> > > > Thanks
> > > >
> > > >
> >
> >
> >
> > --
> > Clebert Suconic
>
-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Andy Yar <an...@gmail.com>.
A quick test using 2.23.1 results in the same error payload being received:

"Message(address='test', durable=True, priority=4,
annotations=AnnotationDict({symbol('x-opt-jms-dest'): byte(0),
symbol('x-opt-jms-msg-type'): byte(0)}),
properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': 67},
body='Conversion to AMQP error: Error reading in simpleString,
length=1330464032 is greater than readableBytes=62')"

Best regards

On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> There's been a few fixes in AMQP Large message.
>
>
> More prominently a fix with the JDBC implementation between 2.17 and HEAD.
>
>
> I would definitely recommend you to upgrade.
>
> On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org> wrote:
> >
> > I would expect this to work. Can you try this on the latest release (i.e.
> > 2.23.1) [1]?
> >
> > If it still doesn't work please open a Jira with all the details necessary
> > to reproduce the problem. An actual test-case would be ideal.
> >
> > Thanks!
> >
> >
> > Justin
> >
> > [1] https://activemq.apache.org/components/artemis/download/
> >
> > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com> wrote:
> >
> > > Hello,
> > > Sending a message as a Large Message via Core protocol and receiving
> > > it via AMQP on Artemis 2.17.0 ends with receiving the following string
> > > as message body:
> > >
> > > "Conversion to AMQP error: Error reading in simpleString, length=y is
> > > greater than readableBytes=x"
> > >
> > > Broker lists the message as Type: Default and Large: True.
> > >
> > > The receiver is basically Qpid's:
> > >
> > > https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> > >
> > > The doc
> > > https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > > states both Core and AMQP support Large Message. Is this approach
> > > valid?
> > >
> > > Thanks
> > >
> > >
>
>
>
> --
> Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Clebert Suconic <cl...@gmail.com>.
There's been a few fixes in AMQP Large message.


More prominently a fix with the JDBC implementation between 2.17 and HEAD.


I would definitely recommend you to upgrade.

On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram <jb...@apache.org> wrote:
>
> I would expect this to work. Can you try this on the latest release (i.e.
> 2.23.1) [1]?
>
> If it still doesn't work please open a Jira with all the details necessary
> to reproduce the problem. An actual test-case would be ideal.
>
> Thanks!
>
>
> Justin
>
> [1] https://activemq.apache.org/components/artemis/download/
>
> On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com> wrote:
>
> > Hello,
> > Sending a message as a Large Message via Core protocol and receiving
> > it via AMQP on Artemis 2.17.0 ends with receiving the following string
> > as message body:
> >
> > "Conversion to AMQP error: Error reading in simpleString, length=y is
> > greater than readableBytes=x"
> >
> > Broker lists the message as Type: Default and Large: True.
> >
> > The receiver is basically Qpid's:
> >
> > https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
> >
> > The doc
> > https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> > states both Core and AMQP support Large Message. Is this approach
> > valid?
> >
> > Thanks
> >
> >



-- 
Clebert Suconic

Re: Artemis - Large Message - Core => AMQP

Posted by Justin Bertram <jb...@apache.org>.
I would expect this to work. Can you try this on the latest release (i.e.
2.23.1) [1]?

If it still doesn't work please open a Jira with all the details necessary
to reproduce the problem. An actual test-case would be ideal.

Thanks!


Justin

[1] https://activemq.apache.org/components/artemis/download/

On Thu, Jul 14, 2022 at 9:20 AM Andy Yar <an...@gmail.com> wrote:

> Hello,
> Sending a message as a Large Message via Core protocol and receiving
> it via AMQP on Artemis 2.17.0 ends with receiving the following string
> as message body:
>
> "Conversion to AMQP error: Error reading in simpleString, length=y is
> greater than readableBytes=x"
>
> Broker lists the message as Type: Default and Large: True.
>
> The receiver is basically Qpid's:
>
> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py
>
> The doc
> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html
> states both Core and AMQP support Large Message. Is this approach
> valid?
>
> Thanks
>
>