You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Arun Koshal <ak...@gmail.com> on 2023/01/19 07:06:40 UTC
Batch send of messages to Azure event hub using Qpid proton
Hi,
I am using the Apache Qpid proton 0.37.0 C++ API to send messages from my
application to Azure Eventhub. The application is able to send one message
at a time, the rate of messages is slightly high, so one message at a time
is not very efficient.
Azure Eventhub supports batches of messages, but I could not figure out a
mechanism to send messages in batches using Qpid proton. Does it support
sending batch messages?
It would be a great help if someone can suggest a way to send batch
messages.
Thanks and regards,
Arun
Re: Batch send of messages to Azure event hub using Qpid proton
Posted by Robbie Gemmell <ro...@gmail.com>.
First, I'd note in case you dont know, that calling 'send' in Proton
container apps in the callbacks doesnt really send anything on the
wire then, that happens when IO is processed after the callbacks
return. You can call send multiple times in a callback if you like to
do multiple sends (first checking available sender credit to see that
there is actually enough to send the messages, and being mindful of
how much is being buffered) rather than doing it once at a time, and
effectively batch sends of individual messages into one IO processing
for efficiency. They are still individual message deliveries on the
wire, but will go out at the 'same time' and could even be in the same
TCP packet if small enough for that.
I know the Azure 'batch messages' in contrast use a different
non-default 'message-format' code and specific payload encoding
structure in the AMQP Transfer frames carrying the delivery, to hold
'multiple messages' back to back in one delivery (or at least they
used to, since I previously advised some ways they could achieve that
in their Java bits years ago using proton-j without it having specific
support). I'm not sure if that formatting code and structure is
actually documented other than the azure client code, you'd need to
check with Microsoft.
If knowing the format, youd then need to be able to set their 'batch
message' message-format code on the delivery frames, and feed the
bytes representing the encoding of your multiple-messages, encoded
into the expected structure. C++ isnt my thing, so I dont know whether
the C++ client can actually do that form of low level work or not (the
underlying C bits should).
On Thu, 19 Jan 2023 at 07:06, Arun Koshal <ak...@gmail.com> wrote:
>
> Hi,
>
> I am using the Apache Qpid proton 0.37.0 C++ API to send messages from my
> application to Azure Eventhub. The application is able to send one message
> at a time, the rate of messages is slightly high, so one message at a time
> is not very efficient.
>
> Azure Eventhub supports batches of messages, but I could not figure out a
> mechanism to send messages in batches using Qpid proton. Does it support
> sending batch messages?
>
> It would be a great help if someone can suggest a way to send batch
> messages.
>
> Thanks and regards,
> Arun
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [EXTERNAL] Batch send of messages to Azure event hub using Qpid proton
Posted by Robbie Gemmell <ro...@gmail.com>.
So from that it appears the Azure batch message format is a 'first
message annotations' included in the 'agregating message', followed by
one or more Data body sections to carry the other individual messages,
with each body section carrying the encoded bytes of an encoded full
amqp message (and all its encoded sections) from the batch...with all
those bytes then sent into a single delivery with the special
message-format code to indicate its an 'azure batch message' rather
than a normal/default AMQP 1.0 message encoding (though, the
components used in the aggregating message are actually a regular
message encoding in themselves).
As I said In my other reply, I dont know if the C++ binding can
actually do that, its a higher level client binding implemented over
the C bits (which should be able to) so I dont know that it actually
exposes defining message-format or the ability to pass the raw
delivery-bytes rather than messages as its rare an application would
use those (whereas the java bits referenced here are all low level
engine bits, used within the higher level Azure client, which again
hides all that).
Robbie
On Thu, 19 Jan 2023 at 21:32, Connie Yau <co...@microsoft.com.invalid> wrote:
>
> Hey,
>
> Batching is supported using proton-j by encoding multiple proton-j messages into a single proton-j Message. Here is a pointer to how we do it in Java:
> https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSender.java#L249
>
> I hope this helps,
> Connie
>
>
>
>
> -----Original Message-----
> From: Arun Koshal <ak...@gmail.com>
> Sent: Wednesday, January 18, 2023 23:07
> To: users@qpid.apache.org
> Cc: Arun Koshal <ak...@gmail.com>
> Subject: [EXTERNAL] Batch send of messages to Azure event hub using Qpid proton
>
> [You don't often get email from akoshal04@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi,
>
> I am using the Apache Qpid proton 0.37.0 C++ API to send messages from my application to Azure Eventhub. The application is able to send one message at a time, the rate of messages is slightly high, so one message at a time is not very efficient.
>
> Azure Eventhub supports batches of messages, but I could not figure out a mechanism to send messages in batches using Qpid proton. Does it support sending batch messages?
>
> It would be a great help if someone can suggest a way to send batch messages.
>
> Thanks and regards,
> Arun
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
RE: [EXTERNAL] Batch send of messages to Azure event hub using Qpid proton
Posted by Connie Yau <co...@microsoft.com.INVALID>.
Hey,
Batching is supported using proton-j by encoding multiple proton-j messages into a single proton-j Message. Here is a pointer to how we do it in Java:
https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSender.java#L249
I hope this helps,
Connie
-----Original Message-----
From: Arun Koshal <ak...@gmail.com>
Sent: Wednesday, January 18, 2023 23:07
To: users@qpid.apache.org
Cc: Arun Koshal <ak...@gmail.com>
Subject: [EXTERNAL] Batch send of messages to Azure event hub using Qpid proton
[You don't often get email from akoshal04@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Hi,
I am using the Apache Qpid proton 0.37.0 C++ API to send messages from my application to Azure Eventhub. The application is able to send one message at a time, the rate of messages is slightly high, so one message at a time is not very efficient.
Azure Eventhub supports batches of messages, but I could not figure out a mechanism to send messages in batches using Qpid proton. Does it support sending batch messages?
It would be a great help if someone can suggest a way to send batch messages.
Thanks and regards,
Arun
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org