You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Marco Guest <ma...@gmail.com> on 2021/05/06 12:48:01 UTC
Providing a custom deallocator for Pulsar::Message created via setAllocatedContent
Hi,
I am implementing a message exchange between C++ applications using Pulsar
technology.
To improve performances I would like to avoid copying my application data
when creating a new Pulsar::Message.
For this reason I would like to use
pulsar::MessageBuilder::setAllocatedContent().
However I’m in trouble currently because I don’t have a way to specify to
pulsar-client-cpp how to deallocate the buffer provided to
setAllocatedContent().
Moreover the current documentation is a bit unclear about this aspect:
“The caller is responsible to ensure the memory buffer is valid until the
message has been persisted (or an error is returned).”
It’s not clear honestly when the message “has been persisted” in my opinion
given that the underlying PulsarMessageImpl is a refcounted data structure
and so it’s hard to tell when its refcount goes to zero (e.g. imagine some
Pulsar::Message being saved into some data structure for later use).
Would you accept a patch that expands the arguments of
pulsar::MessageBuilder::setAllocatedContent() by taking a deallocation
function pointer as well?
Thanks,
Marco
Re: Providing a custom deallocator for Pulsar::Message created via setAllocatedContent
Posted by Marco Guest <ma...@gmail.com>.
Sorry, I did not explained well.
I used with the lambda function a by-copy capture of the pointer to the
allocated buffer of payload. In this manner the callback can free the right
buffer during a sequence of send operation for different payloads.
Il giorno gio 13 mag 2021 alle ore 19:10 Matteo Merli <mm...@apache.org>
ha scritto:
> Yes, my `sizeof(char*)` was really not a good one :)
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
> On Thu, May 13, 2021 at 9:59 AM Marco Guest <ma...@gmail.com>
> wrote:
> >
> > Hi,
> > it worked this way:
> >
> > ===
> > char* payload = new char[1024];
> > Message msg = MessageBuilder()
> > .setAllocatedContent(payload, 1024)
> > .build();
> > producer.sendAsync(msg, [p = payload](Result res, const MessageId&
> messageId) {
> > if (res == ResultOk) {
> > // message was published correctly
> > } else {
> > // publish error
> > }
> > delete[] p;
> > });
> > ===
> >
> > Thank you for the suggestion
> >
> >
> > Il giorno gio 6 mag 2021 alle ore 22:14 Matteo Merli <mm...@apache.org>
> ha scritto:
> >>
> >> Hi,
> >>
> >> the custom deallocator is actually an interesting idea, though the
> >> general pattern to deallocate the memory is something like:
> >>
> >> ===
> >>
> >> char* payload = new char[1024];
> >> Message msg = MessageBuilder()
> >> .setAllocatedContent(payload, sizeof(payload))
> >> .build();
> >> producer.sendAsync(msg, [&](Result res, const MessageId& messageId) {
> >> if (res == ResultOk) {
> >> // message was published correctly
> >> } else {
> >> // publish error
> >> }
> >> delete[] payload;
> >> });
> >> ===
> >>
> >> Does that work for you?
> >>
> >>
> >> --
> >> Matteo Merli
> >> <mm...@apache.org>
> >>
> >>
> >> On Thu, May 6, 2021 at 5:48 AM Marco Guest <ma...@gmail.com>
> wrote:
> >> >
> >> > Hi,
> >> > I am implementing a message exchange between C++ applications using
> Pulsar technology.
> >> > To improve performances I would like to avoid copying my application
> data when creating a new Pulsar::Message.
> >> > For this reason I would like to use
> pulsar::MessageBuilder::setAllocatedContent().
> >> > However I’m in trouble currently because I don’t have a way to
> specify to pulsar-client-cpp how to deallocate the buffer provided to
> setAllocatedContent().
> >> >
> >> > Moreover the current documentation is a bit unclear about this aspect:
> >> > “The caller is responsible to ensure the memory buffer is valid
> until the message has been persisted (or an error is returned).”
> >> > It’s not clear honestly when the message “has been persisted” in my
> opinion given that the underlying PulsarMessageImpl is a refcounted data
> structure and so it’s hard to tell when its refcount goes to zero (e.g.
> imagine some Pulsar::Message being saved into some data structure for later
> use).
> >> >
> >> > Would you accept a patch that expands the arguments of
> pulsar::MessageBuilder::setAllocatedContent() by taking a deallocation
> function pointer as well?
> >> >
> >> > Thanks,
> >> > Marco
> >> >
>
Re: Providing a custom deallocator for Pulsar::Message created via setAllocatedContent
Posted by Matteo Merli <mm...@apache.org>.
Yes, my `sizeof(char*)` was really not a good one :)
--
Matteo Merli
<mm...@apache.org>
On Thu, May 13, 2021 at 9:59 AM Marco Guest <ma...@gmail.com> wrote:
>
> Hi,
> it worked this way:
>
> ===
> char* payload = new char[1024];
> Message msg = MessageBuilder()
> .setAllocatedContent(payload, 1024)
> .build();
> producer.sendAsync(msg, [p = payload](Result res, const MessageId& messageId) {
> if (res == ResultOk) {
> // message was published correctly
> } else {
> // publish error
> }
> delete[] p;
> });
> ===
>
> Thank you for the suggestion
>
>
> Il giorno gio 6 mag 2021 alle ore 22:14 Matteo Merli <mm...@apache.org> ha scritto:
>>
>> Hi,
>>
>> the custom deallocator is actually an interesting idea, though the
>> general pattern to deallocate the memory is something like:
>>
>> ===
>>
>> char* payload = new char[1024];
>> Message msg = MessageBuilder()
>> .setAllocatedContent(payload, sizeof(payload))
>> .build();
>> producer.sendAsync(msg, [&](Result res, const MessageId& messageId) {
>> if (res == ResultOk) {
>> // message was published correctly
>> } else {
>> // publish error
>> }
>> delete[] payload;
>> });
>> ===
>>
>> Does that work for you?
>>
>>
>> --
>> Matteo Merli
>> <mm...@apache.org>
>>
>>
>> On Thu, May 6, 2021 at 5:48 AM Marco Guest <ma...@gmail.com> wrote:
>> >
>> > Hi,
>> > I am implementing a message exchange between C++ applications using Pulsar technology.
>> > To improve performances I would like to avoid copying my application data when creating a new Pulsar::Message.
>> > For this reason I would like to use pulsar::MessageBuilder::setAllocatedContent().
>> > However I’m in trouble currently because I don’t have a way to specify to pulsar-client-cpp how to deallocate the buffer provided to setAllocatedContent().
>> >
>> > Moreover the current documentation is a bit unclear about this aspect:
>> > “The caller is responsible to ensure the memory buffer is valid until the message has been persisted (or an error is returned).”
>> > It’s not clear honestly when the message “has been persisted” in my opinion given that the underlying PulsarMessageImpl is a refcounted data structure and so it’s hard to tell when its refcount goes to zero (e.g. imagine some Pulsar::Message being saved into some data structure for later use).
>> >
>> > Would you accept a patch that expands the arguments of pulsar::MessageBuilder::setAllocatedContent() by taking a deallocation function pointer as well?
>> >
>> > Thanks,
>> > Marco
>> >
Re: Providing a custom deallocator for Pulsar::Message created via setAllocatedContent
Posted by Marco Guest <ma...@gmail.com>.
Hi,
it worked this way:
===
char* payload = new char[1024];
Message msg = MessageBuilder()
.setAllocatedContent(payload, 1024)
.build();
producer.sendAsync(msg, [p = payload](Result res, const MessageId&
messageId) {
if (res == ResultOk) {
// message was published correctly
} else {
// publish error
}
delete[] p;
});
===
Thank you for the suggestion
Il giorno gio 6 mag 2021 alle ore 22:14 Matteo Merli <mm...@apache.org> ha
scritto:
> Hi,
>
> the custom deallocator is actually an interesting idea, though the
> general pattern to deallocate the memory is something like:
>
> ===
>
> char* payload = new char[1024];
> Message msg = MessageBuilder()
> .setAllocatedContent(payload, sizeof(payload))
> .build();
> producer.sendAsync(msg, [&](Result res, const MessageId& messageId) {
> if (res == ResultOk) {
> // message was published correctly
> } else {
> // publish error
> }
> delete[] payload;
> });
> ===
>
> Does that work for you?
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
>
> On Thu, May 6, 2021 at 5:48 AM Marco Guest <ma...@gmail.com> wrote:
> >
> > Hi,
> > I am implementing a message exchange between C++ applications using
> Pulsar technology.
> > To improve performances I would like to avoid copying my application
> data when creating a new Pulsar::Message.
> > For this reason I would like to use
> pulsar::MessageBuilder::setAllocatedContent().
> > However I’m in trouble currently because I don’t have a way to specify
> to pulsar-client-cpp how to deallocate the buffer provided to
> setAllocatedContent().
> >
> > Moreover the current documentation is a bit unclear about this aspect:
> > “The caller is responsible to ensure the memory buffer is valid until
> the message has been persisted (or an error is returned).”
> > It’s not clear honestly when the message “has been persisted” in my
> opinion given that the underlying PulsarMessageImpl is a refcounted data
> structure and so it’s hard to tell when its refcount goes to zero (e.g.
> imagine some Pulsar::Message being saved into some data structure for later
> use).
> >
> > Would you accept a patch that expands the arguments of
> pulsar::MessageBuilder::setAllocatedContent() by taking a deallocation
> function pointer as well?
> >
> > Thanks,
> > Marco
> >
>
Re: Providing a custom deallocator for Pulsar::Message created via setAllocatedContent
Posted by Matteo Merli <mm...@apache.org>.
Hi,
the custom deallocator is actually an interesting idea, though the
general pattern to deallocate the memory is something like:
===
char* payload = new char[1024];
Message msg = MessageBuilder()
.setAllocatedContent(payload, sizeof(payload))
.build();
producer.sendAsync(msg, [&](Result res, const MessageId& messageId) {
if (res == ResultOk) {
// message was published correctly
} else {
// publish error
}
delete[] payload;
});
===
Does that work for you?
--
Matteo Merli
<mm...@apache.org>
On Thu, May 6, 2021 at 5:48 AM Marco Guest <ma...@gmail.com> wrote:
>
> Hi,
> I am implementing a message exchange between C++ applications using Pulsar technology.
> To improve performances I would like to avoid copying my application data when creating a new Pulsar::Message.
> For this reason I would like to use pulsar::MessageBuilder::setAllocatedContent().
> However I’m in trouble currently because I don’t have a way to specify to pulsar-client-cpp how to deallocate the buffer provided to setAllocatedContent().
>
> Moreover the current documentation is a bit unclear about this aspect:
> “The caller is responsible to ensure the memory buffer is valid until the message has been persisted (or an error is returned).”
> It’s not clear honestly when the message “has been persisted” in my opinion given that the underlying PulsarMessageImpl is a refcounted data structure and so it’s hard to tell when its refcount goes to zero (e.g. imagine some Pulsar::Message being saved into some data structure for later use).
>
> Would you accept a patch that expands the arguments of pulsar::MessageBuilder::setAllocatedContent() by taking a deallocation function pointer as well?
>
> Thanks,
> Marco
>