You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Kerry Bonin <ke...@gmail.com> on 2010/03/16 16:58:22 UTC
C++ Messaging API questions...
I'm looking at migrating to messaging, and have a few questions.
r/e Message :
- If headers has an accessor, why doesn't it work? It only returns an empty
map.
- Why isn't timestamp exposed? (Trust & TZ issues aside, it is common to
want this data, why require recreation?)
r/e Receiver :
- Why did we lose the callback pattern? I now have to (at the least) spin a
thread across all my receivers to collect asynchronously.
FWIW, I'd be very happy to help fix these, as I need them myself...
Re: C++ Messaging API questions...
Posted by Kerry Bonin <ke...@gmail.com>.
On Tue, Mar 16, 2010 at 1:29 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 03/16/2010 06:05 PM, Kerry Bonin wrote:
>
>> r/e headers - I do see the same thing you listed (when I use quotation
>> marks, I'm on Win32). Your comment implies the basis for my question -
>> digging into the source I now see properties are internally split into
>> MessageProperties and DeliveryProperties, and I was expecting to see those
>> fields when I called getHeaders().
>>
>
> The messaging API is intended to abstract out protocol specifics. So there
> are explicit setters/getters for attributes of a message (e.g. userid or
> ttl) and these will be mapped on to the underlying protocol used (in the
> case of AMQP 0-10, onto fields in Message- or Delivery- Properties). Then
> there are generic properties available to applications for their own
> purposes. These are name value pairs. In AMQP 0-10 they map onto
> MessageProperites::applicationHeaders.
>
>
> r/e timestamps
>> - One common use case we have is to have clients synchronize distributed
>> data sets, which we do by providing a snapshot via request - response as
>> well as an update broadcast via publish - subscribe. We use timestamps to
>> determine when an update event was sent relative to the snapshot. We were
>> including our own timestamp field in the message body for this, and
>> thought
>> we could use message timestamps.
>> - Another common use case is to broadcast events. We care when these were
>> sent, and historically have included a UTC event time in the message that
>> is
>> used (in part) to identify the event. Again, we could include our own
>> timestamp in the message body for this, but we thought we could use the
>> message timestamp for this - so long as it is the time of creation or
>> first
>> transmission. We haven't seen documentation on the expected behavior of
>> message timestamps across federated and/or clustered systems, or through
>> durable queues.
>>
>
> So the timestamp is set by the sender and refers to the creation of the
> message or to the time of the event the message represents? (i.e. the
> detailed semantics are application specific?)
Yes, that's how we've used message timestamps historically - our send API
would set the timestamp to the current time if it had not been previously
set during message construction, and this value would be propagated to all
recipients. If this concept differs from the meaning of timestamp in AMQP,
we can certainly return to an application field for timestamps. In either
case, it would be useful to clarify formally (somewhere) what timestamps are
used for in AMQP, and if they have any value to an application use it would
seem reasonable to expose them.
>
> r/e the asynchronous APIs, this is more of a nice to have, but we have C++
>> and Python developers who liked not having to create service threads to
>> read
>> asynchronously for the previous API. Event loops are common, but not
>> always
>> used...
>>
>
> I agree its nice to have something there out of the box. However even in
> the old 0-10 specific API the only really usable solution was
> SubscriptionManager::run(). I think you could replace the functionality of
> that method with:
>
> typedef std::map<std::string, MessageListener*> Callbacks;
> Callbacks callbacks;
> //populate per subscription listeners
>
> Listener* default = //default callback
>
> while (true) {
> Receiver r = session.nextReceiver();
> Callbacks::const_iterator i = callbacks.find(r.getName());
> if (i == callbacks.end()) {
> default->received(r.fetch());
> } else {
> i->second->received(r.fetch());
> }
> }
>
> You can also customise that loop more, e.g. by timing out if nothing is
> received after a certain time, or having different strategies for
> dispatching the messages to listeners/callbacks.
>
> The above of course still requires a thread per session, but that is no
> worse than the old API. The plan is to add the ability to tie in waiting on
> one or more connection to the same pattern.
>
>
Re: C++ Messaging API questions...
Posted by Gordon Sim <gs...@redhat.com>.
On 03/16/2010 06:05 PM, Kerry Bonin wrote:
> r/e headers - I do see the same thing you listed (when I use quotation
> marks, I'm on Win32). Your comment implies the basis for my question -
> digging into the source I now see properties are internally split into
> MessageProperties and DeliveryProperties, and I was expecting to see those
> fields when I called getHeaders().
The messaging API is intended to abstract out protocol specifics. So
there are explicit setters/getters for attributes of a message (e.g.
userid or ttl) and these will be mapped on to the underlying protocol
used (in the case of AMQP 0-10, onto fields in Message- or Delivery-
Properties). Then there are generic properties available to applications
for their own purposes. These are name value pairs. In AMQP 0-10 they
map onto MessageProperites::applicationHeaders.
> r/e timestamps
> - One common use case we have is to have clients synchronize distributed
> data sets, which we do by providing a snapshot via request - response as
> well as an update broadcast via publish - subscribe. We use timestamps to
> determine when an update event was sent relative to the snapshot. We were
> including our own timestamp field in the message body for this, and thought
> we could use message timestamps.
> - Another common use case is to broadcast events. We care when these were
> sent, and historically have included a UTC event time in the message that is
> used (in part) to identify the event. Again, we could include our own
> timestamp in the message body for this, but we thought we could use the
> message timestamp for this - so long as it is the time of creation or first
> transmission. We haven't seen documentation on the expected behavior of
> message timestamps across federated and/or clustered systems, or through
> durable queues.
So the timestamp is set by the sender and refers to the creation of the
message or to the time of the event the message represents? (i.e. the
detailed semantics are application specific?)
> r/e the asynchronous APIs, this is more of a nice to have, but we have C++
> and Python developers who liked not having to create service threads to read
> asynchronously for the previous API. Event loops are common, but not always
> used...
I agree its nice to have something there out of the box. However even in
the old 0-10 specific API the only really usable solution was
SubscriptionManager::run(). I think you could replace the functionality
of that method with:
typedef std::map<std::string, MessageListener*> Callbacks;
Callbacks callbacks;
//populate per subscription listeners
Listener* default = //default callback
while (true) {
Receiver r = session.nextReceiver();
Callbacks::const_iterator i = callbacks.find(r.getName());
if (i == callbacks.end()) {
default->received(r.fetch());
} else {
i->second->received(r.fetch());
}
}
You can also customise that loop more, e.g. by timing out if nothing is
received after a certain time, or having different strategies for
dispatching the messages to listeners/callbacks.
The above of course still requires a thread per session, but that is no
worse than the old API. The plan is to add the ability to tie in waiting
on one or more connection to the same pattern.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: C++ Messaging API questions...
Posted by Kerry Bonin <ke...@gmail.com>.
r/e headers - I do see the same thing you listed (when I use quotation
marks, I'm on Win32). Your comment implies the basis for my question -
digging into the source I now see properties are internally split into
MessageProperties and DeliveryProperties, and I was expecting to see those
fields when I called getHeaders().
r/e timestamps
- One common use case we have is to have clients synchronize distributed
data sets, which we do by providing a snapshot via request - response as
well as an update broadcast via publish - subscribe. We use timestamps to
determine when an update event was sent relative to the snapshot. We were
including our own timestamp field in the message body for this, and thought
we could use message timestamps.
- Another common use case is to broadcast events. We care when these were
sent, and historically have included a UTC event time in the message that is
used (in part) to identify the event. Again, we could include our own
timestamp in the message body for this, but we thought we could use the
message timestamp for this - so long as it is the time of creation or first
transmission. We haven't seen documentation on the expected behavior of
message timestamps across federated and/or clustered systems, or through
durable queues.
r/e the asynchronous APIs, this is more of a nice to have, but we have C++
and Python developers who liked not having to create service threads to read
asynchronously for the previous API. Event loops are common, but not always
used...
On Tue, Mar 16, 2010 at 12:20 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 03/16/2010 03:58 PM, Kerry Bonin wrote:
>
>> I'm looking at migrating to messaging, and have a few questions.
>>
>> r/e Message :
>> - If headers has an accessor, why doesn't it work? It only returns an
>> empty
>> map.
>>
>
> It works for me (are you using trunk?):
>
> $ ./examples/messaging/spout --address 'my-queue; {create: always}'
> --property my-property=my-value
> $ ./examples/messaging/drain --address 'my-queue; {create: always}'
> Message(properties={my-property:my-value,
> spout-id:eab53080-8f16-48db-b4cf-5cf9bb200ec0:0}, content='')
>
> The above uses Message::getHeaders(), what do you see when you run those
> examples?
>
> (Note I'd actually like to change that to getProperties() for consistency
> with the python client).
>
>
> - Why isn't timestamp exposed? (Trust& TZ issues aside, it is common to
>> want this data, why require recreation?)
>>
>
> How would you expect the timestamp to be used?
>
>
> r/e Receiver :
>> - Why did we lose the callback pattern? I now have to (at the least) spin
>> a
>> thread across all my receivers to collect asynchronously.
>>
>
> The idea is that some sort of generic dispatcher would be layered on top of
> the current api. The plan is to extend or generalise the
> Session::nextReceiver() pattern so that Sessions and indeed Connections
> could be integrated into a common event loop.
>
>
> FWIW, I'd be very happy to help fix these, as I need them myself...
>>
>
> Excellent, and thanks for the feedback, the API could do with some more
> eyes on it.
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
Re: C++ Messaging API questions...
Posted by Alan Conway <ac...@redhat.com>.
On 03/16/2010 01:20 PM, Gordon Sim wrote:
> On 03/16/2010 03:58 PM, Kerry Bonin wrote:
>> I'm looking at migrating to messaging, and have a few questions.
>>
>> r/e Message :
>> - If headers has an accessor, why doesn't it work? It only returns an
>> empty
>> map.
>
> It works for me (are you using trunk?):
>
> $ ./examples/messaging/spout --address 'my-queue; {create: always}'
> --property my-property=my-value
> $ ./examples/messaging/drain --address 'my-queue; {create: always}'
> Message(properties={my-property:my-value,
> spout-id:eab53080-8f16-48db-b4cf-5cf9bb200ec0:0}, content='')
>
> The above uses Message::getHeaders(), what do you see when you run those
> examples?
>
> (Note I'd actually like to change that to getProperties() for
> consistency with the python client).
>
>> - Why isn't timestamp exposed? (Trust& TZ issues aside, it is common to
>> want this data, why require recreation?)
>
> How would you expect the timestamp to be used?
>
>> r/e Receiver :
>> - Why did we lose the callback pattern? I now have to (at the least)
>> spin a
>> thread across all my receivers to collect asynchronously.
>
> The idea is that some sort of generic dispatcher would be layered on top
> of the current api. The plan is to extend or generalise the
> Session::nextReceiver() pattern so that Sessions and indeed Connections
> could be integrated into a common event loop.
>
IMO we need the API for that sooner rather than later, as it will be a blocker
to porting existing test programs. I will try to take a shot at it this week.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: C++ Messaging API questions...
Posted by Gordon Sim <gs...@redhat.com>.
On 03/16/2010 03:58 PM, Kerry Bonin wrote:
> I'm looking at migrating to messaging, and have a few questions.
>
> r/e Message :
> - If headers has an accessor, why doesn't it work? It only returns an empty
> map.
It works for me (are you using trunk?):
$ ./examples/messaging/spout --address 'my-queue; {create: always}'
--property my-property=my-value
$ ./examples/messaging/drain --address 'my-queue; {create: always}'
Message(properties={my-property:my-value,
spout-id:eab53080-8f16-48db-b4cf-5cf9bb200ec0:0}, content='')
The above uses Message::getHeaders(), what do you see when you run those
examples?
(Note I'd actually like to change that to getProperties() for
consistency with the python client).
> - Why isn't timestamp exposed? (Trust& TZ issues aside, it is common to
> want this data, why require recreation?)
How would you expect the timestamp to be used?
> r/e Receiver :
> - Why did we lose the callback pattern? I now have to (at the least) spin a
> thread across all my receivers to collect asynchronously.
The idea is that some sort of generic dispatcher would be layered on top
of the current api. The plan is to extend or generalise the
Session::nextReceiver() pattern so that Sessions and indeed Connections
could be integrated into a common event loop.
> FWIW, I'd be very happy to help fix these, as I need them myself...
Excellent, and thanks for the feedback, the API could do with some more
eyes on it.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org