You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Gordon Sim <gs...@redhat.com> on 2009/04/06 15:30:24 UTC
[c++]: Sketch of a high-level, protocol independent API
There has been much discussion around the need for a higher-level c++
API that insulates an application from any protocol specific details
while allowing specific features of any given protocol to be exploited
through the abstractions and extension points that this API offers.
I should note that the direct use of the AMQP 0-10 specific API will
of course still be supported.
The API I have in mind is based on the concepts of message sources
(from which messages can be received) and message sinks (to which
messages can be sent).
To send messages on a particular session, the application obtains a
MessageProducer for that session from the chosen sink, through which
it can then send messages. To receive messages, it subscribes the
session to the chosen source. A subscription can be assoicated with a
MessageListener (for 'push' style interaction) or it can create a
MessageReceiver (for 'pull' style interactions).
Two common patterns for sources and sinks are offered though the Queue
and Topic classes which allow sources and sinks to be created for
named queues in a point-to-point pattern, or publish-subscribe
'topics'.
The attached patch offers a working sketch of such an API. The API is
currently in the qpid::client::api namespace, primarily to keep it
separate from the existing AMQP 0-10 specific API.
There is also an implementation of this high-level API in terms of the
existing 0-10 API in the qpid::client::amqp0_10 namespace, and a few
example programs to demonstrate the use of queues and topics
(examples/hla).
This is still very much work in progress intended to demonstrate the
source/sink concept and how that can be mapped to different patterns
of interaction on a given protocol implementation.
A critical aspect that I have yet to address is the nature of the
Message interface (currently this just exposes raw data with no
properties or headers). I think we also need to figure out the common,
protocol independent options that we may want to directly support
e.g. browsing v. consuming, prefetch windows, automatic acking
v. application controlled acking etc. The threading model for connection
processing and session dispatching also still needs to be thought
through in more detail.
All queries, comments, criticisms etc are very welcome.
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by James Mansion <ja...@mansionfamily.plus.com>.
Rafael Schloming wrote:
> (good stuff)
What he said, +1.
Unfortunately, as it were, if you want to go there, don't start from here.
Alan, Gordon:
How are you expecting flow of control to work wrt number of driver
threads and flow of control in any callbacks?
This is arguably a lot more important than whether the API has a C or
C++ flavour.
James
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Rajith Attapattu <ra...@gmail.com>.
I like Rafi's idea.
As I have advocated before, the most compelling reason to make a C API
is the ability to support dynamic languages.
Given the popularity of dynamic languages this would be a very
important step in taking Qpid to the masses.
This approach would allow us to maintain several dynamic language
bindings with reasonable effort after the initial investment of
getting a well defined compact C library.
Projects like Axis2/C has had good sucess using the above approach +
swig for their dynamic language support.
I also like the protocol engine to not impose any threading model
which then gives the particular language binding the freedom to work
with a threading model that best fits it's need. For those familar
with Java, an example would be the javax.net.ssl.SSLEngine which
provides SSL support and is somewhat conceptually similar to what Rafi
is proposing.
Regards,
Rajith
On Fri, Apr 17, 2009 at 9:22 AM, Rafael Schloming <ra...@redhat.com> wrote:
> Alan Conway wrote:
>>
>> Given the current state of affairs I'd say the following is the most
>> likely path to a C API:
>>
>> 1. Clean up, simplify and remove protocol-specifics from the C++ API. Use
>> handles to hide impls (in particular PIMPL the Message class) use a single
>> callback interface (MessageListener)
>>
>> 2. Implement a C API over this C++ API for scripting languages, C projects
>> etc.
>>
>> 3. (maybe) bless the C API and move it inwards to the heart of the client
>> lib, move more of the client lib to C. Not clear (to me) if this is going to
>> be desirable but once we have a C API it becomes easier to weigh the pros &
>> cons.
>
> My reason for being interested in a C API is for something to be embeddable
> from python/ruby/php/perl/etc, as well as something that is suitable for
> embedding inside both a client and a broker, and I'm not convinced the above
> approach is enough to result in something that is useful for that sort of
> thing.
>
> For embeddability what we need is a strong separation between the active and
> passive portions of the code, i.e. the client needs to be structured as a
> passive "protocol engine" and an active "driver". The protocol engine gets
> fed bytes/frames/etc on its input and produces high level events on its
> output, and the driver is responsible for feeding the protocol engine bytes
> and processing any high level events that result.
>
> This split captures all the complex protocol logic into a single embeddable
> piece, but doesn't force a threading and/or (b)locking model onto the
> embedding application, e.g. I can embed the protocol engine easily inside a
> broker and client and use a nonblocking driver in the former case and a
> blocking driver in the latter case.
>
> This also has the benefit of avoiding callbacks across bindings since the
> protocol engine will only ever produce events when it is fed data, so the
> driver can easily check for and dispatch events without relying on
> callbacks. This is a fairly critical feature for embeddability, it means we
> can do things like trivially use swig to expose the protocol engine in just
> about any language, and then integrate with the native threading constructs
> available in that language, e.g. I can use realtime threads in RT Java if I
> want to.
>
> --Rafael
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Rafael Schloming <ra...@redhat.com>.
Alan Conway wrote:
> Given the current state of affairs I'd say the following is the most
> likely path to a C API:
>
> 1. Clean up, simplify and remove protocol-specifics from the C++ API.
> Use handles to hide impls (in particular PIMPL the Message class) use a
> single callback interface (MessageListener)
>
> 2. Implement a C API over this C++ API for scripting languages, C
> projects etc.
>
> 3. (maybe) bless the C API and move it inwards to the heart of the
> client lib, move more of the client lib to C. Not clear (to me) if this
> is going to be desirable but once we have a C API it becomes easier to
> weigh the pros & cons.
My reason for being interested in a C API is for something to be
embeddable from python/ruby/php/perl/etc, as well as something that is
suitable for embedding inside both a client and a broker, and I'm not
convinced the above approach is enough to result in something that is
useful for that sort of thing.
For embeddability what we need is a strong separation between the active
and passive portions of the code, i.e. the client needs to be structured
as a passive "protocol engine" and an active "driver". The protocol
engine gets fed bytes/frames/etc on its input and produces high level
events on its output, and the driver is responsible for feeding the
protocol engine bytes and processing any high level events that result.
This split captures all the complex protocol logic into a single
embeddable piece, but doesn't force a threading and/or (b)locking model
onto the embedding application, e.g. I can embed the protocol engine
easily inside a broker and client and use a nonblocking driver in the
former case and a blocking driver in the latter case.
This also has the benefit of avoiding callbacks across bindings since
the protocol engine will only ever produce events when it is fed data,
so the driver can easily check for and dispatch events without relying
on callbacks. This is a fairly critical feature for embeddability, it
means we can do things like trivially use swig to expose the protocol
engine in just about any language, and then integrate with the native
threading constructs available in that language, e.g. I can use realtime
threads in RT Java if I want to.
--Rafael
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Alan Conway <ac...@redhat.com>.
Gordon Sim wrote:
> James Mansion wrote:
>> Gordon Sim wrote:
>>> James Mansion wrote:
>>>> My question is: why do you feel the need to create something
>>>> materially different, and why in particular does
>>>> a messaging API need to be much more complex for normal use?
>>>
>>> Could you elaborate a little on what aspect(s) you feel is/are much
>>> more complex?
>> Well, I'd look to replace class instances with handles and collaps as
>> many fancy option
>> setting/getting functions to a simple interface that uses string
>> property names and put/get.
>> And I'd expose a C API as much as possible with limited use of
>> callbacks and buffer
>> assignments in the user code rather than having to ask the API to
>> delete buffers (or,
>> worse, assume that the user code and API are sharing a heap manager).
>>
>> I would also look to have the API buffer received messages and use the
>> simplest
>> notification mechanism available. So I'd suggest not starting with
>> any fancy class
>> structure at all, but first ask how you will integrate into
>> applications' event loops.
>>
>> This is something that the MQ API is poor at from memory.
>>
>> Its not necessary for fancy stuff that configures special options to
>> be fast, and having a
>> very stable binary API is handy. Also having an API that can be
>> trivially wrapped
>> with Python CTypes or P/Invoke is handy - not least its also easy to
>> wrap with
>> other tools such as swig.
>
> Ok, so simple for you is avoiding fancy things, not just the ease of
> writing programs to the API? Thats certainly a fair point and I think a
> c API would be a great addition to the project. I'm still keen on a c++
> API myself; it could at some point in the future be implemented on top
> of such a c API.
>
Given the current state of affairs I'd say the following is the most likely path
to a C API:
1. Clean up, simplify and remove protocol-specifics from the C++ API. Use
handles to hide impls (in particular PIMPL the Message class) use a single
callback interface (MessageListener)
2. Implement a C API over this C++ API for scripting languages, C projects etc.
3. (maybe) bless the C API and move it inwards to the heart of the client lib,
move more of the client lib to C. Not clear (to me) if this is going to be
desirable but once we have a C API it becomes easier to weigh the pros & cons.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
James Mansion wrote:
> Gordon Sim wrote:
>> James Mansion wrote:
>>> My question is: why do you feel the need to create something
>>> materially different, and why in particular does
>>> a messaging API need to be much more complex for normal use?
>>
>> Could you elaborate a little on what aspect(s) you feel is/are much
>> more complex?
> Well, I'd look to replace class instances with handles and collaps as
> many fancy option
> setting/getting functions to a simple interface that uses string
> property names and put/get.
> And I'd expose a C API as much as possible with limited use of callbacks
> and buffer
> assignments in the user code rather than having to ask the API to delete
> buffers (or,
> worse, assume that the user code and API are sharing a heap manager).
>
> I would also look to have the API buffer received messages and use the
> simplest
> notification mechanism available. So I'd suggest not starting with any
> fancy class
> structure at all, but first ask how you will integrate into
> applications' event loops.
>
> This is something that the MQ API is poor at from memory.
>
> Its not necessary for fancy stuff that configures special options to be
> fast, and having a
> very stable binary API is handy. Also having an API that can be
> trivially wrapped
> with Python CTypes or P/Invoke is handy - not least its also easy to
> wrap with
> other tools such as swig.
Ok, so simple for you is avoiding fancy things, not just the ease of
writing programs to the API? Thats certainly a fair point and I think a
c API would be a great addition to the project. I'm still keen on a c++
API myself; it could at some point in the future be implemented on top
of such a c API.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by James Mansion <ja...@mansionfamily.plus.com>.
Gordon Sim wrote:
> James Mansion wrote:
>> My question is: why do you feel the need to create something
>> materially different, and why in particular does
>> a messaging API need to be much more complex for normal use?
>
> Could you elaborate a little on what aspect(s) you feel is/are much
> more complex?
Well, I'd look to replace class instances with handles and collaps as
many fancy option
setting/getting functions to a simple interface that uses string
property names and put/get.
And I'd expose a C API as much as possible with limited use of callbacks
and buffer
assignments in the user code rather than having to ask the API to delete
buffers (or,
worse, assume that the user code and API are sharing a heap manager).
I would also look to have the API buffer received messages and use the
simplest
notification mechanism available. So I'd suggest not starting with any
fancy class
structure at all, but first ask how you will integrate into
applications' event loops.
This is something that the MQ API is poor at from memory.
Its not necessary for fancy stuff that configures special options to be
fast, and having a
very stable binary API is handy. Also having an API that can be
trivially wrapped
with Python CTypes or P/Invoke is handy - not least its also easy to
wrap with
other tools such as swig. If you must go straight for an object model,
then do
please evolve the swig specifications at the same time.
I suspect that what you are interested in is what I would choose to use
to implement
such an API internally. Which means that essentially I *do not care*
what it looks like,
and think it needs no API stability at all.
It may be appropriate to bless it for direct use at some future time,
but hopefully that
would be when no-one can be bothered to refactor it any more because its
good
enough. I'd hate to try to go from 0 to a working 'good' object API in
one step.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
James Mansion wrote:
> My question is: why do you feel the need to create something materially
> different, and why in particular does
> a messaging API need to be much more complex for normal use?
Could you elaborate a little on what aspect(s) you feel is/are much more
complex?
> It won't be usable as-is, but the simplicity is simething to aspire to.
I agree that simplicity is important and am very keen on identifying and
eliminating any unnecessary complexity. Your views on the question above
will hopefully move us in that direction.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by James Mansion <ja...@mansionfamily.plus.com>.
Gordon Sim wrote:
> James Mansion wrote:
>> Sure you couldn't have a C API that is closely based on the MQSeries
>> one, with extra tweaks?
>>
>> Or a C++ API that is much like the COM API from MSMQ?
>>
>> Both with extensions as needed.
>>
>> These are the most widely deployed existing systems. Sure you're not
>> going to design a bikeshed?
>
> Are there specific aspects of either of those two APIs that you would
> like to emulate? Or are you just suggesting that we reuse an existing
> API?
Both, but the attraction of MQSeries particularly is the simplicity.
And, to be honest, its the market leader.
My question is: why do you feel the need to create something materially
different, and why in particular does
a messaging API need to be much more complex for normal use?
It won't be usable as-is, but the simplicity is simething to aspire to.
> Were there an existing API open standard for messaging in c++ I would
> be very keen for us to embrace it.
A simple C API would arguably be more useful though, and easier to
explain to users of scripting languages.
I suspect it would be acceptable to implement with C++ and depend on
installed C++ support,
even if that's not ideal long term.
Since when did an API have to be open to be copied anyway?
James
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
James Mansion wrote:
> Sure you couldn't have a C API that is closely based on the MQSeries
> one, with extra tweaks?
>
> Or a C++ API that is much like the COM API from MSMQ?
>
> Both with extensions as needed.
>
> These are the most widely deployed existing systems. Sure you're not
> going to design a bikeshed?
Are there specific aspects of either of those two APIs that you would
like to emulate? Or are you just suggesting that we reuse an existing API?
To be clear, I am not in any way aiming to create a universal standard
API for all c++ messaging. I simply want to help applications using qpid
insulate themselves from details of the protocol as much as possible.
Were there an existing API open standard for messaging in c++ I would be
very keen for us to embrace it.
However I don't myself believe that either of the options you mention
represent such a standard and I don't think they are even open to other
implementations.
The ActiveMQ project has a c++ API, CMS[1], that is essentially a pretty
direct port of the JMS API. I did consider that, but decided against it.
I think what I am proposing will end up being quite similar and I'll be
happy to try and justify the differences before we commit ourselves to
any decision.
[1] http://activemq.apache.org/cms/api_docs/cms-1.2/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by James Mansion <ja...@mansionfamily.plus.com>.
Gordon Sim wrote:
> All queries, comments, criticisms etc are very welcome.
Sure you couldn't have a C API that is closely based on the MQSeries
one, with extra tweaks?
Or a C++ API that is much like the COM API from MSMQ?
Both with extensions as needed.
These are the most widely deployed existing systems. Sure you're not
going to design a bikeshed?
James
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Alan Conway <ac...@redhat.com>.
Gordon Sim wrote:
> Alan Conway wrote:
>> On Wed, 2009-04-15 at 15:37 +0100, Gordon Sim wrote:
>>
>>> I don't see view this work as replacing the existing API. I think it
>>> is simply at a higher level of abstraction.
>>>
>>
>> I see a role for a 'mid-level' API that provides full access to what the
>> protocol can do, without being tied to the protocol details.
>
> That's pretty much what I am aiming for also. We probably need to dig
> into some specifics to understand the different approaches we each have
> in mind.
>
>>> The 0-10 specific API will still be supported and can I think be
>>> considered stable. The initial implementation of the higher level API
>>> I sketched out uses the 0-10 specific API that we have almost without
>>> change.
>>>
>>
>> So are you thinking we'll implement the 0-10 API over the 1.0 protocol?
>
> No.
>
>> I'd rather we get a more "neutral" API that's close enough to cover
>> everything the protocol can do, but not tied to individual protocol
>> commands.
>
> So you feel that my initial proposal is not close enough to allow
> applications to exploit particular aspects of the protocol? What sort of
> things are you thinking of there? There may well be something missing;
> it may also just be the incompleteness of the sketch.
>
Partly that - we need to cover flow control, Futures etc. Partly it's a vague
worry that defining "high level" patterns of use may not provide access to the
full power of the broker. A lot of AMQP's flexibility comes from allowing the
users the freedome to create and bind queues and exchanges (even more so in 1.0
where its easy to compose "links" into networks).
So I'm all for easy to use patterns for the common cases, but I think we still
may still need to provide some more direct access to the notion of
linking/binding (and to do it in a way that works for AMQP 0-10 and 1-0)
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
Alan Conway wrote:
> On Wed, 2009-04-15 at 15:37 +0100, Gordon Sim wrote:
>
>> I don't see view this work as replacing the existing API. I think it is
>> simply at a higher level of abstraction.
>>
>
> I see a role for a 'mid-level' API that provides full access to what the
> protocol can do, without being tied to the protocol details.
That's pretty much what I am aiming for also. We probably need to dig
into some specifics to understand the different approaches we each have
in mind.
>> The 0-10 specific API will still be supported and can I think be
>> considered stable. The initial implementation of the higher level API I
>> sketched out uses the 0-10 specific API that we have almost without change.
>>
>
> So are you thinking we'll implement the 0-10 API over the 1.0 protocol?
No.
> I'd rather we get a more "neutral" API that's close enough to cover
> everything the protocol can do, but not tied to individual protocol
> commands.
So you feel that my initial proposal is not close enough to allow
applications to exploit particular aspects of the protocol? What sort of
things are you thinking of there? There may well be something missing;
it may also just be the incompleteness of the sketch.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Alan Conway <ac...@redhat.com>.
On Wed, 2009-04-15 at 15:37 +0100, Gordon Sim wrote:
> I don't see view this work as replacing the existing API. I think it is
> simply at a higher level of abstraction.
>
I see a role for a 'mid-level' API that provides full access to what the
protocol can do, without being tied to the protocol details.
> The 0-10 specific API will still be supported and can I think be
> considered stable. The initial implementation of the higher level API I
> sketched out uses the 0-10 specific API that we have almost without change.
>
So are you thinking we'll implement the 0-10 API over the 1.0 protocol?
I'd rather we get a more "neutral" API that's close enough to cover
everything the protocol can do, but not tied to individual protocol
commands.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
Alan Conway wrote:
> My first impression is that there are too many steps and too many
> objects involved in sending/receiving messages.
>
> Essentially there are 3 operations:
> - send a message (to somewhere)
> - pull a message (from somewhere)
> - register a listener to have messages pushed (from somewhere)
>
> We need to define "message" and "somewhere" in a portable way. Your
> proposal addresses the "somewhere" part with sources, sinks, producers,
> receivers and listeners.
The 'somewhere' part is really just covered by sources and sinks. I see
the source/sink abstractions as more than just a representation of an
'endpoint'. They can also encapsulate a 'strategy' for incoming/outgoing
message flows.
Producers, receivers and listeners are the objects through which you do
the three operations above. Sinks are 'factories' for creating producers
for a given session; sources are factories for creating receivers or
listener registrations.
> I think we can compress this into fewer classes
> but I don't have a proposal I'm happy with yet.
If we can make things simpler, I'm certainly all for that.
> We also need to cover creating wiring,
I think the concept of creating wiring should be hidden. I think it
tends to be protocol specific and in any case is often not directly
relevant to application code.
Where a protocol requires wiring to be created for e.g. topic
subscriptions I think that should be hidden behind higher level
abstractions (such as the creation of a producer or receiver).
> threading model,
I agree, we need to be more flexible here.
> flow control,
> and accepting/acquiring messages (browsing) etc.
My current thinking is that this sort of thing can be encapsulated as a
specific strategy for incoming or outgoing messages and therefore hidden
behind a sink/source and the producer/receiver it creates.
The sink(s)/source(s) used is/are then part of the configuration of an
application, with most of the code being entirely separated from choices
of these sorts.
> I think we can evolve
> rather than replace the existing API in these areas.
I don't see view this work as replacing the existing API. I think it is
simply at a higher level of abstraction.
The 0-10 specific API will still be supported and can I think be
considered stable. The initial implementation of the higher level API I
sketched out uses the 0-10 specific API that we have almost without change.
> I'll keep mulling it over this week and we can sit down next week. I'll
> try to make a start at cleaning up work at cleaning up the mechanics as
> per my proposal - did you have any comments on it? Here it is again:
It looked good to me. I'll likely have some questions at some point. I
have tried to follow the advice with the work I did so far (in
particular using handles).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Alan Conway <ac...@redhat.com>.
My first impression is that there are too many steps and too many
objects involved in sending/receiving messages.
Essentially there are 3 operations:
- send a message (to somewhere)
- pull a message (from somewhere)
- register a listener to have messages pushed (from somewhere)
We need to define "message" and "somewhere" in a portable way. Your
proposal addresses the "somewhere" part with sources, sinks, producers,
receivers and listeners. I think we can compress this into fewer classes
but I don't have a proposal I'm happy with yet.
We also need to cover creating wiring, threading model, flow control,
and accepting/acquiring messages (browsing) etc. I think we can evolve
rather than replace the existing API in these areas.
I'll keep mulling it over this week and we can sit down next week. I'll
try to make a start at cleaning up work at cleaning up the mechanics as
per my proposal - did you have any comments on it? Here it is again:
===
We need to clean up our public C++ API so that we can:
- identify when we might be making incompatible changes that will
break
existing user code.
- reduce our implementation exposure to incompatible changes.
Here are some thoughts in that direction, feedback appreciated:
* Public API design principles
** Rationale
Any change to code that can be seen by the user's compiler can
potentially break
compatibility.
There are two levels of compatibility:
- Binary compatible: user binaries built against old headers/libs can be
linked
with new libs and will work.
- Source compatible: user source code can be re-compiled without change
and
still work.
Future-proofing the API means:
- clearly identifying public header files - those that are installed and
can be
seen by user's compiler.
- minimizing the amount of code in public header files.
- hiding implementation as much as possible in public header files.
** Design Principles
Public header files live under cpp/include, private header files live
under cpp/src
- Only headers in cpp/include are installed and visible to the user's
compiler.
QPID_*_EXTERN declaration are required on:
- all member functions intended to be called by users.
- all member functions used by another qpid library (e.g. common lib
functions
used by client/broker libs)
See
http://stevehuston.wordpress.com/2009/03/12/lessons-learned-converting-apache-qpid-to-build-dlls-on-windows/
for some discussion on this subject.
Public classes should fall into one of the following 3 categories:
Handle: handle to refcounted object (e.g. Connection, Session)
- pure pointer to impl (PIMPL) idiom: no data except impl pointer, no
virtual
functions, no inlines.
- qpid::client::Handle provides common base class
- defined in client lib, namespace ::qpid::client
Interface: abstract base class, intended for user to subclass (e.g.
MessageListener)
- defined in client lib, namespace ::qpid::client
Value: data type with value semantics (e.g. std::string, FieldTable)
- defined in common lib, namespace ::qpid
- imported into ::qpid::client namespace with using statements.
- more on value types below, this is the area that needs most work.
Note type system is defined in common lib, ::qpid namespace so it can be
shared
by client & broker code.
It is imported into ::qpid::client namespace so a typical client need
only use
the qpid::client namespace.
No boost headers included in public .h files: I think this is
feasible. It would the boost-devel requirement for clients and avoid
incompatibilities due to boost version changes.
* Value Types Proposal
Value types have the most exposed implementation so need to be kept as
simple
and clean as possible.
They should:
- represent strictly the *users* view of the data type: no
encode/decode functions
- have value-semantics - no virtuals, normal copy semantics etc.
- not be tied to a specific AMQP protocol version - this is the C++
view of
the type.
Integral types: typedef int8_t Int8; typedef uint16_6 Uint16; ... etc.
String types: typedef std::string String;
Additional classes: class SequenceNumber; class SequenceSet; class Url
etc.
Discriminated union type: needed by FieldTable and Url. Currently Url
uses a boost::variant which is not extensible, and FieldTable uses
FieldValue which has a somewhat ad-hoc interface.
Propose replacing both with class qpid::Any, modelled after boost::any
with some additional features described below. This provides a
standard-ish (boost::any is proposed for a future C++ standard) API
that allows us to store *any* C++ type without modifications.
FieldTable: propose replacing with
typedef std::map<std::string, Any> Map;
Keep deprecated framing::FieldTable for compatibility.
This allows code like:
map["foo"] = std::string("abc"); // Creates an any containing
std::string
std::string* s = anyCast<std::string>(&map["bar"]);
if (s) { do string stuff }
uint16_t i = anyCast<uint16_t>(&map["x"]); // throw exception if not
a uint16_t
as well as all the standard map iterator stuff.
** Relating C++ types to protocol types.
There is not an exact 1-1 mapping between protocol types and the basic
value types described above. E.g. the protocol's str8, str16, and
str32 all map to std::string.
Where we need a 1-1 mapping (e.g. in a Map) we use **type wrappers**
e.g.
namespace qpid::amqp_0_10 {
struct String8 { String value }; // ctors & conversions to & from
String
omitted
struct String16 { String value };
struct Vbin32 { String value };
etc.
};
So to insert a value in a map that I want encoded as a string with an 8
bit length:
map["foo"] = amqp::0_10::String8("bar");
An Any decoded off the wire will always use the wrapped form of a type.
e.g.
if (anyCast<amqp0_10::String8>(&any)) { ... }
else if (anyCast<amqp0_10::String16>(any)) { ... }
...
To allow the user to avoid these protocol specifics we provide
anyConvert():
if (anyConvert<String>(&any)) { ... }
This will work if the any contains any type that can be converted to
string.
We use **type traits** to associate protocol-specific type codes with C
++ types.
TypeCode<T>::value == type code for type T.
There is a default mapping for unwrapped types with multiple mappings,
e.g.
TypeCode<std::String>::value == typecode for String32
Finally we add one more function to get the type code from an any:
anyTypeCode(any) returns the type code for the any.
To support code like:
switch(anyTypeCode(any)) {
case STR8_TYPE: ...
case UINT16_TYPE: ...
}
}
QUESTION: all of the above should probably be in a
qpid::amqp_<version> namespace? Can we rely on the 0-10 codes not to
change meaning in future versions? 1.0 type system is simpler but is
it backwards compatible?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
Carl Trieloff wrote:
> One comment I would have. In AMQP 1.0 we will have Queues and links,
> which can still handle the 0-10 model. However in an API would we not
> want to drop the noun Exchange and just use BindTo() or LinkTo() so
> that the API will work forwards to AMQP 1.0 with naming?
The noun Exchange does not appear in the core API (i.e. in the
client::api namespsace), nor in the examples that use that API.
It is only used in the AMQP 0-10 specific implementation (i.e. the
client::amqp0_10 namespace), where it makes sense to do so in my view.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Carl Trieloff <cc...@redhat.com>.
Gordon,
One comment I would have. In AMQP 1.0 we will have Queues and links, which
can still handle the 0-10 model. However in an API would we not want to
drop the
noun Exchange and just use BindTo() or LinkTo() so that the API will
work forwards
to AMQP 1.0 with naming?
Carl.
Gordon Sim wrote:
> There has been much discussion around the need for a higher-level c++
> API that insulates an application from any protocol specific details
> while allowing specific features of any given protocol to be exploited
> through the abstractions and extension points that this API offers.
>
> I should note that the direct use of the AMQP 0-10 specific API will
> of course still be supported.
>
> The API I have in mind is based on the concepts of message sources
> (from which messages can be received) and message sinks (to which
> messages can be sent).
>
> To send messages on a particular session, the application obtains a
> MessageProducer for that session from the chosen sink, through which
> it can then send messages. To receive messages, it subscribes the
> session to the chosen source. A subscription can be assoicated with a
> MessageListener (for 'push' style interaction) or it can create a
> MessageReceiver (for 'pull' style interactions).
>
> Two common patterns for sources and sinks are offered though the Queue
> and Topic classes which allow sources and sinks to be created for
> named queues in a point-to-point pattern, or publish-subscribe
> 'topics'.
>
> The attached patch offers a working sketch of such an API. The API is
> currently in the qpid::client::api namespace, primarily to keep it
> separate from the existing AMQP 0-10 specific API.
>
> There is also an implementation of this high-level API in terms of the
> existing 0-10 API in the qpid::client::amqp0_10 namespace, and a few
> example programs to demonstrate the use of queues and topics
> (examples/hla).
>
> This is still very much work in progress intended to demonstrate the
> source/sink concept and how that can be mapped to different patterns
> of interaction on a given protocol implementation.
>
> A critical aspect that I have yet to address is the nature of the
> Message interface (currently this just exposes raw data with no
> properties or headers). I think we also need to figure out the common,
> protocol independent options that we may want to directly support
> e.g. browsing v. consuming, prefetch windows, automatic acking
> v. application controlled acking etc. The threading model for
> connection processing and session dispatching also still needs to be
> thought through in more detail.
>
> All queries, comments, criticisms etc are very welcome.
> ------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
[c++]: Message API
Posted by Jonathan Robie <jo...@redhat.com>.
In addition to streaming, I think we need to make it easier to encode
data using AMQP types (for portability). The framing::Buffer class has a
nice, but incomplete set of methods for doing this:
framing::buffer b;
b.putMediumString("This is a medium string");
b.putShort(i);
b.putDouble(22.0 / i);
But you have to watch out not to overrun the buffer, I haven't figured
out a way to get that buffer into message content without making an
extra copy, and converting a buffer to a string is not the most obvious
way to send encoded binary data.
Some possible solutions:
* Add the putXXX() and getXXX() methods for each type to the Message API
* Create a class for putting and getting binary data that takes a
message, and puts and gets binary data
Message m;
BinaryStuffer bs(m);
bs.putShort(i);
* Add a MapMessage type, as in JavaJMS.
Jonathan
Gordon Sim wrote:
> Jonathan Robie wrote:
>> On the whole, I really like this. Comparing the example programs to
>> the existing examples, I think it does simplify quite a bit.
>>
>> A few thoughts:
>>
>> 1. Streaming data into or out of messages
>>
>> I really want to be able to stream into or out of messages directly.
>>
>> Instead of:
>>
>> message_data << "Message " << i;
>> message.setData(message_data.str());
>>
>> I would like to be able to do something like:
>>
>> message.getData() << "Message " << i;
>
> Yes, I can see that would be convenient.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
Attached is an updated patch with some simplifications and renaming
following reflection on Alan's comments and Rafi's python equivalent.
I've also added a request-response example.
I still need to do more work on the message class itself taking
Jonathan's comments into consideration. Also still outstanding is
addressing issues around threading that have also been discussed here,
details around asynchronous sending etc.
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
Jonathan Robie wrote:
> On the whole, I really like this. Comparing the example programs to the
> existing examples, I think it does simplify quite a bit.
>
> A few thoughts:
>
> 1. Streaming data into or out of messages
>
> I really want to be able to stream into or out of messages directly.
>
> Instead of:
>
> message_data << "Message " << i;
> message.setData(message_data.str());
>
> I would like to be able to do something like:
>
> message.getData() << "Message " << i;
Yes, I can see that would be convenient.
> 2. Having a Subscription object is really very helpful.
>
> 3. These examples don't create queeus and bindings explicitly - is that
> intentional? I assume the existing APIs continue to exist for this, but
> that is not going to bridget API versions.
Right. Binding is protocol specific and consequently is not represented
directly in this API. Some uses are covered in the implementation behind
the scenes, e.g. topic subscriptions that bind a subscription queue to
the exchange representing the topic. Other uses are really best thought
of as configuration and could be accomplished as part of deployment
using configuration tools/scripts.
> 4. I notice you do not use keyword arguments. Is the move away from them
> intentional? Why?
So far I haven't felt the need for them. They were very useful in the
generated signatures reflecting the 0-10 commands where there were often
lots of fields and where specific uses of those commands only needed
different specific subsets of the parameters.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
Jonathan Robie wrote:
> Steve Huston wrote:
>
>>> I think we need a clean way to handle both binary and character data
>>> in a streaming model. That's also very natural when
>>> copying from files to messages.
>>>
>>
>> True... The difficulty is that when you allow binary insertions
>> without the string intermediary, you need to provide ways to manage
>> packing, alignment, and byte ordering too. You end up either
>> reinventing something like CDR or forcing all to network byte order.
>> One advantage to using only strings in the message API is that it
>> throws the binary issues back up to the applications. The disadvantage
>> is that it throws the binary issues back up to the applications ;-)
>>
>
> framing::Buffer already has support for reading and writing many, but
> not all, AMQP types portably, and I think this *is* functionality we
> need to offer our users. We probably do need this for our AMQP types.
I agree that support for something like a JMS 'map message' would be a
valuable addition (possibly also something that represents a simple
sequence of typed values, i.e. like the map message but with no keys).
That is really a bundled special purpose message encoding; the user
would still obviously be free to use their own encodings as needed.
I think the 'streaming' support is a slightly different question (at
least as I understand it) and could mean one of at least two distinct
things. The first is just the convenience of using stream operators to
append to a message, the second is support for sending a large message
without having to have all its content in memory at once.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Jonathan Robie <jo...@redhat.com>.
Steve Huston wrote:
>> I think we need a clean way to handle both binary and
>> character data in a streaming model. That's also very natural when
>> copying from files to messages.
>>
>
> True... The difficulty is that when you allow binary insertions
> without the string intermediary, you need to provide ways to manage
> packing, alignment, and byte ordering too. You end up either
> reinventing something like CDR or forcing all to network byte order.
> One advantage to using only strings in the message API is that it
> throws the binary issues back up to the applications. The disadvantage
> is that it throws the binary issues back up to the applications ;-)
>
framing::Buffer already has support for reading and writing many, but
not all, AMQP types portably, and I think this *is* functionality we
need to offer our users. We probably do need this for our AMQP types.
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
RE: [c++]: Sketch of a high-level, protocol independent API
Posted by Steve Huston <sh...@riverace.com>.
> -----Original Message-----
> Sent: Monday, April 27, 2009 11:20 AM
>
> Alan Conway wrote:
> > Jonathan Robie wrote:
> >> On the whole, I really like this. Comparing the example
> programs to
> >> the existing examples, I think it does simplify quite a bit.
> >>
> >> A few thoughts:
> >>
> >> 1. Streaming data into or out of messages
> >>
> >> I really want to be able to stream into or out of messages
> directly.
> >>
> >> Instead of:
> >>
> >> message_data << "Message " << i;
> >> message.setData(message_data.str());
> >>
> >> I would like to be able to do something like:
> >>
> >> message.getData() << "Message " << i;
> >>
> >
> > The problem here is that there are many different ways that users
> > might want to encode their messages. ostream-style streaming as
> > human-readable strings is great for demos but quite unlikely for a
> > real application. So I would be reluctant to build it into
> the Message
> > class. I think it's best to keep Message's role entirely
> separate from
> > the business of encoding, so users can format their data using any
> > method they like and then attach it to the message.
>
> I don't think of streaming as just a way to format things as
> characters, I think of it primarily as a model for creating things
> incrementally.
Good point. However, if message.getData() returns a string reference,
it will be formatted as characters.
> I come from the "minimize the movement of data" school, and I
> also like my APIs to be clear about when data is managed.
I agree.
> Right now, if I copy a string to a message using setData() it is
> referenced counted and not copied, but if I then append to it
> it is copied.
Note that this behavior is for g++'s string and is not universal, so I
wouldn't count on that when designing an API.
> I think we need a clean way to handle both binary and
> character data in a streaming model. That's also very natural when
> copying from files to messages.
True... The difficulty is that when you allow binary insertions
without the string intermediary, you need to provide ways to manage
packing, alignment, and byte ordering too. You end up either
reinventing something like CDR or forcing all to network byte order.
One advantage to using only strings in the message API is that it
throws the binary issues back up to the applications. The disadvantage
is that it throws the binary issues back up to the applications ;-)
-Steve
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Jonathan Robie <jo...@redhat.com>.
Alan Conway wrote:
> Jonathan Robie wrote:
>> On the whole, I really like this. Comparing the example programs to
>> the existing examples, I think it does simplify quite a bit.
>>
>> A few thoughts:
>>
>> 1. Streaming data into or out of messages
>>
>> I really want to be able to stream into or out of messages directly.
>>
>> Instead of:
>>
>> message_data << "Message " << i;
>> message.setData(message_data.str());
>>
>> I would like to be able to do something like:
>>
>> message.getData() << "Message " << i;
>>
>
> The problem here is that there are many different ways that users
> might want to encode their messages. ostream-style streaming as
> human-readable strings is great for demos but quite unlikely for a
> real application. So I would be reluctant to build it into the Message
> class. I think it's best to keep Message's role entirely separate from
> the business of encoding, so users can format their data using any
> method they like and then attach it to the message.
I don't think of streaming as just a way to format things as characters,
I think of it primarily as a model for creating things incrementally. I
come from the "minimize the movement of data" school, and I also like my
APIs to be clear about when data is managed.
Right now, if I copy a string to a message using setData() it is
referenced counted and not copied, but if I then append to it it is copied.
I think we need a clean way to handle both binary and character data in
a streaming model. That's also very natural when copying from files to
messages.
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Alan Conway <ac...@redhat.com>.
Jonathan Robie wrote:
> On the whole, I really like this. Comparing the example programs to the
> existing examples, I think it does simplify quite a bit.
>
> A few thoughts:
>
> 1. Streaming data into or out of messages
>
> I really want to be able to stream into or out of messages directly.
>
> Instead of:
>
> message_data << "Message " << i;
> message.setData(message_data.str());
>
> I would like to be able to do something like:
>
> message.getData() << "Message " << i;
>
The problem here is that there are many different ways that users might want to
encode their messages. ostream-style streaming as human-readable strings is
great for demos but quite unlikely for a real application. So I would be
reluctant to build it into the Message class. I think it's best to keep
Message's role entirely separate from the business of encoding, so users can
format their data using any method they like and then attach it to the message.
> 4. I notice you do not use keyword arguments. Is the move away from them
> intentional? Why?
Keyword arguments are not common in C++. Being mapped directly from the
protocol, the existing API has functions with a LOT of arguments so the keywords
are very helpful. However a protocol-neutral API designed for C++ programmers
should be able to avoid such long argument lists.
Cheers,
Alan.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Jonathan Robie <jo...@redhat.com>.
On the whole, I really like this. Comparing the example programs to the
existing examples, I think it does simplify quite a bit.
A few thoughts:
1. Streaming data into or out of messages
I really want to be able to stream into or out of messages directly.
Instead of:
message_data << "Message " << i;
message.setData(message_data.str());
I would like to be able to do something like:
message.getData() << "Message " << i;
2. Having a Subscription object is really very helpful.
3. These examples don't create queeus and bindings explicitly - is that
intentional? I assume the existing APIs continue to exist for this, but
that is not going to bridget API versions.
4. I notice you do not use keyword arguments. Is the move away from them
intentional? Why?
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
James Mansion wrote:
> Gordon Sim wrote:
>> Alan Conway wrote:
>>> // Pull style:
>>> LocalQueue lq;
>>> queue.subscribe(lq);
>>> while (...) { Message m = lq.get(); ... }
>>>
>>> // Push style
>>> MyListener l;
>>> queue.subscribe(l);
>>> sessoin.run();
>>> }
>>>
> How does any of this integrate into a single-threaded application's main
> loop?
At present (in both the current and proposed apis) if dispatch to
registered listeners is used then a thread needs to be dedicated to that
(it can be the main thread, e.g. as in the various listener examples).
Using the 'pull' style interface offers a little more flexibility.
I agree the supported threading models is an area that needs more
thought (e.g perhaps offer something like a dispatch(n) call that does
only a limited amount of work, as discussed on another thread by Alan
and Rick).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by James Mansion <ja...@mansionfamily.plus.com>.
Gordon Sim wrote:
> Alan Conway wrote:
>> // Pull style:
>> LocalQueue lq;
>> queue.subscribe(lq);
>> while (...) { Message m = lq.get(); ... }
>>
>> // Push style
>> MyListener l;
>> queue.subscribe(l);
>> sessoin.run();
>> }
>>
How does any of this integrate into a single-threaded application's main
loop?
James
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
Alan Conway wrote:
> Here is a simplified version of the proposal. It involves fewer classes
> but I think is just as flexible and extensible as the original.
>
>
> ------------------------------------------------------------------------
>
> /**
> This is a sketch of some highre-level patterns of use intended as a
> supplement to (not replacement of) existing API. These patterns
> hide wiring and subscription details of setting up common patterns of use.
>
> Main changes from Gordons proposal:
> - Fold MessageProducer into MessageSink
> - No generic MessageSource, subscribe directly to pattern classe objects.
> Listener + Subscription already provide a generic representation of the source.
> - Avoids double layer of factories.
> - Pattern classes are real objects not just static function namespaces.
> - Using LocalQueue rather than MessageReceiver here, we can argue about the name later.
> - Drop bool synchronous arg, sync/async send have different return types, async must return a Future.
>
> ALTERNATE IDEAS:
> Build LocalQueue into Subscription, allow a listener to be attached to a subscription
> after it is created (it will receive any queued messages)
> */
>
> class Future {...}; // Need a future for async send.
>
> class MessageSink {
> // NB: Send can modify the message, e.g. set its routing key or attach properties.
> void send(Message&);
> Future sendAsync(Message&);
> // Should cancel automatically in dtor, do we need an explicit cancel?
> void cancel();
> };
>
> // Re-cast LocalQueue as a Listener so there's only 1 mechanism to deliver messages to app.
> // A single subscribe(Listener) function supports both push and pull.
> class LocalQueue : Listener { ... };
>
> class Topic {
> Topic(Session, name);
> Subscription subscribe(Listener, pattern);
> Subscription subscribe(Listener, pattern1, pattern2);
> ...
> Subscription subscribe(Listener, vector<string> patterns);
>
> MessageSink getSink(pattern);
> };
>
> // Avoid name Queue since that has a different meaning in AMQP than the pattern described here.
> // Note inherits MessageSink since there's only one sync for a SimpleQueue - built in.
> class SimpleQueue : public MessageSink {
> SimpleQueue(Session, name);
> Subscription subscribe(Listener);
>
> // send/sendAsync/cancel from MessageSink
> };
I think explicitly indicating intent to publish is important and I don't
think SimpleQueue above does that.
I think it would also be desirable to be able to write code that is
independent of the specific 'pattern class' used (e.g. can operate with
simple queues or topics).
Other than that I agree with many of your points.
>
> class XQueryEvaluator : public MessageSink {
> XQueryEvaluator(Session, name, ... ?);
> Subscription subscribe(Listener, xquery);
>
> // send/sendAsync/cancel from MessageSink
> };
>
>
> // ==== Example code sketches.
>
>
> // SimpleQueue
> int main() {
> Session s = ...;
> SimpleQueue queue(s, "foo");
>
> // Sending
> for (...) {
> message = ...;
> queue.send(message);
> }
> // Pull style:
> LocalQueue lq;
> queue.subscribe(lq);
> while (...) { Message m = lq.get(); ... }
>
> // Push style
> MyListener l;
> queue.subscribe(l);
> sessoin.run();
> }
>
> // SimpleQueue
> int main() {
> Session s = ...;
> SimpleQueue queue(s, "foo");
> for (...) {
> message = ...;
> queue.send(message);
> }
> // Pull style:
> LocalQueue lq;
> queue.subscribe(lq);
> while (...) { Message m = lq.get(); ... }
>
> // Push style
> MyListener l;
> queue.subscribe(l);
> session.run();
> }
>
> // Topic
>
> int main() {
> Session s;
> Topic topic(s, "myTopic");
>
> // Sending
> sendMessages(topic.getSink("usa.news"), count, "news about usa");
> sendMessages(topic.getSink("europe.news"), count, "news about europe");
>
> // Pull style
> LocalQueue lq;
> topic.subscribe(lq, pattern);
> for (...) { Message m = topic.get(); }
>
> // Push style
> MyListener l;
> queue.subscribe(l);
> session.run();
> }
>
> void sendMessages(MessageSink topic, int count, string news) {
> for (... count ...) { topic.send(Message(news); }
> }
>
>
>
> ------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Alan Conway <ac...@redhat.com>.
Gordon Sim wrote:
> There has been much discussion around the need for a higher-level c++
> API that insulates an application from any protocol specific details
> while allowing specific features of any given protocol to be exploited
> through the abstractions and extension points that this API offers.
>
> I should note that the direct use of the AMQP 0-10 specific API will
> of course still be supported.
>
> The API I have in mind is based on the concepts of message sources
> (from which messages can be received) and message sinks (to which
> messages can be sent).
>
> To send messages on a particular session, the application obtains a
> MessageProducer for that session from the chosen sink, through which
> it can then send messages. To receive messages, it subscribes the
> session to the chosen source. A subscription can be assoicated with a
> MessageListener (for 'push' style interaction) or it can create a
> MessageReceiver (for 'pull' style interactions).
>
> Two common patterns for sources and sinks are offered though the Queue
> and Topic classes which allow sources and sinks to be created for
> named queues in a point-to-point pattern, or publish-subscribe
> 'topics'.
>
> The attached patch offers a working sketch of such an API. The API is
> currently in the qpid::client::api namespace, primarily to keep it
> separate from the existing AMQP 0-10 specific API.
>
> There is also an implementation of this high-level API in terms of the
> existing 0-10 API in the qpid::client::amqp0_10 namespace, and a few
> example programs to demonstrate the use of queues and topics
> (examples/hla).
>
> This is still very much work in progress intended to demonstrate the
> source/sink concept and how that can be mapped to different patterns
> of interaction on a given protocol implementation.
>
> A critical aspect that I have yet to address is the nature of the
> Message interface (currently this just exposes raw data with no
> properties or headers). I think we also need to figure out the common,
> protocol independent options that we may want to directly support
> e.g. browsing v. consuming, prefetch windows, automatic acking
> v. application controlled acking etc. The threading model for connection
> processing and session dispatching also still needs to be thought
> through in more detail.
>
> All queries, comments, criticisms etc are very welcome.
>
>
Here is a simplified version of the proposal. It involves fewer classes but I
think is just as flexible and extensible as the original.
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Aidan Skinner <ai...@gmail.com>.
On Tue, Apr 7, 2009 at 6:39 PM, Gordon Sim <gs...@redhat.com> wrote:
> William Henry wrote:
>>
>> It would be really great if we could define this API as a standard
>> and not just specific to apache/qpid. Is there anywhere this might
>> live? Do others think that would be important?
>
> Standardising is an arduous process. I think we should leave any thought of
> that until user demand for such a standard is clear and explicit.
+1
It might also make sense to see what libamqp produces.
- Aidan
--
Apache Qpid - World Domination through Advanced Message Queueing
http://qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Gordon Sim <gs...@redhat.com>.
William Henry wrote:
> It would be really great if we could define this API as a standard
> and not just specific to apache/qpid. Is there anywhere this might
> live? Do others think that would be important?
Standardising is an arduous process. I think we should leave any thought
of that until user demand for such a standard is clear and explicit.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Carl Trieloff <cc...@redhat.com>.
Jonathan Robie wrote:
> Aidan Skinner wrote:
>> On Tue, Apr 7, 2009 at 7:45 PM, Steve Huston <sh...@riverace.com>
>> wrote:
>>
>>
>>> I don't think trying a formal industry standard is adviseable, but
>>> certainly trying to get some participation from other
>>> AMQP-implementing bodies would be a positive step.
>>>
My view is debate it here, on this list. once we have something we like
we socilize it around.
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Jonathan Robie <jo...@redhat.com>.
Aidan Skinner wrote:
> On Tue, Apr 7, 2009 at 7:45 PM, Steve Huston <sh...@riverace.com> wrote:
>
>
>> I don't think trying a formal industry standard is adviseable, but
>> certainly trying to get some participation from other
>> AMQP-implementing bodies would be a positive step.
>>
>
> +100
>
> - Aidan (who thinks we should do more hugging with other
> implementations generally)
>
+1 (my vote doesn't carry as much weight as Aidan's ...)
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by Aidan Skinner <ai...@gmail.com>.
On Tue, Apr 7, 2009 at 7:45 PM, Steve Huston <sh...@riverace.com> wrote:
> I don't think trying a formal industry standard is adviseable, but
> certainly trying to get some participation from other
> AMQP-implementing bodies would be a positive step.
+100
- Aidan (who thinks we should do more hugging with other
implementations generally)
--
Apache Qpid - World Domination through Advanced Message Queueing
http://qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
RE: [c++]: Sketch of a high-level, protocol independent API
Posted by Steve Huston <sh...@riverace.com>.
> It would be really great if we could define this API as a
> standard and not just specific to apache/qpid. Is there
> anywhere this might live? Do others think that would be important?
I think that's very important, and there was some sentiment from
(potential) users expressed in favor of this sort of idea at the AMQP
face-to-face public meeting last week.
I don't think trying a formal industry standard is adviseable, but
certainly trying to get some participation from other
AMQP-implementing bodies would be a positive step.
-Steve
> ----- "Gordon Sim" <gs...@redhat.com> wrote:
>
> > There has been much discussion around the need for a
> higher-level c++
> > API that insulates an application from any protocol specific
details
> > while allowing specific features of any given protocol to be
> > exploited
> > through the abstractions and extension points that this API
offers.
> >
> > I should note that the direct use of the AMQP 0-10 specific API
will
> > of course still be supported.
> >
> > The API I have in mind is based on the concepts of message sources
> > (from which messages can be received) and message sinks (to which
> > messages can be sent).
> >
> > To send messages on a particular session, the application obtains
a
> > MessageProducer for that session from the chosen sink, through
which
> > it can then send messages. To receive messages, it subscribes the
> > session to the chosen source. A subscription can be
> assoicated with a
> > MessageListener (for 'push' style interaction) or it can create a
> > MessageReceiver (for 'pull' style interactions).
> >
> > Two common patterns for sources and sinks are offered though the
> > Queue
> > and Topic classes which allow sources and sinks to be created for
> > named queues in a point-to-point pattern, or publish-subscribe
> > 'topics'.
> >
> > The attached patch offers a working sketch of such an API.
> The API is
> > currently in the qpid::client::api namespace, primarily to keep it
> > separate from the existing AMQP 0-10 specific API.
> >
> > There is also an implementation of this high-level API in terms of
> > the
> > existing 0-10 API in the qpid::client::amqp0_10 namespace, and a
few
> > example programs to demonstrate the use of queues and topics
> > (examples/hla).
> >
> > This is still very much work in progress intended to demonstrate
the
> > source/sink concept and how that can be mapped to different
patterns
> > of interaction on a given protocol implementation.
> >
> > A critical aspect that I have yet to address is the nature of the
> > Message interface (currently this just exposes raw data with no
> > properties or headers). I think we also need to figure out the
> > common,
> > protocol independent options that we may want to directly support
> > e.g. browsing v. consuming, prefetch windows, automatic acking
> > v. application controlled acking etc. The threading model for
> > connection
> > processing and session dispatching also still needs to be thought
> > through in more detail.
> >
> > All queries, comments, criticisms etc are very welcome.
> >
> >
> >
>
---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [c++]: Sketch of a high-level, protocol independent API
Posted by William Henry <wh...@redhat.com>.
Hi Gordon,
It would be really great if we could define this API as a standard and not just specific to apache/qpid. Is there anywhere this might live? Do others think that would be important?
William
----- "Gordon Sim" <gs...@redhat.com> wrote:
> There has been much discussion around the need for a higher-level c++
> API that insulates an application from any protocol specific details
> while allowing specific features of any given protocol to be
> exploited
> through the abstractions and extension points that this API offers.
>
> I should note that the direct use of the AMQP 0-10 specific API will
> of course still be supported.
>
> The API I have in mind is based on the concepts of message sources
> (from which messages can be received) and message sinks (to which
> messages can be sent).
>
> To send messages on a particular session, the application obtains a
> MessageProducer for that session from the chosen sink, through which
> it can then send messages. To receive messages, it subscribes the
> session to the chosen source. A subscription can be assoicated with a
> MessageListener (for 'push' style interaction) or it can create a
> MessageReceiver (for 'pull' style interactions).
>
> Two common patterns for sources and sinks are offered though the
> Queue
> and Topic classes which allow sources and sinks to be created for
> named queues in a point-to-point pattern, or publish-subscribe
> 'topics'.
>
> The attached patch offers a working sketch of such an API. The API is
> currently in the qpid::client::api namespace, primarily to keep it
> separate from the existing AMQP 0-10 specific API.
>
> There is also an implementation of this high-level API in terms of
> the
> existing 0-10 API in the qpid::client::amqp0_10 namespace, and a few
> example programs to demonstrate the use of queues and topics
> (examples/hla).
>
> This is still very much work in progress intended to demonstrate the
> source/sink concept and how that can be mapped to different patterns
> of interaction on a given protocol implementation.
>
> A critical aspect that I have yet to address is the nature of the
> Message interface (currently this just exposes raw data with no
> properties or headers). I think we also need to figure out the
> common,
> protocol independent options that we may want to directly support
> e.g. browsing v. consuming, prefetch windows, automatic acking
> v. application controlled acking etc. The threading model for
> connection
> processing and session dispatching also still needs to be thought
> through in more detail.
>
> All queries, comments, criticisms etc are very welcome.
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org