C++ reactor based binding

[Cliff Jansen <cl...@gmail.com>]

I have started work on a C++ binding to the Proton reactor event based
model with an eye to providing an API very similar to the python
client.

I have put together a skimpy skeleton of an implementation with enough
corners cut to get a simple HelloWorld example to run, mostly to get
myself acquainted with most of the moving parts.  This can be seen at

  https://github.com/apache/qpid-proton/pull/18

I plan to get some JIRAs going on the topic and do my initial work in
some branch on my github account.  I welcome comments from anyone with
ideas on the shape this should take and welcome any assistance anyone
might wish to provide in it's implementation.

The broad goals are:

  to make it easy to write simple programs
  natural to build out more complicated ones
  very similar API to the Python client
  lean and performant

The choice of C++11 language features to include or leave out is still
a question mark in my mind.  I am currently trying to hedge my bets on
this by introducing such features conservatively (and with plenty of
notice).  The downside is that C++11 language features are not
available in older development systems, i.e. Visual Studio 2008 and
gcc for RHEL 5.  On the other hand, a non-trivial subset is available
in VS2010 and most newer compilers, and this is after all a new C++
API on an "Advanced" message queuing protocol.

Another big question in my mind is the desired level of multi
threading support.  Some event driven systems view threading as the
devil's trickery and eschew thread safety guarantees and multi
threading support entirely.  Perhaps thread safety without multi
threading support could be a valuable feature of the API, allowing the
programmer to ignore the issue (at the expense of some locking
overhead).

Dispatch currently manages some significant multithreading use of
Proton, but it does not use the reactor.  Similar grained multi
threading in the proposed C++ client looks quite possible with minimal
locking overhead (you can set a pn_collector_t per connection and
pn_event_t pools are already managed per collector... nice).  Perhaps
there are other thorny issues I haven't noticed plus there are
remaining known outages in the pn_io_t interface that need fixing.  If
the API is implemented similar to previous qpid C++ client work, using
Handles and PIMPL, presumably the implementation of multithreading
could be deferred without affecting the API (unless the lack of Boost
derails things somehow).

Cliff

Re: C++ reactor based binding

[Alan Conway <ac...@redhat.com>]

On Sun, 2015-04-12 at 13:29 -0700, Cliff Jansen wrote:
> I have started work on a C++ binding to the Proton reactor event based
> model with an eye to providing an API very similar to the python
> client.
> 
> I have put together a skimpy skeleton of an implementation with enough
> corners cut to get a simple HelloWorld example to run, mostly to get
> myself acquainted with most of the moving parts.  This can be seen at
> 
>   https://github.com/apache/qpid-proton/pull/18
> 
This looks good, very clean and obvious to anyone who has seen the
python version.

For exceptions I would recommend keeping it simple and using the
standard:

class ProtonException: public std::runtime_error {
public:
  explicit ProtonException(const string& what);
};

The Qpid C++ exception stuff is too complicated IMO and not a good
example. I don't see any obvious need for any further hierarchy but you
can add it if/when it is needed.

> I plan to get some JIRAs going on the topic and do my initial work in
> some branch on my github account.  I welcome comments from anyone with
> ideas on the shape this should take and welcome any assistance anyone
> might wish to provide in it's implementation.
> 
> The broad goals are:
> 
>   to make it easy to write simple programs
>   natural to build out more complicated ones
>   very similar API to the Python client
>   lean and performant

Looks good on those fronts.

> The choice of C++11 language features to include or leave out is still
> a question mark in my mind.  I am currently trying to hedge my bets on
> this by introducing such features conservatively (and with plenty of
> notice).  The downside is that C++11 language features are not
> available in older development systems, i.e. Visual Studio 2008 and
> gcc for RHEL 5.  On the other hand, a non-trivial subset is available
> in VS2010 and most newer compilers, and this is after all a new C++
> API on an "Advanced" message queuing protocol.

A tricky balance, I tend to be conservative. Middleware libraries get
used in all sorts of places and broad adoption is an important goal. The
ideal is an API that doesn't require the latest features but is easy to
use with the latest features for those who have them.
 
> Another big question in my mind is the desired level of multi
> threading support.  Some event driven systems view threading as the
> devil's trickery and eschew thread safety guarantees and multi
> threading support entirely.  Perhaps thread safety without multi
> threading support could be a valuable feature of the API, allowing the
> programmer to ignore the issue (at the expense of some locking
> overhead).

The core is thread-unaware wrappers and handlers, exactly what you are
doing. Concurrent use of proton builds on top of that.

In go I spin up a goroutine per connection running just the
transport/connection part of the proton event machine. I feed bytes from
a Go net.Conn into the transport to pump the events. Go takes care of
scheduling many goroutines efficiently onto a small thread pool.

In C++ I would have an epoll or whatever feeding a thread pool. There
would be a proton transport/connection instance per connection. Now the
thread pool is doing the job of scheduling activity from many concurrent
connections onto a small thread pool and serializing access to each
proton instance.

The C reactor provides some IO neutral framework for abstracting poll
vs. select etc. but as far as I can see it is not usable in a thread
pool. I may be wrong there.

> Dispatch currently manages some significant multithreading use of
> Proton, but it does not use the reactor.  Similar grained multi
> threading in the proposed C++ client looks quite possible with minimal
> locking overhead (you can set a pn_collector_t per connection and
> pn_event_t pools are already managed per collector... nice).  Perhaps
> there are other thorny issues I haven't noticed plus there are
> remaining known outages in the pn_io_t interface that need fixing.  If
> the API is implemented similar to previous qpid C++ client work, using
> Handles and PIMPL, presumably the implementation of multithreading
> could be deferred without affecting the API (unless the lack of Boost
> derails things somehow).

Dispatch uses the approach above: it does it's own polling in its own
thread pool and serializes into per-connection proton instance. Like I
say maybe the reactor has a role here (dispatch pre-dates the reactor
and my Go binding ignores it out of ignorance) but in any case the basic
pattern is the same.

Bear in mind that communication between proton threads and other
application threads will require some form of locking, in-memory queues
or whatever. In Go my proton event loops are selecting on channels for
reading and writing connection data and a channel for "injecting" work
into the event loop. So any goroutine can put a function call into that
channel with a return channel for results.

In C++ you will also need to write handlers that are thread-safe to some
degree and that allow some form of "injection" of work into the event
loop. If you are writing a broker-like "pure" server that is entirely
driven by connection activity, then the handler itself doesn't have to
be thread safe, just the application resources it uses to respond to
proton events. However in an application with client-like behavior, you
need to push events from the app down to be processed in the handler
thread - possibly with thread safe queues that the handler can check
while its running in the event loop thread.

Again all of the above can definitely be done if you write your own
driver, I'm unclear on whether the reactor can help here in a
multi-threaded context.