Re: Question about pn_reactor and threads.

[Rafael Schloming <rh...@alum.mit.edu>]

What sort of work/connections are you talking about here? Are you talking
about processing AMQP messages in a thread pool or are you talking about
writing some sort of multithreaded I/O handler?

--Rafael

On Thu, Apr 30, 2015 at 2:47 PM, Alan Conway <ac...@redhat.com> wrote:

> Can the proton reactor be used to deliver work from multiple connections
> to a thread pool, where work from a given connection is only handled by
> one thread at a time (so access to each pn_transport and it's stuff is
> serialized)? That is a pretty standard model for servers.
>
> It doesn't look to me like this is the case but I may be missing
> something. If it is the case, what's the pattern for doing it?
>
> Cheers,
> Alan.
>
>

Re: Question about pn_reactor and threads.

[Rafael Schloming <rh...@alum.mit.edu>]

On Wed, May 6, 2015 at 10:59 AM, Alan Conway <ac...@redhat.com> wrote:

> On Thu, 2015-04-30 at 21:51 -0400, Rafael Schloming wrote:
> > What sort of work/connections are you talking about here? Are you talking
> > about processing AMQP messages in a thread pool or are you talking about
> > writing some sort of multithreaded I/O handler?
>
> I'm talking about running multiple proton instances in a thread pool,
> like the Qpid C++ broker does. It uses a thread safe poller (based on
> epoll or somesuch) to dispatch data from many connections to a small
> thread pool. The poller serializes work per connection. There is a
> proton engine per connection so work to each engine is serialized but
> different engines run concurrently. The worker threads feed the proton
> transport, process events, and move messages to and from thread-safe
> queues (which are potentially shared among all connections)
>
> I'm trying to understand if the C reactor can help with that. I have the
> impression that the reactor requires all of the engines and their
> handlers to run in a single thread. From there I could put messages into
> a thread pool to process them concurrently. However the engines share no
> data so there is no reason not to run them in parallel as well.
> Serializing everything thru the reactor doesn't seem to buy me anything,
> but it reduces concurrency and makes the reactor thread a bottleneck.
>

Your assumption that the engines share no state is not valid in general.
Given that the application can attach arbitrary state to connections,
sessions, links, deliveries, etc, it is entirely possible (and quite
common) that a whole group of connections might actually be fairly
intimately intertwined via shared application state. The reactor permits
you to code such an application without having to worry about concurrency.

On the other hand, using a reactor per connection doesn't seem to give
> me anything over just using an engine per connection.
>

In this case it still provides a common event dispatch framework. If you
write your code against the engine directly, then your code is basically
going to be one big event handler for every possible event that might occur
on a connection. This is fine until you start needing different behavior on
different links, say you want to apply one flow control policy on one link
and a different one on another link. You now either need to turn your one
big event handler into spaghetti, or you need a way to associate different
handlers with different links.

You can of course do all this in an ad-hoc way rather than using the
reactor, but the advantage of using the reactor is that by being consistent
about how we do this, we can build up a library of useful handlers that can
work well together.

--Rafael

Re: Question about pn_reactor and threads.

[Alan Conway <ac...@redhat.com>]

On Thu, 2015-04-30 at 21:51 -0400, Rafael Schloming wrote:
> What sort of work/connections are you talking about here? Are you talking
> about processing AMQP messages in a thread pool or are you talking about
> writing some sort of multithreaded I/O handler?

I'm talking about running multiple proton instances in a thread pool,
like the Qpid C++ broker does. It uses a thread safe poller (based on
epoll or somesuch) to dispatch data from many connections to a small
thread pool. The poller serializes work per connection. There is a
proton engine per connection so work to each engine is serialized but
different engines run concurrently. The worker threads feed the proton
transport, process events, and move messages to and from thread-safe
queues (which are potentially shared among all connections)

I'm trying to understand if the C reactor can help with that. I have the
impression that the reactor requires all of the engines and their
handlers to run in a single thread. From there I could put messages into
a thread pool to process them concurrently. However the engines share no
data so there is no reason not to run them in parallel as well.
Serializing everything thru the reactor doesn't seem to buy me anything,
but it reduces concurrency and makes the reactor thread a bottleneck.

On the other hand, using a reactor per connection doesn't seem to give
me anything over just using an engine per connection.

I may have missed something though...
 
Cheers,
Alan.

> 
> --Rafael
> 
> On Thu, Apr 30, 2015 at 2:47 PM, Alan Conway <ac...@redhat.com> wrote:
> 
> > Can the proton reactor be used to deliver work from multiple connections
> > to a thread pool, where work from a given connection is only handled by
> > one thread at a time (so access to each pn_transport and it's stuff is
> > serialized)? That is a pretty standard model for servers.
> >
> > It doesn't look to me like this is the case but I may be missing
> > something. If it is the case, what's the pattern for doing it?
> >
> > Cheers,
> > Alan.
> >
> >