You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by aconway <ac...@redhat.com> on 2015/10/22 23:33:48 UTC
The un-reactor
The proton reactor provides a complete solution for integrating foreign
IO into a single threaded proton event loop. This is useful in
situations where proton is being used in isolation, there is no other
IO handling framework available and everything is single threaded.
However often that is not going to be the case. There are many
existing C and C++ libraries for managing polling of file descriptors
and dispatching of work, and most server applications that might want
to embed proton already have their own frameworks.
So I'm thinking about how we can make integrating proton into an
existing (possibly multi-threaded) framework easier. I think all it
requires is a few simple functions to pull data out of a file
descriptor as events and push data back into a file descriptor, and
make this easier than directly using the collector and transport. I've
done it in Go already but I think it could be captured in C and C++ as
well.
Anyone already done something like this?
Cheers,
Alan.
Re: The un-reactor
Posted by Gordon Sim <gs...@redhat.com>.
On 10/26/2015 04:23 PM, aconway wrote:
> Proton is an AMQP protocol
> engine it is not a general purpose IO and event handling framework. I
> see the value of proton providing a simple IO framework for isolated
> proton projects when there is nothing else available. But if you
> already use tornado, epoll, ACE, boost IO, the Go net package, C++11
> std::async, the proactor pattern etc. etc. then why would you want to
> write code to integrate that with a single-threaded, non-standard, C
> reactor?
+1
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: The un-reactor
Posted by Ken Giusti <kg...@redhat.com>.
----- Original Message -----
> From: "aconway" <ac...@redhat.com>
> To: users@qpid.apache.org
> Sent: Monday, October 26, 2015 12:23:01 PM
> Subject: Re: The un-reactor
>
> On Thu, 2015-10-22 at 22:03 -0700, Cliff Jansen wrote:
> > I'm not sure if this exactly fits your question (especially the
> > multi-threaded part), but the python tornado loop examples run a
> > regular reactor set of events and collectors but driven by the
> > external loop.
> >
> > pn_iohandler is replaced with the TornadoLoophandler which knows how
> > to insert Proton io selectables into the external loop and provide
> > callbacks to do work when the external loop schedules activity on the
> > descriptor.
> >
> > Aside from program configuration on startup, the user code (e.g.
> > hello
> > world) knows nothing about the change and runs just as if under
> > Proton's own selector - the events it sees are the same.
> >
> > Something similar could be done with libuv in Proton-C, certainly
> > with
> > Posix, perhaps with some issues under Windows.
>
> I understand that this can be done, but why? Proton is an AMQP protocol
> engine it is not a general purpose IO and event handling framework. I
> see the value of proton providing a simple IO framework for isolated
> proton projects when there is nothing else available. But if you
> already use tornado, epoll, ACE, boost IO, the Go net package, C++11
> std::async, the proactor pattern etc. etc. then why would you want to
> write code to integrate that with a single-threaded, non-standard, C
> reactor? What is the reactor bringing to the party?
>
> You certainly want proton to handle AMQP for you, but you want to
> integrate it into your existing framework, not the other way around.
> The transport provides a simple bytes in/bytes out interface that is
> exactly what you need, and we can make it a little more user friendly.
>
Not sure if this is pertinent, but pygnus was designed with very little assumption about how the I/O (and timer handling) is achieved. Just as an example, pyngus
1) provides a container-level method for finding all Pyngus connections in the container that need I/O or have pending timers
2) The pyngus Connection wraps both the proton transport and the connection together. It provides APIs to read from and write to the connection, as well as a tick processing routine.
And that's about it. It assumes the application will handle the polling of the container and the actual transfer of the I/O. The application must also handle mapping between the pyngus Connection and the particular I/O mechanism the connection uses (e.g. the proper socket).
The oslo.messaging driver, for example, has a main loop that calls into the container to get all the connections needing processing, maps them to their corresponding socket and passes them to the main blocking select call. If the container returned a list of connections pending a timeout, the next timeout is used to calculate the select's timeout.
For convenience pyngus includes two utility methods for use with sockets - one for reading from the socket, another for writing to the socket. Both take a pyngus connection and a socket-like object as arguments, and handles transferring data between the two. The oslo.messaging driver uses these on return from select, but pyngus proper doesn't require these methods.
> The reactor integration points create more work and more overhead to
> integrate with another IO framework, and if your framework is not
> select-like (e.g. the go net package) then the reactor integration
> points make no sense at all.
>
>
> >
> > Cliff
> >
> > On Thu, Oct 22, 2015 at 2:33 PM, aconway <ac...@redhat.com> wrote:
> > > The proton reactor provides a complete solution for integrating
> > > foreign
> > > IO into a single threaded proton event loop. This is useful in
> > > situations where proton is being used in isolation, there is no
> > > other
> > > IO handling framework available and everything is single threaded.
> > >
> > > However often that is not going to be the case. There are many
> > > existing C and C++ libraries for managing polling of file
> > > descriptors
> > > and dispatching of work, and most server applications that might
> > > want
> > > to embed proton already have their own frameworks.
> > >
> > > So I'm thinking about how we can make integrating proton into an
> > > existing (possibly multi-threaded) framework easier. I think all it
> > > requires is a few simple functions to pull data out of a file
> > > descriptor as events and push data back into a file descriptor, and
> > > make this easier than directly using the collector and transport.
> > > I've
> > > done it in Go already but I think it could be captured in C and C++
> > > as
> > > well.
> > >
> > > Anyone already done something like this?
> > >
> > > Cheers,
> > > Alan.
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > For additional commands, e-mail: users-help@qpid.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: The un-reactor
Posted by aconway <ac...@redhat.com>.
On Thu, 2015-10-22 at 22:03 -0700, Cliff Jansen wrote:
> I'm not sure if this exactly fits your question (especially the
> multi-threaded part), but the python tornado loop examples run a
> regular reactor set of events and collectors but driven by the
> external loop.
>
> pn_iohandler is replaced with the TornadoLoophandler which knows how
> to insert Proton io selectables into the external loop and provide
> callbacks to do work when the external loop schedules activity on the
> descriptor.
>
> Aside from program configuration on startup, the user code (e.g.
> hello
> world) knows nothing about the change and runs just as if under
> Proton's own selector - the events it sees are the same.
>
> Something similar could be done with libuv in Proton-C, certainly
> with
> Posix, perhaps with some issues under Windows.
I understand that this can be done, but why? Proton is an AMQP protocol
engine it is not a general purpose IO and event handling framework. I
see the value of proton providing a simple IO framework for isolated
proton projects when there is nothing else available. But if you
already use tornado, epoll, ACE, boost IO, the Go net package, C++11
std::async, the proactor pattern etc. etc. then why would you want to
write code to integrate that with a single-threaded, non-standard, C
reactor? What is the reactor bringing to the party?
You certainly want proton to handle AMQP for you, but you want to
integrate it into your existing framework, not the other way around.
The transport provides a simple bytes in/bytes out interface that is
exactly what you need, and we can make it a little more user friendly.
The reactor integration points create more work and more overhead to
integrate with another IO framework, and if your framework is not
select-like (e.g. the go net package) then the reactor integration
points make no sense at all.
>
> Cliff
>
> On Thu, Oct 22, 2015 at 2:33 PM, aconway <ac...@redhat.com> wrote:
> > The proton reactor provides a complete solution for integrating
> > foreign
> > IO into a single threaded proton event loop. This is useful in
> > situations where proton is being used in isolation, there is no
> > other
> > IO handling framework available and everything is single threaded.
> >
> > However often that is not going to be the case. There are many
> > existing C and C++ libraries for managing polling of file
> > descriptors
> > and dispatching of work, and most server applications that might
> > want
> > to embed proton already have their own frameworks.
> >
> > So I'm thinking about how we can make integrating proton into an
> > existing (possibly multi-threaded) framework easier. I think all it
> > requires is a few simple functions to pull data out of a file
> > descriptor as events and push data back into a file descriptor, and
> > make this easier than directly using the collector and transport.
> > I've
> > done it in Go already but I think it could be captured in C and C++
> > as
> > well.
> >
> > Anyone already done something like this?
> >
> > Cheers,
> > Alan.
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: The un-reactor
Posted by Cliff Jansen <cl...@gmail.com>.
I'm not sure if this exactly fits your question (especially the
multi-threaded part), but the python tornado loop examples run a
regular reactor set of events and collectors but driven by the
external loop.
pn_iohandler is replaced with the TornadoLoophandler which knows how
to insert Proton io selectables into the external loop and provide
callbacks to do work when the external loop schedules activity on the
descriptor.
Aside from program configuration on startup, the user code (e.g. hello
world) knows nothing about the change and runs just as if under
Proton's own selector - the events it sees are the same.
Something similar could be done with libuv in Proton-C, certainly with
Posix, perhaps with some issues under Windows.
Cliff
On Thu, Oct 22, 2015 at 2:33 PM, aconway <ac...@redhat.com> wrote:
> The proton reactor provides a complete solution for integrating foreign
> IO into a single threaded proton event loop. This is useful in
> situations where proton is being used in isolation, there is no other
> IO handling framework available and everything is single threaded.
>
> However often that is not going to be the case. There are many
> existing C and C++ libraries for managing polling of file descriptors
> and dispatching of work, and most server applications that might want
> to embed proton already have their own frameworks.
>
> So I'm thinking about how we can make integrating proton into an
> existing (possibly multi-threaded) framework easier. I think all it
> requires is a few simple functions to pull data out of a file
> descriptor as events and push data back into a file descriptor, and
> make this easier than directly using the collector and transport. I've
> done it in Go already but I think it could be captured in C and C++ as
> well.
>
> Anyone already done something like this?
>
> Cheers,
> Alan.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org