You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Rafael Schloming <rh...@alum.mit.edu> on 2015/01/14 14:28:50 UTC
c reactor / gordon's examples
Hi Everyone,
I've been doing some work on a C reactor API for proton that is intended to
fit both alongside and underneath what gordon has been doing in pure
python. I have several goals with this work.
- Simplify/enable a reactive style of programming in C that is similar to
what gordon has built out in pure python.
- Provide a C API that translates well into the python/ruby/etc bindings,
so that sophisticated handlers can be written once in C rather than being
duplicated in each scripting language.
- Preserve the extensibility/flexibility that comes with being able to
define custom handlers in the bindings, so python/ruby/etc handlers can
intermix well with C handlers.
- Provide a C API that translates well into javascript via emscripten. In
some ways this is similar to the above goals with the other language
bindings, however I mention it separately because there are additional
memory management constraints for javascript since it has no finalizers.
I believe I've made significant progress towards most of these goals,
although there is still plenty of work left to do. I'd like to share a few
examples both to illustrate where I am with this and to solicit feedback
and/or help.
Let me say up front that these examples aren't intended to be "hello world"
type examples. The focus of this work has really been on the
reactor/handler/event-dispatch infrastructure, and so the example I've
chosen is really intended to illustrate key aspects of how this works. To
do that I've built an example that sets up a recurring task, a server, and
a client, all within the same process and then sends a number of messages
to itself.
I've included the same example twice, once written in C and once written in
the python binding of the C API. Please have a look and let me know what
you think.
--Rafael
Re: c reactor / gordon's examples
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 14, 2015 at 01:34:48PM -0500, Rafael Schloming wrote:
> On Wed, Jan 14, 2015 at 10:46 AM, Darryl L. Pierce <dp...@redhat.com>
> wrote:
> > On Wed, Jan 14, 2015 at 08:28:50AM -0500, Rafael Schloming wrote:
> > <snip>
> > > import sys
> > >
> > > reactor = Reactor(Program())
> > > reactor.acceptor("0.0.0.0", "5672")
> > > reactor.connection(Client("localhost:5672", sys.argv[1:]))
> > > reactor.run()
> >
> > I like the simplicity of how a simple app can be setup. The above can be
> > used inside of an EventMachine block in Ruby. Would we be able to pass a
> > file descriptor to the underlying C code? I'm wondering if we could
> > provide an AMQPServer similar to others provided [1].
> >
>
> The intention is to provide a couple of options here. One is the ability to
> supply an external file descriptor to the C loop as you describe. The other
> option which I'd like to provide is the ability to use the reactor purely
> for the cross language dispatch, and simply write all of the I/O code in
> the embedding language.
>
> Neither of these things will work out of the box right now, however the
> intention is very much to provdie them, e.g. internally the reactor
> provides I/O via what is essentially just a special handler that reacts to
> connection_local_open events and transports and sets up/pumps sockets
> accordingly. Right now this is hard coded to a C handler, but with some
> appropriate accessors we should be able to supply a pure python (or pure
> ruby) handler also.
Excellent, thanks for the response.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: c reactor / gordon's examples
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Wed, Jan 14, 2015 at 10:46 AM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Wed, Jan 14, 2015 at 08:28:50AM -0500, Rafael Schloming wrote:
> <snip>
> > import sys
> >
> > reactor = Reactor(Program())
> > reactor.acceptor("0.0.0.0", "5672")
> > reactor.connection(Client("localhost:5672", sys.argv[1:]))
> > reactor.run()
>
> I like the simplicity of how a simple app can be setup. The above can be
> used inside of an EventMachine block in Ruby. Would we be able to pass a
> file descriptor to the underlying C code? I'm wondering if we could
> provide an AMQPServer similar to others provided [1].
>
The intention is to provide a couple of options here. One is the ability to
supply an external file descriptor to the C loop as you describe. The other
option which I'd like to provide is the ability to use the reactor purely
for the cross language dispatch, and simply write all of the I/O code in
the embedding language.
Neither of these things will work out of the box right now, however the
intention is very much to provdie them, e.g. internally the reactor
provides I/O via what is essentially just a special handler that reacts to
connection_local_open events and transports and sets up/pumps sockets
accordingly. Right now this is hard coded to a C handler, but with some
appropriate accessors we should be able to supply a pure python (or pure
ruby) handler also.
--Rafael
Re: c reactor / gordon's examples
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 14, 2015 at 08:28:50AM -0500, Rafael Schloming wrote:
<snip>
> import sys
>
> reactor = Reactor(Program())
> reactor.acceptor("0.0.0.0", "5672")
> reactor.connection(Client("localhost:5672", sys.argv[1:]))
> reactor.run()
I like the simplicity of how a simple app can be setup. The above can be
used inside of an EventMachine block in Ruby. Would we be able to pass a
file descriptor to the underlying C code? I'm wondering if we could
provide an AMQPServer similar to others provided [1].
[1] https://github.com/eventmachine/eventmachine/tree/master/lib/em/protocols
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: c reactor / gordon's examples
Posted by Rafael Schloming <rh...@alum.mit.edu>.
My intention is to support a variety of scenarios for I/O. I believe it is
a key requirement that we can create sockets/fds in either python or C and
regardless of where they are created, they need to be serviceable by a
single I/O loop written either buy us or by others in C or in pure python.
(I think that covers all the bases, but if I'm missing a key combination,
then let me know.)
I believe the implementation is almost able to support all these scenarios
now, modulo a few details here and there. I will post some code snippets of
all the different permutations fairly soon (probably tomorrow as I am off
today).
--Rafael
On Fri, Jan 16, 2015 at 4:49 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 01/16/2015 08:06 AM, Flavio Percoco wrote:
>
>> One thing that I did want to mention (and forgot, obviously) is that
>> for things like non-blocking IO, we want to make sure the API works
>> well not just with eventlet but also with asyncio[0].
>>
>> Also, it'd be worth taking a look at how the Twisted[1] api works since
>> it's a long-time python project with an event based API like the one
>> you guys are building for proton.
>>
>
>
> Yes, I very much agree on both points. There is a JIRA (with a patch
> attached I believe) for python 3 support, and once that is in then figuring
> out how it would work well with asyncio is one of the things I've had in
> mind. Likewise with twisted.
>
> I knocked a couple of quick examples using the tornado event loop. There
> is still some work there though to better control the write interest in
> AMQP sockets, when that is triggered by non-AMQP events.
>
>
> Hope this helps,
>> Flavio
>>
>> [0] https://docs.python.org/3/library/asyncio.html
>> [1] https://pypi.python.org/pypi/Twisted
>>
>>
>
Re: c reactor / gordon's examples
Posted by Rafael Schloming <rh...@alum.mit.edu>.
My intention is to support a variety of scenarios for I/O. I believe it is
a key requirement that we can create sockets/fds in either python or C and
regardless of where they are created, they need to be serviceable by a
single I/O loop written either buy us or by others in C or in pure python.
(I think that covers all the bases, but if I'm missing a key combination,
then let me know.)
I believe the implementation is almost able to support all these scenarios
now, modulo a few details here and there. I will post some code snippets of
all the different permutations fairly soon (probably tomorrow as I am off
today).
--Rafael
On Fri, Jan 16, 2015 at 4:49 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 01/16/2015 08:06 AM, Flavio Percoco wrote:
>
>> One thing that I did want to mention (and forgot, obviously) is that
>> for things like non-blocking IO, we want to make sure the API works
>> well not just with eventlet but also with asyncio[0].
>>
>> Also, it'd be worth taking a look at how the Twisted[1] api works since
>> it's a long-time python project with an event based API like the one
>> you guys are building for proton.
>>
>
>
> Yes, I very much agree on both points. There is a JIRA (with a patch
> attached I believe) for python 3 support, and once that is in then figuring
> out how it would work well with asyncio is one of the things I've had in
> mind. Likewise with twisted.
>
> I knocked a couple of quick examples using the tornado event loop. There
> is still some work there though to better control the write interest in
> AMQP sockets, when that is triggered by non-AMQP events.
>
>
> Hope this helps,
>> Flavio
>>
>> [0] https://docs.python.org/3/library/asyncio.html
>> [1] https://pypi.python.org/pypi/Twisted
>>
>>
>
Re: c reactor / gordon's examples
Posted by Gordon Sim <gs...@redhat.com>.
On 01/16/2015 08:06 AM, Flavio Percoco wrote:
> One thing that I did want to mention (and forgot, obviously) is that
> for things like non-blocking IO, we want to make sure the API works
> well not just with eventlet but also with asyncio[0].
>
> Also, it'd be worth taking a look at how the Twisted[1] api works since
> it's a long-time python project with an event based API like the one
> you guys are building for proton.
Yes, I very much agree on both points. There is a JIRA (with a patch
attached I believe) for python 3 support, and once that is in then
figuring out how it would work well with asyncio is one of the things
I've had in mind. Likewise with twisted.
I knocked a couple of quick examples using the tornado event loop. There
is still some work there though to better control the write interest in
AMQP sockets, when that is triggered by non-AMQP events.
> Hope this helps,
> Flavio
>
> [0] https://docs.python.org/3/library/asyncio.html
> [1] https://pypi.python.org/pypi/Twisted
>
Re: c reactor / gordon's examples
Posted by Gordon Sim <gs...@redhat.com>.
On 01/16/2015 08:06 AM, Flavio Percoco wrote:
> One thing that I did want to mention (and forgot, obviously) is that
> for things like non-blocking IO, we want to make sure the API works
> well not just with eventlet but also with asyncio[0].
>
> Also, it'd be worth taking a look at how the Twisted[1] api works since
> it's a long-time python project with an event based API like the one
> you guys are building for proton.
Yes, I very much agree on both points. There is a JIRA (with a patch
attached I believe) for python 3 support, and once that is in then
figuring out how it would work well with asyncio is one of the things
I've had in mind. Likewise with twisted.
I knocked a couple of quick examples using the tornado event loop. There
is still some work there though to better control the write interest in
AMQP sockets, when that is triggered by non-AMQP events.
> Hope this helps,
> Flavio
>
> [0] https://docs.python.org/3/library/asyncio.html
> [1] https://pypi.python.org/pypi/Twisted
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: c reactor / gordon's examples
Posted by Flavio Percoco <fl...@redhat.com>.
On 15/01/15 22:54 +0000, Gordon Sim wrote:
>On 01/14/2015 01:28 PM, Rafael Schloming wrote:
>>Hi Everyone,
>>
>>I've been doing some work on a C reactor API for proton that is intended
>>to fit both alongside and underneath what gordon has been doing in pure
>>python. I have several goals with this work.
>>
>> - Simplify/enable a reactive style of programming in C that is similar
>>to what gordon has built out in pure python.
>>
>> - Provide a C API that translates well into the python/ruby/etc
>>bindings, so that sophisticated handlers can be written once in C rather
>>than being duplicated in each scripting language.
>>
>> - Preserve the extensibility/flexibility that comes with being able to
>>define custom handlers in the bindings, so python/ruby/etc handlers can
>>intermix well with C handlers.
>>
>> - Provide a C API that translates well into javascript via emscripten.
>>In some ways this is similar to the above goals with the other language
>>bindings, however I mention it separately because there are additional
>>memory management constraints for javascript since it has no finalizers.
>>
>>I believe I've made significant progress towards most of these goals,
>>although there is still plenty of work left to do. I'd like to share a
>>few examples both to illustrate where I am with this and to solicit
>>feedback and/or help.
>>
>>Let me say up front that these examples aren't intended to be "hello
>>world" type examples. The focus of this work has really been on the
>>reactor/handler/event-dispatch infrastructure, and so the example I've
>>chosen is really intended to illustrate key aspects of how this works.
>>To do that I've built an example that sets up a recurring task, a
>>server, and a client, all within the same process and then sends a
>>number of messages to itself.
>>
>>I've included the same example twice, once written in C and once written
>>in the python binding of the C API. Please have a look and let me know
>>what you think.
>
>Attached is that example using the pure python utilities I've been
>working on. It's similar in many ways, but there are some differences.
>
>Ideally we can start to reconcile these differences, particularly for
>the python bindings, both to avoid the confusion of having slightly
>different variants of the same functionality and to allow more
>flexible combining between c and python (e.g. calling c handlers from
>python, using the c event loop with python handlers etc). I think that
>exercise will also be a useful way of solidifying some of the
>interfaces.
>
>There are a couple of immediate improvements I want to make based on
>your example. The first is having the 'reactor' (I currently call the
>equivalent 'container') as a property of all events. That avoids the
>need to explicitly 'plumb it in' to components. The other is to add a
>handler argument to the schedule call allowing specific timed events
>to be handled by a given piece of logic.
>
>The second point also opens up the wider question of how handler
>scoping works. There are some differences there between our approaches
>that are worth digging into and rationalising.
>
>The first point impacts on a question I've been thinking about which
>is the correct layering (if any). I started to use the term
>'container' for something that had some higher level utility as an
>AMQP container, over and above the 'event loop'. Figuring out what the
>layers are, what they should be called and what exactly their
>interfaces are will be worthwhile.
>
>Anyway, more to come on all this from me, but I wanted to chip in
>briefly. I think we are building some momentum around the approach
>here, the c variant is an interesting addition - nice work - and I
>think also an opportunity to start unifying and consolidating things.
I think the interface looks pretty sane to me. I looked at it a couple
of weeks ago and I was planning to take another look and provide a
more useful feedback.
One thing that I did want to mention (and forgot, obviously) is that
for things like non-blocking IO, we want to make sure the API works
well not just with eventlet but also with asyncio[0].
Also, it'd be worth taking a look at how the Twisted[1] api works since
it's a long-time python project with an event based API like the one
you guys are building for proton.
Hope this helps,
Flavio
[0] https://docs.python.org/3/library/asyncio.html
[1] https://pypi.python.org/pypi/Twisted
--
@flaper87
Flavio Percoco
Re: c reactor / gordon's examples
Posted by Gordon Sim <gs...@redhat.com>.
On 01/14/2015 01:28 PM, Rafael Schloming wrote:
> Hi Everyone,
>
> I've been doing some work on a C reactor API for proton that is intended
> to fit both alongside and underneath what gordon has been doing in pure
> python. I have several goals with this work.
>
> - Simplify/enable a reactive style of programming in C that is similar
> to what gordon has built out in pure python.
>
> - Provide a C API that translates well into the python/ruby/etc
> bindings, so that sophisticated handlers can be written once in C rather
> than being duplicated in each scripting language.
>
> - Preserve the extensibility/flexibility that comes with being able to
> define custom handlers in the bindings, so python/ruby/etc handlers can
> intermix well with C handlers.
>
> - Provide a C API that translates well into javascript via emscripten.
> In some ways this is similar to the above goals with the other language
> bindings, however I mention it separately because there are additional
> memory management constraints for javascript since it has no finalizers.
>
> I believe I've made significant progress towards most of these goals,
> although there is still plenty of work left to do. I'd like to share a
> few examples both to illustrate where I am with this and to solicit
> feedback and/or help.
>
> Let me say up front that these examples aren't intended to be "hello
> world" type examples. The focus of this work has really been on the
> reactor/handler/event-dispatch infrastructure, and so the example I've
> chosen is really intended to illustrate key aspects of how this works.
> To do that I've built an example that sets up a recurring task, a
> server, and a client, all within the same process and then sends a
> number of messages to itself.
>
> I've included the same example twice, once written in C and once written
> in the python binding of the C API. Please have a look and let me know
> what you think.
Attached is that example using the pure python utilities I've been
working on. It's similar in many ways, but there are some differences.
Ideally we can start to reconcile these differences, particularly for
the python bindings, both to avoid the confusion of having slightly
different variants of the same functionality and to allow more flexible
combining between c and python (e.g. calling c handlers from python,
using the c event loop with python handlers etc). I think that exercise
will also be a useful way of solidifying some of the interfaces.
There are a couple of immediate improvements I want to make based on
your example. The first is having the 'reactor' (I currently call the
equivalent 'container') as a property of all events. That avoids the
need to explicitly 'plumb it in' to components. The other is to add a
handler argument to the schedule call allowing specific timed events to
be handled by a given piece of logic.
The second point also opens up the wider question of how handler scoping
works. There are some differences there between our approaches that are
worth digging into and rationalising.
The first point impacts on a question I've been thinking about which is
the correct layering (if any). I started to use the term 'container' for
something that had some higher level utility as an AMQP container, over
and above the 'event loop'. Figuring out what the layers are, what they
should be called and what exactly their interfaces are will be worthwhile.
Anyway, more to come on all this from me, but I wanted to chip in
briefly. I think we are building some momentum around the approach here,
the c variant is an interesting addition - nice work - and I think also
an opportunity to start unifying and consolidating things.
Re: c reactor / gordon's examples
Posted by Gordon Sim <gs...@redhat.com>.
On 01/14/2015 01:28 PM, Rafael Schloming wrote:
> Hi Everyone,
>
> I've been doing some work on a C reactor API for proton that is intended
> to fit both alongside and underneath what gordon has been doing in pure
> python. I have several goals with this work.
>
> - Simplify/enable a reactive style of programming in C that is similar
> to what gordon has built out in pure python.
>
> - Provide a C API that translates well into the python/ruby/etc
> bindings, so that sophisticated handlers can be written once in C rather
> than being duplicated in each scripting language.
>
> - Preserve the extensibility/flexibility that comes with being able to
> define custom handlers in the bindings, so python/ruby/etc handlers can
> intermix well with C handlers.
>
> - Provide a C API that translates well into javascript via emscripten.
> In some ways this is similar to the above goals with the other language
> bindings, however I mention it separately because there are additional
> memory management constraints for javascript since it has no finalizers.
>
> I believe I've made significant progress towards most of these goals,
> although there is still plenty of work left to do. I'd like to share a
> few examples both to illustrate where I am with this and to solicit
> feedback and/or help.
>
> Let me say up front that these examples aren't intended to be "hello
> world" type examples. The focus of this work has really been on the
> reactor/handler/event-dispatch infrastructure, and so the example I've
> chosen is really intended to illustrate key aspects of how this works.
> To do that I've built an example that sets up a recurring task, a
> server, and a client, all within the same process and then sends a
> number of messages to itself.
>
> I've included the same example twice, once written in C and once written
> in the python binding of the C API. Please have a look and let me know
> what you think.
Attached is that example using the pure python utilities I've been
working on. It's similar in many ways, but there are some differences.
Ideally we can start to reconcile these differences, particularly for
the python bindings, both to avoid the confusion of having slightly
different variants of the same functionality and to allow more flexible
combining between c and python (e.g. calling c handlers from python,
using the c event loop with python handlers etc). I think that exercise
will also be a useful way of solidifying some of the interfaces.
There are a couple of immediate improvements I want to make based on
your example. The first is having the 'reactor' (I currently call the
equivalent 'container') as a property of all events. That avoids the
need to explicitly 'plumb it in' to components. The other is to add a
handler argument to the schedule call allowing specific timed events to
be handled by a given piece of logic.
The second point also opens up the wider question of how handler scoping
works. There are some differences there between our approaches that are
worth digging into and rationalising.
The first point impacts on a question I've been thinking about which is
the correct layering (if any). I started to use the term 'container' for
something that had some higher level utility as an AMQP container, over
and above the 'event loop'. Figuring out what the layers are, what they
should be called and what exactly their interfaces are will be worthwhile.
Anyway, more to come on all this from me, but I wanted to chip in
briefly. I think we are building some momentum around the approach here,
the c variant is an interesting addition - nice work - and I think also
an opportunity to start unifying and consolidating things.
Re: c reactor / gordon's examples
Posted by Flavio Percoco <fl...@redhat.com>.
On 14/01/15 08:28 -0500, Rafael Schloming wrote:
>Hi Everyone,
>
>I've been doing some work on a C reactor API for proton that is intended to fit
>both alongside and underneath what gordon has been doing in pure python. I have
>several goals with this work.
>
> - Simplify/enable a reactive style of programming in C that is similar to what
>gordon has built out in pure python.
>
> - Provide a C API that translates well into the python/ruby/etc bindings, so
>that sophisticated handlers can be written once in C rather than being
>duplicated in each scripting language.
>
> - Preserve the extensibility/flexibility that comes with being able to define
>custom handlers in the bindings, so python/ruby/etc handlers can intermix well
>with C handlers.
Is the goal to be able to use swig here?
My question is based on the fact that swig is not as smart as it
should be when it comes to managing references and memory on the
python side. I believe some bugs were found in qpid-proton because of
this and I've hit this same issue in other projects.
When it comes to create python bindings for C code, I normally prefer
to use cffi[0] which is more reliable, the code bloat is less and it
gives you, in my opinion, more control over the C objects.
Other than this, I'm happy to see this happening!
Great work,
Flavio
[0] https://cffi.readthedocs.org/en/release-0.8/
>
> - Provide a C API that translates well into javascript via emscripten. In some
>ways this is similar to the above goals with the other language bindings,
>however I mention it separately because there are additional memory management
>constraints for javascript since it has no finalizers.
>
>I believe I've made significant progress towards most of these goals, although
>there is still plenty of work left to do. I'd like to share a few examples both
>to illustrate where I am with this and to solicit feedback and/or help.
>
>Let me say up front that these examples aren't intended to be "hello world"
>type examples. The focus of this work has really been on the reactor/handler/
>event-dispatch infrastructure, and so the example I've chosen is really
>intended to illustrate key aspects of how this works. To do that I've built an
>example that sets up a recurring task, a server, and a client, all within the
>same process and then sends a number of messages to itself.
>
>I've included the same example twice, once written in C and once written in the
>python binding of the C API. Please have a look and let me know what you think.
--
@flaper87
Flavio Percoco
Re: c reactor / gordon's examples
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Mon, Jan 26, 2015 at 9:43 AM, Alan Conway <ac...@redhat.com> wrote:
> On Mon, 2015-01-19 at 13:57 -0500, Rafael Schloming wrote:
> > On Mon, Jan 19, 2015 at 11:52 AM, Alan Conway <ac...@redhat.com>
> wrote:
> >
> > > On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> > > > Hi Everyone,
> > > >
> > > > I've been doing some work on a C reactor API for proton that is
> > > > intended to fit both alongside and underneath what gordon has been
> > > > doing in pure python. I have several goals with this work.
> > > >
> > > > - Simplify/enable a reactive style of programming in C that is
> > > > similar to what gordon has built out in pure python.
> > > >
> > > > - Provide a C API that translates well into the python/ruby/etc
> > > > bindings, so that sophisticated handlers can be written once in C
> > > > rather than being duplicated in each scripting language.
> > > >
> > > >
> > > > - Preserve the extensibility/flexibility that comes with being able
> > > > to define custom handlers in the bindings, so python/ruby/etc
> handlers
> > > > can intermix well with C handlers.
> > > >
> > > >
> > > > - Provide a C API that translates well into javascript via
> > > > emscripten. In some ways this is similar to the above goals with the
> > > > other language bindings, however I mention it separately because
> there
> > > > are additional memory management constraints for javascript since it
> > > > has no finalizers.
> > > >
> > > >
> > > >
> > > > I believe I've made significant progress towards most of these goals,
> > > > although there is still plenty of work left to do. I'd like to share
> a
> > > > few examples both to illustrate where I am with this and to solicit
> > > > feedback and/or help.
> > >
> > > The goals are spot on, C API looks good at first glance, I'll try to
> > > give it more careful scrutiny. The C handler functions need a void
> > > *context so we can parameterize the callbacks. e.g.
> > >
> > > void task_dispatch(pn_handler_t *handler, pn_event_t *event, void
> > > *context) {
> > > ...
> > > pn_reactor_schedule(reactor, (int)context, handler);
> > > }
> > > ...
> > > pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
> > >
> > > I'm not sure if you intended a 1-1 relationship between handlers and
> > > callback functions. If you do then you could put the context on the
> > > handler rather than as a parameter to the callback - but that would
> > > force the 1-1 relationship in the implementation from the API.
> > >
> >
> > I've had trouble in general using just simple void pointers for contexts
> in
> > more complex scenarios. The memory management semantics quickly get
> pretty
> > cumbersome. My intention was that any state the callback needs should be
> > accessible from either the event itself or from the handler, and there
> are
> > really two ways to add your own state. One being adding "attachments"
> into
> > the object model, and two being extending the handler itself with
> > additional state.
> >
> > The attachments work like this. Most of the core python objects now
> support
> > doing something like this:
> >
> > #define MY_STATE <any-unique-id>
> >
> > pn_record_t *attachments = pn_connection_attachments(conn);
> > pn_record_def(attachments, MY_STATE, PN_VOID);
> > ...
> > pn_record_set(attachments, MY_STATE, foo);
> > ...
> > void *foo = pn_record_get(attachments, MY_STATE);
> >
> > Two key things to note here is that first, when you define the
> attachment,
> > you are explicitly telling proton how to treat the pointer with respect
> to
> > memory management. You can define PN_OBJECT, or PN_WEAKREF attachments
> > also, and even define your own kind of pointer with your own finalization
> > semantics. This is how the python bindings work, the python binding
> defines
> > a PN_PYREF attachment for tracking associations between proton objects
> in C
> > and python objects.
> >
> > Second, there can be multiple independent attachments that won't conflict
> > with each other so long as they use different keys. (I've just been using
> > globally defined memory addresses to give myself guaranteed unique
> numeric
> > ids with no need to have a shared bootstrap process.)
> >
> > In the case of handlers, I've actually supplied another way to add extend
> > the handler with your own state. When you create a handler you can
> request
> > an additional chunk of memory associated with the handler that you can
> use
> > in any way you like, and when you do this you supply a finalizer along
> with
> > it. Your example above could work like this:
> >
> > pn_handler_t *handler = pn_handler_new(task_dispatch, sizeof(int), NULL
> > /*if I were using pointers, I might want a finalizer here*/);
> >
> > You can now access that from the handler via pn_handler_mem:
> >
> > int *my_state = pn_handler_mem(handler);
> >
> > You can check out the pn_flowcontroller handler if you want to see a full
> > working example of this.
>
> Nice! You're way ahead of me :) Agreed on the mem mgmt problems with
> void*, this seems like a nice solution. Attachments do imply an extra
> malloc, but not your second option which I like more. IMO you don't need
> attchments with the second option - you can use the second option to
> store a pointer to anything (or a struct full of pointers to manythings)
> with a proper finalizer and no void* at all. Very neat
The two options are for different scenarios, so I wouldn't necessarily
consider them alternatives. The second option is analogous to subclassing.
You are creating a new kind of object that has additional state, and you
can define whatever state you want and clean it up if necessary, but all
the state defined is under the control of a single piece of code.
Attachments on the other hand allow multiple independent pieces of code to
add state to an existing object without stepping on each others toes. Kind
of like having a map associated with the object and using string keys to
avoid naming collisions.
Say I wanted to augment the pn_flowcontroller handler to dynamically adjust
the window it uses on a per link basis using stats it gathers for that link
over time. In order to do that, my handler is going to have to track extra
state for each link, so of course it is very useful to be able to attach a
reference to the link back to whatever state that handler is tracking. Of
course it's also very useful for the application (which is really just
another handler) to be able to do the same thing, say add a pointer from an
outgoing link back to whatever source of messages it is pulling from (e.g.
a queue or a topic or something). Beyond just the memory management issues,
you can see why having a single context per link is a limitation in this
scenario. You have to globally coordinate exactly how you use that
particular context pointer. Attachments give you a much nicer way to do
this since both handlers can define their own private keys and have
independent back pointers from a link into whatever state the two handlers
are keeping on behalf of that link.
--Rafael
Re: c reactor / gordon's examples
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Mon, Jan 26, 2015 at 9:43 AM, Alan Conway <ac...@redhat.com> wrote:
> On Mon, 2015-01-19 at 13:57 -0500, Rafael Schloming wrote:
> > On Mon, Jan 19, 2015 at 11:52 AM, Alan Conway <ac...@redhat.com>
> wrote:
> >
> > > On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> > > > Hi Everyone,
> > > >
> > > > I've been doing some work on a C reactor API for proton that is
> > > > intended to fit both alongside and underneath what gordon has been
> > > > doing in pure python. I have several goals with this work.
> > > >
> > > > - Simplify/enable a reactive style of programming in C that is
> > > > similar to what gordon has built out in pure python.
> > > >
> > > > - Provide a C API that translates well into the python/ruby/etc
> > > > bindings, so that sophisticated handlers can be written once in C
> > > > rather than being duplicated in each scripting language.
> > > >
> > > >
> > > > - Preserve the extensibility/flexibility that comes with being able
> > > > to define custom handlers in the bindings, so python/ruby/etc
> handlers
> > > > can intermix well with C handlers.
> > > >
> > > >
> > > > - Provide a C API that translates well into javascript via
> > > > emscripten. In some ways this is similar to the above goals with the
> > > > other language bindings, however I mention it separately because
> there
> > > > are additional memory management constraints for javascript since it
> > > > has no finalizers.
> > > >
> > > >
> > > >
> > > > I believe I've made significant progress towards most of these goals,
> > > > although there is still plenty of work left to do. I'd like to share
> a
> > > > few examples both to illustrate where I am with this and to solicit
> > > > feedback and/or help.
> > >
> > > The goals are spot on, C API looks good at first glance, I'll try to
> > > give it more careful scrutiny. The C handler functions need a void
> > > *context so we can parameterize the callbacks. e.g.
> > >
> > > void task_dispatch(pn_handler_t *handler, pn_event_t *event, void
> > > *context) {
> > > ...
> > > pn_reactor_schedule(reactor, (int)context, handler);
> > > }
> > > ...
> > > pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
> > >
> > > I'm not sure if you intended a 1-1 relationship between handlers and
> > > callback functions. If you do then you could put the context on the
> > > handler rather than as a parameter to the callback - but that would
> > > force the 1-1 relationship in the implementation from the API.
> > >
> >
> > I've had trouble in general using just simple void pointers for contexts
> in
> > more complex scenarios. The memory management semantics quickly get
> pretty
> > cumbersome. My intention was that any state the callback needs should be
> > accessible from either the event itself or from the handler, and there
> are
> > really two ways to add your own state. One being adding "attachments"
> into
> > the object model, and two being extending the handler itself with
> > additional state.
> >
> > The attachments work like this. Most of the core python objects now
> support
> > doing something like this:
> >
> > #define MY_STATE <any-unique-id>
> >
> > pn_record_t *attachments = pn_connection_attachments(conn);
> > pn_record_def(attachments, MY_STATE, PN_VOID);
> > ...
> > pn_record_set(attachments, MY_STATE, foo);
> > ...
> > void *foo = pn_record_get(attachments, MY_STATE);
> >
> > Two key things to note here is that first, when you define the
> attachment,
> > you are explicitly telling proton how to treat the pointer with respect
> to
> > memory management. You can define PN_OBJECT, or PN_WEAKREF attachments
> > also, and even define your own kind of pointer with your own finalization
> > semantics. This is how the python bindings work, the python binding
> defines
> > a PN_PYREF attachment for tracking associations between proton objects
> in C
> > and python objects.
> >
> > Second, there can be multiple independent attachments that won't conflict
> > with each other so long as they use different keys. (I've just been using
> > globally defined memory addresses to give myself guaranteed unique
> numeric
> > ids with no need to have a shared bootstrap process.)
> >
> > In the case of handlers, I've actually supplied another way to add extend
> > the handler with your own state. When you create a handler you can
> request
> > an additional chunk of memory associated with the handler that you can
> use
> > in any way you like, and when you do this you supply a finalizer along
> with
> > it. Your example above could work like this:
> >
> > pn_handler_t *handler = pn_handler_new(task_dispatch, sizeof(int), NULL
> > /*if I were using pointers, I might want a finalizer here*/);
> >
> > You can now access that from the handler via pn_handler_mem:
> >
> > int *my_state = pn_handler_mem(handler);
> >
> > You can check out the pn_flowcontroller handler if you want to see a full
> > working example of this.
>
> Nice! You're way ahead of me :) Agreed on the mem mgmt problems with
> void*, this seems like a nice solution. Attachments do imply an extra
> malloc, but not your second option which I like more. IMO you don't need
> attchments with the second option - you can use the second option to
> store a pointer to anything (or a struct full of pointers to manythings)
> with a proper finalizer and no void* at all. Very neat
The two options are for different scenarios, so I wouldn't necessarily
consider them alternatives. The second option is analogous to subclassing.
You are creating a new kind of object that has additional state, and you
can define whatever state you want and clean it up if necessary, but all
the state defined is under the control of a single piece of code.
Attachments on the other hand allow multiple independent pieces of code to
add state to an existing object without stepping on each others toes. Kind
of like having a map associated with the object and using string keys to
avoid naming collisions.
Say I wanted to augment the pn_flowcontroller handler to dynamically adjust
the window it uses on a per link basis using stats it gathers for that link
over time. In order to do that, my handler is going to have to track extra
state for each link, so of course it is very useful to be able to attach a
reference to the link back to whatever state that handler is tracking. Of
course it's also very useful for the application (which is really just
another handler) to be able to do the same thing, say add a pointer from an
outgoing link back to whatever source of messages it is pulling from (e.g.
a queue or a topic or something). Beyond just the memory management issues,
you can see why having a single context per link is a limitation in this
scenario. You have to globally coordinate exactly how you use that
particular context pointer. Attachments give you a much nicer way to do
this since both handlers can define their own private keys and have
independent back pointers from a link into whatever state the two handlers
are keeping on behalf of that link.
--Rafael
Re: c reactor / gordon's examples
Posted by Alan Conway <ac...@redhat.com>.
On Mon, 2015-01-19 at 13:57 -0500, Rafael Schloming wrote:
> On Mon, Jan 19, 2015 at 11:52 AM, Alan Conway <ac...@redhat.com> wrote:
>
> > On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> > > Hi Everyone,
> > >
> > > I've been doing some work on a C reactor API for proton that is
> > > intended to fit both alongside and underneath what gordon has been
> > > doing in pure python. I have several goals with this work.
> > >
> > > - Simplify/enable a reactive style of programming in C that is
> > > similar to what gordon has built out in pure python.
> > >
> > > - Provide a C API that translates well into the python/ruby/etc
> > > bindings, so that sophisticated handlers can be written once in C
> > > rather than being duplicated in each scripting language.
> > >
> > >
> > > - Preserve the extensibility/flexibility that comes with being able
> > > to define custom handlers in the bindings, so python/ruby/etc handlers
> > > can intermix well with C handlers.
> > >
> > >
> > > - Provide a C API that translates well into javascript via
> > > emscripten. In some ways this is similar to the above goals with the
> > > other language bindings, however I mention it separately because there
> > > are additional memory management constraints for javascript since it
> > > has no finalizers.
> > >
> > >
> > >
> > > I believe I've made significant progress towards most of these goals,
> > > although there is still plenty of work left to do. I'd like to share a
> > > few examples both to illustrate where I am with this and to solicit
> > > feedback and/or help.
> >
> > The goals are spot on, C API looks good at first glance, I'll try to
> > give it more careful scrutiny. The C handler functions need a void
> > *context so we can parameterize the callbacks. e.g.
> >
> > void task_dispatch(pn_handler_t *handler, pn_event_t *event, void
> > *context) {
> > ...
> > pn_reactor_schedule(reactor, (int)context, handler);
> > }
> > ...
> > pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
> >
> > I'm not sure if you intended a 1-1 relationship between handlers and
> > callback functions. If you do then you could put the context on the
> > handler rather than as a parameter to the callback - but that would
> > force the 1-1 relationship in the implementation from the API.
> >
>
> I've had trouble in general using just simple void pointers for contexts in
> more complex scenarios. The memory management semantics quickly get pretty
> cumbersome. My intention was that any state the callback needs should be
> accessible from either the event itself or from the handler, and there are
> really two ways to add your own state. One being adding "attachments" into
> the object model, and two being extending the handler itself with
> additional state.
>
> The attachments work like this. Most of the core python objects now support
> doing something like this:
>
> #define MY_STATE <any-unique-id>
>
> pn_record_t *attachments = pn_connection_attachments(conn);
> pn_record_def(attachments, MY_STATE, PN_VOID);
> ...
> pn_record_set(attachments, MY_STATE, foo);
> ...
> void *foo = pn_record_get(attachments, MY_STATE);
>
> Two key things to note here is that first, when you define the attachment,
> you are explicitly telling proton how to treat the pointer with respect to
> memory management. You can define PN_OBJECT, or PN_WEAKREF attachments
> also, and even define your own kind of pointer with your own finalization
> semantics. This is how the python bindings work, the python binding defines
> a PN_PYREF attachment for tracking associations between proton objects in C
> and python objects.
>
> Second, there can be multiple independent attachments that won't conflict
> with each other so long as they use different keys. (I've just been using
> globally defined memory addresses to give myself guaranteed unique numeric
> ids with no need to have a shared bootstrap process.)
>
> In the case of handlers, I've actually supplied another way to add extend
> the handler with your own state. When you create a handler you can request
> an additional chunk of memory associated with the handler that you can use
> in any way you like, and when you do this you supply a finalizer along with
> it. Your example above could work like this:
>
> pn_handler_t *handler = pn_handler_new(task_dispatch, sizeof(int), NULL
> /*if I were using pointers, I might want a finalizer here*/);
>
> You can now access that from the handler via pn_handler_mem:
>
> int *my_state = pn_handler_mem(handler);
>
> You can check out the pn_flowcontroller handler if you want to see a full
> working example of this.
Nice! You're way ahead of me :) Agreed on the mem mgmt problems with
void*, this seems like a nice solution. Attachments do imply an extra
malloc, but not your second option which I like more. IMO you don't need
attchments with the second option - you can use the second option to
store a pointer to anything (or a struct full of pointers to manythings)
with a proper finalizer and no void* at all. Very neat.
Re: c reactor / gordon's examples
Posted by Alan Conway <ac...@redhat.com>.
On Mon, 2015-01-19 at 13:57 -0500, Rafael Schloming wrote:
> On Mon, Jan 19, 2015 at 11:52 AM, Alan Conway <ac...@redhat.com> wrote:
>
> > On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> > > Hi Everyone,
> > >
> > > I've been doing some work on a C reactor API for proton that is
> > > intended to fit both alongside and underneath what gordon has been
> > > doing in pure python. I have several goals with this work.
> > >
> > > - Simplify/enable a reactive style of programming in C that is
> > > similar to what gordon has built out in pure python.
> > >
> > > - Provide a C API that translates well into the python/ruby/etc
> > > bindings, so that sophisticated handlers can be written once in C
> > > rather than being duplicated in each scripting language.
> > >
> > >
> > > - Preserve the extensibility/flexibility that comes with being able
> > > to define custom handlers in the bindings, so python/ruby/etc handlers
> > > can intermix well with C handlers.
> > >
> > >
> > > - Provide a C API that translates well into javascript via
> > > emscripten. In some ways this is similar to the above goals with the
> > > other language bindings, however I mention it separately because there
> > > are additional memory management constraints for javascript since it
> > > has no finalizers.
> > >
> > >
> > >
> > > I believe I've made significant progress towards most of these goals,
> > > although there is still plenty of work left to do. I'd like to share a
> > > few examples both to illustrate where I am with this and to solicit
> > > feedback and/or help.
> >
> > The goals are spot on, C API looks good at first glance, I'll try to
> > give it more careful scrutiny. The C handler functions need a void
> > *context so we can parameterize the callbacks. e.g.
> >
> > void task_dispatch(pn_handler_t *handler, pn_event_t *event, void
> > *context) {
> > ...
> > pn_reactor_schedule(reactor, (int)context, handler);
> > }
> > ...
> > pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
> >
> > I'm not sure if you intended a 1-1 relationship between handlers and
> > callback functions. If you do then you could put the context on the
> > handler rather than as a parameter to the callback - but that would
> > force the 1-1 relationship in the implementation from the API.
> >
>
> I've had trouble in general using just simple void pointers for contexts in
> more complex scenarios. The memory management semantics quickly get pretty
> cumbersome. My intention was that any state the callback needs should be
> accessible from either the event itself or from the handler, and there are
> really two ways to add your own state. One being adding "attachments" into
> the object model, and two being extending the handler itself with
> additional state.
>
> The attachments work like this. Most of the core python objects now support
> doing something like this:
>
> #define MY_STATE <any-unique-id>
>
> pn_record_t *attachments = pn_connection_attachments(conn);
> pn_record_def(attachments, MY_STATE, PN_VOID);
> ...
> pn_record_set(attachments, MY_STATE, foo);
> ...
> void *foo = pn_record_get(attachments, MY_STATE);
>
> Two key things to note here is that first, when you define the attachment,
> you are explicitly telling proton how to treat the pointer with respect to
> memory management. You can define PN_OBJECT, or PN_WEAKREF attachments
> also, and even define your own kind of pointer with your own finalization
> semantics. This is how the python bindings work, the python binding defines
> a PN_PYREF attachment for tracking associations between proton objects in C
> and python objects.
>
> Second, there can be multiple independent attachments that won't conflict
> with each other so long as they use different keys. (I've just been using
> globally defined memory addresses to give myself guaranteed unique numeric
> ids with no need to have a shared bootstrap process.)
>
> In the case of handlers, I've actually supplied another way to add extend
> the handler with your own state. When you create a handler you can request
> an additional chunk of memory associated with the handler that you can use
> in any way you like, and when you do this you supply a finalizer along with
> it. Your example above could work like this:
>
> pn_handler_t *handler = pn_handler_new(task_dispatch, sizeof(int), NULL
> /*if I were using pointers, I might want a finalizer here*/);
>
> You can now access that from the handler via pn_handler_mem:
>
> int *my_state = pn_handler_mem(handler);
>
> You can check out the pn_flowcontroller handler if you want to see a full
> working example of this.
Nice! You're way ahead of me :) Agreed on the mem mgmt problems with
void*, this seems like a nice solution. Attachments do imply an extra
malloc, but not your second option which I like more. IMO you don't need
attchments with the second option - you can use the second option to
store a pointer to anything (or a struct full of pointers to manythings)
with a proper finalizer and no void* at all. Very neat.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: c reactor / gordon's examples
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Mon, Jan 19, 2015 at 11:52 AM, Alan Conway <ac...@redhat.com> wrote:
> On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> > Hi Everyone,
> >
> > I've been doing some work on a C reactor API for proton that is
> > intended to fit both alongside and underneath what gordon has been
> > doing in pure python. I have several goals with this work.
> >
> > - Simplify/enable a reactive style of programming in C that is
> > similar to what gordon has built out in pure python.
> >
> > - Provide a C API that translates well into the python/ruby/etc
> > bindings, so that sophisticated handlers can be written once in C
> > rather than being duplicated in each scripting language.
> >
> >
> > - Preserve the extensibility/flexibility that comes with being able
> > to define custom handlers in the bindings, so python/ruby/etc handlers
> > can intermix well with C handlers.
> >
> >
> > - Provide a C API that translates well into javascript via
> > emscripten. In some ways this is similar to the above goals with the
> > other language bindings, however I mention it separately because there
> > are additional memory management constraints for javascript since it
> > has no finalizers.
> >
> >
> >
> > I believe I've made significant progress towards most of these goals,
> > although there is still plenty of work left to do. I'd like to share a
> > few examples both to illustrate where I am with this and to solicit
> > feedback and/or help.
>
> The goals are spot on, C API looks good at first glance, I'll try to
> give it more careful scrutiny. The C handler functions need a void
> *context so we can parameterize the callbacks. e.g.
>
> void task_dispatch(pn_handler_t *handler, pn_event_t *event, void
> *context) {
> ...
> pn_reactor_schedule(reactor, (int)context, handler);
> }
> ...
> pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
>
> I'm not sure if you intended a 1-1 relationship between handlers and
> callback functions. If you do then you could put the context on the
> handler rather than as a parameter to the callback - but that would
> force the 1-1 relationship in the implementation from the API.
>
I've had trouble in general using just simple void pointers for contexts in
more complex scenarios. The memory management semantics quickly get pretty
cumbersome. My intention was that any state the callback needs should be
accessible from either the event itself or from the handler, and there are
really two ways to add your own state. One being adding "attachments" into
the object model, and two being extending the handler itself with
additional state.
The attachments work like this. Most of the core python objects now support
doing something like this:
#define MY_STATE <any-unique-id>
pn_record_t *attachments = pn_connection_attachments(conn);
pn_record_def(attachments, MY_STATE, PN_VOID);
...
pn_record_set(attachments, MY_STATE, foo);
...
void *foo = pn_record_get(attachments, MY_STATE);
Two key things to note here is that first, when you define the attachment,
you are explicitly telling proton how to treat the pointer with respect to
memory management. You can define PN_OBJECT, or PN_WEAKREF attachments
also, and even define your own kind of pointer with your own finalization
semantics. This is how the python bindings work, the python binding defines
a PN_PYREF attachment for tracking associations between proton objects in C
and python objects.
Second, there can be multiple independent attachments that won't conflict
with each other so long as they use different keys. (I've just been using
globally defined memory addresses to give myself guaranteed unique numeric
ids with no need to have a shared bootstrap process.)
In the case of handlers, I've actually supplied another way to add extend
the handler with your own state. When you create a handler you can request
an additional chunk of memory associated with the handler that you can use
in any way you like, and when you do this you supply a finalizer along with
it. Your example above could work like this:
pn_handler_t *handler = pn_handler_new(task_dispatch, sizeof(int), NULL
/*if I were using pointers, I might want a finalizer here*/);
You can now access that from the handler via pn_handler_mem:
int *my_state = pn_handler_mem(handler);
You can check out the pn_flowcontroller handler if you want to see a full
working example of this.
--Rafael
Re: c reactor / gordon's examples
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Mon, Jan 19, 2015 at 11:52 AM, Alan Conway <ac...@redhat.com> wrote:
> On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> > Hi Everyone,
> >
> > I've been doing some work on a C reactor API for proton that is
> > intended to fit both alongside and underneath what gordon has been
> > doing in pure python. I have several goals with this work.
> >
> > - Simplify/enable a reactive style of programming in C that is
> > similar to what gordon has built out in pure python.
> >
> > - Provide a C API that translates well into the python/ruby/etc
> > bindings, so that sophisticated handlers can be written once in C
> > rather than being duplicated in each scripting language.
> >
> >
> > - Preserve the extensibility/flexibility that comes with being able
> > to define custom handlers in the bindings, so python/ruby/etc handlers
> > can intermix well with C handlers.
> >
> >
> > - Provide a C API that translates well into javascript via
> > emscripten. In some ways this is similar to the above goals with the
> > other language bindings, however I mention it separately because there
> > are additional memory management constraints for javascript since it
> > has no finalizers.
> >
> >
> >
> > I believe I've made significant progress towards most of these goals,
> > although there is still plenty of work left to do. I'd like to share a
> > few examples both to illustrate where I am with this and to solicit
> > feedback and/or help.
>
> The goals are spot on, C API looks good at first glance, I'll try to
> give it more careful scrutiny. The C handler functions need a void
> *context so we can parameterize the callbacks. e.g.
>
> void task_dispatch(pn_handler_t *handler, pn_event_t *event, void
> *context) {
> ...
> pn_reactor_schedule(reactor, (int)context, handler);
> }
> ...
> pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
>
> I'm not sure if you intended a 1-1 relationship between handlers and
> callback functions. If you do then you could put the context on the
> handler rather than as a parameter to the callback - but that would
> force the 1-1 relationship in the implementation from the API.
>
I've had trouble in general using just simple void pointers for contexts in
more complex scenarios. The memory management semantics quickly get pretty
cumbersome. My intention was that any state the callback needs should be
accessible from either the event itself or from the handler, and there are
really two ways to add your own state. One being adding "attachments" into
the object model, and two being extending the handler itself with
additional state.
The attachments work like this. Most of the core python objects now support
doing something like this:
#define MY_STATE <any-unique-id>
pn_record_t *attachments = pn_connection_attachments(conn);
pn_record_def(attachments, MY_STATE, PN_VOID);
...
pn_record_set(attachments, MY_STATE, foo);
...
void *foo = pn_record_get(attachments, MY_STATE);
Two key things to note here is that first, when you define the attachment,
you are explicitly telling proton how to treat the pointer with respect to
memory management. You can define PN_OBJECT, or PN_WEAKREF attachments
also, and even define your own kind of pointer with your own finalization
semantics. This is how the python bindings work, the python binding defines
a PN_PYREF attachment for tracking associations between proton objects in C
and python objects.
Second, there can be multiple independent attachments that won't conflict
with each other so long as they use different keys. (I've just been using
globally defined memory addresses to give myself guaranteed unique numeric
ids with no need to have a shared bootstrap process.)
In the case of handlers, I've actually supplied another way to add extend
the handler with your own state. When you create a handler you can request
an additional chunk of memory associated with the handler that you can use
in any way you like, and when you do this you supply a finalizer along with
it. Your example above could work like this:
pn_handler_t *handler = pn_handler_new(task_dispatch, sizeof(int), NULL
/*if I were using pointers, I might want a finalizer here*/);
You can now access that from the handler via pn_handler_mem:
int *my_state = pn_handler_mem(handler);
You can check out the pn_flowcontroller handler if you want to see a full
working example of this.
--Rafael
Re: c reactor / gordon's examples
Posted by Alan Conway <ac...@redhat.com>.
On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> Hi Everyone,
>
> I've been doing some work on a C reactor API for proton that is
> intended to fit both alongside and underneath what gordon has been
> doing in pure python. I have several goals with this work.
>
> - Simplify/enable a reactive style of programming in C that is
> similar to what gordon has built out in pure python.
>
> - Provide a C API that translates well into the python/ruby/etc
> bindings, so that sophisticated handlers can be written once in C
> rather than being duplicated in each scripting language.
>
>
> - Preserve the extensibility/flexibility that comes with being able
> to define custom handlers in the bindings, so python/ruby/etc handlers
> can intermix well with C handlers.
>
>
> - Provide a C API that translates well into javascript via
> emscripten. In some ways this is similar to the above goals with the
> other language bindings, however I mention it separately because there
> are additional memory management constraints for javascript since it
> has no finalizers.
>
>
>
> I believe I've made significant progress towards most of these goals,
> although there is still plenty of work left to do. I'd like to share a
> few examples both to illustrate where I am with this and to solicit
> feedback and/or help.
The goals are spot on, C API looks good at first glance, I'll try to
give it more careful scrutiny. The C handler functions need a void
*context so we can parameterize the callbacks. e.g.
void task_dispatch(pn_handler_t *handler, pn_event_t *event, void *context) {
...
pn_reactor_schedule(reactor, (int)context, handler);
}
...
pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
I'm not sure if you intended a 1-1 relationship between handlers and
callback functions. If you do then you could put the context on the
handler rather than as a parameter to the callback - but that would
force the 1-1 relationship in the implementation from the API.
>
>
> Let me say up front that these examples aren't intended to be "hello
> world" type examples. The focus of this work has really been on the
> reactor/handler/event-dispatch infrastructure, and so the example I've
> chosen is really intended to illustrate key aspects of how this works.
> To do that I've built an example that sets up a recurring task, a
> server, and a client, all within the same process and then sends a
> number of messages to itself.
>
> I've included the same example twice, once written in C and once
> written in the python binding of the C API. Please have a look and let
> me know what you think.
>
>
> --Rafael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
Re: c reactor / gordon's examples
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 14, 2015 at 08:28:50AM -0500, Rafael Schloming wrote:
<snip>
> import sys
>
> reactor = Reactor(Program())
> reactor.acceptor("0.0.0.0", "5672")
> reactor.connection(Client("localhost:5672", sys.argv[1:]))
> reactor.run()
I like the simplicity of how a simple app can be setup. The above can be
used inside of an EventMachine block in Ruby. Would we be able to pass a
file descriptor to the underlying C code? I'm wondering if we could
provide an AMQPServer similar to others provided [1].
[1] https://github.com/eventmachine/eventmachine/tree/master/lib/em/protocols
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: c reactor / gordon's examples
Posted by Alan Conway <ac...@redhat.com>.
On Wed, 2015-01-14 at 08:28 -0500, Rafael Schloming wrote:
> Hi Everyone,
>
> I've been doing some work on a C reactor API for proton that is
> intended to fit both alongside and underneath what gordon has been
> doing in pure python. I have several goals with this work.
>
> - Simplify/enable a reactive style of programming in C that is
> similar to what gordon has built out in pure python.
>
> - Provide a C API that translates well into the python/ruby/etc
> bindings, so that sophisticated handlers can be written once in C
> rather than being duplicated in each scripting language.
>
>
> - Preserve the extensibility/flexibility that comes with being able
> to define custom handlers in the bindings, so python/ruby/etc handlers
> can intermix well with C handlers.
>
>
> - Provide a C API that translates well into javascript via
> emscripten. In some ways this is similar to the above goals with the
> other language bindings, however I mention it separately because there
> are additional memory management constraints for javascript since it
> has no finalizers.
>
>
>
> I believe I've made significant progress towards most of these goals,
> although there is still plenty of work left to do. I'd like to share a
> few examples both to illustrate where I am with this and to solicit
> feedback and/or help.
The goals are spot on, C API looks good at first glance, I'll try to
give it more careful scrutiny. The C handler functions need a void
*context so we can parameterize the callbacks. e.g.
void task_dispatch(pn_handler_t *handler, pn_event_t *event, void *context) {
...
pn_reactor_schedule(reactor, (int)context, handler);
}
...
pn_handler_t *handler = pn_handler(task_dispatch, (void*)1000);
I'm not sure if you intended a 1-1 relationship between handlers and
callback functions. If you do then you could put the context on the
handler rather than as a parameter to the callback - but that would
force the 1-1 relationship in the implementation from the API.
>
>
> Let me say up front that these examples aren't intended to be "hello
> world" type examples. The focus of this work has really been on the
> reactor/handler/event-dispatch infrastructure, and so the example I've
> chosen is really intended to illustrate key aspects of how this works.
> To do that I've built an example that sets up a recurring task, a
> server, and a client, all within the same process and then sends a
> number of messages to itself.
>
> I've included the same example twice, once written in C and once
> written in the python binding of the C API. Please have a look and let
> me know what you think.
>
>
> --Rafael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org