You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Ted Ross <tr...@redhat.com> on 2012/11/14 16:54:45 UTC
Using script-native drivers for Proton messenger bindings
One of the problems we've had for a long time with wrapped bindings is
the mismatch of threading models. For example, a Ruby program using an
extension module written in C/C++ will hang if a call into the extension
blocks. This is because the extension uses pthreads and the host
program uses "greenthreads" (i.e. some layered threading mechanism in
the scripting language). This is also an issue for Python (and probably
every other scripting language).
The proper solution for proton/messenger (which necessarily has a few
blocking calls) is to provide (or allow the developer to provide) a
driver written in the scripting language using the same threading
library that their program uses.
The proton code structure allows for such a strategy, but the packaging
does not. I think that the proton project should make the necessary
adjustments to encourage developers to contribute "native" drivers for
Ruby, Python-eventlet, etc.
I'm interested in others' thoughts on this and what would be necessary
to make this happen.
-Ted
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Using script-native drivers for Proton messenger bindings
Posted by Ted Ross <tr...@redhat.com>.
On 11/14/2012 12:24 PM, Andrew Stitcher wrote:
> On Wed, 2012-11-14 at 17:49 +0100, Rafael Schloming wrote:
>> ...
>> How does the packaging prevent this?
> Not to put words in Ted's mouth, but I think what he is talking about
> here is that the messenger API code itself depends on the driver code.
>
> Andrew
>
>
Yes, that is correct. If you are using just the engine API, you can
ignore the built-in driver and provide your own. But for messenger,
this is not possible.
-Ted
Re: Using script-native drivers for Proton messenger bindings
Posted by Andrew Stitcher <as...@redhat.com>.
On Wed, 2012-11-14 at 17:49 +0100, Rafael Schloming wrote:
> ...
> How does the packaging prevent this?
Not to put words in Ted's mouth, but I think what he is talking about
here is that the messenger API code itself depends on the driver code.
Andrew
Re: Using script-native drivers for Proton messenger bindings
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Wed, Nov 14, 2012 at 10:36 PM, Ted Ross <tr...@redhat.com> wrote:
>
> On 11/14/2012 12:56 PM, Rafael Schloming wrote:
>
>> On Wed, Nov 14, 2012 at 6:37 PM, Ted Ross <tr...@redhat.com> wrote:
>>
>> On 11/14/2012 11:49 AM, Rafael Schloming wrote:
>>>
>>> How does the packaging prevent this?
>>>>
>>>>
>>>> 1) Develop the non-blocking aspects of the API. This might just
>>>> involve a
>>>> few additions to the current API or possibly adding a distinct
>>>> non-blocking
>>>> API which would be a layer underneath the current API. Either way, this
>>>> non-blocking API then becomes the top half of the messenger engine.
>>>>
>>>> This is good, but I claim that the blocking/non-blocking of the
>>> messenger
>>> API is an orthogonal concern. The style of API used is related to the
>>> programming model used in the application. I think that users will want
>>> to
>>> have the "native" driver and use the blocking messenger API.
>>>
>>
>> Yes, but unless I'm missing something, the blocking calls will need to be
>> "native" as well since they will need to call into the native driver to
>> block, and so they will need to be built on top of a non blocking API.
>>
>
> I'll use a concrete example. Assume we have a python program using the
> current messaging API (which has three blocking calls). Assume further
> that the driver is written in Python and uses a blocking mechanism
> appropriate for the in-use threading library. Since the only place proton
> *ever* blocks is in pn_driver_wait, and pn_driver_wait is written in
> python, everything works correctly even though the program uses the
> "blocking" messenger api.
Yes, but those 3 blocking messenger calls are implemented in C. They do
call pn_driver_wait, as you say, but the call the C pn_driver_wait. For
what you're describing to work you'd need to somehow get them to call the
python pn_driver_wait. That's why I think you'd need to re-implement the
blocking messenger calls in the python part of the binding for this to work
(so you're not calling python -> C -> python, or ruby -> C -> ruby). I
think re-implementing the blocking messenger calls can be made quite
trivial if the non blocking API is constructed appropriately, and I'm
guessing easier than constructing the above call chain in each binding.
--Rafael
Re: Using script-native drivers for Proton messenger bindings
Posted by Ted Ross <tr...@redhat.com>.
On 11/14/2012 12:56 PM, Rafael Schloming wrote:
> On Wed, Nov 14, 2012 at 6:37 PM, Ted Ross <tr...@redhat.com> wrote:
>
>> On 11/14/2012 11:49 AM, Rafael Schloming wrote:
>>
>>> How does the packaging prevent this?
>>>
>>>
>>> 1) Develop the non-blocking aspects of the API. This might just
>>> involve a
>>> few additions to the current API or possibly adding a distinct
>>> non-blocking
>>> API which would be a layer underneath the current API. Either way, this
>>> non-blocking API then becomes the top half of the messenger engine.
>>>
>> This is good, but I claim that the blocking/non-blocking of the messenger
>> API is an orthogonal concern. The style of API used is related to the
>> programming model used in the application. I think that users will want to
>> have the "native" driver and use the blocking messenger API.
>
> Yes, but unless I'm missing something, the blocking calls will need to be
> "native" as well since they will need to call into the native driver to
> block, and so they will need to be built on top of a non blocking API.
I'll use a concrete example. Assume we have a python program using the
current messaging API (which has three blocking calls). Assume further
that the driver is written in Python and uses a blocking mechanism
appropriate for the in-use threading library. Since the only place
proton *ever* blocks is in pn_driver_wait, and pn_driver_wait is written
in python, everything works correctly even though the program uses the
"blocking" messenger api.
-Ted
Re: Using script-native drivers for Proton messenger bindings
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Wed, Nov 14, 2012 at 6:37 PM, Ted Ross <tr...@redhat.com> wrote:
>
> On 11/14/2012 11:49 AM, Rafael Schloming wrote:
>
>>
>> How does the packaging prevent this?
>>
>>
>> 1) Develop the non-blocking aspects of the API. This might just
>> involve a
>> few additions to the current API or possibly adding a distinct
>> non-blocking
>> API which would be a layer underneath the current API. Either way, this
>> non-blocking API then becomes the top half of the messenger engine.
>>
>
> This is good, but I claim that the blocking/non-blocking of the messenger
> API is an orthogonal concern. The style of API used is related to the
> programming model used in the application. I think that users will want to
> have the "native" driver and use the blocking messenger API.
Yes, but unless I'm missing something, the blocking calls will need to be
"native" as well since they will need to call into the native driver to
block, and so they will need to be built on top of a non blocking API.
--Rafael
Re: Using script-native drivers for Proton messenger bindings
Posted by Ted Ross <tr...@redhat.com>.
On 11/14/2012 11:49 AM, Rafael Schloming wrote:
>
> How does the packaging prevent this?
>
>
> 1) Develop the non-blocking aspects of the API. This might just involve a
> few additions to the current API or possibly adding a distinct non-blocking
> API which would be a layer underneath the current API. Either way, this
> non-blocking API then becomes the top half of the messenger engine.
This is good, but I claim that the blocking/non-blocking of the
messenger API is an orthogonal concern. The style of API used is
related to the programming model used in the application. I think that
users will want to have the "native" driver and use the blocking
messenger API.
>
> 2) Develop the bottom half API for the messenger engine. This would in
> part simply be an API that would provide access to a set of transport
> objects along with connection details. This would presumably also need to
> expose a way for the messenger to listen (or ask the driver to listen) on a
> given host/port also in order to support the full range of behaviour that
> messenger currently does. This latter part might be as simple as providing
> access to the list of subscriptions.
>
> --Rafael
>
Re: Using script-native drivers for Proton messenger bindings
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Wed, Nov 14, 2012 at 4:54 PM, Ted Ross <tr...@redhat.com> wrote:
> One of the problems we've had for a long time with wrapped bindings is the
> mismatch of threading models. For example, a Ruby program using an
> extension module written in C/C++ will hang if a call into the extension
> blocks. This is because the extension uses pthreads and the host program
> uses "greenthreads" (i.e. some layered threading mechanism in the scripting
> language). This is also an issue for Python (and probably every other
> scripting language).
>
> The proper solution for proton/messenger (which necessarily has a few
> blocking calls) is to provide (or allow the developer to provide) a driver
> written in the scripting language using the same threading library that
> their program uses.
>
> The proton code structure allows for such a strategy, but the packaging
> does not. I think that the proton project should make the necessary
> adjustments to encourage developers to contribute "native" drivers for
> Ruby, Python-eventlet, etc.
>
How does the packaging prevent this?
I'm interested in others' thoughts on this and what would be necessary to
> make this happen.
I think what's needed is to "engine-ify" Messenger, which involves two
steps:
1) Develop the non-blocking aspects of the API. This might just involve a
few additions to the current API or possibly adding a distinct non-blocking
API which would be a layer underneath the current API. Either way, this
non-blocking API then becomes the top half of the messenger engine.
2) Develop the bottom half API for the messenger engine. This would in
part simply be an API that would provide access to a set of transport
objects along with connection details. This would presumably also need to
expose a way for the messenger to listen (or ask the driver to listen) on a
given host/port also in order to support the full range of behaviour that
messenger currently does. This latter part might be as simple as providing
access to the list of subscriptions.
--Rafael
Re: Using script-native drivers for Proton messenger bindings
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Nov 14, 2012 at 10:54:45AM -0500, Ted Ross wrote:
> One of the problems we've had for a long time with wrapped bindings
> is the mismatch of threading models. For example, a Ruby program
> using an extension module written in C/C++ will hang if a call into
> the extension blocks. This is because the extension uses pthreads
> and the host program uses "greenthreads" (i.e. some layered
> threading mechanism in the scripting language). This is also an
> issue for Python (and probably every other scripting language).
>
> The proper solution for proton/messenger (which necessarily has a
> few blocking calls) is to provide (or allow the developer to
> provide) a driver written in the scripting language using the same
> threading library that their program uses.
>
> The proton code structure allows for such a strategy, but the
> packaging does not. I think that the proton project should make the
> necessary adjustments to encourage developers to contribute "native"
> drivers for Ruby, Python-eventlet, etc.
>
> I'm interested in others' thoughts on this and what would be
> necessary to make this happen.
I'm not sure I follow what is meant by "the packaging does not". Can you
be more specific there?
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Using script-native drivers for Proton messenger bindings
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Nov 14, 2012 at 10:54:45AM -0500, Ted Ross wrote:
> One of the problems we've had for a long time with wrapped bindings
> is the mismatch of threading models. For example, a Ruby program
> using an extension module written in C/C++ will hang if a call into
> the extension blocks. This is because the extension uses pthreads
> and the host program uses "greenthreads" (i.e. some layered
> threading mechanism in the scripting language). This is also an
> issue for Python (and probably every other scripting language).
>
> The proper solution for proton/messenger (which necessarily has a
> few blocking calls) is to provide (or allow the developer to
> provide) a driver written in the scripting language using the same
> threading library that their program uses.
>
> The proton code structure allows for such a strategy, but the
> packaging does not. I think that the proton project should make the
> necessary adjustments to encourage developers to contribute "native"
> drivers for Ruby, Python-eventlet, etc.
>
> I'm interested in others' thoughts on this and what would be
> necessary to make this happen.
I'm not sure I follow what is meant by "the packaging does not". Can you
be more specific there?
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/