Embeddable Protocol Engine (was Re: Accepting large contributions (was Re: 10000 msgs limit per session))

[Rafael Schloming <ra...@redhat.com>]

Aidan Skinner wrote:
> On Thu, Dec 3, 2009 at 10:11 PM, James Mansion
> <ja...@mansionfamily.plus.com> wrote:
> 
>> Aidan Skinner wrote:
>>> It's particularly important where we're importing something which
>>> duplicates (fully or partially) existing functionality, if only so
>>> that the situation is sufficiently clear to people trying to make an
>>> informed decision about what best suits their needs.
>>>
>> Perhaps the QPID project itself could focus on just C++ for all the protocol
>> handling and use swig (or similar) to create wrappers, so the code volume to
>> support multiple client languages will be much smaller.  Then native clients
>> can be completely independant projects.
> 
> Eh, I don't really want to get rid of the actively maintained clients
> we already have. In particular, Java and C# derive enormous benefits
> from purely managed code in terms of portability, JITability etc.
> 
> Having said that, I can see definite value in a C wire library and
> autogenerated bindings using something like swig or
> gobject-introspection. C++ is a *total* pain to bind.
> Gouge-your-eyes-out bad.

Not to hijack another hijacked thread, but...

One of my goals with the API work we've been doing has actually been to 
lay the groundwork for this exact strategy, i.e. creating an embeddable 
C library as a means of sustainably maintaining clients in all the 
languages we currently strive to support, as well as opening the door 
for many more.

Beyond the general C++ binding issues that Aidan alludes to (as well as 
the general C++ portability issues that Andrew frequently complains 
about given the slightest opportunity ;), it wouldn't be as useful as 
you might at first think to autogenerate bindings to the low level C++ 
client since you'd still actually have a fair amount to build out in the 
target language in order to make a usable messaging API for that 
language, not to mention the fact that the binding would be protocol 
version specific, leaving even more to build out in order to deal with 
different AMQP versions and extensions.

Also the C++ client as it currently stands is understandably designed 
more as a client than as an embeddable component. It does it's own 
threading and I/O behind the scenes, and that does not always interact 
in an ideal way with a constrained host environment.

What we've tried to do with the API design is to provide a much smaller 
version independent interface to equivalent functionality. The relevance 
in this case being that the autogenerated bindings for this API would be 
much more directly usable in the target language.

I've also been experimenting with moving towards the "protocol engine" 
concept in the python implementation of the API. The idea being to 
create a strong separation between the protocol machinery (aka the 
protocol engine) and the I/O and threading used around it to construct a 
client.

This basically involves defining an API (or SPI really) to the bottom 
end of the client in addition to the usual API at the top end. This SPI 
can be defined based on the observation that whenever the application 
interacts with the top end of the API, e.g. sending a message, 
acknowledging a message, increasing the capacity of a receiver, etc, 
these actions translate into a very small set of possible actions at the 
bottom end of the client. Any API action results in either more bytes to 
be written onto the wire, more space for bytes to be read from the wire, 
  or both.

This is sort of like having a pluggable network driver except turned 
inside out, the protocol engine is actually what you're embedding inside 
the network driver, rather than the network driver being embedded inside 
a client.

This allows for a very simple SPI to be defined, the benefit being that 
everything that sits between the API and SPI is pure protocol machinery 
and doesn't need to have threading, I/O, or really almost any external 
dependencies, yet it can still encapsulate all the complexity of the 
different framing, encoding, and low level protocol interactions across 
all the various protocol versions.

This "protocol engine" chunk is really what would be ideal to implement 
in very portable C code and then embed as the basis for clients in 
pretty much any swiggable language. It would be relatively trivial to 
write an I/O driver in the target environment, and the result would 
avoid all the usual pitfalls of just embedding another client wholesale, 
e.g. hidden threads, lack of portability, nonstandard SSL configuration, 
weird exception hierarchies, no integration with standard language I/O 
libraries, etc.

--Rafael


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Embeddable Protocol Engine (was Re: Accepting large contributions (was Re: 10000 msgs limit per session))

[Rajith Attapattu <ra...@gmail.com>]

A huge +1.
I mooted this idea a looong time ago :)
As mentioned at that time Apache Axis2/C project used this strategy to
successfully support web services APIs for a number of languages.
They had the core protocol engine written in C and then swig bindings
for c++, python, ruby, javascript, php and perl.

Rajith

On Fri, Dec 4, 2009 at 11:29 AM, Rafael Schloming <ra...@redhat.com> wrote:
> Aidan Skinner wrote:
>>
>> On Thu, Dec 3, 2009 at 10:11 PM, James Mansion
>> <ja...@mansionfamily.plus.com> wrote:
>>
>>> Aidan Skinner wrote:
>>>>
>>>> It's particularly important where we're importing something which
>>>> duplicates (fully or partially) existing functionality, if only so
>>>> that the situation is sufficiently clear to people trying to make an
>>>> informed decision about what best suits their needs.
>>>>
>>> Perhaps the QPID project itself could focus on just C++ for all the
>>> protocol
>>> handling and use swig (or similar) to create wrappers, so the code volume
>>> to
>>> support multiple client languages will be much smaller.  Then native
>>> clients
>>> can be completely independant projects.
>>
>> Eh, I don't really want to get rid of the actively maintained clients
>> we already have. In particular, Java and C# derive enormous benefits
>> from purely managed code in terms of portability, JITability etc.
>>
>> Having said that, I can see definite value in a C wire library and
>> autogenerated bindings using something like swig or
>> gobject-introspection. C++ is a *total* pain to bind.
>> Gouge-your-eyes-out bad.
>
> Not to hijack another hijacked thread, but...
>
> One of my goals with the API work we've been doing has actually been to lay
> the groundwork for this exact strategy, i.e. creating an embeddable C
> library as a means of sustainably maintaining clients in all the languages
> we currently strive to support, as well as opening the door for many more.
>
> Beyond the general C++ binding issues that Aidan alludes to (as well as the
> general C++ portability issues that Andrew frequently complains about given
> the slightest opportunity ;), it wouldn't be as useful as you might at first
> think to autogenerate bindings to the low level C++ client since you'd still
> actually have a fair amount to build out in the target language in order to
> make a usable messaging API for that language, not to mention the fact that
> the binding would be protocol version specific, leaving even more to build
> out in order to deal with different AMQP versions and extensions.
>
> Also the C++ client as it currently stands is understandably designed more
> as a client than as an embeddable component. It does it's own threading and
> I/O behind the scenes, and that does not always interact in an ideal way
> with a constrained host environment.
>
> What we've tried to do with the API design is to provide a much smaller
> version independent interface to equivalent functionality. The relevance in
> this case being that the autogenerated bindings for this API would be much
> more directly usable in the target language.
>
> I've also been experimenting with moving towards the "protocol engine"
> concept in the python implementation of the API. The idea being to create a
> strong separation between the protocol machinery (aka the protocol engine)
> and the I/O and threading used around it to construct a client.
>
> This basically involves defining an API (or SPI really) to the bottom end of
> the client in addition to the usual API at the top end. This SPI can be
> defined based on the observation that whenever the application interacts
> with the top end of the API, e.g. sending a message, acknowledging a
> message, increasing the capacity of a receiver, etc, these actions translate
> into a very small set of possible actions at the bottom end of the client.
> Any API action results in either more bytes to be written onto the wire,
> more space for bytes to be read from the wire,  or both.
>
> This is sort of like having a pluggable network driver except turned inside
> out, the protocol engine is actually what you're embedding inside the
> network driver, rather than the network driver being embedded inside a
> client.
>
> This allows for a very simple SPI to be defined, the benefit being that
> everything that sits between the API and SPI is pure protocol machinery and
> doesn't need to have threading, I/O, or really almost any external
> dependencies, yet it can still encapsulate all the complexity of the
> different framing, encoding, and low level protocol interactions across all
> the various protocol versions.
>
> This "protocol engine" chunk is really what would be ideal to implement in
> very portable C code and then embed as the basis for clients in pretty much
> any swiggable language. It would be relatively trivial to write an I/O
> driver in the target environment, and the result would avoid all the usual
> pitfalls of just embedding another client wholesale, e.g. hidden threads,
> lack of portability, nonstandard SSL configuration, weird exception
> hierarchies, no integration with standard language I/O libraries, etc.
>
> --Rafael
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>



-- 
Regards,

Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org