Re: Embeddable Protocol Engine (was Re: Accepting large contributions (was Re: 10000 msgs limit per session))

[Rajith Attapattu <ra...@gmail.com>]

A huge +1.
I mooted this idea a looong time ago :)
As mentioned at that time Apache Axis2/C project used this strategy to
successfully support web services APIs for a number of languages.
They had the core protocol engine written in C and then swig bindings
for c++, python, ruby, javascript, php and perl.

Rajith

On Fri, Dec 4, 2009 at 11:29 AM, Rafael Schloming <ra...@redhat.com> wrote:
> Aidan Skinner wrote:
>>
>> On Thu, Dec 3, 2009 at 10:11 PM, James Mansion
>> <ja...@mansionfamily.plus.com> wrote:
>>
>>> Aidan Skinner wrote:
>>>>
>>>> It's particularly important where we're importing something which
>>>> duplicates (fully or partially) existing functionality, if only so
>>>> that the situation is sufficiently clear to people trying to make an
>>>> informed decision about what best suits their needs.
>>>>
>>> Perhaps the QPID project itself could focus on just C++ for all the
>>> protocol
>>> handling and use swig (or similar) to create wrappers, so the code volume
>>> to
>>> support multiple client languages will be much smaller.  Then native
>>> clients
>>> can be completely independant projects.
>>
>> Eh, I don't really want to get rid of the actively maintained clients
>> we already have. In particular, Java and C# derive enormous benefits
>> from purely managed code in terms of portability, JITability etc.
>>
>> Having said that, I can see definite value in a C wire library and
>> autogenerated bindings using something like swig or
>> gobject-introspection. C++ is a *total* pain to bind.
>> Gouge-your-eyes-out bad.
>
> Not to hijack another hijacked thread, but...
>
> One of my goals with the API work we've been doing has actually been to lay
> the groundwork for this exact strategy, i.e. creating an embeddable C
> library as a means of sustainably maintaining clients in all the languages
> we currently strive to support, as well as opening the door for many more.
>
> Beyond the general C++ binding issues that Aidan alludes to (as well as the
> general C++ portability issues that Andrew frequently complains about given
> the slightest opportunity ;), it wouldn't be as useful as you might at first
> think to autogenerate bindings to the low level C++ client since you'd still
> actually have a fair amount to build out in the target language in order to
> make a usable messaging API for that language, not to mention the fact that
> the binding would be protocol version specific, leaving even more to build
> out in order to deal with different AMQP versions and extensions.
>
> Also the C++ client as it currently stands is understandably designed more
> as a client than as an embeddable component. It does it's own threading and
> I/O behind the scenes, and that does not always interact in an ideal way
> with a constrained host environment.
>
> What we've tried to do with the API design is to provide a much smaller
> version independent interface to equivalent functionality. The relevance in
> this case being that the autogenerated bindings for this API would be much
> more directly usable in the target language.
>
> I've also been experimenting with moving towards the "protocol engine"
> concept in the python implementation of the API. The idea being to create a
> strong separation between the protocol machinery (aka the protocol engine)
> and the I/O and threading used around it to construct a client.
>
> This basically involves defining an API (or SPI really) to the bottom end of
> the client in addition to the usual API at the top end. This SPI can be
> defined based on the observation that whenever the application interacts
> with the top end of the API, e.g. sending a message, acknowledging a
> message, increasing the capacity of a receiver, etc, these actions translate
> into a very small set of possible actions at the bottom end of the client.
> Any API action results in either more bytes to be written onto the wire,
> more space for bytes to be read from the wire,  or both.
>
> This is sort of like having a pluggable network driver except turned inside
> out, the protocol engine is actually what you're embedding inside the
> network driver, rather than the network driver being embedded inside a
> client.
>
> This allows for a very simple SPI to be defined, the benefit being that
> everything that sits between the API and SPI is pure protocol machinery and
> doesn't need to have threading, I/O, or really almost any external
> dependencies, yet it can still encapsulate all the complexity of the
> different framing, encoding, and low level protocol interactions across all
> the various protocol versions.
>
> This "protocol engine" chunk is really what would be ideal to implement in
> very portable C code and then embed as the basis for clients in pretty much
> any swiggable language. It would be relatively trivial to write an I/O
> driver in the target environment, and the result would avoid all the usual
> pitfalls of just embedding another client wholesale, e.g. hidden threads,
> lack of portability, nonstandard SSL configuration, weird exception
> hierarchies, no integration with standard language I/O libraries, etc.
>
> --Rafael
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>



-- 
Regards,

Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org