Re: remote service admin with libffi

[Bjoern Petri <bj...@sundevil.de>]

Hi Pepijn,

I actually have to admit that I've been a bit jealous when looking how 
easy the serialization can
be done with Java. For C there is always additional effort in adapting 
the proxy/endpoint
bundle. So, if this works as expected it would be from great benefit!

Looking at the code you've implemented I would assume that we can be 
even quite
flexible (e.g. variadic functions). The only constraint there is using 
the Java bytecode method
signature as schema. That's why it also might be an idea to also support 
an evolving schema?!

I am also wondering how to handle complex data types like hashmaps or 
arraylists. We usually
store void* pointers in there, so we cannot easily create a deep copy. 
Any ideas on that?

Last, this will also not work with the current amdatu remote service 
implementation,
so we should ensure that we can still do it the 'old-fashioned' way - 
but that's probably
what you had in mind anyway.


Regards,
   Bjorn



On 2015-05-31 21:20, Pepijn Noltes wrote:
> Hi All,
> 
> I pushed a new branch to Apache Celix (rsa-ffi) which contains a 
> initial
> setup for a remote service admin based on libffi [1].
> libffi (MIT License) is a c library for foreign function interface and 
> is
> used, among other, by python to invoke c functions. This is done 
> runtime by
> constructing calling conventions (ABI). libffi has been around for 
> quite
> some time and supports a broad range of configurations.
> 
> Although not design for this purpose libffi is very use-able within 
> Apache
> Celix to support dynamic import/export of services. In the branch I
> realized this by provided an additional property for remote services.
> 
> For example the calculator service is register with:
> ["add(DD)D", "sub(DD)D", "sqrt(DD)D"]
> as property values.
> This describes the services using a format inspired (and largly
> overlapping) with Java bytecode method signature. This description 
> (schema)
> is the only thing the ffi rsa needs to make endpoints or proxies. Note 
> that
> at the moment only double as argument type is supported.
> 
> For endpoints ffi_cal structs are created for invoking the actual 
> service
> methods and for proxies a ffi closure object are created to "catch" the
> methods invoked.
> 
> In the branch there is also a dyn_type & json_serliazer component which
> goes even furher and uses libffi types knowledge (alignment, size) to
> runtime construct and manipulate (complex) types based on a schema.
> 
> For example:
> The following schema "{DI} a b"
> describes the following struct:
> struct arg {
>   double a;
>   int32_t b;
> };
> 
> My goal is to combine the rsa and serializer part to support dynamic
> proxy/endpoints and serialization as long as there is a runtime schema 
> of
> the services. So for example a json based Apache Avro schema. The 
> dyn_type
> code works on a specific schema (again inspired by java bytecode method
> signature) to avoid support only a single serialization schema.
> As result only the types and services header have to be generated, 
> based on
> a schema (e.g. Apache Avro), and no additional bundles are needed to
> support remoting those services.
> 
> The branch contains a struct_test executable which runs a few examples 
> of
> dynamic serialization based on a schema describing the types. For info 
> if
> you install pahole (linux only?) you can verify if dyn_type (-> libffi)
> creates the correct struct layout.
> 
> Any thoughts/comments ?
> 
> [1] https://sourceware.org/libffi
> 
> Greetings,
> Pepijn

Re: remote service admin with libffi

[Pepijn Noltes <pe...@gmail.com>]

Hi,


On Wed, Jun 3, 2015 at 9:02 AM Bjoern Petri <bj...@sundevil.de>
wrote:

>
>
> Hi Pepijn,
>
> I actually have to admit that I've been a bit jealous when looking how
> easy the serialization can
> be done with Java. For C there is always additional effort in adapting
> the proxy/endpoint
> bundle. So, if this works as expected it would be from great benefit!
>
> Looking at the code you've implemented I would assume that we can be
> even quite
> flexible (e.g. variadic functions). The only constraint there is using
> the Java bytecode method
> signature as schema. That's why it also might be an idea to also support
> an evolving schema?!
>

I agree and evolving schema would be very welcome and in my opinion also
feasible.  I have explicitly chosen a descriptor/schema close to how the
data will look like in C with the idea "higher" level schema can be used to
generate the "close to C" schema.


>
> I am also wondering how to handle complex data types like hashmaps or
> arraylists. We usually
> store void* pointers in there, so we cannot easily create a deep copy.
> Any ideas on that?
>

That why i like Apache Avro concept that a schema is always present when
(de)serializing data. That way the serializer knows which data the pointers
should point to and thus can make deep copies.

In the current examples I known use a corba inspired sequence (struct with
capacitiy, length and buffer field).


>
> Last, this will also not work with the current amdatu remote service
> implementation,
> so we should ensure that we can still do it the 'old-fashioned' way -
> but that's probably
> what you had in mind anyway.
>

Yes, correct. Although at the moment for method with only simple types
(e.g. calculator) this should work (because the method signature with is
data is send when invoking a remote service), but I did not test this yet.

Note that the only change needed is that  a java remote service also
provides it's schema as a property. So this is a small change needed to C
Java interoperability.

If we combine the concept of evolving schemas with remote service on the
concept Apache Avro uses (schema runtime present for provider and consumer
and runtime compare for compatiblity) then making schema's runtime
available would benefit both implementations.


>
>
> Regards,
>    Bjorn
>
>
>
> On 2015-05-31 21:20, Pepijn Noltes wrote:
> > Hi All,
> >
> > I pushed a new branch to Apache Celix (rsa-ffi) which contains a
> > initial
> > setup for a remote service admin based on libffi [1].
> > libffi (MIT License) is a c library for foreign function interface and
> > is
> > used, among other, by python to invoke c functions. This is done
> > runtime by
> > constructing calling conventions (ABI). libffi has been around for
> > quite
> > some time and supports a broad range of configurations.
> >
> > Although not design for this purpose libffi is very use-able within
> > Apache
> > Celix to support dynamic import/export of services. In the branch I
> > realized this by provided an additional property for remote services.
> >
> > For example the calculator service is register with:
> > ["add(DD)D", "sub(DD)D", "sqrt(DD)D"]
> > as property values.
> > This describes the services using a format inspired (and largly
> > overlapping) with Java bytecode method signature. This description
> > (schema)
> > is the only thing the ffi rsa needs to make endpoints or proxies. Note
> > that
> > at the moment only double as argument type is supported.
> >
> > For endpoints ffi_cal structs are created for invoking the actual
> > service
> > methods and for proxies a ffi closure object are created to "catch" the
> > methods invoked.
> >
> > In the branch there is also a dyn_type & json_serliazer component which
> > goes even furher and uses libffi types knowledge (alignment, size) to
> > runtime construct and manipulate (complex) types based on a schema.
> >
> > For example:
> > The following schema "{DI} a b"
> > describes the following struct:
> > struct arg {
> >   double a;
> >   int32_t b;
> > };
> >
> > My goal is to combine the rsa and serializer part to support dynamic
> > proxy/endpoints and serialization as long as there is a runtime schema
> > of
> > the services. So for example a json based Apache Avro schema. The
> > dyn_type
> > code works on a specific schema (again inspired by java bytecode method
> > signature) to avoid support only a single serialization schema.
> > As result only the types and services header have to be generated,
> > based on
> > a schema (e.g. Apache Avro), and no additional bundles are needed to
> > support remoting those services.
> >
> > The branch contains a struct_test executable which runs a few examples
> > of
> > dynamic serialization based on a schema describing the types. For info
> > if
> > you install pahole (linux only?) you can verify if dyn_type (-> libffi)
> > creates the correct struct layout.
> >
> > Any thoughts/comments ?
> >
> > [1] https://sourceware.org/libffi
> >
> > Greetings,
> > Pepijn
>