[lucy-dev] Interface-based callbacks for Perl bindings

[Nick Wellnhofer <we...@aevum.de>]

Lucifers,

This is just a braindump on how the callback mechanism could be changed to be 
interface-based without the need to fully support interfaces in Clownfish. 
With regard to the Perl bindings, this would also allow to use arbitrary Perl 
classes that don't inherit from Clownfish::Obj. So there's no need for 
inside-out instance variables which would be a big plus.

I take the Analyzer class as an example. Currently, custom analyzers are 
implemented by subclassing and overriding the `Transform` method. With an 
interface-based approach, we'd introduce a purely abstract class `Transformer`

     public abstract class Transformer {
         public abstract incremented Inversion*
         Transform(Transformer *self, Inversion *inv);
     }

and add a transformer variable to Analyzer:

     public class Analyzer {
         Transformer *transformer;

         public inert Analyzer*
         init(Analyzer *self, Transformer *transformer);
     }

So my approach would be to add a modifier to the Transformer class

     public abstract *create_host_interface* class Transformer

which would autogenerate a class equivalent to

     public class HostTransformer extends Transformer {
         void *host_interface_obj; // SV* for Perl
     }

and set a new property `host_interface` in the `TRANSFORMER` class singleton 
pointing to `HOSTRANSFORMER`.

When Analyzer_init is called from Perl, the following piece of code in 
XSBind_maybe_sv_to_cfish_obj would handle the conversion to a HostTransformer:

     Obj*
     XSBind_maybe_sv_to_cfish_obj(SV *sv, Class *klass) {
         Obj *retval;

         // ...

         if (klass->host_interface
             && sv_isobject(sv)
             && !sv_derived_from(...)
         ) {
             HV *stash = SvSTASH(SvRV(sv));
             StackString *class_name
                 = SSTR_WRAP_UTF8(HvNAME(stash), NvNAMELEN(stash));
             Class *singleton
                 = Class_singleton((String*)class_name,
                                   klass->host_interface);
             retval = Class_Make_Obj(singleton);
             retval->host_interface_obj = sv;
             SvREFCNT_inc(sv);
             return retval;
         }

         // ...
     }

Additionally, the following functions would be autogenerated:

     Inversion*
     HostTransformer_Transform_OVERRIDE(HostTransformer *self,
                                        Inversion *inv) {
         // Autogenerated callback using self->host_interface_obj
     }

     void
     HostTransformer_Destroy_IMP(HostTransformer *self) {
         SvREFCNT_dec((SV*)self->host_interface_obj);
         SUPER_DESTROY(...);
     }

This means that every time a callback object is passed from Perl to Clownfish, 
a new HostTransformer object is created, i.e. the HostTransformer is not 
cached. But I don't think this would cause a performance problem.

Another benefit is that the number of generated OVERRIDE functions would be 
reduced drastically. Also, the Perl constructor bindings could be simplified 
because they wouldn't have to check for Perl subclasses anymore.

I'm probably missing some things but I wanted share my initial thoughts with 
the list.

Nick

Re: [lucy-dev] Interface-based callbacks for Perl bindings

[Nick Wellnhofer <we...@aevum.de>]

On 05/12/2014 00:08, Marvin Humphrey wrote:
> On Thu, Dec 4, 2014 at 2:04 PM, Nick Wellnhofer <we...@aevum.de> wrote:
>
>>> I guess you need need a separate Transformer class because you want to
>>> exclude concrete methods on Analyzer like Transform_Text()?
>>
>> Yes.
>
> Why is this a requirement?
>
> Why can't the Clownfish version of "interface" allow both abstract and
> concrete methods?

The idea I posted isn't meant to implement interfaces that can be used by 
Clownfish internally. It's only an approach how to implement an interface-like 
callback mechanism that doesn't rely on subclassing and makes only a few 
changes to the current architecture. At least in the Perl case, it doesn't 
make much sense to allow default implementations for this kind of interface 
because then the Perl class would have to subclass a Clownfish class. This is 
what I was trying to avoid.

If we implement interfaces to be used by Clownfish internally, I agree that 
they should support default implementations.

Nick

Re: [lucy-dev] Interface-based callbacks for Perl bindings

[Marvin Humphrey <ma...@rectangular.com>]

On Thu, Dec 4, 2014 at 2:04 PM, Nick Wellnhofer <we...@aevum.de> wrote:

>> I guess you need need a separate Transformer class because you want to
>> exclude concrete methods on Analyzer like Transform_Text()?
>
> Yes.

Why is this a requirement?

Why can't the Clownfish version of "interface" allow both abstract and
concrete methods?

Java 8 allows such methods, called "virtual extension methods", "default
methods" or "defender methods".

http://viralpatel.net/blogs/java-8-default-methods-tutorial/

Scala also allows such methods in its traits.  (Scala traits may also have
member variables but that would be much harder to implement and I don't
want to go there with Clownfish.)

Marvin Humphrey

Re: [lucy-dev] Interface-based callbacks for Perl bindings

[Nick Wellnhofer <we...@aevum.de>]

On 04/12/2014 22:27, Marvin Humphrey wrote:
> From C-space, we only need something we can invoke Clownfish-style methods on.
> It's easy to generate a Clownfish wrapper around any kind of Perl object to
> achieve that.

That's what `HostTransformer` in my example is for.

> I have to admit that I don't understand everything about the proposal.  I
> guess you need need a separate Transformer class because you want to exclude
> concrete methods on Analyzer like Transform_Text()?

Yes.

> In any case, I hope that the eventual goal is to support stuff like this:
>
>      package DummyAnalyzer;
>
>      sub new { return bless {}, __PACKAGE__ }
>
>      sub transform {
>          my ($self, $batch) = @_;
>          return $batch;
>      }
>
>      sub dump { return { _class => __PACKAGE} }
>      sub load { return __PACKAGE__->new() }
>
>      package main;
>
>      my $type = Lucy::Plan::FullTextType->new(
>          analyzer => DummyAnalyzer->new,
>      );
>
> Note that DummyAnalyzer...
>
> *   is implemented as a blessed hash
> *   does not have "Lucy::Analysis::Analyzer" in @ISA
> *   does not call SUPER::new
> *   does not implement DESTROY

Yes, that's the goal.

> Right now, if we supply DummyAnalyzer object as an argument to FullTextType's
> constructor, we'll get an exception because DummyAnalyzer does not inherit
> from Analyzer.  But what we could do instead is wrap the blessed hash in a
> Clownfish object which knows how to call back into Perl for each method.

Yes, the wrapper is `HostTransformer`.

Nick

Re: [lucy-dev] Interface-based callbacks for Perl bindings

[Marvin Humphrey <ma...@rectangular.com>]

On Wed, Dec 3, 2014 at 9:26 AM, Nick Wellnhofer <we...@aevum.de> wrote:

> With regard to the Perl bindings, this would also allow to use
> arbitrary Perl classes that don't inherit from Clownfish::Obj. So there's no
> need for inside-out instance variables which would be a big plus.

Awesome -- making it possible to use the full range of host OO options would be
a great improvement, and we should do everything we can to make that happen!

> I take the Analyzer class as an example.

You know, Analyzer doesn't even have any member variables.  It's crying out to
be an interface.

>From C-space, we only need something we can invoke Clownfish-style methods on.
It's easy to generate a Clownfish wrapper around any kind of Perl object to
achieve that.

(Once you remove the requirement to duplicate the layout of inherited member
variables in memory, life gets easier.)

>     public class HostTransformer extends Transformer {
>         void *host_interface_obj; // SV* for Perl
>     }

I think we need to tweak this idea for Go and other languages that may use
compacting garbage collection.

It is not valid to pass any Go data structure into C-space, so keeping a
reference in host_interface_obj is not possible.  However, we could keep an
integer ID of some sort and use that to look up a host object in a Go-space
data structure.

> I'm probably missing some things but I wanted share my initial thoughts with
> the list.

I have to admit that I don't understand everything about the proposal.  I
guess you need need a separate Transformer class because you want to exclude
concrete methods on Analyzer like Transform_Text()?

In any case, I hope that the eventual goal is to support stuff like this:

    package DummyAnalyzer;

    sub new { return bless {}, __PACKAGE__ }

    sub transform {
        my ($self, $batch) = @_;
        return $batch;
    }

    sub dump { return { _class => __PACKAGE} }
    sub load { return __PACKAGE__->new() }

    package main;

    my $type = Lucy::Plan::FullTextType->new(
        analyzer => DummyAnalyzer->new,
    );

Note that DummyAnalyzer...

*   is implemented as a blessed hash
*   does not have "Lucy::Analysis::Analyzer" in @ISA
*   does not call SUPER::new
*   does not implement DESTROY

Right now, if we supply DummyAnalyzer object as an argument to FullTextType's
constructor, we'll get an exception because DummyAnalyzer does not inherit
from Analyzer.  But what we could do instead is wrap the blessed hash in a
Clownfish object which knows how to call back into Perl for each method.

Marvin Humphrey