[lucy-dev] Go bindings -- composition over inheritance

[Marvin Humphrey <ma...@rectangular.com>]

On Mon, Nov 17, 2014 at 1:12 PM, Marvin Humphrey <ma...@rectangular.com> wrote:

> *   Clownfish abstract classes are mapped to Go interfaces.
> *   Clownfish concrete classes are mapped to Go struct pointers.

> *   Adapting Clownfish's subtyping for Go is hard, because Go does not
>     support inheritance.  This is especially a problem with functionality
>     which can only be unlocked by overriding methods on concrete classes
>     like QueryParser, IndexManager and Schema.

To illustrate the challenge posed by Go's lack of support for inheritance...
If we wrap Query in a Go "struct", then it would not be valid to use a
*TermQuery Go struct pointer in a context where a *Query Go struct pointer is
called for -- e.g.  passing a *TermQuery to a method which expects a *Query.
However, if we make Query a Go "interface", then it *would* be valid to use a
*TermQuery in a Query context.

For abstract classes like Query or Analyzer, making them Go interfaces works
fine.  However, there are unresolved difficulties for concrete classes which
we expect people to subclass.

One such class is IndexManager.  In order to control how segments get recycled
in Lucy, you need to subclass IndexManager and override Recycle() (as
described in Lucy::Docs::Cookbook::FastUpdates).  However, as things are now,
that won't be possible from Lucy's Go bindings:

    http://play.golang.org/p/4GLVD1Gg6N

    prog.go:31: cannot use manager (type *FastAddManager) as type
    *IndexManager in argument to NewIndexer [process exited with non-zero
    status]

Change IndexManager to an interface and the code at least compiles:

    http://play.golang.org/p/ues1hRQp8y
    https://paste.apache.org/iwCi     (diff)

A more elegant solution, though, is to refactor IndexManager to avoid
inheritance and rely only on interface-based polymorphism and composition.
For example, IndexManager could call Recycle() on an object which
implements an "IndexDeletionPolicy" interface.  Once IndexManager has-a
IndexDeletionPolicy, you change IndexManager's behavior by supplying an
IndexDeletionPolicy object which does what you want -- eliminating the need to
subclass IndexManager.

The compositional approach would also be perfectly natural and idiomatic for
other languages.

    # Python:
    manager = lucy.IndexManager.new()
    manager.set_deletion_policy(MyDeletionPolicy.new())
    indexer = lucy.Indexer.new(index="path/to/index", manager=manager)

    # Java
    IndexManager manager = new IndexManager();
    manager.setDeletionPolicy(new MyDeletionPolicy());
    Indexer indexer = new Indexer("path/to/index", manager);

    # Perl
    my $manager = Lucy::Index::IndexManager->new;
    $manager->set_deletion_policy(MyDeletionPolicy->new);
    my $indexer = Lucy::Index::Indexer->new(
        index   => 'path/to/index',
        manager => $manager,
    );

One thing I've come to appreciate while working on the Go bindings is that the
composition model is more universal -- besides its popularity among elite
hackers, it's more compatible with more object models than inheritance.  It's
made me contemplate whether composition might be a more suitable model on
which to base a project like Clownfish.

Marvin Humphrey

Re: [lucy-dev] Go bindings -- composition over inheritance

[Marvin Humphrey <ma...@rectangular.com>]

On Wed, Dec 3, 2014 at 7:03 AM, Nick Wellnhofer <we...@aevum.de> wrote:

> Don't we run into exactly the same problem as with IndexManager when writing
> a custom Query or Analyzer in Go?

If we implement IndexManager as a Go struct, we will have the problem
of not being able to use the subtype *MyIndexManager in the context of
*IndexManager.

In contrast, if we implement Query as a Go _interface_, we definitely
will _not_ have the analogous problem of not being able to use a
*MyQuery Go struct pointer in a Query context.  We may want to monkey
with Query's design a bit to make implementing its interface easier,
though.

There may be other challenges; let me get a little further so I can
provide a more informed reply.

Marvin Humphrey

Re: [lucy-dev] Go bindings -- composition over inheritance

[Nick Wellnhofer <we...@aevum.de>]

On 03/12/2014 03:47, Marvin Humphrey wrote:
> For abstract classes like Query or Analyzer, making them Go interfaces works
> fine.  However, there are unresolved difficulties for concrete classes which
> we expect people to subclass.

Don't we run into exactly the same problem as with IndexManager when writing a 
custom Query or Analyzer in Go? Query isn't even marked as abstract [1] and 
both classes define some non-abstract methods. From my understanding, a 
Clownfish class could only be "subclassed" from Go if it contains only 
abstract methods and no ivars which would make it an interface.

> One thing I've come to appreciate while working on the Go bindings is that the
> composition model is more universal -- besides its popularity among elite
> hackers, it's more compatible with more object models than inheritance.  It's
> made me contemplate whether composition might be a more suitable model on
> which to base a project like Clownfish.

I wouldn't question the core architecture of Clownfish. Composition works so 
well in Go because anonymous struct fields are promoted and their methods are 
added to the structs method set when determining the interfaces it implements. 
This makes composition feel like inheritance in many cases. But all of this 
would be hard replicate in C.

The actual problem is the way we handle callbacks from Clownfish back to the 
host language. The current implementation is based on subclassing and makes a 
lot of assumptions:

* The host language has some concept of classes and inheritance.
* The host language can somehow subclass Clownfish classes.
* A constructor can call the constructor of the parent class.
* The host language has an introspection feature to list the
   methods of a class.

This works for Perl, Python, and Ruby, but some of these points would make it 
hard to even support a dynamically typed language like JavaScript in an 
idiomatic way.

So we really should change the whole callback mechanism to be based on 
interfaces. This also means to change the parts of Lucy where callbacks are 
involved. But I don't see a need to throw Clownfish's internal object model 
overboard.

Nick


[1] Probably only by omission. The "abstract" modifier for classes is purely 
informational, AFAICS.

Re: [lucy-dev] Go bindings -- composition over inheritance

[Peter Karman <pe...@peknet.com>]

Marvin Humphrey wrote on 12/2/14, 6:47 PM:

>
> The compositional approach would also be perfectly natural and idiomatic for
> other languages.

that was very helpful, esp with the examples. thanks.


-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [lucy-dev] Go bindings -- composition over inheritance

[Nick Wellnhofer <we...@aevum.de>]

On 05/12/2014 17:17, Nick Wellnhofer wrote:
> Otherwise, I don't see how objects could be upcasted.

I meant "downcasted".

Nick

Re: [lucy-dev] Go bindings -- composition over inheritance

[Nick Wellnhofer <we...@aevum.de>]

On 03/12/2014 03:47, Marvin Humphrey wrote:
> On Mon, Nov 17, 2014 at 1:12 PM, Marvin Humphrey <ma...@rectangular.com> wrote:
>
>> *   Clownfish abstract classes are mapped to Go interfaces.
>> *   Clownfish concrete classes are mapped to Go struct pointers.

Shouldn't all (non-final, at least) Clownfish classes be mapped to Go 
interfaces? Otherwise, I don't see how objects could be upcasted. This would 
mean that for every class, there'd be an interface and a struct containing the 
implementation. For the implementing structs, it should work to put a 
cfish_Obj pointer in the Obj struct and the parent struct in the structs of 
subclasses?

type Obj interface {
     ToPtr() unsafe.Pointer
}
type ObjImpl struct {
     ref *C.cfish_Obj
}

type Searcher interface {
     Obj
     Hits(...)
}
type SearcherImpl struct {
     ObjImpl
}

type IndexSearcher interface {
     Searcher
}
type IndexSearcherImpl struct {
     SearcherImpl
}

Nick