Re: Native callbacks

[Marvin Humphrey <ma...@rectangular.com>]

On Apr 26, 2008, at 12:55 PM, Marvin Humphrey wrote:

> This isn't a truly hard problem. There are definitely many ways to  
> make this introspection data available using C.   But all the ways  
> I've thought of so far are ugly.

I've gotten this working.  On its own, it's mildly ugly -- not too bad.

Things are improving step by step. At first, you couldn't override  
anything.  Then you could implement abstract methods using the host  
language.  Now you can override public methods.

There's one significant problem left that I'm aware of: Perl's SUPER  
mechanism is broken with the vtable override.

Say you use the host language to override a method which was  
implemented in C.  The original binding invoked the implementing  
function via the vtable.  The override has now changed that vtable  
pointer to be a callback.  If you invoke SUPER from the subclass, it  
invokes the callback, which invokes the native method again, and you  
end up in an infinite loop.

   sub serialize {
      my ( $self, $outstream ) = @_;
      $self->SUPER::serialize($outstream); # BOOM! Infinite recursion.
      ...
   }

The solution, I believe, is to have the XS bindings not invoke via the  
vtable, but instead invoke the implementing function directly.   
Unfortunately, that means a lot more binding code, because each  
override requires its own binding.

Now:

    MODULE = KinoSearch   PACKAGE = KinoSearch::Index::PostingList

    chy_u32_t
    get_doc_num(self)
       kino_PostingList *self;
    CODE:
       RETVAL = Kino_PList_Get_Doc_Num(self);
    OUTPUT: RETVAL

Fixed:

    MODULE = KinoSearch   PACKAGE = KinoSearch::Index::PostingList

    chy_u32_t
    get_doc_num(self)
       kino_PostingList *self;
    CODE:
       RETVAL = kino_PList_get_doc_num(self);
    OUTPUT: RETVAL

    MODULE = KinoSearch   PACKAGE = KinoSearch::Index::SegPostingList

    chy_u32_t
    get_doc_num(self)
       kino_SegPostingList *self;
    CODE:
       RETVAL = kino_SegPList_get_doc_num(self);
    OUTPUT: RETVAL

    MODULE = KinoSearch   PACKAGE = KinoSearch::Index::MultiPostingList

    chy_u32_t
    get_doc_num(self)
       kino_MultiPostingList *self;
    CODE:
       RETVAL = kino_MultiPList_get_doc_num(self);
    OUTPUT: RETVAL

Fortunately, most of the binding code gets auto-generated these days,  
so once the auto-generation code gets patched the suffering will  
primarily take the form of slower compiles rather than increased  
maintenance costs.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Re: Native callbacks

[Marvin Humphrey <ma...@rectangular.com>]

On Apr 30, 2008, at 10:58 PM, Marvin Humphrey wrote:

> There's one significant problem left that I'm aware of: Perl's SUPER  
> mechanism is broken with the vtable override.

> The solution, I believe, is to have the XS bindings not invoke via  
> the vtable, but instead invoke the implementing function directly.   
> Unfortunately, that means a lot more binding code, because each  
> override requires its own binding.


I finally got around to implementing this, and it turns out to have  
been surprisingly easy.  The commit diff was only 137 lines long.

http://www.rectangular.com/pipermail/kinosearch-commits/2008-June/001277.html

The size of the autogenerated binding file, KinoSearch.xs, went from  
13442 to 18401 lines.  On my Core 2 Duo MacBook Pro, compile time  
increased from about 15 to 20 seconds for that file, and from about 55  
to 60 seconds for all of KS.  That's perfectly acceptable, and a  
smaller increase than I'd expected.

The behavior of vtable methods is now virtually indistinguishable from  
that of native Perl methods from the point of view of the library  
user.  The only difference is that because the vtable pointers for  
subclasses are set when the DynamicVirtualTable object is created (the  
first time it is needed by a new object), dynamically generated  
methods created after that time won't trigger an override.  That's an  
esoteric use case, and I don't think it's even worth mentioning in the  
documentation.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/