[lucy-dev] Cross-parcel method aliases in binding code

[Marvin Humphrey <ma...@rectangular.com>]

Greets,

Summary of #lucy_dev IRC conversation:

There's a problem in the way that we specify Perl bindings for clownfish
methods.  The default is to lowercase the name of the Clownfish method (`Load`
=> 'load') but we also allow aliases (`Load` => `_load`).  That
information is not preserved once the shared object is built, so derived
parcels are not getting it and generate the default instead.

We need a way to pass this metadata, e.g. from Clownfish to Lucy.  The easiest
technique seems to be to embed it in the shared object and have CFC `use
Clownfish` then extract it.

Marvin Humphrey

nwellnhof joined the chat room.
[3:14pm] nwellnhof: creamygoodness: I hit an interesting problem:
Method aliases don't work across parcels.
[3:14pm] nwellnhof: This breaks 'load' in the Perl bindings.
[3:14pm] nwellnhof: $binding->bind_method( alias => '_load', method => 'Load' );
[3:15pm] nwellnhof: ^^^ isn't used in the Lucy parcel.
[3:15pm] creamygoodness: nwellnhof: It generates the wrong symbol?
[3:15pm] creamygoodness: Oh!
[3:15pm] creamygoodness: I see.
[3:16pm] nwellnhof: The Lucy parcel doesn't know about aliases defined
in the Clownfish parcel.
[3:16pm] creamygoodness: Gotcha.
[3:16pm] creamygoodness: So it autogenerates a binding
[3:17pm] creamygoodness: but with the ordinary symbol, not the alias.
[3:17pm] nwellnhof: Yes.
[3:17pm] creamygoodness: So my immediate thought is that we need to
bake that information into the parent parcel.
[3:18pm] nwellnhof: That's what I thought, too.
[3:18pm] creamygoodness: heh.
[3:18pm] • creamygoodness waits for the shoe to drop
[3:19pm] creamygoodness: You worked on a lot of metadata-ish stuff for Method.
[3:19pm] creamygoodness: But that's all core, not binding.
[3:19pm] creamygoodness: This isn't stored in the .cfh headers.
[3:21pm] nwellnhof: We'd need some kind of metadata store for host
language bindings.
[3:22pm] creamygoodness: An interesting problem: does it go in the
shared object, or in the headers?
[3:23pm] nwellnhof: Or somewhere else?
[3:23pm] creamygoodness: Maybe!
[3:23pm] creamygoodness: So CFC is what's going to have to extract this data.
[3:24pm] nwellnhof: It gets that data from the Perl build process.
[3:25pm] nwellnhof: And has to store it somewhere for other parcels to read.
[3:25pm] creamygoodness: Is it a problem having CFC examine shared
objects in a variety of formats?
[3:25pm] creamygoodness: I wonder if we can rule out the shared object
storage for that reason.
[3:26pm] nwellnhof: No, that's actually a good idea. But pretty
complex, I think.
[3:26pm] creamygoodness: The problem is that we'd have to load a
shared object into a CFC process.
[3:26pm] creamygoodness: It's a little weird for a compiler to have to
load a shared object, isn't it?
[3:27pm] nwellnhof: I already thought about putting a lot more
informations in the shared objects. So we can extend parcels without
the .cfh files.
[3:27pm] creamygoodness: Oh, interesting idea!
[3:28pm] creamygoodness: That will make them bigger, but does that
matter if we never actually use that part of the data during ordinary
runtime?
[3:29pm] creamygoodness: I guess it could depending on the size of a
virtual memory page (if the page size is big enough).
[3:29pm] nwellnhof: I don't think we need much additional space. The
most important stuff is method signatures.
[3:29pm] creamygoodness: But even then it will only consume main
memory, not CPU cache.
[3:31pm] creamygoodness: So in a perfect world, would you not even
store away the .cfh files -- instead extracting them on the fly from
the shared object metadata?
[3:32pm] creamygoodness: That would limit our ability to pass along
things like inline functions.
[3:32pm] creamygoodness: (Think Clownfish::Util::NumberUtils)
[3:33pm] nwellnhof: My idea was to only pass the generated C headers
and the shared object.
[3:34pm] nwellnhof: This would work well for the C bindings. Maybe not
for other host languages.
[3:35pm] nwellnhof: But the solution for the Perl method aliases could
be much easier.
[3:36pm] nwellnhof: The Lucy build could simply 'use Clownfish' and
extract the aliases from there.
[3:37pm] creamygoodness: That would probably rule out cross-compiling
bindings, e.g. generate Perl bindings while running a version of CFC
bound to Python.  Dunno if that's important.
[3:38pm] creamygoodness: BTW, can I have your permission to paste this
conversation into a mail to lucy-dev? I'll send and summarize.
[3:41pm] creamygoodness: However, if we can't do that, than any
information specified in the Perl bindings has to serialized, written
to a file, stored and then parsed later.
[3:42pm] creamygoodness: It's ironic, I'd hoped to avoid the
complexity of a binding specification interface-description-language.
[3:43pm] creamygoodness: There's a lot you can do dynamically and
programmatically that is a PITA if you've got to write out files.
[3:43pm] nwellnhof: Paste it to lucy-dev if you like.
[3:44pm] nwellnhof: The easiest solution is probably to generate Perl
code with the metadata. But yes, that would be host-language specific
[3:45pm] creamygoodness: OK, I'm now leaning in that direction.
[3:46pm] nwellnhof: But that's the only problem I encountered. The
rest of the Clownfish separation seems to work.
[3:46pm] creamygoodness: Awesome. Truly awesome.
[3:46pm] nwellnhof: With some minor fixes...
[3:47pm] nwellnhof: I'm gonna push the code in the next days.
[3:47pm] creamygoodness: Sweet!

Re: [lucy-dev] Cross-parcel method aliases in binding code

[Marvin Humphrey <ma...@rectangular.com>]

On Sat, Jun 8, 2013 at 5:48 AM, Nick Wellnhofer <we...@aevum.de> wrote:
> On Jun 7, 2013, at 02:00 , Marvin Humphrey <ma...@rectangular.com> wrote:
> Looking at the code in CFCPerlClass_method_bindings where we need the
> metadata from other parcels, it turns out that the same problem applies to
> "excluded" methods.

That makes sense.  Really, it could be anything that we put into the Perl
binding specification -- i.e. anything in
`$REPOS/perl/buildlib/Lucy/Build/Binding/*.pm`.

For instance, say that we inherited method documentation -- that would also be
lost.

The issue is that we generate the method binding spec programatically rather
than via a spec file.  It's more flexible, but introduces some complications.

I don't imagine we'll ultimately decide to do this, but we should at least
consider that installing those Lucy::Build::Binding::* modules would be one
path to solving this problem.

> I think the best place to store that information is in the Method objects.
> The bootstraping code in boot.c could add method aliases and exclusions like
> this:
>
>     VTable_Add_Host_Method_Alias(LUCY_OBJ, "Load", "_load");
>     VTable_Exclude_Host_Method(LUCY_HASH, "Store");

+1

Adding this information to the introspection data makes sense regardless of
how we solve the larger problem of preserving binding spec data.

> By the way, I'm really in favor of renaming Clownfish::VTable to
> Clownfish::Class.

+1

For previous discussion, see <http://markmail.org/message/bdxdrcyxgbum627g>,
near the end.

Marvin Humphrey

Re: [lucy-dev] Cross-parcel method aliases in binding code

[Nick Wellnhofer <we...@aevum.de>]

On Jun 7, 2013, at 02:00 , Marvin Humphrey <ma...@rectangular.com> wrote:

> There's a problem in the way that we specify Perl bindings for clownfish
> methods.  The default is to lowercase the name of the Clownfish method (`Load`
> => 'load') but we also allow aliases (`Load` => `_load`).  That
> information is not preserved once the shared object is built, so derived
> parcels are not getting it and generate the default instead.
> 
> We need a way to pass this metadata, e.g. from Clownfish to Lucy.  The easiest
> technique seems to be to embed it in the shared object and have CFC `use
> Clownfish` then extract it.

Looking at the code in CFCPerlClass_method_bindings where we need the metadata from other parcels, it turns out that the same problem applies to "excluded" methods.

I think the best place to store that information is in the Method objects. The bootstraping code in boot.c could add method aliases and exclusions like this:

    VTable_Add_Host_Method_Alias(LUCY_OBJ, "Load", "_load");
    VTable_Exclude_Host_Method(LUCY_HASH, "Store");

Then the Lucy build process could use the VTable registry to tell CFC about the metadata. If we do it in Perl, we don't have to care about dynamically loading the shared object:

    use Clownfish;

    my @vtables = Clownfish::VTable->vtables_from_parcel('Clownfish');
    for my $vtable (@vtables) {
        my $class = Clownfish::CFC::Model::Class->fetch_singleton($vtable->get_name);
        for my $method (@{ $vtable->get_methods }) {
            if ($method->is_excluded_from_host) {
                $class->exclude_host_method($method->get_name);
            }
            else {
                my $alias = $method->get_host_alias;
                if (defined($alias)) {
                    $class->add_host_method_alias($method->get_name, $alias);
                }
            }
        }
    }

We'd need a few extensions to our metadata system but that should be useful later. As I indicated on IRC, I've been thinking about storing method signatures in the Method objects. That should make it possible to extend Clownfish parcels without the parcel's CFH files. Only the DSO and the generated C headers would be required.

By the way, I'm really in favor of renaming Clownfish::VTable to Clownfish::Class.

Nick