You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Logan Bell <lo...@apache.org> on 2012/11/02 07:08:10 UTC
[lucy-dev] Ruby Next Steps
Lucy Devs,
To summarize this is where we're at in regard to binding ruby to Lucy:
1. We have a rakefile clownfish/runtime/ruby that builds builds lemon,
clownfish, and CFC.c in the ext folder. This CFC.c contains bindings for
Ruby for the following clownfish objects: Clownfish::CFC::Model::Hierarchy
and Clownfish::CFC::Binding::Core.
2. We also have a rakefile in clownfish/compiler/ruby that invokes the
aforementioned runtime rake file, builds charmonizer and the ruby
charmonizer.rb file, and invokes the Clownfish::CFC::Model::Hierarchy
and Clownfish::CFC::Binding::Core ruby objects to generate the autogen
folder. This rakefile also builds, utilizing Rake::ExtensionTask, the Bind
code in the ext folder which contain routines to convert Clownfish
charbufs/varrays to ruby objects.
3. There is also a Rakefile in root/ruby that I believe is compiling Lucy
itself. After taking a cursory glance at it appears to do be compiling
charmonizer and clownfish too, so I think this build script may need to be
revisited and/or refactored at a later point.
With my limited scope and understanding, I believe going forward the next
logical step would be to start creating a Clownfish::CFC::Binding::Ruby
that implements a write_ext_typemap method which in turn would call out to
CFCRubyTypeMap_write_ext_typemap.
Does this seem correct to others who have gone down the path of trying to
bind another language? Any other parting words of wisdom and suggestions
would be highly appreciated.
Thanks,
Logan
Re: [lucy-dev] Ruby Next Steps
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Nov 6, 2012 at 11:03 PM, Logan Bell <lo...@gmail.com> wrote:
> So, with that, correct me if I'm wrong, but it sounds like the
> immediate next steps (outside of the charmonizer issue) is the
> following:
>
> 1. Create a CFCRubyType.h/c with the associated functions
> CFCRubyTypeMap_from_ruby/CFCRubyTypeMap_to_ruby
Sounds good.
> 2. Map these in the CFC.c file in compiler/ruby/ext/Clownfish
We aren't necessarily going to need to access these from Ruby, so I wouldn't
bother just yet.
The functions `CFCPerlTypeMap_to_perl` and `CFCPerlTypeMap_from_perl` are only
used in CFCPerlMethod.c and CFCPerlClass.c. The analogous `to_ruby` and
`from_ruby` functions will presumably end up being used in the future files
"CFCRubyMethod.c" and "CFCRubyClass.c".
> After these steps, again referring back to
> Clownfish::CFC::Perl::Build, it calls out to the binding object the
> following methods: write_boot and write_bindings. I'm imagining, this
> would be where the real work will begin by creating CFCRuby.c to
> implement these calls.
Yes.
> So if I'm correct the first step is to start
> fashioning write_boot, which I believe is the boot strapping code for
> clownfish?
Conceptually, it writes the code that must be run when the XS module is
loaded (wrapped up in a function named `lucy_Lucy_bootstrap`):
MODULE = Lucy PACKAGE = Lucy
BOOT:
lucy_Lucy_bootstrap();
Run the following commands:
cd $REPOS/perl/
perl Build.PL
./Build clownfish
Then look at these files, which are what gets written by `write_boot()`
autogen/include/lucy_boot.h
autogen/source/lucy_boot.c
I think for starters it would be OK if the boot.c file you're generating more
or less looks like this:
#include "lucy_boot.h"
#include "parcel.h"
void
lucy_Lucy_bootstrap() {
lucy_bootstrap_parcel();
}
Marvin Humphrey
Re: [lucy-dev] Ruby Next Steps
Posted by Logan Bell <lo...@gmail.com>.
Thank you for such a detailed response! Please see mine below.
On Mon, Nov 5, 2012 at 6:52 PM, Marvin Humphrey <ma...@rectangular.com> wrote:
>
> On Thu, Nov 1, 2012 at 11:08 PM, Logan Bell <lo...@apache.org> wrote:
> > To summarize this is where we're at in regard to binding ruby to Lucy:
> >
> > 1. We have a rakefile clownfish/runtime/ruby that builds builds lemon,
> > clownfish, and CFC.c in the ext folder. This CFC.c contains bindings for
> > Ruby for the following clownfish objects:
> > Clownfish::CFC::Model::Hierarchy
> > and Clownfish::CFC::Binding::Core.
> >
> > 2. We also have a rakefile in clownfish/compiler/ruby that invokes the
> > aforementioned runtime rake file, builds charmonizer and the ruby
> > charmonizer.rb file, and invokes the Clownfish::CFC::Model::Hierarchy
> > and Clownfish::CFC::Binding::Core ruby objects to generate the autogen
> > folder. This rakefile also builds, utilizing Rake::ExtensionTask, the
> > Bind
> > code in the ext folder which contain routines to convert Clownfish
> > charbufs/varrays to ruby objects.
>
> Nice work. :)
>
> FWIW, recent changes to CFCVersion.h have broken the Ruby build -- we
> started
> pound-including "charmony.h" (which isn't there) rather than "stdint.h".
> Thus, the very next step will be to integrate Charmonizer.
>
> That will involve compiling clownfish/compiler/common/charmonizer.c and
> running the resulting executable to produce charmony.h (see the "charmony"
> task in ruby/Rakefile), and then adding some include dirs to feed
> charmony.h
> to CFCVersion.h.
Got it, I will look into this.
>
> > 3. There is also a Rakefile in root/ruby that I believe is compiling
> > Lucy
> > itself. After taking a cursory glance at it appears to do be compiling
> > charmonizer and clownfish too, so I think this build script may need to
> > be
> > revisited and/or refactored at a later point.
> >
> > With my limited scope and understanding, I believe going forward the
> > next
> > logical step would be to start creating a Clownfish::CFC::Binding::Ruby
> > that implements a write_ext_typemap method which in turn would call out
> > to
> > CFCRubyTypeMap_write_ext_typemap.
>
> The purpose of CFCPerlTypeMap_write_ext_typemap() function is to create
> the
> file named "typemap" needed by Perl's XS compiler, xsubpp. I've had
> limited
> experience with the Ruby C API, but to the best of my knowledge there's no
> file which is analogous to "typemap".
>
> The xsubpp compiler uses "typemap" when it compiles "Foo.xs" to "Foo.c"
>
> For instance, consider the following XS function[1]:
>
> lucy_Obj*
> fetch(self, tick)
> lucy_VArray *self;
> uint32_t tick;
> CODE:
> RETVAL = Lucy_VA_Fetch(self, tick);
> OUTPUT: RETVAL
>
> It needs to convert a `lucy_VArray*` from Perl to C on input, and then
> convert
> a `lucy_Obj*` return value into something Perl can deal with on output .
> To
> determine how it does that, it uses the following entries from the typemap
> file:
>
> LUCY_VARRAY_
> $var = (lucy_VArray*)XSBind_sv_to_cfish_obj($arg, LUCY_VARRAY,
> NULL);
>
> ...
>
> LUCY_OBJ_
> $arg = (SV*)Cfish_Obj_To_Host((cfish_Obj*)$var);
>
> Here's the C code which xsubpp spits out; if you look close, you can see
> where
> the typemap entries were used:
>
> XS(XS_Clownfish__VArray_fetch); /* prototype to pass
> -Wmissing-prototypes */
> XS(XS_Clownfish__VArray_fetch)
> {
> dXSARGS;
> if (items != 2) {
> croak_xs_usage(cv, "self, tick");
> }
> {
> lucy_VArray * self =
> (lucy_VArray*)XSBind_sv_to_cfish_obj(ST(0), LUCY_VARRAY, NULL);
> uint32_t tick = (uint32_t)SvUV(ST(1));
> lucy_Obj * RETVAL;
> #line 3813 "lib/Clownfish.xs"
> RETVAL = Lucy_VA_Fetch(self, tick);
> #line 4325 "lib/Clownfish.c"
> ST(0) = (SV*)Cfish_Obj_To_Host((cfish_Obj*)RETVAL);
>
> sv_2mortal(ST(0));
> }
> XSRETURN(1);
> }
>
> To the best of my knowledge, Ruby doesn't have anything like "typemap" --
> because Ruby extensions are written directly in C, and there's no
> intermediate
> stage like a .xs file to be compiled by something like xsubpp.
>
> Therefore, I don't think we will need write_ext_typemap(). HOWEVER...
>
> If you look at CFCPerlTypeMap.h, there are three functions:
>
> /** Return an expression which converts from a Perl scalar to a
> variable
> * of the specified type.
> *
> * @param type A Clownfish::CFC::Model::Type, which will be used to
> select
> * the mapping code.
> * @param xs_var The C name of the Perl scalar from which we are
> * extracting a value.
> */
> char*
> CFCPerlTypeMap_from_perl(struct CFCType *type, const char *xs_var);
>
> /** Return an expression converts from a variable of type `type` to a
> Perl
> * scalar.
> *
> * @param type A Clownfish::CFC::Model::Type, which will be used to
> select
> * the mapping code.
> * @param cf_var The name of the variable from which we are extracting
> a
> * value.
> */
> char*
> CFCPerlTypeMap_to_perl(struct CFCType *type, const char *cf_var);
>
> /** Auto-generate a "typemap" file that adheres to the conventions
> * documented in "perlxs".
> *
> * We generate this file on the fly rather than maintain a static copy
> * because we want an entry for each Clownfish type so that we can
> * differentiate between them when checking arguments. Keeping the
> * entries up-to-date manually as classes come and go would be a pain.
> *
> * @param hierarchy A Clownfish::CFC::Model::Hierarchy.
> */
> void
> CFCPerlTypeMap_write_xs_typemap(struct CFCHierarchy *hierarchy);
>
> We need to adapt the first two routines for Ruby (but not the third).
>
> Marvin Humphrey
>
> [1] This XS code ought to work fine, but it's not what we actually use; I
> took
> some liberties and cleaned it up a bit for the purposes of
> illustration.
Fascinating. I believe I got off course while looking through
Clownfish::CFC::Perl::Build and noticed Clownfish::CFC::Binding::Perl
and the first method it was firing off was write_xs_typemap. Further
with a combination of spelunking I saw the functions you mentioned in
CFCPerlTypeMap.h, which map to the perl equivalent: to_perl/from_perl.
Since they all happened to be associated in one header file I assumed
they all needed to be implemented. Well that assumption was wrong :),
but I feel a bit more enlightened now.
So, with that, correct me if I'm wrong, but it sounds like the
immediate next steps (outside of the charmonizer issue) is the
following:
1. Create a CFCRubyType.h/c with the associated functions
CFCRubyTypeMap_from_ruby/CFCRubyTypeMap_to_ruby
2. Map these in the CFC.c file in compiler/ruby/ext/Clownfish
After these steps, again referring back to
Clownfish::CFC::Perl::Build, it calls out to the binding object the
following methods: write_boot and write_bindings. I'm imagining, this
would be where the real work will begin by creating CFCRuby.c to
implement these calls. So if I'm correct the first step is to start
fashioning write_boot, which I believe is the boot strapping code for
clownfish? Or if not, could you elaborate a bit on what it does?
Thanks again for your detailed and helpful response. Look forward to
more conversations as we try to put this together.
Thanks,
Logan
Re: [lucy-dev] Ruby Next Steps
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Thu, Nov 1, 2012 at 11:08 PM, Logan Bell <lo...@apache.org> wrote:
> To summarize this is where we're at in regard to binding ruby to Lucy:
>
> 1. We have a rakefile clownfish/runtime/ruby that builds builds lemon,
> clownfish, and CFC.c in the ext folder. This CFC.c contains bindings for
> Ruby for the following clownfish objects: Clownfish::CFC::Model::Hierarchy
> and Clownfish::CFC::Binding::Core.
>
> 2. We also have a rakefile in clownfish/compiler/ruby that invokes the
> aforementioned runtime rake file, builds charmonizer and the ruby
> charmonizer.rb file, and invokes the Clownfish::CFC::Model::Hierarchy
> and Clownfish::CFC::Binding::Core ruby objects to generate the autogen
> folder. This rakefile also builds, utilizing Rake::ExtensionTask, the Bind
> code in the ext folder which contain routines to convert Clownfish
> charbufs/varrays to ruby objects.
Nice work. :)
FWIW, recent changes to CFCVersion.h have broken the Ruby build -- we started
pound-including "charmony.h" (which isn't there) rather than "stdint.h".
Thus, the very next step will be to integrate Charmonizer.
That will involve compiling clownfish/compiler/common/charmonizer.c and
running the resulting executable to produce charmony.h (see the "charmony"
task in ruby/Rakefile), and then adding some include dirs to feed charmony.h
to CFCVersion.h.
> 3. There is also a Rakefile in root/ruby that I believe is compiling Lucy
> itself. After taking a cursory glance at it appears to do be compiling
> charmonizer and clownfish too, so I think this build script may need to be
> revisited and/or refactored at a later point.
>
> With my limited scope and understanding, I believe going forward the next
> logical step would be to start creating a Clownfish::CFC::Binding::Ruby
> that implements a write_ext_typemap method which in turn would call out to
> CFCRubyTypeMap_write_ext_typemap.
The purpose of CFCPerlTypeMap_write_ext_typemap() function is to create the
file named "typemap" needed by Perl's XS compiler, xsubpp. I've had limited
experience with the Ruby C API, but to the best of my knowledge there's no
file which is analogous to "typemap".
The xsubpp compiler uses "typemap" when it compiles "Foo.xs" to "Foo.c"
For instance, consider the following XS function[1]:
lucy_Obj*
fetch(self, tick)
lucy_VArray *self;
uint32_t tick;
CODE:
RETVAL = Lucy_VA_Fetch(self, tick);
OUTPUT: RETVAL
It needs to convert a `lucy_VArray*` from Perl to C on input, and then convert
a `lucy_Obj*` return value into something Perl can deal with on output . To
determine how it does that, it uses the following entries from the typemap
file:
LUCY_VARRAY_
$var = (lucy_VArray*)XSBind_sv_to_cfish_obj($arg, LUCY_VARRAY, NULL);
...
LUCY_OBJ_
$arg = (SV*)Cfish_Obj_To_Host((cfish_Obj*)$var);
Here's the C code which xsubpp spits out; if you look close, you can see where
the typemap entries were used:
XS(XS_Clownfish__VArray_fetch); /* prototype to pass -Wmissing-prototypes */
XS(XS_Clownfish__VArray_fetch)
{
dXSARGS;
if (items != 2) {
croak_xs_usage(cv, "self, tick");
}
{
lucy_VArray * self =
(lucy_VArray*)XSBind_sv_to_cfish_obj(ST(0), LUCY_VARRAY, NULL);
uint32_t tick = (uint32_t)SvUV(ST(1));
lucy_Obj * RETVAL;
#line 3813 "lib/Clownfish.xs"
RETVAL = Lucy_VA_Fetch(self, tick);
#line 4325 "lib/Clownfish.c"
ST(0) = (SV*)Cfish_Obj_To_Host((cfish_Obj*)RETVAL);
sv_2mortal(ST(0));
}
XSRETURN(1);
}
To the best of my knowledge, Ruby doesn't have anything like "typemap" --
because Ruby extensions are written directly in C, and there's no intermediate
stage like a .xs file to be compiled by something like xsubpp.
Therefore, I don't think we will need write_ext_typemap(). HOWEVER...
If you look at CFCPerlTypeMap.h, there are three functions:
/** Return an expression which converts from a Perl scalar to a variable
* of the specified type.
*
* @param type A Clownfish::CFC::Model::Type, which will be used to select
* the mapping code.
* @param xs_var The C name of the Perl scalar from which we are
* extracting a value.
*/
char*
CFCPerlTypeMap_from_perl(struct CFCType *type, const char *xs_var);
/** Return an expression converts from a variable of type `type` to a Perl
* scalar.
*
* @param type A Clownfish::CFC::Model::Type, which will be used to select
* the mapping code.
* @param cf_var The name of the variable from which we are extracting a
* value.
*/
char*
CFCPerlTypeMap_to_perl(struct CFCType *type, const char *cf_var);
/** Auto-generate a "typemap" file that adheres to the conventions
* documented in "perlxs".
*
* We generate this file on the fly rather than maintain a static copy
* because we want an entry for each Clownfish type so that we can
* differentiate between them when checking arguments. Keeping the
* entries up-to-date manually as classes come and go would be a pain.
*
* @param hierarchy A Clownfish::CFC::Model::Hierarchy.
*/
void
CFCPerlTypeMap_write_xs_typemap(struct CFCHierarchy *hierarchy);
We need to adapt the first two routines for Ruby (but not the third).
Marvin Humphrey
[1] This XS code ought to work fine, but it's not what we actually use; I took
some liberties and cleaned it up a bit for the purposes of illustration.