Inlining XS code

[Marvin Humphrey <ma...@rectangular.com>]

Greets,

If you're not interested in Lucy's Perl bindings, you can skip this  
post...

The most powerful, flexible way to glue C and Perl together is a  
suite of C macros and code-generation tools collectively known as  
"XS".  Another popular option is Inline::C, which allows you to embed  
C code directly within perl code. (Peeking underneath the hood  
reveals that Inline::C is actually an XS code generator).  You can  
also use SWIG, but that's not common.

Ordinarily, XS resides in dedicated .xs files.  There is no one place  
that these files are "supposed" to go -- while Perl modules  
themselves typically end up in $DIST_ROOT/lib, XS files might end up  
in $DIST_ROOT, or in $DIST_ROOT/xs, or alongside their companion Perl  
modules in $DIST_ROOT/lib, or somewhere else.

KinoSearch does something a little bit unconventional with regards to  
its XS code: it's stored Inline::C-style, within the Perl modules  
after an __END__ token, delimited by __XS__ and __POD__ tokens.   
(Example at <http://xrl.us/vgri>.)  At build time, the XS code is  
extracted and concatenated into a single KinoSearch.xs file, which is  
subsequently compiled using standard Perl building tools.

There are two main reasons for doing things this way.

First, having XS code right next to the perl code it services is a  
real boon.  It's easier to move subroutine implementations from one  
space to the other, as there are 3 files (.c, .h and .pm) to deal  
with instead of 4 (with the addition of the .xs file).  The XS and  
Perl are often tightly coupled; for example, the default values for  
an XS subroutine might be managed via a hash created in Perl-space.   
And even when there is no coupling, it is useful to have all the  
subroutines available from a given perl package concentrated in one  
place.

Second, all that XS code really does belong in one giant file.  It  
would be highly annoying if KS adopted a 1:1 pairing of Perl modules  
to XS files, because XS takes a lllloooonnngggg time to compile.  I  
tried breaking things up into 6 packages a while back but quickly  
reverted because of exploding compilation times.  However, at 4400  
lines and growing, a monolithic KinoSearch.xs file is cumbersome to  
edit.

The downside of inlining the XS code is that the style is a bit  
unusual: an XS hacker doesn't automatically know what to make of  
a .pm file like that.  I'm not sure that's a big deal, though,  
because there are so few XS hackers and they're all savvy enough to  
grok what's going on after a short explanation -- learning XS is a  
daunting, difficult task and I'd guess that the number of people who  
know it is in the low thousands, max.    Plus, after a significant  
push last year the XS code in KinoSearch is now 90% glue -- the  
algorithmic stuff is all in either C or Perl.  That will be the case  
with Lucy as well.

Nevertheless, I figured I'd try the more conventional approach of  
using a dedicated XS file for Lucy, so I set up KS to use one as an  
experiment.  I've been unhappy with the results.  The inlining is  
just more development-friendly.

So unless someone here raises an objection, I plan to duplicate that  
style in Lucy's Perl bindings.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Re: Inlining XS code

[Peter Karman <pe...@peknet.com>]

Marvin Humphrey scribbled on 3/26/07 9:37 PM:

> Peter... Dave and I have been collaborating fruitfully, but this isn't 
> an area of expertise for him.  If I start presenting these tools here 
> and explaining the rationales behind them, are you willing to read the 
> emails and offer critiques/suggestions, or at least nod your head OK?  
> ;)  I'm not looking for deep debugging, rather high-level design 
> discussion.
> 

I'll try. Life (work) is about to get insane again for me, so I may be terse or 
tardy in response, but I do read this list pretty thoroughly.

Is there a lucy-commit list so I know when stuff is going in to svn?

-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: Inlining XS code

[Marvin Humphrey <ma...@rectangular.com>]

On Mar 26, 2007, at 11:07 AM, Peter Karman wrote:

> I like the current convention of putting XS in with the Perl.

Groovy.

I'm now about to start contributing a bunch of utility stuff that's  
used by KinoSearch's XS bindings.

Peter... Dave and I have been collaborating fruitfully, but this  
isn't an area of expertise for him.  If I start presenting these  
tools here and explaining the rationales behind them, are you willing  
to read the emails and offer critiques/suggestions, or at least nod  
your head OK?  ;)  I'm not looking for deep debugging, rather high- 
level design discussion.

At the least, I'd like to get the thoughts behind this stuff entered  
into the email record.  The material is commented well enough for me  
to understand what I did six months ago, but XS is so esoteric and as  
I wrote everything in isolation, I think some aspects might prove  
difficult for other maintainers to unravel.  Forcing myself to  
justify it would be a good exercise, and if we come up with  
improvements, more's the better.

The material I'm talking about is most of the content from two files  
from the KS repository (<http://www.rectangular.com/svn/kinosearch/ 
trunk/>)...

   $REPOS_ROOT/perl/xshelper/XSHelper.c
   $REPOS_ROOT/perl/xshelper/XSHelper.h

... plus parts of the Module::Build subclass at...

   $REPOS_ROOT/perl/buildlib/Lucy/Build.pm

... particularly the ACTION_write_typemap subroutine.

(Yes, KS svn trunk now has Lucy::Build in it, to aid synchronization.)

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Re: Inlining XS code

[Peter Karman <pe...@peknet.com>]

Peter Karman scribbled on 3/26/07 1:07 PM:
> 
> 
> Marvin Humphrey scribbled on 3/26/07 5:28 AM:
> 
>>
>> So unless someone here raises an objection, I plan to duplicate that 
>> style in Lucy's Perl bindings.
>>
> 
> I like the current convention of putting XS in with the Perl.
> 

oh, and in case that wasn't clear, I'm agreeing with Marvin. ;)

-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: Inlining XS code

[Peter Karman <pe...@peknet.com>]

Marvin Humphrey scribbled on 3/26/07 5:28 AM:

> 
> So unless someone here raises an objection, I plan to duplicate that 
> style in Lucy's Perl bindings.
> 

I like the current convention of putting XS in with the Perl.

-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com