[lucy-dev] Dependencies

[Marvin Humphrey <ma...@rectangular.com>]

On Sun, May 15, 2011 at 10:47:24PM -0700, Nathan Kurz wrote:
> On Thu, May 12, 2011 at 9:33 PM, Marvin Humphrey <ma...@rectangular.com> wrote:
> > The apreq harness isn't a good replacement candidate in its present
> > form, though, because it requires APR.
> 
> I tend to like the approach of having everything self-contained, but
> if there were candidates I would think the the APR would be high on
> the list.

There are two TAP-producing test frameworks within Lucy right now: one
belonging to Charmonizer, and one belonging to Lucy/Clownfish.  That passage
you've quoted refers to replacing the Charmonizer test framework, not the
Lucy/Clownfish test framework.

Like Clownfish, Charmonizer has been designed as a distinct layer within the
Lucy project, mainly for separation of concerns.  It might potentially
become usable outside of Lucy at some point[1].  If that day ever comes, it
would be absurd for a C configuration prober to require APR as a 
prerequisite.  :)

So, I stand by my statement above that the apreq harness is not a good
replacement candidate for Charmonizer's test framework as long as it requires
APR.

However, I realize that you're speaking of adding APR as a dependency for Lucy
at large.  OK, but what concrete advantages would that give us?  APR is big
and complicated.  It took me 10 minutes to build APR and run its test suite
just now on my MacBook Pro.  It failed.

APR is more of a base platform for development rather than something you add
to get specific targeted features.  The ill-fated Lucene4C project was built
on top of it.  Lucy isn't.

It would take an awful lot to persuade me that taking on APR as a dependency
at this point was worthwhile.  

> Also, I would guess that a developer only test-suite wouldn't be much of a
> hindrance to adoption.

Run our tests only during development?  I disagree strongly with that
proposition, even if it's only Charmonizer's tests.   Our users need to know
when Lucy won't work effectively on their platform.  We need to know, too.

Developer-only tests can be useful and important -- Lucy's "test_valgrind"
build target is a good example.  But easy tests ought to be run everywhere we
can run them.

> Is there a formal policy on avoiding outside dependencies, or is this
> just a general preference?

I'm pretty sure that there is no formal ASF policy, though from what I've seen
most ASF projects bundle their dependencies unless licensing restrictions make
that impossible.  For Lucy, the policy is whatever we decide on as a community
and vote for as a PPMC. 

Here's my personal perspective:

Lucy is a low-level library.  Our goal should be to make it so rock solid and
portable that other projects will not hesitate to add it as a dependency.

Lucy should be easy to install.  Historically, KinoSearch has been easy to
install via the CPAN client (modulo its own bugs) because all our dependencies
were extremely reliable and portable and CPAN did a good job of resolving
them.  Since we are preparing to expand outside the purview of Perl/CPAN we
should bundle all the dependencies we add from here on out.

Lucy (and Clownfish) are designed to live within the context of a host
language and its library directories, rather than within the context of an
operating system and its shared library directories (e.g. /usr/local/lib).
There can be multiple installs of Lucy on a system if you have multiple
installs of Perl (and eventually Ruby, Python, etc).  Any dependency we add
has to be able to live harmoniously within this environment.

> Are you planning to stay the course of independence, or is there a point
> you'd consider utilizing outside libraries?

We're bundling the Snowball stemming libraries already.  We'll be bundling the
Lemon parser generator soon.

These two dependencies share a few common traits:

 * They're bare .c/.h files.
 * They are extremely reliable and trouble free[2].
 * Their authors aren't narrow-minded GCC/Linux parochialists (or Windows
   parochialists, for that matter).  Both libraries are standard C and work
   fine across multiple compilers and operating systems.

I'm not opposed to dependencies per se, but they must meet high standards for
portability, reliability and ease-of-integration.

Marvin Humphrey


[1] Charmonizer is actually somewhat closer to being spun off from Lucy than
    Clownfish.

[2] I don't know for certain that Lemon itself is trouble free, but SQLite's
    parser, which is built with Lemon, surely qualifies.